CN103269448A - Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm - Google Patents
Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm Download PDFInfo
- Publication number
- CN103269448A CN103269448A CN2013101997164A CN201310199716A CN103269448A CN 103269448 A CN103269448 A CN 103269448A CN 2013101997164 A CN2013101997164 A CN 2013101997164A CN 201310199716 A CN201310199716 A CN 201310199716A CN 103269448 A CN103269448 A CN 103269448A
- Authority
- CN
- China
- Prior art keywords
- video
- audio
- frame
- adjustment
- ratio
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention relates to a method for achieving synchronization of an audio and a video on the basis of the RTP/RTCP feedback early-warning algorithm. The method for achieving the synchronization of the audio and the video on the basis of the RTP/RTCP feedback early-warning algorithm includes a first step of collecting audio and video signals and encoding the audio and video signals, a second step of encapsulating the audio and video signals, adding corresponding timestamp information to the encapsulated audio and video signals, and sending the encapsulated audio and video signals, a third step of decoding the audio and video signals according to the timestamp information of an RTP package and playing the audio and video signals, and a fourth step of sending playing synchronization information back to a server through the RTCP protocol, achieving the synchronization of the audio and the video through sending-strategy adjustment conducted by the server according to feedback information. The method for achieving the synchronization of the audio and the video on the basis of the RTP/RTCP feedback early-warning algorithm has the advantages of being low in complexity, and suitable for multimedia communication with high packet loss probability, large delay variation and resource constrain.
Description
Technical field
The invention belongs to the audio-visual synchronization technical field, relate to a kind of audio and video synchronization method.
Background technology
Realtime transmission protocol RTP: be a host-host protocol going up multimedia data stream at Internet, issued as RFC1889 by IETF (Internet engineering duty group).RTP be defined in one to one or the transmission situation of one-to-many under work, its objective is provides temporal information and realizes that stream is synchronously.RTCP Real-time Transport Control Protocol RTCP: be in charge of transmission quality exchange of control information between the current application process.During the RTP session, each participant periodically transmits the RTCP bag, contains the quantity of data packets that has sent, the statistics of losing such as quantity of data packets in the bag.H.264 standard is the defined up-to-date video encoding standard of ITU-T, is known as ISO/IEC14496-10 or MPEG-4 AVC, is the new product by the common exploitation of video coding expert group of dynamic image expert group and International Telecommunications Union.
In recent years, both at home and abroad the researcher at medium, media player, video conference, distributed interactive multimedia system etc. different application the multimedia simultaneous techniques has been carried out research widely, but all be based on different applied environments, up to now, general synchronization mechanism and pattern are not arranged as yet.
Summary of the invention
The present invention is directed to the deficiencies in the prior art, provide a kind of feedback warning algorithm based on RTP/RTCP in Streaming Media H.264, to realize the method for audio-visual synchronization.
The technical scheme that technical solution problem of the present invention is taked is:
Step (1). the audio-video signal collection.
Step (2). the audio-video signal coding.
Step (3). the package of audio-video signal, stamp corresponding timestamp information and transmission.
Step Zou (4). according to timestamp information in the RTP bag audio-video signal is decoded, play.
Step Zou (5). will play synchronizing information and utilize rtcp protocol to send back server end, service end sends strategy according to the feedback information adjustment and realizes audio-visual synchronization, adjusts and sends strategy specifically:
According to the auditory properties of people's ear, the audio frequency and video deviation is divided into four zones:
A. synchronization zone: deviation-60ms～+ 60ms;
B. early warning adjustment district: deviation-120ms～-60ms and 60ms～120ms;
C. adjust the district: deviation-160ms～-120ms and 120ms～160ms;
D. asynchronous district: deviation＜-160ms and deviation〉160ms.
For the synchronization zone normal play.
For early warning adjustment district, method of adjustment is as follows:
If frame of video falls behind, then carry out following steps 1-3:
Step 1 checks the audio frequency and video buffer performance;
Step 2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, is only play key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 40%-60%;
Step 3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%.
For adjusting the district, arrived the edge of critical zone, be about to occur the audio frequency and video asynchrony phenomenon; Suppose that frame of video falls behind, adjustment algorithm is as follows:
Step 4, the broadcast strategy of adjustment player is only play key frame;
Step 5, the transmission ratio of adjustment server end audio/video frames, concrete method of adjustment is as follows:
Step 5-1 checks the audio frequency and video buffer performance;
Step 5-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 60%-70%;
Step 5-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%.
For asynchronous district, adjustment algorithm is as follows:
Step 6, the broadcast strategy of adjustment client if frame of video falls behind, is directly abandoning, if frame of video is leading, then takes the strategy of replaying;
Step 7, the transmission ratio of adjustment server end, concrete method of adjustment is as follows:
Step 7-1 checks the audio frequency and video buffer performance;
Step 7-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 70%-85%;
Step 7-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%.
Beneficial effect of the present invention:
(1) vision signal of same district is not taked the different disposal mode, improved video quality;
(2) have lower complexity, can be applicable to high packet loss, big, the resource-constrained multimedia communication of delay variation.
Description of drawings
Fig. 1 is the flow chart of video monitoring system;
Fig. 2 is early warning adjustment algorithm flow chart;
Fig. 3 is for adjusting district's algorithm flow chart;
Fig. 4 audio frequency and video deviation region figure.
Concrete execution mode:
Below in conjunction with accompanying drawing the present invention is described further.
As shown in Figure 1, the inventive method may further comprise the steps:
Step (1). the collection of beginning audio-video signal.
Start the sub-thread of audio collection and the sub-thread of video acquisition, initialization sound card equipment, video capture device.Specifically comprise the size that the audio/video coding buffering area is set, audio sample rate, sound channel, coding figure place etc.Frame of video form (width, highly, frame per second etc.), the application video data buffer also is mapped to user's space.Initiate a message and await a response to the sub-thread of coding after having gathered a frame.
Step (2). with the audio, video data coding of gathering.
Start the sub-thread of audio/video coding, the initialization codes parameter.Audio frame coded format, sample rate etc.Take buffer data away after receiving the signal that collecting thread sends, and send it back the information of answering, collecting thread continues to gather audio, video data.The sub-thread dispatching api interface of audio/video coding function is encoded to data.Encode behind the frame data and to send message and await a response to sending sub-thread.
Step (3). call corresponding built-in function to the package of audio-video signal, and stamp corresponding timestamp information, send audio, video data according to sending strategy.
Start the sub-thread of transmission, initialization time stabs, and creates the rtp session object, and IP and the listening port of long-range RTP client is set, and loadtype is set, and obtains Synchronization Source etc.To coded data packing and stamp corresponding timestamp information, send then.
Step Zou (4). client is decoded, is play audio-video signal according to timestamp information in the RTP bag.
Client is taken out data from receiving in the buffering area, carries out RTP and unpacks, and analyzes voice data or video data, puts into corresponding play buffer then.Utilize the time tag in the RTP timestamp to set up the absolute time axis information, with audiotime message and the comparison of video time information of same absolute time information and broadcast, difference is fed back to server end by RTCP, server is made corresponding adjustment.Wherein Fig. 4 is audio frequency and video deviation region figure, the synchronization zone (60ms～+ 60ms), early warning adjustment district (120ms～-60ms and 60ms～120ms), adjust the district (160ms～-120ms and 120ms～160ms), asynchronous district (＜-160ms and 160ms).
Step Zou (5). will play synchronizing information and utilize rtcp protocol to send back server end, service end sends strategy according to the feedback information adjustment and realizes audio-visual synchronization.
The algorithm pattern that the first, Fig. 2,3 synchronizing signals are not being taked simultaneously.Fig. 2 is early warning adjustment algorithm flow chart, supposes that frame of video falls behind, and checks the audio frequency and video buffer performance, and this moment, general screen buffer occupancy was relatively low.
The second, if screen buffer takies higher (more than or equal to 50%), adjust broadcast strategy, only play key frame, it goes without doing adjusts for transmitting terminal.If the screen buffer occupancy is lower, be lower than 50%, check the audio buffer occupancy, the transmitting terminal frame of video sends ratio if the audio buffer occupancy, improves (frame of video improves 40% to 60%) more than or equal to 50%.
The 3rd if the occupancy of audio/video frames play buffer, slightly improves (10% to %20) frame of video all less than 50% to send ratio just passable.
Fig. 3 adjusts audio frequency and video transmission ratio flow chart for adjusting district's algorithm, for adjusting the district, has arrived the edge of critical zone, is about to occur the audio frequency and video asynchrony phenomenon.Suppose that frame of video falls behind, adjustment algorithm is as follows:
First: adjust the broadcast strategy of player, only play key frame.
Second: adjust the transmission ratio of server end audio/video frames, the method in concrete method of adjustment and early warning adjustment district is similar.
At first, check the audio frequency and video buffer performance, this moment, general screen buffer occupancy was relatively low.
Secondly, if screen buffer takies higher (more than or equal to 50%), adjust broadcast strategy, only play key frame, it goes without doing adjusts for transmitting terminal.If the screen buffer occupancy is lower, be lower than 50%, check the audio buffer occupancy, the transmitting terminal frame of video sends ratio if the audio buffer occupancy, significantly improves (frame of video improves 60% to 70%) more than or equal to 50%.
At last, frame of video transmission ratio is just passable if the occupancy of audio/video frames play buffer, slightly improves (10% to %20) all less than 50%.
For asynchronous district, adjustment algorithm also is in two steps:
The first, the broadcast strategy of adjustment client directly abandons if frame of video falls behind, if take the strategy of replaying in advance.
The second, the transmission ratio of adjustment server end, the method in concrete grammar and early warning adjustment district is similar.
At first, check the audio frequency and video buffer performance, this moment, general screen buffer occupancy was relatively low.
Secondly, if screen buffer takies higher (more than or equal to 50%), adjust broadcast strategy, only play key frame, it goes without doing adjusts for transmitting terminal.If the screen buffer occupancy is lower, be lower than 50%, check the audio buffer occupancy, the transmitting terminal frame of video sends ratio if the audio buffer occupancy, significantly improves (frame of video improves 70% to 85%) more than or equal to 50%.
At last, frame of video transmission ratio is just passable if the occupancy of audio/video frames play buffer, slightly improves (10% to %20) all less than 50%.
When the audio frame backwardness, when frame of video was leading, the same Noodles of processing method seemingly.Because video data is much larger than voice data, the situation that audio frame falls behind can not produce substantially, so the kind situation is only done simple introduction.Judge earlier which audio frequency and video distinguished at, and then go to check the occupancy of audio frequency and video buffering area, the transmission strategy of the strategy when adjudicating client terminal playing the most afterwards and server.
Claims (2)
1. realize audio and video synchronization method based on RTP/RTCP feedback warning algorithm, it is characterized in that this method may further comprise the steps:
Step (1). gather audio-video signal;
Step (2). the coding audio-video signal, stamp corresponding timestamp information;
Step (3). package, transmission audio-video signal;
Step Zou (4). according to timestamp information in the RTP bag audio-video signal is decoded, play;
Step Zou (5). will play synchronizing information and utilize rtcp protocol to send back server end, service end sends strategy according to the feedback information adjustment and realizes audio-visual synchronization.
2. synchronisation control means according to claim 1 is characterized in that: adjust in the step (5) and send strategy specifically:
According to the auditory properties of people's ear, the audio frequency and video deviation is divided into four zones:
A. synchronization zone: deviation-60ms～+ 60ms;
B. early warning adjustment district: deviation-120ms～-60ms and 60ms～120ms;
C. adjust the district: deviation-160ms～-120ms and 120ms～160ms;
D. asynchronous district: deviation＜-160ms and deviation〉160ms;
For the synchronization zone normal play;
For early warning adjustment district, method of adjustment is as follows:
If frame of video falls behind, then carry out following steps 1-3:
Step 1 checks the audio frequency and video buffer performance;
Step 2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, is only play key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 40%-60%;
Step 3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%;
For adjusting the district, arrived the edge of critical zone, be about to occur the audio frequency and video asynchrony phenomenon; Suppose that frame of video falls behind, adjustment algorithm is as follows:
Step 4, the broadcast strategy of adjustment player is only play key frame;
Step 5, the transmission ratio of adjustment server end audio/video frames, concrete method of adjustment is as follows:
Step 5-1 checks the audio frequency and video buffer performance;
Step 5-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 60%-70%;
Step 5-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%;
For asynchronous district, adjustment algorithm is as follows:
Step 6, the broadcast strategy of adjustment client if frame of video falls behind, is directly abandoning, if frame of video is leading, then takes the strategy of replaying;
Step 7, the transmission ratio of adjustment server end, concrete method of adjustment is as follows:
Step 7-1 checks the audio frequency and video buffer performance;
Step 7-2 if the screen buffer occupancy, is adjusted broadcast strategy more than or equal to 50%, only plays key frame, and it goes without doing adjusts for transmitting terminal; If the screen buffer occupancy is lower than 50%, check the audio buffer occupancy, if, then improving the transmitting terminal frame of video more than or equal to 50%, the audio buffer occupancy sends ratio, the frame of video ratio is improved 70%-85%;
Step 7-3 sends ratio if the occupancy of audio/video frames play buffer, improves the 10%-%20 frame of video all less than 50%.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013101997164A CN103269448A (en) | 2013-05-24 | 2013-05-24 | Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013101997164A CN103269448A (en) | 2013-05-24 | 2013-05-24 | Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103269448A true CN103269448A (en) | 2013-08-28 |
Family
ID=49013053
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013101997164A Pending CN103269448A (en) | 2013-05-24 | 2013-05-24 | Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103269448A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104683823A (en) * | 2013-11-29 | 2015-06-03 | 红板凳科技股份有限公司 | Multi-screen linked audio and video synchronizing system |
| CN107517401A (en) * | 2016-06-15 | 2017-12-26 | 成都鼎桥通信技术有限公司 | multimedia data playing method and device |
| CN113286184A (en) * | 2018-10-17 | 2021-08-20 | 上海赛连信息科技有限公司 | Lip sound synchronization method for respectively playing audio and video on different devices |
| CN114584811A (en) * | 2022-05-09 | 2022-06-03 | 江西师范大学 | Method and system for synchronizing streaming media video based on RTP (real-time transport protocol) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1655547A (en) * | 2004-09-09 | 2005-08-17 | 上海川海信息科技有限公司 | A speed control method in stream media transmission system |
| US20050281246A1 (en) * | 2004-06-22 | 2005-12-22 | Lg Electronics Inc. | Synchronizing video/audio data of mobile communication terminal |
| CN102868939A (en) * | 2012-09-10 | 2013-01-09 | 杭州电子科技大学 | Method for synchronizing audio/video data in real-time video monitoring system |
-
2013
- 2013-05-24 CN CN2013101997164A patent/CN103269448A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050281246A1 (en) * | 2004-06-22 | 2005-12-22 | Lg Electronics Inc. | Synchronizing video/audio data of mobile communication terminal |
| CN1655547A (en) * | 2004-09-09 | 2005-08-17 | 上海川海信息科技有限公司 | A speed control method in stream media transmission system |
| CN102868939A (en) * | 2012-09-10 | 2013-01-09 | 杭州电子科技大学 | Method for synchronizing audio/video data in real-time video monitoring system |
Non-Patent Citations (5)
| Title |
|---|
| 南春辉、李博、武颖: "动态网络环境下的音视频同步技术", 《计算机系统应用》 * |
| 南春辉、李博、武颖: "动态网络环境下的音视频同步技术", 《计算机系统应用》, vol. 21, no. 11, 31 December 2012 (2012-12-31) * |
| 柴若楠、曾文献、张鹏云: "音视频同步技术综述", 《计算机系统应用》 * |
| 王凤纯、鲁静: "基于RTP/RTCP的音视频同步方法研究", 《软件》 * |
| 许延、常义林、刘增基: "一种新的媒体内同步控制算法", 《计算机研究与发展》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104683823A (en) * | 2013-11-29 | 2015-06-03 | 红板凳科技股份有限公司 | Multi-screen linked audio and video synchronizing system |
| CN107517401A (en) * | 2016-06-15 | 2017-12-26 | 成都鼎桥通信技术有限公司 | multimedia data playing method and device |
| CN113286184A (en) * | 2018-10-17 | 2021-08-20 | 上海赛连信息科技有限公司 | Lip sound synchronization method for respectively playing audio and video on different devices |
| CN113286184B (en) * | 2018-10-17 | 2024-01-30 | 上海赛连信息科技有限公司 | Lip synchronization method for respectively playing audio and video on different devices |
| CN114584811A (en) * | 2022-05-09 | 2022-06-03 | 江西师范大学 | Method and system for synchronizing streaming media video based on RTP (real-time transport protocol) |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100579238C (en) | Synchronous playing method for audio and video buffer | |
| US9973345B2 (en) | Calculating and signaling segment availability times for segments of media data | |
| RU2408158C2 (en) | Synchronisation of sound and video | |
| CN105704580B (en) | A kind of video transmission method | |
| CN103546662A (en) | Audio and video synchronizing method in network monitoring system | |
| CN105915904A (en) | Video stream Qos control method for broadband trunking call service | |
| WO2012034442A1 (en) | System and method for realizing synchronous transmission and reception of scalable video coding service | |
| US20100034256A1 (en) | Video frame/encoder structure to increase robustness of video delivery | |
| CN101699867A (en) | Dynamic adjustment method of video data transmission rate | |
| CN102905128A (en) | Code rate controlling method of coding and decoding processor in wireless video transmission process | |
| CN104079870A (en) | Video monitoring method and system for single-channel video and multiple-channel audio frequency | |
| CN102970585B (en) | Method for quick channel switching of streaming media | |
| CN103269448A (en) | Method for achieving synchronization of audio and video on the basis of RTP/RTCP feedback early-warning algorithm | |
| CN104270594A (en) | Data packet sending and receiving method and device | |
| CN104683823A (en) | Multi-screen linked audio and video synchronizing system | |
| CN109040818A (en) | Audio and video synchronization method, storage medium, electronic equipment and system when live streaming | |
| CN103826084A (en) | Audio encoding method | |
| KR20110040687A (en) | Network device, information processing apparatus, stream switching method, information processing method, program, and content distribution system | |
| US9313508B1 (en) | Feeding intra-coded video frame after port reconfiguration in video telephony | |
| TWI491218B (en) | Media relay video communication | |
| CN103024369A (en) | Transmitting end, terminal, system and method for multiplexing of hierarchical coding | |
| CN102377977A (en) | Method, device and system for processing video in video call process | |
| CN109104635A (en) | The method and system of instant delivery screen picture | |
| CN106254963A (en) | A kind of method of real-time synchronization transmission AV signal | |
| CN101273631A (en) | Multi-party video communication media flow control system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130828 |
|
| RJ01 | Rejection of invention patent application after publication |