WO2017067489A1 - 机顶盒音视频同步的方法及装置、存储介质 - Google Patents
机顶盒音视频同步的方法及装置、存储介质 Download PDFInfo
- Publication number
- WO2017067489A1 WO2017067489A1 PCT/CN2016/102775 CN2016102775W WO2017067489A1 WO 2017067489 A1 WO2017067489 A1 WO 2017067489A1 CN 2016102775 W CN2016102775 W CN 2016102775W WO 2017067489 A1 WO2017067489 A1 WO 2017067489A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame
- video
- audio
- output
- rate
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/24—Systems for the transmission of television signals using pulse code modulation
- H04N7/52—Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal
- H04N7/54—Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal the signals being synchronous
Definitions
- the present invention relates to the field of smart set top boxes, and in particular, to a method and device for synchronizing audio and video of a set top box, and a storage medium.
- the current set-top box Due to the demand for high-definition video quality and the resource limitation of the central processing unit (CPU), the current set-top box usually adopts a hardware decoding method, and needs to synchronize the audio and video frames before the decoded audio and video frames are sent to the display module. Processing, that is, controlling the frame rate of the currently sent display frame to ensure that the play time difference of the audio and video frames is small.
- this hardware decoding method has a large error in the time stamp obtained after decoding the audio and video frames, and the synchronization effect of the audio and video is not good.
- embodiments of the present invention are directed to a method and apparatus for synchronizing audio and video of a set top box and a storage medium to improve the effect of audio and video synchronization.
- an embodiment of the present invention provides a method for synchronizing audio and video of a set top box, including:
- the video includes a first frame of audio frames and a first frame of video frames;
- the calculating an ideal frame rate and a compensation frame rate of the sequence of video frames includes:
- the acquiring the repeated frame rate of the video frame sequence according to the ideal frame rate and the compensation frame rate includes:
- the audio frame and the video frame in the current synchronization queue are synchronously processed and output according to the repeated frame rate, including:
- the current head node of the video post-processing module is entered into the current synchronization queue, and the number of repeated frames is equal to the repeated frame rate;
- the repeated frame rate is less than or equal to zero, discarding the current video frame, and synchronously outputting the audio frame in the current synchronization queue and a video frame other than the current video frame;
- the repeated frame rate is equal to the ideal frame rate, synchronously output the audio frame and the video frame in the current synchronization queue at an ideal frame rate;
- the method further includes:
- the method further includes:
- the synchronization scheme is a slow synchronization scheme or a fast synchronization scheme
- the preset first frame audio and video output synchronization scheme is a slow synchronization scheme, first output the first frame video frame, and then synchronously output other video frames in the synchronization queue and the audio frame;
- the preset first frame audio and video output synchronization scheme is a fast synchronization scheme, the first frame audio frame and the first frame video frame are simultaneously output.
- the simultaneously outputting the first frame audio frame and the first frame video frame include:
- the first frame video frame waits, and no output is performed
- the first frame audio frame is not output, and the difference between the timestamp of the first frame audio frame and the timestamp of the first frame video frame is not within the preset time difference,
- One frame When the video frame is faster than the first frame audio frame, the first frame video frame waits, and when the first frame video frame is slower than the first frame audio frame, the first frame video frame is discarded;
- the method further includes: before determining whether to output the first frame audio and video in the current synchronization queue according to the first frame audio and video output identifier bit, the method further includes:
- the operation of determining whether the first frame audio and video in the current synchronization queue is output is performed.
- an embodiment of the present invention provides an apparatus for synchronizing audio and video of a set top box, where the apparatus includes:
- a determining module configured to determine that the first frame audio and video has been output in the current synchronization queue
- An acquiring module configured to: when the determining module determines that the first frame audio and video has been output in the current synchronization queue, acquire a timestamp of a video frame sequence in the current synchronization queue, and perform the timestamp on the timestamp Repairing and predicting; the first frame audio and video includes a first frame audio frame and a first frame video frame;
- a calculation module configured to calculate an ideal frame rate and a compensation frame rate of the sequence of video frames
- the acquiring module is configured to acquire a repetition frame rate of the video frame sequence according to the ideal frame rate and the compensation frame rate, and to perform audio frame and video in the current synchronization queue according to the repeated frame rate.
- the frame is synchronized and output.
- the calculating module is configured to calculate the video by using a low frame rate interpolation frame algorithm and a high frame rate frame loss algorithm according to a current display module refresh rate and a time stamp of the video frame sequence.
- the ideal frame rate of the sequence of frames; the time stamp and audio according to the sequence of video frames The difference between the time stamps of the sequence of frames, the compensated frame rate of the sequence of video frames is calculated.
- the obtaining module is configured to:
- the obtaining module is configured to:
- the current head node of the video post-processing module is entered into the current synchronization queue, and the number of repeated frames is equal to the repeated frame rate;
- the repeated frame rate is less than or equal to zero, discarding the current video frame, and synchronously outputting the audio frame in the current synchronization queue and a video frame other than the current video frame;
- the repeated frame rate is equal to the ideal frame rate, synchronously output the audio frame and the video frame in the current synchronization queue at an ideal frame rate;
- the device further includes: a determining module, configured to determine, according to the first frame audio and video output flag, whether to output the first frame audio and video in the current synchronization queue; if the first frame If the audio and video output identification position is no, it is determined that the output is not the first frame audio and video; if the first frame audio and video output identification position is YES, it is determined that the first frame audio and video is output.
- a determining module configured to determine, according to the first frame audio and video output flag, whether to output the first frame audio and video in the current synchronization queue; if the first frame If the audio and video output identification position is no, it is determined that the output is not the first frame audio and video; if the first frame audio and video output identification position is YES, it is determined that the first frame audio and video is output.
- the acquiring module is further configured to: when the determining module determines that the first frame audio and video is not output in the current synchronization queue, according to the preset first frame audio and video output synchronization The scheme outputs the first frame audio and video; wherein the preset first frame audio and video output synchronization scheme is a slow synchronization scheme or a fast synchronization scheme;
- the acquiring module first outputs the first frame video frame, and then uses other video frames and audio frames in the synchronization queue. Synchronous output;
- the acquiring module simultaneously outputs the first frame audio frame and the first frame video frame.
- the obtaining module is configured to:
- the first frame video frame waits, and no output is performed
- the first frame audio frame is not output, and the difference between the timestamp of the first frame audio frame and the timestamp of the first frame video frame is not within the preset time difference,
- the first frame of the video frame waits, and when the first frame of the video frame is slower than the first frame of the audio frame, the first frame of video is discarded frame;
- the determining module is further configured to:
- the processing module performs the determining the first in the current synchronization queue. Whether the frame audio and video output is output.
- an embodiment of the present invention provides a computer storage medium, where the computer-executable medium is stored, where the computer-executable instructions are used to perform audio-video synchronization of a set-top box provided by the first aspect of the present invention. method.
- the method and device for synchronizing the audio and video of the set top box when determining that the first frame of audio and video has been outputted in the current synchronization queue, acquiring the timestamp of the video frame sequence in the current synchronization queue, And repairing and predicting the timestamp;
- the first frame audio and video includes a first frame audio frame and a first frame video frame; calculating an ideal frame rate and a compensation frame rate of the video frame sequence; a frame rate and the compensation frame rate, acquiring a repetition frame rate of the video frame sequence; and synchronizing and outputting the audio frame and the video frame in the current synchronization queue according to the repeated frame rate; thus, reducing Acquire the error of the time stamp of the audio and video frames, thereby improving the synchronization effect of the audio and video.
- FIG. 1 is a flowchart of a method for synchronizing audio and video of a set top box according to an embodiment of the present invention
- FIG. 2 is a schematic structural diagram of an apparatus for synchronizing audio and video of a set top box according to an embodiment of the present invention.
- the time stamp of the audio frame is used as the time standard.
- the frame rate of the video frame is mainly adjusted according to the time difference value of the audio and video to wait for the audio frame or discard the video frame. Therefore, the synchronization of the video frame and the audio frame is realized, thereby reducing the error of acquiring the time stamp of the audio and video frame, and improving the synchronization effect of the audio and video.
- FIG. 1 is a flowchart of a method for synchronizing audio and video of a set top box according to an embodiment of the present invention. As shown in FIG. 1, the method for synchronizing audio and video of a set top box provided by this embodiment may include:
- Step 101 Determine, when the first frame audio and video is output in the current synchronization queue, obtain a timestamp of the video frame sequence in the current synchronization queue, and repair and predict the timestamp; the first frame
- the audio and video includes a first frame of audio frames and a first frame of video frames.
- the time stamp of the video frame sequence needs to be repaired and predicted.
- the normal pts sequence has an increasing feature, namely Pts1 ⁇ pts2 ⁇ pts3 ⁇ pts4, if the pts sequence that does not satisfy this condition is judged to be an abnormal pts.
- three abnormal forms are described.
- Abnormal form 1 pts1 ⁇ pts2, pts2>pts4>pts3, when the live channel loopback (film source looping, head and tail connection point), this form will occur; this kind of exception needs to clear some internal variables of the synchronization module.
- pts prediction if the interlaced video is deinterlaced, the frame rate is doubled, that is, each original field becomes one frame after interpolation, and the original frame becomes two frames, one frame adopts the original frame.
- the pts are taken as pts, and the pts of the other frame takes the pts mean of the current original frame and the next original frame as the predicted pts.
- the first frame audio and video before determining that the first frame audio and video has been output in the current synchronization queue, it is further determined whether the first frame audio and video is output in the current synchronization queue according to the first frame audio and video output flag; If the position of the one-frame audio and video output flag is no, it is determined that the output is not the first frame audio and video; if the first frame audio and video output flag position is YES, it is determined that the first frame audio and video is output.
- the first frame audio and video output identification position is 1, it indicates that the first frame audio and video has been output.
- the synchronization queue is about to output not the first frame audio and video
- the first frame of audio and video is not output.
- the first frame of audio and video is about to be outputted in the synchronization queue; or when the first frame of the audio and video output flag is set to 0, it indicates that the first frame of audio and video has been Output, set to 1 indicates that the first frame of audio and video is not output.
- This embodiment does not specifically limit this.
- the display queue Before determining whether the first frame audio and video is output in the current synchronization queue according to the first frame audio and video output flag, it is also required to determine whether the number of input frames in the current synchronization queue meets the preset number of frames, and determine the display queue. Whether it is overflowed; when the number of input frames in the current synchronization queue satisfies the preset number of frames, and the display queue does not overflow, performing the determining whether the first frame audio and video in the current synchronization queue is output operating.
- the number of required frames is greater than or equal to 3, and in other cases, the number of frames is required to be greater than or equal to 2.
- the first frame audio and video is output according to a preset first frame audio and video output synchronization scheme, and the current processing flow is ended.
- the preset first frame audio and video output synchronization scheme is preset according to the specific needs of the customer. When it is determined that the first frame audio and video is output in the current synchronization queue, the preset first frame tone may be used.
- the video output synchronization scheme outputs the first frame of audio and video.
- the preset first frame audio and video output synchronization scheme is a slow synchronization scheme or a fast synchronization scheme; if the preset first frame audio and video output synchronization scheme is a slow synchronization scheme, the output is first output.
- the first frame of the video frame, and the other video frames in the synchronization queue are synchronously outputted with the audio frame, that is, the slow synchronization scheme requires that the first frame of the video frame is output immediately, and then passes through the first frame of the video frame.
- the synchronization scheme synchronizes the video frame with the audio frame; if the preset first frame audio and video output synchronization scheme is a fast synchronization scheme, simultaneously outputting the first frame audio frame and the first frame video frame, ie, In the fast synchronization scheme, audio and video are required to be simultaneously output and synchronized at the time of output.
- simultaneously outputting the first frame audio frame and the first frame video frame may include: if the first frame audio frame is not detected, the first frame video frame is waiting, No output is performed; if it is detected that the first frame audio frame has arrived, determining that the first frame audio frame is No output; if the first frame audio frame is not output, and the difference between the timestamp of the first frame audio frame and the timestamp of the first frame video frame is within a preset time difference, the output is a first frame of audio and video, and the first frame audio and video output identification position is no; if the first frame audio frame is not output, and the first frame audio frame has a timestamp and the first frame of the video frame If the difference of the timestamp is not within the preset time difference, when the first frame video frame is faster than the first frame audio frame, the first frame video frame waits, in the first frame video When the frame is slower than the first frame audio frame, discarding the first frame video frame; if the first frame audio frame has been output, outputting the
- the first frame of the video frame is forcibly output.
- Step 102 Calculate an ideal frame rate and a compensation frame rate of the sequence of video frames.
- the ideal frame rate of the video frame sequence is calculated by the low frame rate interpolation frame algorithm and the high frame rate frame loss algorithm;
- the difference between the timestamp of the sequence of video frames and the timestamp of the sequence of audio frames is calculated as the compensated frame rate extra_repeat of the sequence of video frames.
- the high frame rate frame loss algorithm used in this embodiment is: three frames skip the current frame across two cycles, and the display module delay is 20 ms as an example: obtaining a synchronous input queue (video post-processing)
- the three frames of pts are recorded as pts0, pts1, and pts2, respectively corresponding to the synchronous output queue tail frame pts, the synchronous input queue header frame pts, and the synchronous input queue second frame pts; if (pts2-pts0)>2*20ms, Then, the video frame corresponding to pts1 is ready to be discarded; otherwise, it is considered to be a low frame rate source, and the algorithm is exited; if the number of frames in the queue is insufficient for N frames (experience value), the video frame is not allowed to be dropped at this time by inserting a small amount of audio.
- the data implements the effect of the audio waiting for the video, otherwise the current video frame is discarded
- the low frame rate interpolation frame algorithm used in this embodiment it is required to ensure that the delivery frequency cannot exceed the refresh rate of the display module, thereby ensuring each by inserting repeated frames. The valid frame stays on the screen for the right time.
- the ideal frame rate only considers the influence of the refresh rate of the display module, to ensure that the decoding and playback of the video can be smooth, and the synchronization between the audio and video is not considered, the synchronization of the video to the audio is performed on the video frame according to the audio playback speed. Inserting a frame or dropping a frame is called compensating the frame rate extra_repeat.
- Step 103 Acquire an repeated frame rate of the video frame sequence according to the ideal frame rate and the compensation frame rate.
- Step 104 Synchronize and output the audio frame and the video frame in the current synchronization queue according to the repeated frame rate.
- the current head node of the video post-processing module is entered into the current synchronization queue, and the number of repeated frames is equal to the repeated frame rate; if the repeated frame rate is less than or equal to zero, the current a video frame, synchronously outputting the audio frame in the current synchronization queue and a video frame other than the current video frame; if the repeated frame rate is equal to the ideal frame rate, synchronously outputting the current synchronization at an ideal frame rate The audio frame and the video frame in the queue; if the repeated frame rate is greater than the ideal frame rate, inserting the compensation frame rate repeating frame into the video frame, so that the current synchronization queue The audio frame and the video frame are output synchronously.
- the technical solution of the embodiment can reduce the error of acquiring the time stamp of the audio and video frame, and improve the synchronization effect of the audio and video.
- FIG. 2 is a schematic structural diagram of an apparatus for synchronizing audio and video of a set top box according to an embodiment of the present invention. As shown in FIG. 2, the apparatus provided in this embodiment may include:
- the determining module 11 is configured to determine that the first frame audio and video has been output in the current synchronization queue
- the obtaining module 12 is configured to: when the determining module determines that the first frame audio and video has been output in the current synchronization queue, acquire a timestamp of the video frame sequence in the current synchronization queue, and obtain the timestamp Performing repair and prediction; the first frame audio and video includes a first frame audio frame and a first frame video frame;
- the calculating module 13 is configured to calculate an ideal frame rate and a compensation frame rate of the video frame sequence
- the acquiring module 12 is configured to acquire a repetition frame rate of the video frame sequence according to the ideal frame rate and the compensation frame rate, and to perform audio frame and the audio frame in the current synchronization queue according to the repeated frame rate.
- the video frames are synchronized and output.
- the calculating module 13 is configured to calculate an ideal frame rate of the video frame sequence by using a low frame rate interpolation frame algorithm and a high frame rate frame loss algorithm according to a refresh rate of the current display module and a time stamp of the video frame sequence; And calculating a compensated frame rate of the sequence of video frames according to a difference between a timestamp of the sequence of video frames and a timestamp of the sequence of audio frames.
- the obtaining module 12 is configured to add the ideal frame rate and the compensation frame rate to obtain a repeated frame rate of the video frame sequence; if the repeated frame rate is greater than zero, the video post-processing module The current head node enters the current synchronization queue, and the number of repeated frames is equal to the repeated frame rate; if the repeated frame rate is less than or equal to zero, the current video frame is discarded, and the current synchronization team is synchronously output.
- the audio frame in the column and a video frame other than the current video frame if the repeated frame rate is equal to the ideal frame rate, synchronously outputting the audio frame and video in the current synchronization queue at an ideal frame rate a frame; if the repeated frame rate is greater than the ideal frame rate, inserting the frame rate repeating frame into the video frame, so that the audio frame and the video frame in the current synchronization queue Synchronous output.
- the device may further include: a determining module, configured to determine, according to the first frame audio and video output flag, whether to output the first frame audio and video in the current synchronization queue; if the first frame audio and video output identification position is Otherwise, it is determined that the first frame audio and video is not output; if the first frame audio and video output flag position is YES, it is determined that the first frame audio and video is output.
- a determining module configured to determine, according to the first frame audio and video output flag, whether to output the first frame audio and video in the current synchronization queue; if the first frame audio and video output identification position is Otherwise, it is determined that the first frame audio and video is not output; if the first frame audio and video output flag position is YES, it is determined that the first frame audio and video is output.
- the obtaining module 12 may be further configured to: when the determining module 11 determines that the first frame audio and video is not output in the current synchronization queue, output the first according to a preset first frame audio and video output synchronization scheme a frame audio and video; wherein the preset first frame audio and video output synchronization scheme is a slow synchronization scheme or a fast synchronization scheme; if the preset first frame audio and video output synchronization scheme is a slow synchronization scheme, the acquiring module 12: first outputting the first frame video frame, and then outputting the other video frames in the synchronization queue and the audio frame synchronously; if the preset first frame audio and video output synchronization scheme is a fast synchronization scheme, The obtaining module 12 simultaneously outputs the first frame audio frame and the first frame video frame.
- the acquiring module 12 is configured to: if it is detected that the first frame audio frame has not arrived, the first frame video frame waits, and no output is performed; If a frame of the audio frame has arrived, determining whether the first frame of the audio frame is output; if the first frame of the audio frame is not output, and the timestamp of the first frame of the audio frame and the time of the first frame of the video frame If the difference between the stamps is within a preset time difference, the first frame audio and video is output, and the first frame audio and video output identification position is output; if the first frame audio frame is not output, and the When the difference between the timestamp of the one frame of the audio frame and the timestamp of the first frame of the video frame is not within the preset time difference, when the first frame of the video frame is faster than the first frame of the audio frame, The first frame of video frames Waiting, when the first frame video frame is slower than the first frame audio frame, discarding the first frame video frame; if the first frame audio frame has
- the determining module may be further configured to: determine whether the number of input frames in the current synchronization queue meets a preset number of frames, and determine whether the display queue overflows; and when the determining module determines the number of input frames in the current synchronization queue When the preset frame number is satisfied, and the display queue does not overflow, the processing module performs an operation of determining whether the first frame audio and video in the current synchronization queue is output.
- the apparatus for synchronizing audio and video of the set-top box in this embodiment may be used to perform the technical solution of the method embodiment shown in FIG. 1 , and the implementation principle and technical effects thereof are similar, and details are not described herein again.
- the determining module 11, the obtaining module 12 and the computing module 13 may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array located on the user terminal. (FPGA) and other device implementations.
- CPU central processing unit
- MPU microprocessor
- DSP digital signal processor
- FPGA field programmable gate array
- the method for synchronizing audio and video of the set top box is implemented in the form of a software function module, and is sold or used as a standalone product, it may also be stored in a computer readable storage medium.
- the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product stored in a storage medium, including a plurality of instructions.
- a computer device (which may be a personal computer, server, or network device, etc.) is caused to perform all or part of the methods described in various embodiments of the present invention.
- the foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read only memory (ROM), a magnetic disk, or an optical disk.
- program codes such as a USB flash drive, a mobile hard disk, a read only memory (ROM), a magnetic disk, or an optical disk.
- the embodiment of the present invention further provides a computer storage medium, where the computer storage medium stores computer executable instructions, and the computer executable instructions are used to execute the present invention.
- embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention can take the form of a hardware embodiment, a software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) including computer usable program code.
- the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
- the apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.
- These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device.
- the instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.
- the first frame audio and video when determining that the first frame audio and video has been output in the current synchronization queue, acquiring a timestamp of the video frame sequence in the current synchronization queue, and repairing and predicting the timestamp;
- the first frame audio and video includes a first frame audio frame and a first frame video frame; calculating an ideal frame rate and a compensation frame rate of the video frame sequence; acquiring the video according to the ideal frame rate and the compensation frame rate a repetition frame rate of the frame sequence; according to the repeated frame rate, the audio frame and the video frame in the current synchronization queue are synchronously processed and output; thus, the error of the timestamp of acquiring the audio and video frame can be reduced, thereby improving the sound The sync effect of the video.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Television Receiver Circuits (AREA)
Abstract
本发明实施例提供一种机顶盒音视频同步的方法，包括：确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧；计算所述视频帧序列的理想帧率和补偿帧率；根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。本发明实施例同时还提供一种机顶盒音视频同步的装置、存储介质。
Description
本发明涉及智能机顶盒技术领域，尤其涉及一种机顶盒音视频同步的方法及装置、存储介质。
随着宽带技术的普及，数字机顶盒越来越成为智能家居研究的热点。目前数字机顶盒已经被推广应用，因此，机顶盒的音视频同步效果的好坏直接影响到机顶盒用户的用户体验。
由于高清视频质量的需求和中央处理器(Center Processing Unit，CPU)的资源限制，目前的机顶盒通常采用硬件解码的方式，需要在完成解码的音视频帧送到显示模块之前对音视频帧做同步处理，即，控制当前送显帧的帧率，以确保音视频帧的播放时间差较小。然而，这种硬件解码方式对音视频帧解码后获取的时间戳存在较大的误差，音视频的同步效果不好。
发明内容
有鉴于此，本发明实施例期望提供一种机顶盒音视频同步的方法及装置、存储介质，以提高音视频同步的效果。
本发明实施例的技术方案是这样实现的：
第一方面，本发明实施例提供一种机顶盒音视频同步的方法，包括：
确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；其中，所述第一帧音视频包括第一帧音频帧和第一帧视频帧；
计算所述视频帧序列的理想帧率和补偿帧率；
根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；
根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
在本发明的其他实施例中，所述计算所述视频帧序列的理想帧率和补偿帧率，包括：
根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率；
根据所述视频帧序列的时间戳与音频帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率。
在本发明的其他实施例中，所述根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率，包括：
将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率。
在本发明的其他实施例中，所述根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出，包括：
若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；
若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当前同步队列中的所述音频帧和除当前视频帧外的视频帧；
若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；
若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述补偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
在本发明的其他实施例中，所述确定当前同步队列中已输出所述第一
帧音视频之前，所述方法还包括：
根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频；
若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；
若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
在本发明的其他实施例中，所述方法还包括：
确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频；其中，所述预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；
若所述预设第一帧音视频输出同步方案为慢同步方案，则先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧进行同步输出；
若所述预设第一帧音视频输出同步方案为快同步方案，则同时输出所述第一帧音频帧和所述第一帧视频帧。
在本发明的其他实施例中，所述同时输出所述第一帧音频帧和所述第一帧视频帧，包括：
若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；
若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是否输出；
若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为否；
若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧
视频帧比所述第一帧音频帧快时，所述第一帧视频帧等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；
若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为否。
在本发明的其他实施例中，所述根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频之前，所述方法还包括：
判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；
当所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
第二方面，本发明实施例提供一种机顶盒音视频同步的装置，所述装置包括：
确定模块，配置为确定当前同步队列中已输出所述第一帧音视频；
获取模块，配置为在所述确定模块确定所述当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧；
计算模块，配置为计算所述视频帧序列的理想帧率和补偿帧率；
所述获取模块，配置为根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
在本发明的其他实施例中，所述计算模块配置为根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率；根据所述视频帧序列的时间戳与音频
帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率。
在本发明的其他实施例中，所述获取模块配置为：
将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率。
在本发明的其他实施例中，所述获取模块配置为：
若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；
若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当前同步队列中的所述音频帧和除当前视频帧外的视频帧；
若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；
若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述补偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
在本发明的其他实施例中，所述装置还包括：判断模块，配置为根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频；若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
在本发明的其他实施例中，所述获取模块还配置为：在所述确定模块确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频；其中，所述预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；
若所述预设第一帧音视频输出同步方案为慢同步方案，则所述获取模块先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧
进行同步输出；
若所述预设第一帧音视频输出同步方案为快同步方案，则所述获取模块同时输出所述第一帧音频帧和所述第一帧视频帧。
在本发明的其他实施例中，所述获取模块配置为：
若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；
若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是否输出；
若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为否；
若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧视频帧比所述第一帧音频帧快时，所述第一帧视频帧等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；
若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为否。
在本发明的其他实施例中，所述判断模块还配置为：
判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；
当所述判断模块判断所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，所述处理模块执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
第三方面，本发明实施例提供一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，该计算机可执行指令用于执行本发明第一方面实施例提供的机顶盒音视频同步的方法。
本发明实施例提供的机顶盒音视频同步的方法及装置、存储介质，通过确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧；计算所述视频帧序列的理想帧率和补偿帧率；根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出；如此，能减少获取音视频帧的时间戳的误差，从而提高音视频的同步效果。
图1为本发明实施例提供的机顶盒音视频同步的方法的流程图；
图2为本发明实施例提供的机顶盒音视频同步的装置的结构示意图。
在本发明的各实施例中，以音频帧的时间戳作为时间标准，在同步过程中，主要根据音视频的时间戳差值调整视频帧的帧率，以对音频帧进行等待或丢弃视频帧，从而实现视频帧与音频帧的同步，从而减少了获取音视频帧的时间戳的误差，提高了音视频的同步效果。
图1为本发明实施例提供的机顶盒音视频同步的方法的流程图。如图1所示，本实施例提供的机顶盒音视频同步的方法可以包括：
步骤101、确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧。
若当前同步队列输出的不是第一帧音视频，即第一帧音视频已经输出，则需要对视频帧序列的时间戳进行修复和预测。
这里，在播放过程中，正常的pts序列具有不断递增的特点，即
pts1<pts2<pts3<pts4，如果不满足此条件的pts序列都被判断为异常pts。本实施例中举出三种异常形态进行说明。
异常形态1：pts1<pts2,pts2>pts4>pts3，当直播频道出现回环(片源循环播，首尾连接点)，会出现此种形态；这种异常下需要将同步模块一些内部变量清零，表示重新开始播放，不去修复pts；异常形态2：pts1<pts2<pts3,p3>pts4>p2；这种形态下需要修复pts3，修复方法是pts3＝pts2+delay；Delay表示理想的帧间pts差；异常形态3：p3<p1<p2<p4；这种形态下需要修复pts3，修复方法是pts3＝pts2+delay；其中，delay表示理想的帧间pts差，通过统计前100帧的视频前后两帧的pts差值的分布，取最大概率的pts差值作为理想delay。
对于pts预测：如果将隔行视频经过去隔行处理之后，帧率翻倍，即原来的每一场经过插值之后变成了一帧，原来的每一帧就变成两帧，一帧采用原始帧的pts作为pts，另外一帧的pts取当前原始帧与下个原始帧的pts均值作为预测的pts。
本实施例中，在确定当前同步队列中已输出第一帧音视频之前，还需要根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频；若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
例如，可以设置当第一帧音视频输出标识位置为1时，表明所述第一帧音视频已输出，此时当同步队列中即将输出的不是第一帧音视频，置为0时表明所述第一帧音视频未输出，此时当同步队列中即将输出的是第一帧音视频；或者可以设置当第一帧音视频输出标识位置为0时，表明所述第一帧音视频已输出，置为1时表明所述第一帧音视频未输出。本实施例对此不做具体限制。
在根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频之前，还需要判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；当所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
判断当前同步队列中的输入帧数是否满足预设帧数，本实施例中，对于隔行片源逐行输出来说，要求帧数大于等于3，其他情况下则要求帧数大于等于2。
进一步地，确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频，结束本次处理流程。
需要说明的是，该预设第一帧音视频输出同步方案是根据客户的具体需求预先设置好的，当判断出当前同步队列中输出第一帧音视频时，可以按照预设第一帧音视频输出同步方案对第一帧音视频进行输出。
在本发明的其他实施例中，该预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；若所述预设第一帧音视频输出同步方案为慢同步方案，则先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧进行同步输出，即，慢同步方案要求第一帧视频帧立即输出，在输出第一帧视频帧后再通过同步方案将视频帧与音频帧同步；若所述预设第一帧音视频输出同步方案为快同步方案，则同时输出所述第一帧音频帧和所述第一帧视频帧，即，在快同步方案中，要求音视频同时输出且在输出时即同步。
在快同步方案中，同时输出所述第一帧音频帧和所述第一帧视频帧，可以包括：若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是
否输出；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为否；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧视频帧比所述第一帧音频帧快时，所述第一帧视频帧等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为否。
另外需说明的是，如果超过1.5s音视频没有同步或者音频帧没有到来，则将第一帧视频帧强制输出。
步骤102、计算所述视频帧序列的理想帧率和补偿帧率。
本步骤中，根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率idea_repeat；根据所述视频帧序列的时间戳与音频帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率extra_repeat。
在本发明的其他实施例中，本实施例中采用的高帧率丢帧算法为：三帧跨两周期丢当前帧，以显示模块延时20ms为例：获取同步输入队列(视频后处理)顺序出来的三帧pts，记为pts0、pts1、pts2，分别对应同步输出队列尾帧pts、同步输入队列头帧pts、同步输入队列第二帧pts；若(pts2-pts0)>2*20ms，则准备丢掉pts1对应的视频帧；否则认为是低帧率片源，退出算法；若显示队列中压的帧数不够N帧(经验值)，则不允许丢掉视频帧此时通过插入少量的音频数据实现音频等待视频的效果，否则丢掉当前视频帧。
在本发明的其他实施例中，本实施例中采用的低帧率插帧算法中，需要保证送显频率不能超出显示模块的刷新率，从而通过插重复帧保证每个
有效帧在屏幕上停留合适的时间。
由于理想帧率只是考虑了显示模块的刷新率的影响，以确保视频的解码与播放能够流畅，并未考虑音视频之间的同步，将视频同步到音频就是根据音频的播放快慢对视频帧进行插帧或者丢帧，称为补偿帧率extra_repeat。
步骤103、根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率。
将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率all_repeat；例如，all_repeat＝idea_repeat+extra_repeat。
步骤104、根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当前同步队列中的所述音频帧和除当前视频帧外的视频帧；若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述补偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
即，若all_repeat>0，表示视频后处理模块当前头结点入同步队列，重复帧数是all_repeat，如果all_repeat＝idea_repeat表示以理想帧率播放，如果all_repeat>idea_repeat表示视频比音频快，相比较于理想帧率需要另插入extra_repeat个重复帧对帧率进行补偿，确保音视频能够同步；若all_repeat<＝0,表示视频比音频慢，需要将当前帧丢弃，对于当前帧可以插多个重复帧但最多能丢一帧。
在实际应用中，为防止画面出现冻结，当extra_repeat<0即判断丢为丢
帧，此时显示模块的显示队列数据不足，则将extra_repeat＝1；为防止画面卡顿，最多连续丢3帧；为防止重复帧过多画面顿住，对于一个帧节点入同步队列，最多插3个重复帧；为保证播放开始前几帧画面播放流畅，在播放开始的前几帧extra_repeat最多插1个重复帧进行补偿。
本实施例的技术方案，可以减少获取音视频帧的时间戳的误差，提高音视频的同步效果。
图2为本发明实施例提供的机顶盒音视频同步的装置的结构示意图。如图2所示，本实施例提供的装置可以包括：
确定模块11，配置为确定当前同步队列中已输出所述第一帧音视频；
获取模块12，配置为在所述确定模块确定所述当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧；
计算模块13，配置为计算所述视频帧序列的理想帧率和补偿帧率；
所述获取模块12，配置为根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
所述计算模块13，配置为根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率；根据所述视频帧序列的时间戳与音频帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率。
所述获取模块12，配置为将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率；若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当前同步队
列中的所述音频帧和除当前视频帧外的视频帧；若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
进一步地，所述装置还可以包括判断模块，配置为根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频；若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
进一步地，所述获取模块12还可以配置为在确定模块11确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频；其中，所述预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；若所述预设第一帧音视频输出同步方案为慢同步方案，则所述获取模块12先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧进行同步输出；若所述预设第一帧音视频输出同步方案为快同步方案，则所述获取模块12同时输出所述第一帧音频帧和所述第一帧视频帧。
在本发明的其他实施例中，所述获取模块12配置为：若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是否输出；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为已输出；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧视频帧比所述第一帧音频帧快时，所述第一帧视频帧
等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为已输出。
所述判断模块还可以配置为：判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；当所述判断模块判断所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，所述处理模块执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
本实施例的机顶盒音视频同步的装置，可以用于执行上述图1所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
在实际应用中，该确定模块11、获取模块12和计算模块13可由位于用户终端上的中央处理器(CPU)、微处理器(MPU)、数字信号处理器(DSP)或现场可编程门阵列(FPGA)等器件实现。
需要说明的是，本发明实施例中，如果以软件功能模块的形式实现上述的机顶盒音视频同步的方法，并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本发明各个实施例所述方法的全部或部分。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read Only Memory)、磁碟或者光盘等各种可以存储程序代码的介质。这样，本发明实施例不限制于任何特定的硬件和软件结合。
相应地，本发明实施例再提供一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，该计算机可执行指令用于执行本发明实
施例中机顶盒音视频同步的方法。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用硬件实施例、软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器和光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
以上所述，仅为本发明的较佳实施例而已，并非用于限定本发明的保护范围。
本发明实施例中，确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧和第一帧视频帧；计算所述视频帧序列的理想帧率和补偿帧率；根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出；如此，能减少获取音视频帧的时间戳的误差，从而提高音视频的同步效果。
Claims (17)
- 一种机顶盒音视频同步的方法，所述方法包括：确定当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；其中，所述第一帧音视频包括第一帧音频帧和第一帧视频帧；计算所述视频帧序列的理想帧率和补偿帧率；根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
- 根据权利要求1所述的方法，其中，所述计算所述视频帧序列的理想帧率和补偿帧率，包括：根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率；根据所述视频帧序列的时间戳与音频帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率。
- 根据权利要求1所述的方法，其中，所述根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率，包括：将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率。
- 根据权利要求1所述的方法，其中，所述根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出，包括：若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当 前同步队列中的所述音频帧和除当前视频帧外的视频帧；若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述补偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
- 根据权利要求1所述的方法，其中，所述确定当前同步队列中已输出所述第一帧音视频之前，所述方法还包括：根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频；若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
- 根据权利要求1所述的方法，其中，所述方法还包括：确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频；其中，所述预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；若所述预设第一帧音视频输出同步方案为慢同步方案，则先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧进行同步输出；若所述预设第一帧音视频输出同步方案为快同步方案，则同时输出所述第一帧音频帧和所述第一帧视频帧。
- 根据权利要求6所述的方法，其中，所述同时输出所述第一帧音频帧和所述第一帧视频帧，包括：若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是否输出；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为否；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧视频帧比所述第一帧音频帧快时，所述第一帧视频帧等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为否。
- 根据权利要求5所述的方法，其中，所述根据第一帧音视频输出标识位判断所述当前同步队列中是否输出第一帧音视频之前，所述方法还包括：判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；当所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
- 一种机顶盒音视频同步的装置，所述装置包括：确定模块，配置为确定当前同步队列中已输出所述第一帧音视频；获取模块，配置为在所述确定模块确定所述当前同步队列中已输出所述第一帧音视频时，获取所述当前同步队列中的视频帧序列的时间戳，并对所述时间戳进行修复和预测；所述第一帧音视频包括第一帧音频帧 和第一帧视频帧；计算模块，配置为计算所述视频帧序列的理想帧率和补偿帧率；所述获取模块，配置为根据所述理想帧率和所述补偿帧率，获取所述视频帧序列的重复帧率；根据所述重复帧率，对所述当前同步队列中的音频帧和视频帧进行同步处理并输出。
- 根据权利要求9所述的装置，其中，所述计算模块配置为：根据当前显示模块的刷新率和所述视频帧序列的时间戳，通过低帧率插帧算法和高帧率丢帧算法计算所述视频帧序列的理想帧率；根据所述视频帧序列的时间戳与音频帧序列的时间戳之间的差值，计算所述视频帧序列的补偿帧率。
- 根据权利要求9所述的装置，其中，所述获取模块配置为：将所述理想帧率与所述补偿帧率相加，得到所述视频帧序列的重复帧率。
- 根据权利要求9所述的装置，其中，所述获取模块配置为：若所述重复帧率大于零，则将视频后处理模块的当前头结点入所述当前同步队列，且重复帧数等于所述重复帧率；若所述重复帧率小于等于零，则丢弃当前视频帧，同步输出所述当前同步队列中的所述音频帧和除当前视频帧外的视频帧；若所述重复帧率等于所述理想帧率，则以理想帧率同步输出所述当前同步队列中的所述音频帧和视频帧；若所述重复帧率大于所述理想帧率，则向所述视频帧中插入所述补偿帧率个重复帧，以使所述当前同步队列中的所述音频帧和所述视频帧同步输出。
- 根据权利要求9所述的装置，其中，所述装置还包括：判断模块，配置为根据第一帧音视频输出标识位判断所述当前同步队列中是否 输出第一帧音视频；若所述第一帧音视频输出标识位置为否，则判定输出的不是所述第一帧音视频；若所述第一帧音视频输出标识位置为是，则判定输出的是所述第一帧音视频。
- 根据权利要求9所述的装置，其中，所述获取模块还配置为：在所述确定模块确定所述当前同步队列中未输出所述第一帧音视频时，按照预设第一帧音视频输出同步方案输出所述第一帧音视频；其中，所述预设第一帧音视频输出同步方案为慢同步方案或者快同步方案；若所述预设第一帧音视频输出同步方案为慢同步方案，则所述获取模块先输出所述第一帧视频帧，再将所述同步队列中的其他视频帧与音频帧进行同步输出；若所述预设第一帧音视频输出同步方案为快同步方案，则所述获取模块同时输出所述第一帧音频帧和所述第一帧视频帧。
- 根据权利要求14所述的装置，其中，所述获取模块配置为：若检测到所述第一帧音频帧未到，则所述第一帧视频帧等待，不进行输出；若检测到所述第一帧音频帧已到，判断所述第一帧音频帧是否输出；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值在预设时间差范围内，则输出所述第一帧音视频，并将第一帧音视频输出标识位置为否；若所述第一帧音频帧未输出，且所述第一帧音频帧的时间戳与所述第一帧视频帧的时间戳的差值不在所述预设时间差范围内，则在所述第一帧视频帧比所述第一帧音频帧快时，所述第一帧视频帧等待，在所述第一帧视频帧比所述第一帧音频帧慢时，丢弃所述第一帧视频帧；若所述第一帧音频帧已输出，则输出所述第一帧视频帧，并将第一帧音视频输出标识位置为否。
- 根据权利要求13所述的装置，其中，所述判断模块还配置为：判断所述当前同步队列中的输入帧数是否满足预设帧数，并判断显示队列是否溢出；当所述判断模块判断所述当前同步队列中的输入帧数满足所述预设帧数，且所述显示队列未溢出时，所述处理模块执行所述判断所述当前同步队列中的第一帧音视频是否输出的操作。
- 一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，该计算机可执行指令用于执行权利要求1至8任一项所述的机顶盒音视频同步的方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510691001.X | 2015-10-22 | ||
| CN201510691001.XA CN106612452B (zh) | 2015-10-22 | 2015-10-22 | 机顶盒音视频同步的方法及装置 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2017067489A1 true WO2017067489A1 (zh) | 2017-04-27 |
Family
ID=58556676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2016/102775 WO2017067489A1 (zh) | 2015-10-22 | 2016-10-20 | 机顶盒音视频同步的方法及装置、存储介质 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106612452B (zh) |
| WO (1) | WO2017067489A1 (zh) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157228A (zh) * | 2021-02-01 | 2021-07-23 | 中国船舶重工集团公司第七0九研究所 | 一种多源帧率互异型高帧率显示控制装置及方法 |
| CN113949898A (zh) * | 2021-10-13 | 2022-01-18 | 北京奇艺世纪科技有限公司 | 多媒体处理方法、装置、设备和存储介质 |
| CN114268830A (zh) * | 2021-12-06 | 2022-04-01 | 咪咕文化科技有限公司 | 云导播同步方法、装置、设备及存储介质 |
| CN114302169A (zh) * | 2021-12-24 | 2022-04-08 | 威创集团股份有限公司 | 一种画面同步录制方法、装置、系统及计算机存储介质 |
| CN114512139A (zh) * | 2022-04-18 | 2022-05-17 | 杭州星犀科技有限公司 | 多路音频混音的处理方法、系统、混音处理器和存储介质 |
| CN115334344A (zh) * | 2022-08-08 | 2022-11-11 | 青岛海信宽带多媒体技术有限公司 | 一种应用于智能机顶盒的频道切换方法及装置 |
| CN116233472A (zh) * | 2023-05-08 | 2023-06-06 | 湖南马栏山视频先进技术研究院有限公司 | 一种音频视频同步的方法和云端处理系统 |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109711421A (zh) * | 2017-10-25 | 2019-05-03 | 腾讯科技（深圳）有限公司 | 一种数据处理方法和装置 |
| CN108379832B (zh) * | 2018-01-29 | 2021-03-30 | 珠海金山网络游戏科技有限公司 | 一种游戏同步方法和装置 |
| CN108495164B (zh) * | 2018-04-09 | 2021-01-29 | 珠海全志科技股份有限公司 | 音视频同步处理方法及装置、计算机装置及存储介质 |
| CN111355989B (zh) * | 2018-12-21 | 2023-08-08 | 深圳市中兴微电子技术有限公司 | 一种帧率控制方法及相关设备 |
| CN112825562B (zh) * | 2019-11-21 | 2022-11-01 | 杭州海康威视数字技术股份有限公司 | 一种视频帧补偿方法、装置、以及视频处理芯片 |
| CN113596549B (zh) * | 2020-10-13 | 2023-09-22 | 杭州涂鸦信息技术有限公司 | 基于不同参考时钟的音视频同步方法、装置和计算机设备 |
| CN113573119B (zh) * | 2021-06-15 | 2022-11-29 | 荣耀终端有限公司 | 多媒体数据的时间戳生成方法及装置 |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771869A (zh) * | 2008-12-30 | 2010-07-07 | 深圳市万兴软件有限公司 | 一种音视频编解码装置及方法 |
| EP2334049A2 (en) * | 2009-12-14 | 2011-06-15 | QNX Software Systems GmbH & Co. KG | Synchronization of video presentation by video cadence modification |
| CN102685437A (zh) * | 2012-02-03 | 2012-09-19 | 深圳市创维群欣安防科技有限公司 | 视频图像补偿方法及监视器 |
| CN103581730A (zh) * | 2013-10-28 | 2014-02-12 | 南京熊猫电子股份有限公司 | 在数字机顶盒上实现音视频同步的方法 |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101207822A (zh) * | 2006-12-22 | 2008-06-25 | 上海晨兴电子科技有限公司 | 流媒体终端的音频视频同步的方法 |
| CN103167342B (zh) * | 2013-03-29 | 2016-07-13 | 天脉聚源(北京)传媒科技有限公司 | 一种音视频同步处理装置及方法 |
| US20150062353A1 (en) * | 2013-08-30 | 2015-03-05 | Microsoft Corporation | Audio video playback synchronization for encoded media |
| CN104378675B (zh) * | 2014-12-08 | 2019-07-30 | 厦门雅迅网络股份有限公司 | 一种多通道音视频同步播放处理方法 |
| CN104618786B (zh) * | 2014-12-22 | 2018-01-05 | 深圳市腾讯计算机系统有限公司 | 音视频同步方法和装置 |
-
2015
- 2015-10-22 CN CN201510691001.XA patent/CN106612452B/zh active Active
-
2016
- 2016-10-20 WO PCT/CN2016/102775 patent/WO2017067489A1/zh active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771869A (zh) * | 2008-12-30 | 2010-07-07 | 深圳市万兴软件有限公司 | 一种音视频编解码装置及方法 |
| EP2334049A2 (en) * | 2009-12-14 | 2011-06-15 | QNX Software Systems GmbH & Co. KG | Synchronization of video presentation by video cadence modification |
| CN102685437A (zh) * | 2012-02-03 | 2012-09-19 | 深圳市创维群欣安防科技有限公司 | 视频图像补偿方法及监视器 |
| CN103581730A (zh) * | 2013-10-28 | 2014-02-12 | 南京熊猫电子股份有限公司 | 在数字机顶盒上实现音视频同步的方法 |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157228A (zh) * | 2021-02-01 | 2021-07-23 | 中国船舶重工集团公司第七0九研究所 | 一种多源帧率互异型高帧率显示控制装置及方法 |
| CN113157228B (zh) * | 2021-02-01 | 2022-07-05 | 中国船舶重工集团公司第七0九研究所 | 一种多源帧率互异型高帧率显示控制装置及方法 |
| CN113949898A (zh) * | 2021-10-13 | 2022-01-18 | 北京奇艺世纪科技有限公司 | 多媒体处理方法、装置、设备和存储介质 |
| CN113949898B (zh) * | 2021-10-13 | 2024-03-08 | 北京奇艺世纪科技有限公司 | 多媒体处理方法、装置、设备和存储介质 |
| CN114268830A (zh) * | 2021-12-06 | 2022-04-01 | 咪咕文化科技有限公司 | 云导播同步方法、装置、设备及存储介质 |
| CN114268830B (zh) * | 2021-12-06 | 2024-05-24 | 咪咕文化科技有限公司 | 云导播同步方法、装置、设备及存储介质 |
| CN114302169A (zh) * | 2021-12-24 | 2022-04-08 | 威创集团股份有限公司 | 一种画面同步录制方法、装置、系统及计算机存储介质 |
| CN114512139A (zh) * | 2022-04-18 | 2022-05-17 | 杭州星犀科技有限公司 | 多路音频混音的处理方法、系统、混音处理器和存储介质 |
| CN115334344A (zh) * | 2022-08-08 | 2022-11-11 | 青岛海信宽带多媒体技术有限公司 | 一种应用于智能机顶盒的频道切换方法及装置 |
| CN115334344B (zh) * | 2022-08-08 | 2023-08-18 | 青岛海信宽带多媒体技术有限公司 | 一种应用于智能机顶盒的频道切换方法及装置 |
| CN116233472A (zh) * | 2023-05-08 | 2023-06-06 | 湖南马栏山视频先进技术研究院有限公司 | 一种音频视频同步的方法和云端处理系统 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106612452A (zh) | 2017-05-03 |
| CN106612452B (zh) | 2019-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017067489A1 (zh) | 机顶盒音视频同步的方法及装置、存储介质 | |
| CN110139148B (zh) | 一种视频切换清晰度的方法及相关装置 | |
| TWI788744B (zh) | 動態縮減替換內容之播放以幫助對齊替換內容之結束與已替換內容之結束 | |
| US11812103B2 (en) | Dynamic playout of transition frames while transitioning between playout of media streams | |
| JP4983923B2 (ja) | デコーダ装置、およびデコード方法 | |
| WO2018120557A1 (zh) | 一种音视频同步处理的方法和装置、存储介质 | |
| CN106791994B (zh) | 一种低延时快速开播方法及装置 | |
| CN109660805B (zh) | 解码播放中音视频同步优化方法、存储介质、设备及系统 | |
| CN111131874B (zh) | 一种解决h.265码流随机接入点播放卡顿的方法、设备及计算机存储介质 | |
| US9615130B2 (en) | Method and device for processing multimedia frame and storage medium | |
| CN106331820B (zh) | 音视频的同步处理方法和装置 | |
| JP4987034B2 (ja) | 映像表示装置 | |
| JP5539276B2 (ja) | ファイル位置高速検索方法 | |
| WO2014115295A1 (ja) | ビデオ表示装置及びビデオ表示方法 | |
| JP5465278B2 (ja) | 映像表示装置 | |
| CN107087210B (zh) | 基于缓存时间判断视频播放状态的方法及终端 | |
| JP2009218696A (ja) | データ処理装置及びデータ処理方法 | |
| KR20090020324A (ko) | 멀티미디어 재생장치 및 오디오/비디오 데이터의 처리방법 | |
| JP2009060324A (ja) | タイムスタンプ付加装置およびタイムスタンプ付加方法、並びにプログラム |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16856920 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 16856920 Country of ref document: EP Kind code of ref document: A1 |