CN115243089A - Audio and video synchronous rendering method and device and electronic equipment - Google Patents
Audio and video synchronous rendering method and device and electronic equipment Download PDFInfo
- Publication number
- CN115243089A CN115243089A CN202210912502.6A CN202210912502A CN115243089A CN 115243089 A CN115243089 A CN 115243089A CN 202210912502 A CN202210912502 A CN 202210912502A CN 115243089 A CN115243089 A CN 115243089A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- player
- time stamp
- played
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 79
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 36
- 238000006243 chemical reaction Methods 0.000 claims abstract description 17
- 238000012544 monitoring process Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims 1
- 238000004806 packaging method and process Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000001960 triggered effect Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4398—Processing of audio elementary streams involving reformatting operations of audio signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The application discloses an audio and video synchronous rendering method and device and electronic equipment, wherein the method comprises the following steps: decoding a video timestamp and video image data from the video data to be decoded through a decoding sub-thread of the player, and performing format conversion on the audio data to be played through a rotor packaging sub-thread of the player; the main thread of the player uses a playing component to play the audio data to be played after format conversion; monitoring an audio time stamp of currently played audio data through a timeupdate event by a main thread of a player; and rendering the video image data according to the video time stamp and the audio time stamp through a main thread of a player. The fluency of audio and video synchronous rendering is improved, the playing is smoother and smoother, and the pause rate is reduced.
Description
Technical Field
The invention relates to the technical field of video processing, in particular to a method and a device for synchronously rendering audio and video and electronic equipment.
Background
High Efficiency Video Coding (HEVC) is a successor to the ITU-T H.264/MPEG-4AVC standard. Compared with h.264, HEVC has a higher compression rate, which means that the picture quality of h.265 is clearer with the same code rate, and the higher compression rate can use lower storage and transmission costs. The code rate is also called bit rate, and means the number of bits transmitted per second, and the higher the bit rate is, the more data is transmitted per second, and the clearer the image quality is.
Most of the existing Web-end HEVC playing devices in the market adopt a mode of aligning video to audio, and perform synchronous rendering playing by using audio as a reference time axis, so that how to perform accurate synchronous rendering of video becomes a problem to be solved urgently under the condition of normal audio playing.
Disclosure of Invention
The application aims to provide an audio and video synchronous rendering method and device and electronic equipment. The method is used for solving the problem that the video is accurately synchronously rendered without blockage under the condition of normal audio playing.
In a first aspect, an embodiment of the present application provides an audio and video synchronous rendering method, where the method includes:
decoding a video time stamp and video image data from video data to be decoded through a decoding sub-thread of the player, and monitoring an audio time stamp of currently played audio data through a main thread of the player;
and rendering the video image data according to the video time stamp and the audio time stamp through a main thread of a player.
In some possible embodiments, the method further comprises: carrying out format conversion on audio data to be played through a rotor package program of the player;
the main thread of the player plays the audio data to be played after format conversion by using a playing component;
monitoring an audio time stamp of currently played audio data through a main thread of a player, wherein the audio time stamp comprises the following steps:
and the main thread of the player monitors the audio time stamp of the currently played audio data through the timeupdate event.
In some possible embodiments, rendering, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp includes:
and through a main thread of the player, determining that the audio time stamp of the current audio data to be played is the same as the audio time stamp of the last audio data to be played, and directly rendering the video image data.
In some possible embodiments, rendering, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp includes:
determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through a main thread of a player;
and if the video time stamp is less than or equal to the audio time stamp, rendering the video image data.
In some possible embodiments, rendering, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp includes:
determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through a main thread of the player,
and if the video time stamp is larger than the audio time stamp, delaying rendering of the video image data, and rendering the video image data when the video time stamp is smaller than or equal to the audio time stamp currently acquired by the sub thread.
In a second aspect, an embodiment of the present application provides an audio and video synchronization rendering apparatus, where the apparatus includes:
the decoding module is used for decoding a video time stamp and video image data from video data to be decoded through a decoding sub-thread of the player and monitoring an audio time stamp of currently played audio data through a main thread of the player;
and the rendering module is used for rendering the video image data according to the video time stamp and the audio time stamp through a main thread of a player.
In a third aspect, an embodiment of the present application provides an electronic device, including at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform the audio-video synchronous rendering method provided by the first aspect.
In a fourth aspect, an embodiment of the present application provides a computer storage medium, where a computer program is stored, and the computer program is configured to enable a computer to execute the audio and video synchronous rendering method provided in the first aspect.
In the embodiment of the application, in order to solve the problem that videos are accurately rendered synchronously under the condition that the audios are normally played, the embodiment of the application utilizes the performance advantage of the audio timeupdate event callback to replace a conventionally used settimeout timing round training mode to perform audio and video synchronous rendering. The advantage of the callback of the audio timeout event is that the triggering frequency of the audio timeout event can ensure that the audio timeout event is triggered 4-66 times per second, the audio timeout event can be automatically triggered as long as the audio timestamp is updated, the audio timestamp is independently acquired, the process that the audio timestamp passes through is more uniform and fine, the audio timestamp and the video timestamp are more stably obtained, the synchronization is carried out in the fastest time without being influenced by the idle state of a player main thread, video frame accumulation cannot be caused, and the video pause phenomenon caused by the delay of a video in a live broadcast scene is avoided.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of an audio/video synchronous rendering method according to an embodiment of the present application;
fig. 2 is a relational diagram of a decoding sub-thread, a converter-packaging sub-thread and a player main thread in the audio and video synchronous rendering method according to an embodiment of the present application;
fig. 3 is a detailed flowchart schematic diagram of an audio and video synchronous rendering method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an audio/video synchronous rendering device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described in detail and clearly with reference to the accompanying drawings. In the description of the embodiments of the present application, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" in the text is only an association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: three cases of a alone, a and B both, and B alone exist, and in addition, "a plurality" means two or more than two in the description of the embodiments of the present application.
In the description of the embodiments of the present application, the term "plurality" means two or more unless otherwise specified, and other terms and the like should be understood similarly, and the preferred embodiments described herein are only for the purpose of illustrating and explaining the present application, and are not intended to limit the present application, and features in the embodiments and examples of the present application may be combined with each other without conflict.
To further illustrate the technical solutions provided by the embodiments of the present application, the following detailed description is made with reference to the accompanying drawings and the detailed description. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or figures, more or fewer operation steps may be included in the method based on conventional or non-inventive labor. In steps where no necessary causal relationship exists logically, the order of execution of these steps is not limited to the order of execution provided by the embodiments of the present application. The method can be executed in the order of the embodiments or the method shown in the drawings or in parallel in the actual process or the control device.
In view of the problem of how to accurately perform synchronous rendering of video under the condition of normal audio playing in the related art. The application provides an audio and video synchronous rendering method and device and electronic equipment, which can not cause video frame accumulation and avoid video blocking caused by delay of videos in live scenes.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The audio and video synchronous rendering method in the embodiment of the present application is described in detail below with reference to the accompanying drawings. The Video synchronization rendering method is suitable for the Audio and Video synchronization rendering of encoding and playing devices such as a web-end HEVC (Video Coding, VVC, also called H.266), a digital Audio and Video Coding and decoding Standard (AVS), a 2 nd generation digital Audio and Video Coding and decoding Standard (AVS 2 ), a 3 rd generation digital Audio and Video Coding and decoding Standard (AVS 3 ) and the like.
A javascript timer is used in the existing web-side audio and video synchronous rendering technology, and generally sets two parameters, wherein the parameter is a function expected to be executed after expiration time (delay milliseconds); parameter two is the number of milliseconds of delay (one second equals 1000 milliseconds) after which the invocation of the function will occur; the timer may execute a function or a specified piece of code after the timer expires. No matter what the set delay time is, the actual delay time may be longer than the expected value, and there are many reasons for this, so that a synchronization timer may be triggered only once (for example, more than 1s, generally, two video images are considered to be stuck after an interval of more than 200 ms) for a long time, which causes a phenomenon that the played images are stuck, and a large number of decoded video images are accumulated, but since the audio is continuously updated, a large number of expired video image data has to be discarded for audio-video synchronization, and the stuck rate is gradually increased, which affects the visual experience of the user.
Fig. 1 shows a schematic flow diagram of an audio and video synchronous rendering method provided in an embodiment of the present application, including:
step 101: and decoding the video time stamp and the video image data from the video data to be decoded through a decoding sub-thread of the player, and monitoring the audio time stamp of the currently played audio data through a main thread of the player.
The player is provided with a main thread and a plurality of sub-threads, the main thread and the plurality of sub-threads are established in the initial state of the player, the plurality of sub-threads serve as the main thread, data processed by the sub-threads can be sent to the main thread, and audio and video synchronization is carried out through the main thread.
When the application is applied to the audio and video synchronous rendering of the Web-end HEVC player, an audio time axis formed by audio time stamps is taken as a main axis to carry out audio and video synchronous playing, namely, a process of synchronizing video to audio.
The decoding sub-thread first downloads the video stream, decapsulates the audio and video, and obtains the basic information of the audio and video, such as the length, the width, the video format, the video length, and the like, through the decapsulation. The resulting decapsulated h.265 video data needs to be further decoded to obtain video image data and a video timestamp for this video frame.
And the decoding sub-thread decodes the video data to be decoded, and calculates to obtain the video timestamp and the video image data. The video image data refers to YUV data of a video image, the YUV data and the video timestamp of the video image are decoded by the decoding thread, and the video timestamp and the YUV data are stored in a queue to wait for the use of the main thread.
As an optional implementation, the method further comprises: carrying out format conversion on audio data to be played through a rotor package sub-thread of the player; the main thread of the player uses a playing component to play the audio data to be played after format conversion; monitoring an audio time stamp of currently played audio data through a main thread of a player, wherein the audio time stamp comprises the following steps: and the main thread of the player monitors the audio time stamp of the currently played audio data through the timeupdate event.
The method comprises the steps that a rotor packaging sub-thread of a player obtains current audio data to be played, the audio data are converted and packaged into audio data in an mp4a format which can be identified by audio elements (element labels for playing audio on a browser), the audio data are transmitted to the audio elements, the main thread of the player uses a playing assembly to play the audio data to be played after format conversion, and the playing assembly is the audio elements. Currenttime update triggers a timeupdate event to get an audio timestamp. That is, the audio timestamp needs to be obtained through an audio _ currenttime attribute, where the attribute is to call an audio timestamp returned once, and the main thread of the player listens to the audio timestamp of the currently played audio data through a timeupdate event of the audio element.
Step 102: and rendering the video image data according to the video time stamp and the audio time stamp through a main thread of the player.
Specifically, after the decoding sub-thread processes the video data to be decoded to obtain the video time stamp and the video image data, and the main thread of the player monitors the audio time stamp of the currently played audio data, the main thread of the player renders the video image data.
As an optional implementation, rendering, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp includes: and through a main thread of the player, determining that the audio time stamp of the current audio data to be played is the same as the audio time stamp of the last audio data to be played, and directly rendering the video image data.
Specifically, if the audio time stamp of the audio data to be played currently is the same as the audio time stamp of the audio data played last, that is, the audio time stamp of the audio is not changed, the video image data is directly and continuously rendered.
As an optional implementation manner, rendering, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp includes: determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through a main thread of the player;
rendering the video image data if the video timestamp is less than or equal to the audio timestamp;
and if the video time stamp is larger than the audio time stamp, delaying rendering of the video image data, and rendering the video image data when the video time stamp is smaller than or equal to the audio time stamp currently acquired by the sub thread.
Specifically, if the audio time stamp of the currently-to-be-played audio data is different from the audio time stamp of the audio data played last, the sizes of the video time stamp and the audio time stamp need to be determined.
And the audio and video synchronous rendering adopts a strategy of synchronizing video to audio, namely normally playing the audio, continuously checking the video frame of the head of the queue in the video queue to be rendered, comparing the video timestamp of the video frame with the current audio timestamp to be played, and rendering the video image data corresponding to the video frame of the head of the queue to be rendered if the video timestamp is less than or equal to the audio timestamp. And if not, re-entering the audio timeupdate event callback processing function to acquire the audio time stamp in real time, and rendering the video screen image data corresponding to the frame of video when the condition is met (the video time stamp is less than or equal to the audio time stamp currently acquired by the sub-thread). If the queue to be rendered is empty, then a fixed time (e.g., 10 ms) is waited to resume the aforementioned operations.
Fig. 2 is a relational graph of a decoding sub-thread, a rotor packaging sub-thread and a player main thread in the audio and video synchronous rendering method;
the decoding sub-thread is used for decoding a video time stamp and YUV data of a video image from the video data to be decoded;
the rotor package thread is used for carrying out format conversion on the audio data to be played if the video to be played is the audio data which is not supported by the AudioElement.
And the main thread of the player is used for monitoring the audio time stamp of the currently played audio data through the timeupdate event and rendering the video image data according to the video time stamp and the audio time stamp.
Fig. 3 is a detailed flowchart of an audio and video synchronous rendering method.
Step 301, decoding a video timestamp and video image data from video data to be decoded by a decoding sub-thread of a player;
307, delaying rendering of the video image data, and returning to the step 301;
The method and the device utilize the performance advantage of the callback of the audio timeupdate event to replace the conventional settimeout timing round training mode for audio and video synchronous rendering. The advantage of the callback of the audio timeupdate event lies in that the triggering frequency of the audio timeupdate event can ensure that the audio timeupdate event is triggered 4-66 times per second, the audio timeupdate event can be automatically triggered as long as the audio timestamp is updated, the audio timestamp is independently acquired, the process that the audio timestamp passes through is carried out more uniformly and finely, more stably, the audio timestamp is obtained, the video timestamp is synchronized in the fastest time without being influenced by the idle state of the main thread of the player, video frame accumulation can not be caused, the smoothness of audio and video synchronous rendering is further enhanced, the video pause phenomenon caused by the delay of the video in a live scene is avoided, the playing is smoother, and the visual experience of a pause rate user is better.
Example 2
Based on the same inventive concept, the present application further provides an audio/video synchronous rendering device, as shown in fig. 4, the device includes:
the decoding module 401 is configured to decode a video timestamp and video image data from video data to be decoded through a decoding sub-thread of the player, and monitor an audio timestamp of currently played audio data through a main thread of the player;
a rendering module 402, configured to render, by a main thread of a player, the video image data according to the video timestamp and the audio timestamp.
Optionally, the apparatus further comprises:
a format conversion module 403, configured to perform format conversion on the audio data to be played through a converter-wrapper thread of the player;
a playing module 404, configured to play the format-converted audio data to be played by using a playing component by a main thread of the player;
the decoding module 401 is specifically configured to monitor an audio timestamp of currently played audio data through a timeupdate event by the main thread of the player.
Optionally, the rendering module 402 is specifically configured to: and determining that the audio time stamp of the current audio data to be played is the same as the audio time stamp of the last audio data to be played through a main thread of the player, and directly rendering the video image data.
Optionally, the rendering module 402 is specifically configured to: determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through a main thread of the player;
and if the video time stamp is less than or equal to the audio time stamp, rendering the video image data.
Optionally, the rendering module 402 is specifically configured to: determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through a main thread of the player,
if the video time stamp is larger than the audio time stamp, the video image data is delayed to be rendered, the audio time stamp is monitored in real time, and the video image data is rendered when the video time stamp is smaller than or equal to the audio time stamp currently acquired by the sub thread.
After introducing the audio-video synchronous rendering method and apparatus according to the exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application is introduced next.
As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. The memory stores program code, and when the program code is executed by the processor, the program code causes the processor to execute the steps of the audio and video synchronous rendering method according to various exemplary embodiments of the present application described above in the present specification.
The electronic device 130 according to this embodiment of the present application, i.e., the above-described av synchronized rendering device, is described below with reference to fig. 5. The electronic device 130 shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 5, the electronic device 130 is represented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 that couples various system components including the memory 132 and the processor 131.
The memory 132 may include readable media in the form of volatile memory, such as Random Access Memory (RAM) 1321 and/or cache memory 1322, and may further include Read Only Memory (ROM) 1323.
The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 130, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 130 to communicate with one or more other electronic devices. Such communication may occur via input/output (I/O) interfaces 135. Also, the electronic device 130 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 over the bus 133. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In some possible embodiments, various aspects of an av synchronized rendering method provided by the present application may also be implemented in the form of a program product, which includes program code for causing a computer device to perform the steps of an av synchronized rendering method according to various exemplary embodiments of the present application described above in this specification, when the program product is run on the computer device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product for monitoring of the embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present application is not so limited, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external electronic devices (e.g., through the internet using an internet service provider).
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and block diagrams, and combinations of flows and blocks in the flow diagrams and block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. An audio and video synchronous rendering optimization method is characterized by comprising the following steps:
decoding a video timestamp and video image data from video data to be decoded through a decoding sub-thread of the player;
monitoring an audio time stamp of currently played audio data through a main thread of a player;
and rendering the video image data according to the video time stamp and the audio time stamp through a main thread of the player.
2. The method of claim 1, wherein the method further comprises:
carrying out format conversion on audio data to be played through a rotor package program of the player;
the main thread of the player uses a playing component to play the audio data to be played after format conversion;
monitoring an audio time stamp of currently played audio data through a main thread of a player, wherein the monitoring comprises the following steps:
and the main thread of the player monitors the audio time stamp of the currently played audio data through the timeupdate event.
3. The method of claim 1, wherein rendering the video image data according to the video timestamp and the audio timestamp by a main thread of the player comprises:
and through the main thread of the player, determining that the audio time stamp of the current audio data to be played is the same as the audio time stamp of the last audio data to be played, and directly rendering the video image data.
4. The method of claim 1, wherein rendering the video image data according to the video timestamp and the audio timestamp by a main thread of the player comprises:
determining that the audio time stamp of the current audio data to be played is different from the audio time stamp of the last audio data to be played through the main thread of the player;
and if the video time stamp is less than or equal to the audio time stamp, rendering the video image data.
5. The method of claim 1, wherein rendering the video image data according to the video timestamp and the audio timestamp by a main thread of the player comprises:
determining, by a main thread of the player, that an audio timestamp of audio data to be currently played is different from an audio timestamp of audio data that was last played,
and if the video time stamp is larger than the audio time stamp, delaying rendering of the video image data, and rendering the video image data when the video time stamp is smaller than or equal to the audio time stamp currently acquired by the sub thread.
6. An audio-video synchronized rendering apparatus, the apparatus comprising:
the decoding module is used for decoding a video timestamp and video image data from the video data to be decoded through a decoding sub-thread of the player; monitoring an audio time stamp of currently played audio data through a main thread of a player;
and the rendering module is used for rendering the video image data according to the video time stamp and the audio time stamp through a main thread of the player.
7. The apparatus of claim 6, wherein the apparatus further comprises:
the format conversion module is used for carrying out format conversion on the audio data to be played through a converter package thread of the player;
the playing module is used for playing the audio data to be played after format conversion by using a playing component by a main thread of the player;
the decoding module is specifically configured to monitor an audio timestamp of currently played audio data through a timeupdate event by the main thread of the player.
8. The apparatus of claim 6, wherein the rendering module is specifically configured to:
and through the main thread of the player, determining that the audio time stamp of the current audio data to be played is the same as the audio time stamp of the last audio data to be played, and directly rendering the video image data.
9. An electronic device comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
10. A computer storage medium, characterized in that the computer storage medium stores a computer program for causing a computer to perform the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210912502.6A CN115243089B (en) | 2022-07-30 | 2022-07-30 | Audio and video synchronous rendering method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210912502.6A CN115243089B (en) | 2022-07-30 | 2022-07-30 | Audio and video synchronous rendering method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115243089A true CN115243089A (en) | 2022-10-25 |
| CN115243089B CN115243089B (en) | 2024-01-02 |
Family
ID=83677477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210912502.6A Active CN115243089B (en) | 2022-07-30 | 2022-07-30 | Audio and video synchronous rendering method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115243089B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090284601A1 (en) * | 2008-05-15 | 2009-11-19 | Jayakrishnan Kumar Eledath | Apparatus for intelligent and autonomous video content generation and streaming |
| EP2334077A1 (en) * | 2009-11-03 | 2011-06-15 | Research in Motion Limited | System and method for dynamic post-processing on a mobile device |
| CN102868939A (en) * | 2012-09-10 | 2013-01-09 | 杭州电子科技大学 | Method for synchronizing audio/video data in real-time video monitoring system |
| CN108377409A (en) * | 2018-03-05 | 2018-08-07 | 广东欧珀移动通信有限公司 | A kind of seamless play method, terminal device and the storage medium of multimedia file |
| CN109327899A (en) * | 2018-10-11 | 2019-02-12 | 恒大法拉第未来智能汽车（广东）有限公司 | A kind of method, apparatus and system synchronization time of mobile unit synchronization time |
| CN110533969A (en) * | 2019-08-05 | 2019-12-03 | 深圳市编玩边学教育科技有限公司 | A kind of teaching programming end and system |
| CN111641838A (en) * | 2020-05-13 | 2020-09-08 | 深圳市商汤科技有限公司 | Browser video playing method and device and computer storage medium |
| CN113225598A (en) * | 2021-05-07 | 2021-08-06 | 上海一谈网络科技有限公司 | Method, device and equipment for synchronizing audio and video of mobile terminal and storage medium |
| CN114339415A (en) * | 2021-12-23 | 2022-04-12 | 天翼云科技有限公司 | Client video playing method and device, electronic equipment and readable medium |
| CN114449162A (en) * | 2021-12-22 | 2022-05-06 | 天翼云科技有限公司 | Method and device for playing panoramic video, computer equipment and storage medium |
-
2022
- 2022-07-30 CN CN202210912502.6A patent/CN115243089B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090284601A1 (en) * | 2008-05-15 | 2009-11-19 | Jayakrishnan Kumar Eledath | Apparatus for intelligent and autonomous video content generation and streaming |
| EP2334077A1 (en) * | 2009-11-03 | 2011-06-15 | Research in Motion Limited | System and method for dynamic post-processing on a mobile device |
| CN102868939A (en) * | 2012-09-10 | 2013-01-09 | 杭州电子科技大学 | Method for synchronizing audio/video data in real-time video monitoring system |
| CN108377409A (en) * | 2018-03-05 | 2018-08-07 | 广东欧珀移动通信有限公司 | A kind of seamless play method, terminal device and the storage medium of multimedia file |
| CN109327899A (en) * | 2018-10-11 | 2019-02-12 | 恒大法拉第未来智能汽车（广东）有限公司 | A kind of method, apparatus and system synchronization time of mobile unit synchronization time |
| CN110533969A (en) * | 2019-08-05 | 2019-12-03 | 深圳市编玩边学教育科技有限公司 | A kind of teaching programming end and system |
| CN111641838A (en) * | 2020-05-13 | 2020-09-08 | 深圳市商汤科技有限公司 | Browser video playing method and device and computer storage medium |
| CN113225598A (en) * | 2021-05-07 | 2021-08-06 | 上海一谈网络科技有限公司 | Method, device and equipment for synchronizing audio and video of mobile terminal and storage medium |
| CN114449162A (en) * | 2021-12-22 | 2022-05-06 | 天翼云科技有限公司 | Method and device for playing panoramic video, computer equipment and storage medium |
| CN114339415A (en) * | 2021-12-23 | 2022-04-12 | 天翼云科技有限公司 | Client video playing method and device, electronic equipment and readable medium |
Non-Patent Citations (2)
| Title |
|---|
| TONI LIVAJA ET AL.: "A method for synchronization of audio and video signals", 《2018 IEEE 8TH INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - BERLIN (ICCE-BERLIN)》, pages 1 - 5 * |
| 马宏雷 等: "非对称无线传输信道的音视频同步优化设计", 《计算机仿真》, vol. 33, no. 02, pages 221 - 226 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115243089B (en) | 2024-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10154320B2 (en) | Dynamic time synchronization | |
| US20210168418A1 (en) | Method and device for controlling video transcoding code rate | |
| US8613013B2 (en) | Ad splicing using re-quantization variants | |
| WO2019170073A1 (en) | Media playback | |
| US20070147517A1 (en) | Video processing system capable of error resilience and video processing method for same | |
| CN101710997A (en) | MPEG-2 (Moving Picture Experts Group-2) system based method and system for realizing video and audio synchronization | |
| JPH11122608A (en) | Picture coding synchronization circuit and its method | |
| CN104363411A (en) | Video playing method and video playing device in video monitoring system | |
| US20240056637A1 (en) | Performing Trick Plays of Segmented Video Content | |
| CN113938470A (en) | Method and device for playing RTSP data source by browser and streaming media server | |
| CN107613302B (en) | Decoding method and device, storage medium and processor | |
| CN113141525B (en) | Online video cut-off continuous playing method and system | |
| CN115243089B (en) | Audio and video synchronous rendering method and device and electronic equipment | |
| CN103179449A (en) | Media file playing method, electronic device and virtual machine framework | |
| US20220256215A1 (en) | Systems and methods for adaptive output | |
| CN110855645B (en) | Streaming media data playing method and device | |
| US20100076944A1 (en) | Multiprocessor systems for processing multimedia data and methods thereof | |
| CN1826802A (en) | Method and system for detecting weak or invalid signals in data streams | |
| CN115278288B (en) | Display processing method and device, computer equipment and readable storage medium | |
| CN114401445B (en) | Video processing control method, device, monitoring equipment, client and storage medium | |
| CN113824715B (en) | Method and device for playing real-time video stream | |
| US20220329903A1 (en) | Media content distribution and playback | |
| CN117998149A (en) | Method, device and equipment for controlling review of video monitoring on-demand file | |
| CN117834992A (en) | Method, equipment and storage medium for playing Web player at multiple times of speed | |
| CN115776567A (en) | Video data stream decoding method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |