CN112218115A - Control method and device for streaming media audio and video synchronization and computer equipment - Google Patents
Control method and device for streaming media audio and video synchronization and computer equipment Download PDFInfo
- Publication number
- CN112218115A CN112218115A CN202011021295.2A CN202011021295A CN112218115A CN 112218115 A CN112218115 A CN 112218115A CN 202011021295 A CN202011021295 A CN 202011021295A CN 112218115 A CN112218115 A CN 112218115A
- Authority
- CN
- China
- Prior art keywords
- audio data
- audio
- timestamp information
- application program
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23106—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The application relates to a control method and device for streaming media audio and video synchronization, computer equipment and a storage medium. The method comprises the following steps: when an audio signal is acquired, triggering a controller to start carrying audio data corresponding to the audio signal; when the transportation is finished, acquiring timestamp information corresponding to the audio data, wherein the timestamp information is determined based on the time when the hardware generates the interrupt; caching the audio data and the corresponding timestamp information to a cache library; and when the application program acquires the audio data from the cache library, sending the audio data carrying the timestamp information to the application program. By adopting the method, the application program can be ensured to normally carry out synchronous processing of audio and video, thereby effectively solving the problem that the audio and video in the streaming media can not be synchronized.
Description
Technical Field
The present application relates to the multimedia technology field, and in particular, to a method and an apparatus for controlling audio and video synchronization of a streaming media, a computer device, and a storage medium.
Background
With the development of multimedia technology, streaming media push stream not only has vigorous requirements on traditional applications, but also has new requirements in industries such as video conferences, distance education, digital courts and the like, and has higher requirements on terminals, for example, a digital court user expects that a manufacturer can provide a real-time optical disc recording function, and when the judgment is finished, the recorded optical disc is immediately taken out for sealing. Users expect that multimedia applications provide higher definition and more paths of streaming media streaming capabilities to meet different application scene requirements of users.
However, in the currently commonly used multi-core embedded system, because the performance of the CPU of the embedded system is limited, the load of the CPU is also increasing, which causes a pause phenomenon in the system, the application program does not acquire the audio packets in time, the audio packets are accumulated in the driver, and the audio and video cannot be synchronized easily.
Disclosure of Invention
Therefore, it is necessary to provide a control method, an apparatus, a computer device and a storage medium for streaming media audio/video synchronization, which can ensure synchronous transmission of audio/video data of streaming media.
A control method for audio and video synchronization of streaming media comprises the following steps:
when an audio signal is acquired, triggering a controller to start carrying audio data corresponding to the audio signal;
when the transportation is finished, acquiring timestamp information corresponding to the audio data, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
caching the audio data and the corresponding timestamp information to a cache library;
and when the application program acquires the audio data from the cache library, sending the audio data carrying the timestamp information to the application program.
In one embodiment, before the audio signal is acquired, the method further includes:
updating the audio parameters to obtain updated audio parameters; the audio parameters include a channel number;
prior to the caching the audio data and the corresponding timestamp information into a cache bank, the method further comprises:
and adding the timestamp information to an idle channel corresponding to the audio data.
In one embodiment, before the triggering controller starts to carry the audio data corresponding to the audio signal, the method further includes:
determining the data volume corresponding to each sampling point according to the audio parameters; the audio parameters comprise sampling frequency, sampling digit and channel number;
acquiring the number of sampling points in a time interval according to the time interval when hardware is interrupted;
and determining the audio data volume to be carried according to the data volume corresponding to each sampling point and the number of the sampling points.
In one embodiment, the caching the audio data and the corresponding timestamp information into a cache library includes:
and caching the audio data and the timestamp information to a cache library according to the sequence of receiving the audio signals.
In one embodiment, after acquiring the timestamp information corresponding to the audio data when the transport is completed, the method further includes:
detecting the number of channels corresponding to the audio data;
when detecting that a redundant channel exists in a channel corresponding to the audio data, storing timestamp information corresponding to the audio data into the redundant channel;
and when detecting that no redundant channel exists in the channel corresponding to the audio data, updating the number of the channels corresponding to the audio data, and storing the timestamp information corresponding to the audio data into the updated channel.
In one embodiment, the obtaining of the timestamp information corresponding to the audio data when the transport is completed includes:
and triggering and generating an interruption of the completion of the transportation when the transportation is completed, wherein the interruption is generated by hardware based on a fixed frequency signal.
A control device for audio-video synchronization of streaming media, the device comprising:
the carrying module is used for triggering the controller to start carrying the audio data corresponding to the audio signal when the audio signal is obtained;
the acquisition module is used for acquiring timestamp information corresponding to the audio data when the transport is finished, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
the cache module is used for caching the audio data and the corresponding timestamp information to a cache library;
and the sending module is used for sending the audio data carrying the timestamp information to the application program when the application program obtains the audio data from the cache library.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
when an audio signal is acquired, triggering a controller to start carrying audio data corresponding to the audio signal;
when the transportation is finished, acquiring timestamp information corresponding to the audio data, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
caching the audio data and the corresponding timestamp information to a cache library;
and when the application program acquires the audio data from the cache library, sending the audio data carrying the timestamp information to the application program.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
when an audio signal is acquired, triggering a controller to start carrying audio data corresponding to the audio signal;
when the transportation is finished, acquiring timestamp information corresponding to the audio data, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
caching the audio data and the corresponding timestamp information to a cache library;
and when the application program acquires the audio data from the cache library, sending the audio data carrying the timestamp information to the application program.
According to the control method and device for the audio and video synchronization of the streaming media, the computer equipment and the storage medium, when the audio signal is obtained, the controller is triggered to start to carry the audio data corresponding to the audio signal. And when the transportation is finished, acquiring time stamp information corresponding to the audio data, wherein the time stamp information is determined based on the time when the hardware generates the interrupt, and caching the audio data and the corresponding time stamp information into a cache library. And when the application program acquires the audio data from the buffer library, sending the audio data carrying the timestamp information to the application program. Therefore, even if the system is jammed and the application program continuously acquires the audio data, the timestamp information cached in the audio data is determined based on the time of interruption generated by hardware, so that the timestamp information can truly reflect the interval of the audio data, the application program can be ensured to normally carry out audio and video synchronous processing, and the problem that audio and video cannot be synchronized in the streaming media is effectively solved.
Drawings
Fig. 1 is a schematic flow chart of a control method for audio and video synchronization of streaming media in an embodiment;
FIG. 2 is a flowchart illustrating the step of detecting the number of channels corresponding to audio data according to an embodiment;
fig. 3A is a schematic flowchart of a control method for audio and video synchronization of streaming media in another embodiment;
fig. 3B is a schematic flow chart of system logic processing for audio and video synchronization of streaming media in an embodiment;
fig. 4 is a block diagram of a control device for audio and video synchronization of streaming media in one embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In an embodiment, as shown in fig. 1, a control method for audio and video synchronization of streaming media is provided, and this embodiment is illustrated by applying the method to a terminal, it is to be understood that the method may also be applied to a server, and may also be applied to a system including the terminal and the server, and is implemented by interaction between the terminal and the server. In this embodiment, the method includes the steps of:
and 102, when the audio signal is acquired, triggering the controller to start to convey the audio data corresponding to the audio signal.
With the development of multimedia technology, streaming media streaming is not only in vigorous demand in traditional applications, but also in industries such as video conferences, distance education, digital courtrooms and the like, and has higher requirements on the system, and users expect that the system can provide streaming media streaming capability with higher definition and more channels so as to meet the requirements of different application scenes of the users. For example, currently common streaming media applications include, but are not limited to, live fighting fish, live tiger-teeth, Tencent video, enterprise WeChat, etc., which support the function of barrage messages. The user can also utilize the platform to carry out operations such as video live conference and the like through different types of intelligent mobile office platforms. For example, an enterprise may register with different intelligent mobile office platforms by filling in information such as enterprise name, industry type, personnel size, administrative password, contact name, etc. The administrator can import the address book of the enterprise employee in the management background, and the employee can receive the activation short message. After the staff completes the filling of the real name and the verification of the mobile phone number, the staff can log in and use the functions of the video live conference and the like in the system, such as common platform nails with the video live conference function.
The user can start a certain application program into a page corresponding to the application program by clicking the specific application program in the main interface of the mobile terminal device, or the user can directly log in a certain specific platform page by triggering operation, and the user can select a corresponding video live broadcast scene according to different requirements. Specifically, after a user starts live broadcasting through a trigger operation, when the terminal acquires an audio signal through the audio controller, the trigger controller starts to carry audio data corresponding to the audio signal. Streaming Media (Streaming Media) is a technology for compressing a series of Media data and then transmitting the compressed Media data in a Streaming manner in a network in a segmented manner to realize real-time transmission of video and audio for viewing on the network. That is, streaming media refers to a new media delivery method and may include audio stream, video stream, text stream, image stream, animation stream, etc. Audio signals (audio signals) are regular carriers of frequency, amplitude variation information with speech, music and sound effects. The audio signal in this application refers to converts the analog signal who gathers into corresponding digital signal through analog-to-digital conversion chip, sends digital signal to audio controller. In the driving layer, when the audio controller acquires the audio signal, namely a digital signal, the controller is triggered to start to convey audio data corresponding to the audio signal. The audio controller collects audio data corresponding to the audio signal into the central processing unit.
And 104, acquiring time stamp information corresponding to the audio data when the transportation is finished, wherein the time stamp information is determined based on the time when the hardware generates the interrupt.
When the controller finishes the transportation, the time stamp information corresponding to the audio data of the current transportation is acquired, and the time stamp information is determined based on the time when the hardware generates the interrupt. The time stamp refers to data generated by using a digital signature technology, and a signature object comprises information such as original file information, signature parameters, signature time and the like. The time stamp of the audio-video data may be divided into a Decoding Time Stamp (DTS) and a Presentation Time Stamp (PTS), i.e., time stamps indicating the decoding and presentation of a certain frame data with respect to a start time, respectively. Because the audio parameters are preset by the user, when the audio parameters are determined, the audio signals are generated according to the independent clock. The independent clock refers to hardware interrupt generated by a hardware device according to a preset frequency, the time of the independent clock is not connected with a system, and the independent clock is generated by the hardware device, so that the fluctuation of the acquisition time caused by the load change of the system is avoided. Specifically, in the system kernel space, when the controller completes the transport, an interrupt is triggered to be generated, and the system central processing unit acquires the timestamp information corresponding to the current audio data based on the hardware interrupt, that is, the timestamp information is determined based on the time when the hardware generates the interrupt.
After the terminal acquires the timestamp information corresponding to the audio data, the terminal caches the acquired audio data and the corresponding timestamp information to a cache library. Specifically, in the kernel space, after the terminal acquires the timestamp information corresponding to the audio data through the system function, the terminal stores the acquired timestamp information in the description information of the corresponding audio data. For example, the terminal may add the timestamp information to a redundant channel of the audio data and cache the audio data and corresponding timestamp information to a cache library.
And step 108, when the application program obtains the audio data from the buffer library, sending the audio data carrying the timestamp information to the application program.
After the terminal caches the audio data and the corresponding timestamp information in the cache library, the application program can obtain the audio data through the API. Among them, the API (Application Programming Interface) is some predefined functions or appointments for linking different components of the software system. Specifically, in the application space, when different application programs obtain audio data from the buffer library, the terminal sends the audio data carrying the timestamp information to the corresponding application program.
In this embodiment, when the audio signal is acquired, the controller is triggered to start to carry the audio data corresponding to the audio signal. And when the transportation is finished, acquiring time stamp information corresponding to the audio data, wherein the time stamp information is determined based on the time when the hardware generates the interrupt, and caching the audio data and the corresponding time stamp information into a cache library. And when the application program acquires the audio data from the buffer library, sending the audio data carrying the timestamp information to the application program. Therefore, even if the system is jammed and the application program continuously acquires the audio data, the time stamp information cached in the audio data is determined based on the time of interruption generated by hardware, so that the time stamp information can truly reflect the interval of the audio data, the application program can be ensured to normally carry out audio and video synchronous processing, and the problem that audio and video cannot be synchronized in the streaming media is effectively solved.
In an embodiment, before the audio signal is acquired, the method further includes a step of updating the audio parameter, which specifically includes:
and updating the audio parameters to obtain the updated audio parameters, wherein the audio parameters comprise the number of channels.
And adding the time stamp information to the idle channel corresponding to the audio data.
Before the terminal acquires the audio signal or when the application program acquires the audio data from the buffer library, the audio parameter may be updated to obtain an updated audio parameter, where the audio parameter includes the number of channels. For example, in the kernel space of the terminal, ti8168 may be used as a processor, which is a multi-channel high-definition soc system chip integrated with an A8 processor running linux, and the processor is used for collecting audio data. After the audio analog signal is converted into a digital signal through an AD (analog-to-digital) conversion chip, the digital signal acquires corresponding data to the central processing unit through the controller. In order not to affect the integrity of the audio data, when the application program acquires the audio packets, the amount of the interactive audio data can be increased appropriately by adjusting the audio parameters, for example, in a multi-channel scenario, the time stamp information can be saved by increasing the number of channels or using redundant channels. That is, the terminal may update the number of channels in the audio parameter to obtain the updated number of channels, and the terminal adds the timestamp information to a newly added idle channel corresponding to the audio data. Therefore, the time stamp processing is carried out from the source of the audio data, namely, the time based on hardware interruption, the dependency on the system is reduced, the accuracy of the time stamp is ensured, and the follow-up audio and video synchronization is guaranteed.
In one embodiment, before triggering the controller to start carrying the audio data corresponding to the audio signal, the method further includes a step of determining an amount of audio data to be carried, which specifically includes:
and determining the data volume corresponding to each sampling point according to the audio parameters, wherein the audio parameters comprise sampling frequency, sampling bit number and channel number.
And acquiring the number of sampling points in the time interval according to the time interval when the hardware is interrupted.
And determining the audio data volume to be carried according to the data volume corresponding to each sampling point and the number of the sampling points.
When the controller collects the audio data, the collection of the audio data is based on the digital audio interface time sequence, and when the digital audio interface time sequence is determined, audio parameters such as sampling digit, channel number and the like can be determined. When an application program acquires data through an application program interface, the data is generally interacted with an audio data volume corresponding to the number of sampling points, that is, the data volume generated in the interval time of the hardware interrupt. For example, a video live conference generally uses a 48k sampling frequency, a 32-bit sampling bit number, and the number of channels is not fixed, and different sampling frequencies, sampling bit numbers, and channel numbers can be set according to different application scenarios. Specifically, when the controller collects audio data, the terminal may determine a data amount corresponding to each sampling point according to an audio parameter, where the audio parameter includes a sampling frequency, a sampling bit number, and a channel number. And the terminal acquires the number of sampling points in the time interval according to the time interval when the hardware is interrupted. And the terminal determines the audio data volume to be carried according to the data volume corresponding to each sampling point and the number of the sampling points. Therefore, according to the interrupt generated by hardware, namely the trigger signal of the fixed frequency, the timestamp information is obtained based on the signal of the fixed frequency, and the timestamp information is stored in the audio data, so that the accuracy of the timestamp is ensured, and the follow-up audio and video synchronization is guaranteed.
In an embodiment, as shown in fig. 2, after acquiring timestamp information corresponding to audio data when the transport is completed, the method further includes a step of detecting a number of channels corresponding to the audio data, and specifically includes:
When the application program obtains the audio data from the buffer library, the terminal may detect the number of channels corresponding to the audio data. And when the terminal detects that a redundant channel exists in the channel corresponding to the audio data, storing the timestamp information corresponding to the audio data into the redundant channel. And when the terminal detects that no redundant channel exists in the channel corresponding to the audio data, updating the number of the channels corresponding to the audio data, and storing the timestamp information corresponding to the audio data into the updated channel. For example, the audio parameters originally set in the system are 2 channels, 32-bit sampling bit width, 48000 sampling frequency, and 960 sampling points, and then the number of audio packets acquired at a time is: 7680 bytes 2 × 4 × 960. In order to store the time stamp data, that is, when the terminal detects that there is no redundant channel in the channel corresponding to the audio data, the terminal may update the number of channels corresponding to the audio data, and store the time stamp information corresponding to the audio data in the updated channel. For example, the terminal updates the original 2 channels to 4 channels, and through software adaptation, the hardware still collects data according to the original 2 channels, but the data reported to the application program is doubled, that is, corresponding timestamp information can be filled in any position of the redundant channel. When the terminal detects that a redundant channel exists in the channel corresponding to the audio data, namely under the condition of multiple channels, the terminal can store the timestamp information corresponding to the audio data into the unused redundant channel and report the timestamp information to the corresponding application program. Compared with the traditional audio and video processing mode, when the system is blocked, the application program can be caused to acquire the audio data untimely, after the audio data is acquired once, the audio data is immediately acquired for the second time, the interval between two operations is very small, the acquired timestamp information is very close, the application program can be caused to have abnormal conditions such as audio and video synchronization failure when the application program uses the timestamp to carry out audio and video synchronization, and the time interval does not really reflect the audio data is mainly used as the timestamp. In the application, adaptation is performed when a cache space is allocated, and the main purpose is to reserve a memory space for storing the timestamp. Like this the audio data that application program acquireed at every turn just contained corresponding time stamp information, for traditional mode, need when distributing audio data buffering, suitably increase the buffer memory district, reserve the space of preserving time stamp information, audio data acquisition is still carried out according to original chronogenesis to hardware, when guaranteeing audio data integrality, can ensure the accuracy of time stamp, guarantee that application program can normally carry out audio and video expert synchronous processing, thereby the effectual unable synchronous problem of audio and video in the streaming media of having solved.
In an embodiment, as shown in fig. 3A, a control method for audio and video synchronization of a streaming media is provided, and this embodiment is illustrated by applying the method to a terminal, it can be understood that the method may also be applied to a server, and may also be applied to a system including the terminal and the server, and is implemented by interaction between the terminal and the server. In this embodiment, the method includes the steps of:
And 304, after the transportation is finished, triggering to generate an interruption of the transportation, and acquiring time stamp information corresponding to the audio data, wherein the time stamp information is determined based on the time when the hardware generates the interruption.
And 308, when the application program acquires the audio data from the buffer library, sending the audio data carrying the timestamp information to the application program.
And when the controller acquires the audio signal, triggering the controller to start carrying the audio data corresponding to the audio signal. And after the controller finishes the transportation, triggering to generate an interruption of the completed transportation, acquiring timestamp information corresponding to the audio data based on the time of the interruption of the completed transportation generated by the hardware, and adding the acquired timestamp information to an idle channel corresponding to the audio data. The central processing unit caches the audio data and the timestamp information to a cache library according to the sequence of receiving the audio signals. And when the application program acquires the audio data from the buffer library, the central processing unit sends the audio data carrying the timestamp information to the application program. For example, taking an ALSA-based audio time stamp processing procedure as an example, ALSA is an abbreviation of Advanced Linux Sound Architecture, and an Advanced Linux Sound Architecture is an abbreviation, and provides audio and MIDI (Musical Instrument Digital Interface) support on a Linux operating system. As shown in fig. 3B, a schematic diagram of system logic processing for audio and video synchronization of streaming media is shown. In order to ensure the clock synchronization of the whole audio system, the bit clock and the frame synchronization clock of the audio are provided by the central processing unit. The ADC chip, i.e., the analog-to-digital conversion chip, converts the analog signal into a digital signal, and sends the digital signal to an internal module Mcasp of the central processing unit, which is an audio processing controller on the central processing unit and is also called a multi-channel audio access interface. After the audio signal is acquired, the Mcasp controller triggers the EDMA transport event AREVT, i.e., the EDAM is started to start transporting the audio data. AREVT is a trigger event for EDMA and can notify EDMA to carry out data transportation. EDMA is an important technology for fast data exchange in a Digital Signal Processor (DSP), has the capacity of background batch data transmission independent of a CPU, can meet the requirement of high-speed data transmission in real-time image processing, and is used for exchanging data of a peripheral and an internal memory. And triggering and generating an EDAM (electronic design automation) transport completion interrupt after EDMA (enhanced data access) transport is completed, and stamping the audio data by the system based on the interrupt. That is, the time stamp information corresponding to the audio data acquired by the terminal is determined based on the time when the hardware generates the interrupt. The central processing unit caches the audio data and the timestamp information to a cache library according to the sequence of receiving the audio signals. And when the application program acquires the audio data from the buffer library, the central processing unit sends the audio data carrying the timestamp information to the application program. Therefore, timestamp reporting based on alsa is realized, a foundation is laid for audio and video synchronization, and stable and reliable operation of live broadcast and recorded broadcast functions is realized.
In this embodiment, through carry out the timestamp processing when audio data caches, make the audio data in the cache all carry time stamp information, even the system appears the card pause, the unable timely data of taking is seen to the application, when appearing the condition that acquires audio data in succession, because the timestamp that carries in the audio data is based on the time of hardware interrupt is acquireed, it is lower to the lazy nature of system, therefore can ensure the accuracy of timestamp information, thereby can guarantee that the application can normally carry out audio and video expert synchronous processing, the effectual unable synchronous problem of audio and video in the streaming media of having solved.
It should be understood that although the various steps in the flow charts of fig. 1-3 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-3 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 4, there is provided a control device for audio-video synchronization of streaming media, including: a carrying module 402, an obtaining module 404, a buffering module 406, and a sending module 408, wherein:
the carrying module 402 is configured to trigger the controller to start carrying audio data corresponding to the audio signal when the audio signal is acquired.
An obtaining module 404, configured to obtain timestamp information corresponding to the audio data when the transportation is completed, where the timestamp information is determined based on a time when the hardware generates the interrupt.
The buffer module 406 is configured to buffer the audio data and the corresponding timestamp information into a buffer library.
The sending module 408 is configured to send the audio data carrying the timestamp information to the application program when the application program obtains the audio data from the buffer library.
In one embodiment, the apparatus further comprises: an update module and an add module.
The updating module is used for updating the audio parameters to obtain updated audio parameters, and the audio parameters comprise the number of channels. The adding module is used for adding the timestamp information to an idle channel corresponding to the audio data.
In one embodiment, the apparatus further comprises: and determining a module.
The determining module is used for determining the data volume corresponding to each sampling point according to the audio parameters, and the audio parameters comprise sampling frequency, sampling bit number and channel number. The acquisition module is further used for acquiring the number of sampling points in the time interval according to the time interval when the hardware is interrupted. The determining module is further used for determining the audio data volume to be carried according to the data volume corresponding to each sampling point and the number of the sampling points.
In one embodiment, the buffering module is further configured to buffer the audio data and the timestamp information into a buffer library according to an order in which the audio signals are received.
In one embodiment, the apparatus further comprises: and a detection module.
The detection module is used for detecting the number of channels corresponding to the audio data. When detecting that a redundant channel exists in a channel corresponding to the audio data, storing timestamp information corresponding to the audio data into the redundant channel; and when detecting that no redundant channel exists in the channel corresponding to the audio data, updating the number of the channels corresponding to the audio data, and storing the timestamp information corresponding to the audio data into the updated channel.
In one embodiment, the apparatus further comprises: and generating a module.
The generation module is used for triggering and generating an interruption of the completion of the transportation after the transportation is completed, wherein the interruption is generated by hardware based on a fixed frequency signal.
For specific limitations of the control device for audio-video synchronization of streaming media, reference may be made to the above limitations of the control method for audio-video synchronization of streaming media, and details are not described herein again. All or part of each module in the streaming media audio and video synchronization control device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 5. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to realize a control method of the audio and video synchronization of the streaming media. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the steps of the above-described method embodiments being implemented when the computer program is executed by the processor.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A control method for audio and video synchronization of streaming media comprises the following steps:
when an audio signal is acquired, triggering a controller to start carrying audio data corresponding to the audio signal;
when the transportation is finished, acquiring timestamp information corresponding to the audio data, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
caching the audio data and the corresponding timestamp information to a cache library;
and when the application program acquires the audio data from the cache library, sending the audio data carrying the timestamp information to the application program.
2. The method of claim 1, wherein before the audio signal is acquired, the method further comprises:
updating the audio parameters to obtain updated audio parameters; the audio parameters include a channel number;
prior to the caching the audio data and the corresponding timestamp information into a cache bank, the method further comprises:
and adding the timestamp information to an idle channel corresponding to the audio data.
3. The method of claim 1, wherein before the triggering controller begins to convey audio data corresponding to the audio signal, the method further comprises:
determining the data volume corresponding to each sampling point according to the audio parameters; the audio parameters comprise sampling frequency, sampling digit and channel number;
acquiring the number of sampling points in a time interval according to the time interval when hardware is interrupted;
and determining the audio data volume to be carried according to the data volume corresponding to each sampling point and the number of the sampling points.
4. The method of claim 1, wherein caching the audio data and the corresponding timestamp information in a cache library comprises:
and caching the audio data and the timestamp information to a cache library according to the sequence of receiving the audio signals.
5. The method according to claim 1, wherein after acquiring the time stamp information corresponding to the audio data when the transport is completed, the method further comprises:
detecting the number of channels corresponding to the audio data;
when detecting that a redundant channel exists in a channel corresponding to the audio data, storing timestamp information corresponding to the audio data into the redundant channel;
and when detecting that no redundant channel exists in the channel corresponding to the audio data, updating the number of the channels corresponding to the audio data, and storing the timestamp information corresponding to the audio data into the updated channel.
6. The method according to claim 1, wherein the acquiring time stamp information corresponding to the audio data when the transport is completed comprises:
and triggering and generating an interruption of the completion of the transportation when the transportation is completed, wherein the interruption is generated by hardware based on a fixed frequency signal.
7. A control device for audio and video synchronization of streaming media is characterized by comprising:
the carrying module is used for triggering the controller to start carrying the audio data corresponding to the audio signal when the audio signal is obtained;
the acquisition module is used for acquiring timestamp information corresponding to the audio data when the transport is finished, wherein the timestamp information is determined based on the time when the hardware generates the interrupt;
the cache module is used for caching the audio data and the corresponding timestamp information to a cache library;
and the sending module is used for sending the audio data carrying the timestamp information to the application program when the application program obtains the audio data from the cache library.
8. The apparatus for controlling audio-video synchronization of streaming media according to claim 7, wherein said apparatus further comprises:
and the adding module is used for adding the timestamp information to an idle channel corresponding to the audio data.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011021295.2A CN112218115B (en) | 2020-09-25 | 2020-09-25 | Control method and device for streaming media audio and video synchronization and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011021295.2A CN112218115B (en) | 2020-09-25 | 2020-09-25 | Control method and device for streaming media audio and video synchronization and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112218115A true CN112218115A (en) | 2021-01-12 |
| CN112218115B CN112218115B (en) | 2022-07-29 |
Family
ID=74052312
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011021295.2A Active CN112218115B (en) | 2020-09-25 | 2020-09-25 | Control method and device for streaming media audio and video synchronization and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112218115B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113132672A (en) * | 2021-03-24 | 2021-07-16 | 联想(北京)有限公司 | Data processing method and video conference equipment |
| CN113784073A (en) * | 2021-09-28 | 2021-12-10 | 深圳万兴软件有限公司 | Method, device and related medium for synchronizing sound and picture of sound recording and video recording |
| CN114339353A (en) * | 2021-12-31 | 2022-04-12 | 晶晨半导体科技(北京)有限公司 | Audio and video synchronization method and device, electronic equipment and computer readable storage medium |
| WO2024061005A1 (en) * | 2022-09-23 | 2024-03-28 | 天翼数字生活科技有限公司 | Read processing method and apparatus for audio and video buffer |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040236583A1 (en) * | 1998-11-16 | 2004-11-25 | Yoshiaki Tanaka | Audio signal processing apparatus |
| JP2005130388A (en) * | 2003-10-27 | 2005-05-19 | Yamaha Corp | Audio signal reproducing apparatus |
| CN1960228A (en) * | 2006-08-22 | 2007-05-09 | 中兴通讯股份有限公司 | Method for multiplexing aerial channels of mobile multimedia broadcast |
| CN101001485A (en) * | 2006-10-23 | 2007-07-18 | 中国传媒大学 | Finite sound source multi-channel sound field system and sound field analogy method |
| CN101043265A (en) * | 2007-04-27 | 2007-09-26 | 华为技术有限公司 | Method for realizing multimedia broadcasting and multicasting service data synchronized transmission |
| US20080037954A1 (en) * | 2006-05-15 | 2008-02-14 | Microsoft Corporation | Automatic Video Glitch Detection and Audio-Video Synchronization Assessment |
| CN103561291A (en) * | 2013-10-31 | 2014-02-05 | 腾讯科技（武汉）有限公司 | Video channel distribution management method, relevant device and communication system |
| US20150016626A1 (en) * | 2003-07-28 | 2015-01-15 | Sonos, Inc. | Switching Between a Directly Connected and a Networked Audio Source |
| CN104378675A (en) * | 2014-12-08 | 2015-02-25 | 厦门雅迅网络股份有限公司 | Multichannel audio-video synchronized playing processing method |
| CN105306110A (en) * | 2015-09-18 | 2016-02-03 | 深圳市冠旭电子有限公司 | Methods and system for realizing music synchronous play |
| CN105847873A (en) * | 2016-05-16 | 2016-08-10 | 西安电子科技大学 | High definition video code stream and data multiplexing system and method for surveillance application |
| CN106412662A (en) * | 2016-09-20 | 2017-02-15 | 腾讯科技（深圳）有限公司 | Timestamp distribution method and device |
| CN106937137A (en) * | 2015-12-30 | 2017-07-07 | 惠州市伟乐科技股份有限公司 | A kind of synchronous method of multi-channel digital audio coding audio-visual |
| CN108055566A (en) * | 2017-12-26 | 2018-05-18 | 郑州云海信息技术有限公司 | Method, apparatus, equipment and the computer readable storage medium of audio-visual synchronization |
| CN108965971A (en) * | 2018-07-27 | 2018-12-07 | 北京数码视讯科技股份有限公司 | MCVF multichannel voice frequency synchronisation control means, control device and electronic equipment |
| CN109152041A (en) * | 2017-06-16 | 2019-01-04 | 华为技术有限公司 | Method, terminal device and the network equipment of information transmission |
| CN109729277A (en) * | 2018-11-19 | 2019-05-07 | 魔门塔(苏州)科技有限公司 | Multi-sensor collection timestamp synchronizing device |
| WO2019226760A1 (en) * | 2018-05-24 | 2019-11-28 | Google Llc | Methods, systems, and media for synchronizing audio and video content on multiple media devices |
| KR102090070B1 (en) * | 2018-10-31 | 2020-03-17 | 카테노이드 주식회사 | Streaming server, client terminal and audio/video live streaming system using the same |
| CN111447396A (en) * | 2020-03-06 | 2020-07-24 | 视联动力信息技术股份有限公司 | Audio and video transmission method and device, electronic equipment and storage medium |
-
2020
- 2020-09-25 CN CN202011021295.2A patent/CN112218115B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040236583A1 (en) * | 1998-11-16 | 2004-11-25 | Yoshiaki Tanaka | Audio signal processing apparatus |
| US20150016626A1 (en) * | 2003-07-28 | 2015-01-15 | Sonos, Inc. | Switching Between a Directly Connected and a Networked Audio Source |
| JP2005130388A (en) * | 2003-10-27 | 2005-05-19 | Yamaha Corp | Audio signal reproducing apparatus |
| US20080037954A1 (en) * | 2006-05-15 | 2008-02-14 | Microsoft Corporation | Automatic Video Glitch Detection and Audio-Video Synchronization Assessment |
| CN1960228A (en) * | 2006-08-22 | 2007-05-09 | 中兴通讯股份有限公司 | Method for multiplexing aerial channels of mobile multimedia broadcast |
| CN101001485A (en) * | 2006-10-23 | 2007-07-18 | 中国传媒大学 | Finite sound source multi-channel sound field system and sound field analogy method |
| CN101043265A (en) * | 2007-04-27 | 2007-09-26 | 华为技术有限公司 | Method for realizing multimedia broadcasting and multicasting service data synchronized transmission |
| CN103561291A (en) * | 2013-10-31 | 2014-02-05 | 腾讯科技（武汉）有限公司 | Video channel distribution management method, relevant device and communication system |
| CN104378675A (en) * | 2014-12-08 | 2015-02-25 | 厦门雅迅网络股份有限公司 | Multichannel audio-video synchronized playing processing method |
| CN105306110A (en) * | 2015-09-18 | 2016-02-03 | 深圳市冠旭电子有限公司 | Methods and system for realizing music synchronous play |
| CN106937137A (en) * | 2015-12-30 | 2017-07-07 | 惠州市伟乐科技股份有限公司 | A kind of synchronous method of multi-channel digital audio coding audio-visual |
| CN105847873A (en) * | 2016-05-16 | 2016-08-10 | 西安电子科技大学 | High definition video code stream and data multiplexing system and method for surveillance application |
| CN106412662A (en) * | 2016-09-20 | 2017-02-15 | 腾讯科技（深圳）有限公司 | Timestamp distribution method and device |
| CN109152041A (en) * | 2017-06-16 | 2019-01-04 | 华为技术有限公司 | Method, terminal device and the network equipment of information transmission |
| CN108055566A (en) * | 2017-12-26 | 2018-05-18 | 郑州云海信息技术有限公司 | Method, apparatus, equipment and the computer readable storage medium of audio-visual synchronization |
| WO2019226760A1 (en) * | 2018-05-24 | 2019-11-28 | Google Llc | Methods, systems, and media for synchronizing audio and video content on multiple media devices |
| CN108965971A (en) * | 2018-07-27 | 2018-12-07 | 北京数码视讯科技股份有限公司 | MCVF multichannel voice frequency synchronisation control means, control device and electronic equipment |
| KR102090070B1 (en) * | 2018-10-31 | 2020-03-17 | 카테노이드 주식회사 | Streaming server, client terminal and audio/video live streaming system using the same |
| CN109729277A (en) * | 2018-11-19 | 2019-05-07 | 魔门塔(苏州)科技有限公司 | Multi-sensor collection timestamp synchronizing device |
| CN111447396A (en) * | 2020-03-06 | 2020-07-24 | 视联动力信息技术股份有限公司 | Audio and video transmission method and device, electronic equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| RIZWANA SHAHER BANO: "COTS based multichannel FM receiver and recorder using SDR concept", 《2013 IEEE 20TH INTERNATIONAL CONFERENCE ON ELECTRONICS, CIRCUITS, AND SYSTEMS (ICECS)》 * |
| 蔡李非: "基于FPGA的多声道音频系统的设计与实现", 《中国优秀硕士学位论文全文库》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113132672A (en) * | 2021-03-24 | 2021-07-16 | 联想(北京)有限公司 | Data processing method and video conference equipment |
| CN113132672B (en) * | 2021-03-24 | 2022-07-26 | 联想(北京)有限公司 | Data processing method and video conference equipment |
| CN113784073A (en) * | 2021-09-28 | 2021-12-10 | 深圳万兴软件有限公司 | Method, device and related medium for synchronizing sound and picture of sound recording and video recording |
| CN114339353A (en) * | 2021-12-31 | 2022-04-12 | 晶晨半导体科技(北京)有限公司 | Audio and video synchronization method and device, electronic equipment and computer readable storage medium |
| CN114339353B (en) * | 2021-12-31 | 2023-09-29 | 晶晨半导体科技(北京)有限公司 | Audio/video synchronization method and device, electronic equipment and computer readable storage medium |
| WO2024061005A1 (en) * | 2022-09-23 | 2024-03-28 | 天翼数字生活科技有限公司 | Read processing method and apparatus for audio and video buffer |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112218115B (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112218115B (en) | Control method and device for streaming media audio and video synchronization and computer equipment | |
| US11146502B2 (en) | Method and apparatus for allocating resource | |
| CN108664331B (en) | Distributed data processing method and device, electronic equipment and storage medium | |
| US20170123713A1 (en) | Device and process for data storage and read/write efficiency | |
| CN104539977A (en) | Live broadcast previewing method and device | |
| US20230285854A1 (en) | Live video-based interaction method and apparatus, device and storage medium | |
| CN105786539B (en) | File downloading method and device | |
| US10755707B2 (en) | Selectively blacklisting audio to improve digital assistant behavior | |
| CN115328742B (en) | Container information monitoring method and device, storage medium and electronic equipment | |
| CN111818383B (en) | Video data generation method, system, device, electronic equipment and storage medium | |
| CN111614954A (en) | Index acquisition processing method and device for streaming media, computer and storage medium | |
| US10432490B2 (en) | Monitoring single content page application transitions | |
| US20150006724A1 (en) | System, method, and computer program product for providing test logic to user devices | |
| CN116627333A (en) | Log caching method and device, electronic equipment and computer readable storage medium | |
| CN111198853A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US11895203B2 (en) | Methods and apparatus to collect media metrics on computing devices | |
| CN109194971A (en) | A kind of generation method and device for multimedia file | |
| CN111726650B (en) | Video live broadcast method and device and computer storage medium | |
| CN111031354B (en) | Multimedia playing method, device and storage medium | |
| CN113438250A (en) | Abnormal event processing method and equipment | |
| CN111382381B (en) | User data sharing method and device, mobile terminal and storage medium | |
| CN111310175A (en) | iOS application safety monitoring and protecting method and device based on plug-in | |
| CN109308778A (en) | Mobile detection alarm method, device, acquisition equipment and storage medium | |
| CN114827674B (en) | Network resource display method, device and system | |
| CN106454512A (en) | Player event statistical method and device player and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Control method, device, and computer equipment for streaming audio and video synchronization Effective date of registration: 20230912 Granted publication date: 20220729 Pledgee: Guangxi Guangtou Zhanxin Investment Fund Partnership Enterprise (L.P.) Pledgor: IFREECOMM TECHNOLOGY Co.,Ltd. Registration number: Y2023980056247 |