CN117097936B - Audio and video synchronization method and device, electronic equipment and storage medium - Google Patents
Audio and video synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117097936B CN117097936B CN202311354870.4A CN202311354870A CN117097936B CN 117097936 B CN117097936 B CN 117097936B CN 202311354870 A CN202311354870 A CN 202311354870A CN 117097936 B CN117097936 B CN 117097936B
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- data
- module
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000005540 biological transmission Effects 0.000 claims abstract description 73
- 230000001360 synchronised effect Effects 0.000 claims abstract description 32
- 230000008569 process Effects 0.000 claims abstract description 28
- 230000002159 abnormal effect Effects 0.000 claims abstract description 17
- 230000006835 compression Effects 0.000 claims abstract description 16
- 238000007906 compression Methods 0.000 claims abstract description 16
- 238000004364 calculation method Methods 0.000 claims description 24
- 238000013139 quantization Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 230000007717 exclusion Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 4
- 230000003139 buffering effect Effects 0.000 claims description 3
- 230000001934 delay Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims 2
- 238000012545 processing Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention relates to the technical field of audio and video synchronization, in particular to an audio and video synchronization method, an audio and video synchronization device, electronic equipment and a storage medium, which comprise the following steps: a media capturing module: respectively acquiring audio and video data through a microphone and camera equipment connected with the system; and the audio and video coding module is used for: compressing and encoding the obtained original audio and video data; and a data transmission module: and collecting performance information of the audio and video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process. Calculating an anomaly coefficient; and a data receiving module: setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay; and the synchronous playing module is used for: after the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are played simultaneously. The invention can ensure that the audio and video are kept synchronous in the playing process by synchronously adjusting the time difference, and improves the processing efficiency and quality.
Description
Technical Field
The present invention relates to the field of audio and video synchronization technologies, and in particular, to an audio and video synchronization method, an audio and video synchronization device, an electronic device, and a storage medium.
Background
With the vigorous development of security and protection industry, the monitoring industry has spread over the fields of education, public security, banks and the like and plays a great role, meanwhile, the audio and video compression technology is increasingly mature, and the high-performance audio and video compression algorithm enables network monitoring to be possible.
In the existing network monitoring system, the pure tone asynchronous phenomenon often occurs to the audio and video information received by the client end due to network delay, congestion and the like, so that the experience of a user is greatly reduced, the audio and video synchronization algorithm is various, complex and low in efficiency at present, and is difficult to apply to the network monitoring system with limited resources, and the existing audio and video synchronization systems in the market at present have the defects of poor multipath audio synchronization effect, more jitter and more timing problems, and are difficult to meet the requirement of high-quality audio and video synchronization.
Disclosure of Invention
The invention aims to solve the problems in the background technology and provides an audio and video synchronization method, an audio and video synchronization device, electronic equipment and a storage medium.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
the method, the device, the electronic equipment and the storage medium for audio and video synchronization comprise a media capturing module, an audio and video coding module, a data transmission module, a data receiving module, a synchronous playing module and a shared memory module;
a media capturing module: respectively acquiring audio and video data through a microphone and camera equipment connected with the system;
and the audio and video coding module is used for: compressing and encoding the obtained original audio and video data;
and a data transmission module: collecting performance information of an audio/video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process, and calculating the abnormal coefficients;
and a data receiving module: setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay;
and the synchronous playing module is used for: after the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are played simultaneously.
And the shared memory module is used for: a memory region for sharing is created, and the shared memory region is mapped into the address space of each participating process.
In the method, the device, the electronic equipment and the storage medium for synchronizing the audio and video, the media capturing module firstly determines the audio and video capturing equipment to be used, wherein the audio and video capturing equipment comprises a microphone and a camera equipment which are connected in a system, after the configuration of the capturing equipment is completed, a capturing instruction is generated, after the media capturing module receives the capturing instruction, the designated capturing equipment is opened to respectively acquire the audio and video data, the acquired audio and video data is compressed, and the compressed audio and video data is written into the shared memory.
In the above method, device, electronic device and storage medium for audio/video synchronization, the audio/video encoding module is responsible for compressing and encoding the obtained original audio/video data, and the specific operation steps are as follows:
s1, acquiring original audio and video data from a media capturing module;
s2, preprocessing operation of noise reduction, redundant information removal and resolution adjustment is carried out on the audio and video data;
s3, coding video data through a video coding standard, dividing the acquired original video data into a plurality of blocks with equal size, performing color space conversion and predictive filtering on each block, and performing color space conversion on each blockAnd (3) carrying out quantization to obtain a frequency domain coefficient, wherein the specific calculation mode is as follows:
，
wherein round represents rounding, coe represents transformed coefficients, qs represents quantization step size, sc represents quantization matrix value, prediction is performed by using information of blocks coded around the current block to obtain a prediction residual, and a specific calculation formula of intra-frame prediction is as follows:
，
wherein u and v represent predictive modesWhere Y represents a predicted value, P represents a reference pixel,for the weight, the specific calculation mode of the prediction residual is as follows:
，
wherein, R represents the pixel value of the current block, the information of the adjacent frames coded before is utilized to estimate motion, a motion vector is obtained, the motion vector is used for carrying out motion compensation on the current block, and the specific calculation mode of the motion compensation is as follows:
，
wherein SAD represents a block difference function, p represents a current block, q represents a reference block, and the current frame is compared with other encoded frames to obtain a prediction residual, and a specific calculation formula of inter-frame prediction is as follows:
，
wherein MV is a motion vector, and interp represents an interpolation function, entropy coding is carried out on the obtained prediction residual, a decompressor carries out inverse quantization and inverse transformation on the compressed data to obtain a reconstructed frequency domain coefficient, and the reconstructed frequency domain coefficient is subjected to inverse transformation to obtain a final reconstructed image;
s4, compressing the audio and video data, firstly opening an audio and video equipment node, initializing the audio and video equipment, acquiring a frame of image, and if the image is successfully acquired, compressing the data, otherwise, re-acquiring the data, stamping a time stamp on the compressed data, and writing the compressed data into a shared memory.
In the above method, device, electronic device and storage medium for audio/video synchronization, the data transmission module acquires performance information of the audio/video compression packet during transmission, where the performance information includes an anomaly coefficient during transmission, and the anomaly coefficient during transmission is set as NM, and specifically the steps of obtaining are as follows:
s1, acquiring an optimal transmission time range of an audio and video in a transmission process, and calibrating the optimal transmission time range asWhereinAnd (3) withRepresenting the maximum transmission time and the minimum transmission time in the audio and video transmission process respectively;
s2, acquiring actual transmission speeds of the data transmission module at different moments in different time, and marking the actual reading speed asX represents the number of the actual transmission speed at different times in different times, x=1, 2, 3.
S3, calibrating the actual speed smaller than the optimal transmission range asJ represents a number of an actual transmission speed smaller than the optimal transmission range, j=1, 2, 3.
S4, calculating an abnormal coefficient in the transmission process, wherein a specific calculation formula is as follows:
，
s5, eliminating asynchronism generated in the audio and video transmission process, when the audio and video buffer occupiesIs the audio data and video buffer takes upVideo data, audio compression generates time delayVideo compression generates time delayThe specific calculation mode is as follows:
，
time asynchronization is reduced by buffering.
In the above method, device, electronic equipment and storage medium for audio and video synchronization, the data receiving module sets a second level buffer, the first level buffer is used for eliminating the disorder of the data stream, the second level buffer is used for eliminating the jitter phenomenon of the audio and video data generated by time delay, the second level buffer contains time stamp, serial number, data size information and PT value type according to the data characteristics of the received RTP packet, the receiving buffer is designed into a linked list, it is composed of a plurality of nodes, each node stores the length of the audio and video data, time stamp and serial number information, the initialization buffer can specify the maximum length and the current occupied length, the buffer is full of the buffer and can automatically mark that the buffer is ready, according to the mark, the audio and video playing thread triggers the playing of the audio and video, the method of the access is to store the data at the end of the linked list, the second level buffer is set with a threshold value M, the value M is larger than the expected maximum jitter time, the value of M can be set by itself according to the actual situation, the specific calculation method for compensating the network time delay is as follows:
，
wherein,respectively, the maximum and minimum delays of the network, r representing the data transmission rate.
In the above method, device, electronic device and storage medium for audio/video synchronization, when the playing starts, the synchronous playing module finds the video frames with the same playing time according to the playing time of the first frame of audio data, and plays the video frames simultaneously, which determines the common time reference of audio/video playing, then, when N frames of audio data are played, one time of synchronization adjustment is performed, the playing time T (N) of the latest played audio frame is compared with the playing time T (vd) of the latest played video frame, when the time difference between the two is less than 0.08S, no adjustment is needed, when the time difference between the two is greater than 0.08S, synchronous adjustment is performed, and when the time difference is greater than 0.16S, the audio/video playing is in an out-of-step state, so that:
，
t，
wherein n1 and n2 are integers, r represents the frame rate of normal playing of the video,representing the delay of the video image.
In the method, the device, the electronic equipment and the storage medium for synchronizing the audio and video, the shared memory module needs to create a memory area for sharing, the shared memory area is mapped into a process address space, each process participating in sharing needs to map the shared memory into an own address space, when the shared memory is mapped into the address space of the process, the process can directly read and write the content of the shared memory, and the data exchange and sharing can be realized between different processes through the read and write of the shared memory, and when the read and write operation of the shared memory is performed, a synchronization and mutual exclusion mechanism is used for controlling the access of the process to the shared memory.
Compared with the prior art, the audio and video synchronization method, the device, the electronic equipment and the storage medium have the advantages that:
according to the invention, the synchronization mechanism based on the time stamp is added in the system, so that the audio and video synchronization effect is better, the problem of jitter can be processed in real time by arranging the secondary buffer in the receiving module, and the time difference is synchronously adjusted by the synchronous playing module, so that the audio and video can be ensured to be kept synchronous in the playing process, and the processing efficiency and quality are improved.
Drawings
Fig. 1 is a schematic flow chart of a method, an apparatus, an electronic device and a storage medium for audio and video synchronization according to the present invention;
fig. 2 is a block diagram of a method, an apparatus, and an electronic device for audio and video synchronization according to the present invention;
FIG. 3 is a block diagram of a storage medium for audio/video synchronization according to the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments.
In the description of the present invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "top," "bottom," "inner," "outer," and the like indicate or are based on the orientation or positional relationship shown in the drawings, merely to facilitate description of the present invention and to simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Referring to fig. 1-2, a method, an apparatus, an electronic device, and a storage medium for audio/video synchronization include: the device comprises a media capturing module, an audio and video encoding module, a data transmission module, a data receiving module and a synchronous playing module;
a media capturing module: respectively acquiring audio and video data through a microphone and camera equipment connected with the system;
and the audio and video coding module is used for: compressing and encoding the obtained original audio and video data;
and a data transmission module: collecting performance information of an audio/video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process, and calculating the abnormal coefficients;
and a data receiving module: setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay;
and the synchronous playing module is used for: after the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are played simultaneously.
101. Respectively acquiring audio and video data through a microphone and camera equipment connected with the system;
in this embodiment, a specific description is provided of a media capturing module, where the media capturing module first determines an audio and video capturing device to be used, including a microphone and a camera device connected by a system, after the capturing device is configured, a capturing instruction is generated, after receiving the capturing instruction, the media capturing module opens a designated capturing device to respectively obtain audio and video data, compresses the obtained audio and video data, and writes the compressed audio and video data into a shared memory.
102. Compressing and encoding the obtained original audio and video data;
in this embodiment, an audio/video encoding module is specifically needed to be described, and the audio/video encoding module is responsible for compressing and encoding the obtained original audio/video data, and the specific operation steps are as follows:
s1, acquiring original audio and video data from a media capturing module;
s2, preprocessing operation of noise reduction, redundant information removal and resolution adjustment is carried out on the audio and video data;
s3, coding video data through a video coding standard, dividing the acquired original video data into a plurality of blocks with equal size, performing color space conversion and predictive filtering on each block, and performing 4 on each blockAnd (3) carrying out quantization to obtain a frequency domain coefficient, wherein the specific calculation mode is as follows:
，
wherein round represents rounding, coe represents transformed coefficients, qs represents quantization step size, sc represents quantization matrix value, prediction is performed by using information of blocks coded around the current block to obtain a prediction residual, and a specific calculation formula of intra-frame prediction is as follows:
，
where u and v represent prediction modes, Y represents a predicted value, P is a reference pixel,for the weight, the specific calculation mode of the prediction residual is as follows:
，
wherein, R represents the pixel value of the current block, the information of the adjacent frames coded before is utilized to estimate motion, a motion vector is obtained, the motion vector is used for carrying out motion compensation on the current block, and the specific calculation mode of the motion compensation is as follows:
，
wherein SAD represents a block difference function, p represents a current block, q represents a reference block, and the current frame is compared with other encoded frames to obtain a prediction residual, and a specific calculation formula of inter-frame prediction is as follows:
，
wherein MV is a motion vector, and interp represents an interpolation function, entropy coding is carried out on the obtained prediction residual, a decompressor carries out inverse quantization and inverse transformation on the compressed data to obtain a reconstructed frequency domain coefficient, and the reconstructed frequency domain coefficient is subjected to inverse transformation to obtain a final reconstructed image;
s4, compressing the audio and video data, firstly opening an audio and video equipment node, initializing the audio and video equipment, acquiring a frame of image, and if the image is successfully acquired, compressing the data, otherwise, re-acquiring the data, stamping a time stamp on the compressed data, and writing the compressed data into a shared memory.
103. Collecting performance information of an audio/video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process, and calculating the abnormal coefficients;
in this embodiment, a specific description is provided of a data transmission module, where the data transmission module acquires performance information of an audio and video compression packet during transmission, where the performance information includes an abnormal coefficient in a transmission process, and the abnormal coefficient in the transmission process is set to NM, and specifically the steps of obtaining are as follows:
s1, acquiring an optimal transmission time range of an audio and video in a transmission process, and calibrating the optimal transmission time range asWhereinAnd (3) withRepresenting the maximum transmission time and the minimum transmission time in the audio and video transmission process respectively;
s2, acquiring actual transmission speeds of the data transmission module at different moments in different time, and marking the actual reading speed asX represents the number of the actual transmission speed at different times in different times, x=1, 2, 3.
S3, calibrating the actual speed smaller than the optimal transmission range asJ represents a number of an actual transmission speed smaller than the optimal transmission range, j=1, 2, 3.
S4, calculating an abnormal coefficient in the transmission process, wherein a specific calculation formula is as follows:
，
s5, eliminating asynchronism generated in the audio and video transmission process, when the audio and video buffer occupiesIs the audio data and video buffer takes upVideo data, audio compression generates time delayVideo compression generates time delayThe specific calculation mode is as follows:
，
time asynchronization is reduced by buffering.
104. Setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay;
in this embodiment, a specific description needs to be provided about a data receiving module, where the data receiving module sets a second level buffer, where the second level buffer is used to eliminate disorder of a data stream, the second level buffer is used to eliminate jitter phenomenon generated by delay of audio and video data, the second level buffer includes a timestamp, a serial number, data size information, and a PT value type according to data characteristics of a received RTP packet, the receiving buffer is designed into a linked list, and is composed of a plurality of nodes, each node stores a length of audio and video data, a timestamp, and serial number information, an initialization buffer can specify a maximum length and a length currently occupied by the node, the buffer is full of data, the buffer is ready for automatic marking, an audio and video playing thread triggers playing of audio and video according to the mark, the method of accessing is to store data at a linked list end, and to fetch data at a linked list end, the second level buffer sets a threshold value M according to the data characteristics of the received RTP packet, the value M can be set according to an actual situation, and a specific calculation method of compensating network delay jitter is as follows:
，
wherein,respectively, the maximum and minimum delays of the network, r representing the data transmission rate.
105. After the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are played simultaneously.
In this embodiment, a specific description needs to be provided about a synchronous playing module, where when playing starts, the synchronous playing module finds a video frame with the same playing time according to the playing time of the first frame of audio data, and plays the video frame at the same time, which determines a common time reference for playing audio and video, then, when playing N frames of audio data, performs a synchronous adjustment, compares the playing time T (N) of the latest playing audio frame with the playing time T (vd) of the latest playing video frame, when the time difference between the two is less than 0.08S, does not need to perform adjustment, when the time difference between the two is greater than 0.08S, performs synchronous adjustment, and when the time difference is greater than 0.16S, the audio and video playing is in a step-out state, so that:
，
t，
wherein n1 and n2 are integers, r represents the frame rate of normal playing of the video,representing the delay of the video image.
In the invention, a media capturing module acquires continuous audio samples from audio input equipment such as a microphone and the like, a camera in the media capturing module is dragged to acquire continuous video frames, the media capturing module carries out noise suppression and noise reduction pretreatment on the acquired audio and video data, and meanwhile, the media capturing module caches the acquired audio and video data.
Referring to fig. 3, an audio-video synchronized storage medium includes: and the shared memory module is used for creating a memory area for sharing and mapping the shared memory area into the address space of each participating process.
In this embodiment, it should be specifically noted that, the shared memory module needs to create a memory area for sharing, map the shared memory area to a process address space, each process participating in sharing needs to map the shared memory to its own address space, when the shared memory is mapped to the process address space, the process can directly read and write the content of the shared memory, and exchange and share data between different processes can be implemented by reading and writing the shared memory, and when performing the reading and writing operation of the shared memory, a synchronization and mutual exclusion mechanism is used to control the access of the process to the shared memory.
Furthermore, a mutual exclusion mechanism is created in the shared memory area and used for controlling the access of the processes to the shared resource, and each process needs to perform synchronous operation before accessing the shared resource, so that only one process can access the shared resource, after the processes obtain the permission of the mutual exclusion mechanism, the processes access the shared resource, and in the shared memory, the processes can perform read-write operation on the shared resource, so that the consistency and the correctness of data are ensured.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.
Claims (5)
1. An audio-video synchronization apparatus, comprising:
a media capturing module: respectively acquiring audio and video data through a microphone and camera equipment connected with the system;
and the audio and video coding module is used for: the obtained original audio and video data are compressed and encoded, and the specific operation steps of the audio and video encoding module are as follows:
s1, acquiring original audio and video data from a media capturing module;
s2, preprocessing operation of noise reduction, redundant information removal and resolution adjustment is carried out on the audio and video data;
s3, coding video data through a video coding standard, dividing the acquired original video data into a plurality of blocks with equal size, performing color space conversion and predictive filtering on each block, and performing 4 on each blockAnd (3) carrying out quantization to obtain a frequency domain coefficient, wherein the specific calculation mode is as follows:
，
wherein round represents rounding, coe represents transformed coefficients, qs represents quantization step size, sc represents quantization matrix value, prediction is performed by using information of blocks coded around the current block to obtain a prediction residual, and a specific calculation formula of intra-frame prediction is as follows:
，
where u and v represent prediction modes, x represents a variable of pixel values of the reference block, Y represents pixel values in the current block, Y represents a prediction value,is a parameter that determines the location of the reference pixel, P is the reference pixel,/>as the weight, the weight coefficientThe specific calculation mode for correcting the prediction error and the prediction residual error is as follows:
，
wherein, R represents the pixel value of the current block, the information of the adjacent frames coded before is utilized to estimate motion, a motion vector is obtained, the motion vector is used for carrying out motion compensation on the current block, and the specific calculation mode of the motion compensation is as follows:
，
wherein SAD represents a block difference function, p represents a current block, q represents a reference block, and the current frame is compared with other encoded frames to obtain a prediction residual, and a specific calculation formula of inter-frame prediction is as follows:
，
wherein MV is a motion vector, and interp represents an interpolation function, entropy coding is carried out on the obtained prediction residual, a decompressor carries out inverse quantization and inverse transformation on the compressed data to obtain a reconstructed frequency domain coefficient, and the reconstructed frequency domain coefficient is subjected to inverse transformation to obtain a final reconstructed image;
s4, compressing the audio and video data, firstly opening an audio and video equipment node, initializing the audio and video equipment, acquiring a frame of image, and if the image is successfully acquired, compressing the data, otherwise, re-acquiring the data, stamping a time stamp on the compressed data, and writing the compressed data into a shared memory;
and a data transmission module: the method comprises the steps of collecting performance information of an audio and video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process, calculating the abnormal coefficients, setting the abnormal coefficients in the transmission process as NM, and specifically obtaining the following steps:
s1, acquiring an optimal transmission time range of an audio and video in a transmission process, and calibrating the optimal transmission time range asWherein->And->Representing the maximum transmission time and the minimum transmission time in the audio and video transmission process respectively;
s2, acquiring actual transmission speeds of the data transmission module at different moments in different time, and marking the actual reading speed asX represents the number of the actual transmission speed at different times in different times, x=1, 2, 3.
S3, calibrating the actual speed smaller than the optimal transmission range asJ represents a number of an actual transmission speed smaller than the optimal transmission range, j=1, 2, 3.
S4, calculating an abnormal coefficient in the transmission process, wherein a specific calculation formula is as follows:
，
s5, eliminating asynchronism generated in the audio and video transmission process, when the audio and video buffer occupiesIs of (a) soundFrequency data, video buffer memory occupies +.>Video data of (a) audio compression generating delay +.>Video compression generates delay ∈ ->The specific calculation mode is as follows:
，
time asynchronization is reduced through buffering;
and a data receiving module: setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay;
and the synchronous playing module is used for: after the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are simultaneously played, when the playing starts, video frames with the same playing time are found according to the playing time of the first frame of audio data, and are simultaneously played, the common time reference of the audio and video playing is determined, then, when N frames of audio data are played, the synchronous adjustment is carried out once, the playing time T (N) of the audio frame which is played recently and the playing time T (vd) of the video frame which is played recently are compared, when the time difference between the two is smaller than 0.08S, the adjustment is not needed, when the time difference between the two is larger than 0.08S, the synchronous adjustment is carried out, and when the time difference is larger than 0.16S, the audio and video playing is in a step-out state, so that:
，
n2=|t，
wherein n1 and n2 are integers, and r represents video positiveThe frame rate of the regular play-out,t represents the delay of the video image;
the data receiving module is provided with a secondary buffer, the secondary buffer is used for eliminating disorder of data flow, the secondary buffer comprises a time stamp, a serial number and data size information according to the data characteristics of a received RTP packet, the PT value type is designed into a linked list, the data receiving module consists of a plurality of nodes, each node stores the length of audio and video data, the time stamp and the serial number information, the maximum length and the currently occupied length of the buffer can be designated by initializing the buffer, the buffer is fully filled with the data and can automatically mark that the buffer is ready, an audio and video playing thread triggers playing of the audio and video according to the mark, the data is stored at the tail of the linked list, the data is fetched at the head of the linked list, the secondary buffer is provided with a threshold value M which is larger than the expected maximum jitter time, and the value of M can be set automatically according to the actual situation, and the specific calculation method for compensating network delay jitter is as follows:
，
wherein,respectively, the maximum and minimum delays of the network, r representing the data transmission rate.
2. The apparatus for audio-video synchronization according to claim 1, wherein: the media capturing module firstly determines the audio and video capturing equipment to be used, generates a capturing instruction after the configuration of the capturing equipment is completed, and opens the designated capturing equipment to acquire audio and video data respectively after receiving the capturing instruction.
3. The audio-video synchronization method is applied to the audio-video synchronization device according to any one of claims 1-2, and is characterized in that: the method comprises the following steps:
101. respectively acquiring audio and video data through a microphone and camera equipment connected with the system;
102. compressing and encoding the obtained original audio and video data;
103. collecting performance information of an audio/video compression packet during transmission, wherein the performance information comprises abnormal coefficients in the transmission process, and calculating the abnormal coefficients;
104. setting a secondary buffer memory to eliminate jitter phenomenon of audio and video data caused by delay;
105. after the time difference generated during synchronous playing of the audio and video data is monitored, synchronous adjustment is carried out, so that the audio and video data are played simultaneously.
4. An electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes the method for synchronizing audio and video according to claim 3 by calling the computer program stored in the memory.
5. The audio-video synchronization storage medium is applied to the audio-video synchronization device according to any one of claims 1-2 and to the electronic device according to claim 4, and is characterized in that: the shared memory system comprises a shared memory module, wherein the shared memory module needs to create a memory area for sharing, the shared memory area is mapped into a process address space, each process participating in sharing maps the shared memory into the own address space, when the shared memory is mapped into the process address space, the process can directly read and write the content of the shared memory, the data exchange and sharing can be realized among different processes through the read and write of the shared memory, and when the read and write operation of the shared memory is carried out, a synchronous and mutual exclusion mechanism is used for controlling the access of the process to the shared memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311354870.4A CN117097936B (en) | 2023-10-19 | 2023-10-19 | Audio and video synchronization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311354870.4A CN117097936B (en) | 2023-10-19 | 2023-10-19 | Audio and video synchronization method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117097936A CN117097936A (en) | 2023-11-21 |
| CN117097936B true CN117097936B (en) | 2024-02-06 |
Family
ID=88775561
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311354870.4A Active CN117097936B (en) | 2023-10-19 | 2023-10-19 | Audio and video synchronization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117097936B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN109963184A (en) * | 2017-12-14 | 2019-07-02 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and electronic equipment of audio-video network broadcasting |
| CN112423074A (en) * | 2020-11-11 | 2021-02-26 | 广州华多网络科技有限公司 | Audio and video synchronization processing method and device, electronic equipment and storage medium |
-
2023
- 2023-10-19 CN CN202311354870.4A patent/CN117097936B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN109963184A (en) * | 2017-12-14 | 2019-07-02 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and electronic equipment of audio-video network broadcasting |
| CN112423074A (en) * | 2020-11-11 | 2021-02-26 | 广州华多网络科技有限公司 | Audio and video synchronization processing method and device, electronic equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| 一种基于Android的音视频同步算法设计;王辉;田鹏辉;;工业仪表与自动化装置(第04期);全文 * |
| 在接收端实现流媒体同步控制的方法;赵淑兰;宋人杰;李广仁;;东北电力大学学报;30(第02期);第50-53页 * |
| 基于多媒体信息交互系统的音视频同步方法;陈伟;刘永红;高菡;;电子技术与软件工程(第08期);全文 * |
| 邓中亮等.基于H.264的视频编/解码与控制技术.北京邮电大学出版社,2010,第1-3章. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117097936A (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6803964B1 (en) | Method and apparatus for processing digital data | |
| JP4281309B2 (en) | Image processing apparatus, image processing method, image frame data storage medium, and computer program | |
| CA2154856C (en) | Dram mapping for a digital video decompression processor | |
| US8675728B2 (en) | Transmitting apparatus and method, and receiving apparatus and method | |
| JP2014523712A (en) | Adaptive setting of reference frame buffer based on camera and background movement | |
| WO2006066195A2 (en) | Local macroblock information buffer | |
| US9509940B2 (en) | Image output device, image output method, and recording medium | |
| KR20210002459A (en) | Systems and methods for synchronizing audio and video | |
| EP3748983A1 (en) | Video playback method, terminal apparatus, and storage medium | |
| WO2023017928A1 (en) | Video recording method and device | |
| CN117097936B (en) | Audio and video synchronization method and device, electronic equipment and storage medium | |
| WO2021057686A1 (en) | Video decoding method and apparatus, video encoding method and apparatus, storage medium and electronic device | |
| US5760847A (en) | Processing moving image data based on an arbitrarily designated frame | |
| DE69736693T2 (en) | Image signal processing apparatus and method | |
| CN114938461A (en) | Video processing method, device and equipment and readable storage medium | |
| KR100970992B1 (en) | System and method for multiplexing stereoscopic high-definition video through gpu acceleration and transporting the video with light-weight compression and storage media having program source thereof | |
| KR100891116B1 (en) | Apparatus and method for bandwidth aware motion compensation | |
| US11825088B2 (en) | Adaptively encoding video frames based on complexity | |
| JP6483850B2 (en) | Data processing method and apparatus | |
| CN114727116A (en) | Encoding method and device | |
| JP2003230148A (en) | Image data coding unit | |
| CN113691814A (en) | Video encoding method, video encoding device, electronic device, and storage medium | |
| CN112995610A (en) | Method for application in shared in-existence multi-channel video monitoring | |
| CN108124155A (en) | A kind of bit rate control method, device and electronic equipment | |
| CN113840166A (en) | Method and system for synchronizing audio and video mixing of multi-path streaming media |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |