CN106686438B - method, device and system for synchronously playing audio images across equipment - Google Patents
method, device and system for synchronously playing audio images across equipment Download PDFInfo
- Publication number
- CN106686438B CN106686438B CN201611251322.9A CN201611251322A CN106686438B CN 106686438 B CN106686438 B CN 106686438B CN 201611251322 A CN201611251322 A CN 201611251322A CN 106686438 B CN106686438 B CN 106686438B
- Authority
- CN
- China
- Prior art keywords
- audio data
- equipment
- delay time
- playing
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
Abstract
The embodiment of the invention provides a method, a device and a system for synchronously playing audio images of cross-equipment.
Description
Technical Field
the present invention relates to the field of audio image synchronization technologies, and in particular, to a method, an apparatus, and a system for cross-device audio image synchronization playback.
background
video watching on a large-screen device such as a smart television or a set-top box is becoming popular, however, in some occasions, for example, in order to avoid affecting work or study of others, a user needs to play audio on the large-screen device through other devices such as an earphone.
In the prior art, audio on a large screen device is usually played directly through other devices such as wired connection or bluetooth, but for a user, the direct wired connection is limited by the length of a wire during use, so that the use is inconvenient, and the bluetooth device has hardware limitation. Therefore, it is desirable to provide a simple and versatile method for synchronous playing of audio images across devices.
Disclosure of Invention
the embodiment of the invention aims to provide a method, a device and a system for synchronously playing cross-device audio images, so as to realize synchronous playing of audio images among different devices. The specific technical scheme is as follows:
A method for synchronous playing of audio images across devices, comprising:
the method comprises the steps that a first device obtains audio data locally and sends the audio data to a second device;
And the first equipment acquires audio delay time, and plays an image corresponding to the audio data after the audio delay time is reached.
The method as above, wherein the first device locally acquires audio data and sends the audio data to the second device, comprising:
the first equipment closes local sound output and obtains the audio data through system recording;
and the first equipment encodes the audio data and then sends the encoded audio data to the second equipment.
The method as above, wherein the first device obtaining an audio delay time comprises:
The first equipment is used for timing to obtain coding delay time; said encoding delay time is the time taken by said first device to encode a first pulse code modulation, PCM, packet of said audio data;
the first device sends a network sending delay request to the second device, obtains the average network delay time for sending the audio data between the first device and the second device, and obtains the decoding delay time, the caching delay time and the playing delay time of the second device;
and the first device calculates the sum of the coding delay time, the average network delay time, the decoding delay time of the second device, the caching delay time and the playing delay time to obtain the audio delay time.
the method as described above, where the obtaining, by the first device, an average network delay time for transmitting the audio data with the second device by transmitting a network transmission delay request to the second device, includes:
The first device records the sending time of the network sending delay request sent to the second device and the receiving time of the network sending delay response sent by the second device; wherein, the network sending delay response includes the decoding delay time, the caching delay time and the playing delay time of the second device;
and the first equipment calculates the average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time.
the method as described above, wherein the first device obtains an audio delay time, and after playing the image corresponding to the audio data after reaching the audio delay time, the method further includes:
the first equipment receives a playing pause command sent by the second equipment and pauses playing of the image;
And the first equipment sends a pause playing response to the second equipment and sends silent audio data to the second equipment.
the method as described above, wherein after the first device sends a pause playback response to the second device and pauses playback of the image, further comprising:
the first equipment receives a playing resuming command sent by the second equipment and resumes playing the image;
and the first equipment sends a play resuming response to the second equipment and resumes sending normal audio data to the second equipment.
The method as described above, wherein the first device obtains an audio delay time, and after playing the image corresponding to the audio data after reaching the audio delay time, the method further includes:
The first equipment receives a dragging command sent by the second equipment; the dragging command comprises a fast forward command or a fast backward command; the fast forward command is used for fast forward playing the image, and the fast backward command is used for fast backward playing the image;
the first device sends a dragging response to the second device;
and the first equipment acquires the audio data again through system recording according to the dragging command and sends the audio data to the second equipment.
The method as above, wherein before the first device locally acquires the audio data and sends the audio data to the second device, further comprising:
the first equipment receives a synchronous connection establishment request sent by the second equipment; wherein, the synchronous connection establishment request comprises the internet protocol IP address information and the port information of the second equipment;
And the first equipment establishes synchronous connection with the second equipment according to the IP address information and the port information.
A method for synchronous playing of audio images across devices, comprising:
The second equipment receives the audio data sent by the first equipment;
The second device decodes and buffers the audio data;
and the second equipment plays the audio data after reaching the preset cache delay time.
The method as described above, further comprising:
the second equipment decodes the audio data and counts time to obtain decoding delay time; the decoding delay time is the time taken by the second device to decode a first pulse code modulation, PCM, packet of the audio data.
The method as described above, wherein before the second device plays the audio data after reaching the preset buffering delay time, the method further includes:
The second equipment receives a network transmission delay request sent by the first equipment;
The second equipment sends a network sending delay response to the first equipment; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
The method as described above, wherein after the second device plays the audio data after reaching the preset buffering delay time, the method further includes:
The second device records a system time P from the beginning of playing the audio data;
the second device clocks each PCM packet of the audio datacounting playing timeWherein α is the sampling rate of the audio data, β is the coding frame length of the audio data, qiindicating the time taken to play the ith said PCM packet;
the second equipment is according to the system time P and the playing time QnAnd judging whether to play the current PCM data packet.
the method as above, wherein the second device is based on the system time P and the play time Qnjudging whether to play the current PCM data packet or not, including:
If the second device judgesdiscarding the current PCM packet;
If the second device judgesPlaying the current PCM data packet;
the second device updates the playing time to
Where α is the audio sampling rate and β is the coding frame length.
The method as described above, wherein after the second device plays the audio data after reaching the preset buffering delay time, the method further includes:
the second equipment sends a pause playing command to the first equipment;
the second equipment receives a pause playing response and silent audio data sent by the first equipment;
The second device plays the silent audio data.
The method as described above, wherein after the second device plays the silent audio data, further comprising:
the second equipment sends a play resuming command to the first equipment;
And the second equipment receives the play resuming response sent by the first equipment and resumes playing the audio data.
the method as described above, wherein after the second device plays the audio data after reaching the preset buffering delay time, the method further includes:
the second equipment sends a dragging command to the first equipment, wherein the dragging command comprises a fast forward command or a fast backward command;
The second equipment receives the dragging response sent by the first equipment;
the second device empties the cached audio data;
and the second equipment receives the audio data sent by the first equipment again and buffers the audio data.
the method as above, wherein before the second device receives the audio data sent by the first device, the method further includes:
The second equipment sends a synchronous connection establishment request to the first equipment; wherein, the synchronous connection establishment request comprises the internet protocol IP address information and the port information of the second equipment; the synchronous connection establishment request is used for requesting the establishment of synchronous connection with the first equipment.
an apparatus for synchronized playback of audio images across devices, comprising:
The acquisition and sending module is used for acquiring the audio data locally and sending the audio data to the second equipment; acquiring audio delay time;
and the image playing module is used for playing the image corresponding to the audio data after the audio delay time is reached.
the apparatus as described above, wherein the acquiring and sending module is specifically configured to:
closing local sound output, and acquiring the audio data through system recording; the audio data are sent to the second equipment after being coded;
Timing to obtain coding delay time; said encoding delay time is the time taken by said first device to encode a first pulse code modulation, PCM, packet of said audio data;
recording the sending time of the network sending delay request sent to the second equipment and the receiving time of the network sending delay response sent by the second equipment; wherein, the network sending delay response includes the decoding delay time, the caching delay time and the playing delay time of the second device;
calculating to obtain the average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time;
And calculating the sum of the coding delay time, the average network delay time, the decoding delay time, the caching delay time and the playing delay time to obtain the audio delay time.
the apparatus as described above, further comprising:
The receiving module is used for receiving a pause playing command sent by the second equipment;
The acquisition and sending module is further configured to send a play pause response and silent audio data to the second device;
the image playing module is also used for pausing the playing of the image.
An apparatus for synchronized playback of audio images across devices, comprising:
The receiving module is used for receiving audio data sent by first equipment;
The buffer module is used for decoding and buffering the audio data;
and the audio playing module is used for playing the audio data after the preset cache delay time is reached.
The apparatus as described above, further comprising:
the timing module is used for timing to obtain decoding delay time when the cache module decodes the audio data; said decoding delay time is the time taken by said second device to decode a first pulse code modulation, PCM, packet of said audio data;
The receiving module is further configured to receive a network transmission delay request sent by the first device;
The device further comprises: a sending module, configured to send a network sending delay response to the first device; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
The apparatus as described above, wherein the timing module is further configured to:
Recording system time P when the audio playing module starts playing the audio data;
timing of PCM packets for each play of said audio dataCounting playing timewherein α is the sampling rate of the audio data, β is the coding frame length of the audio data, qiindicating the time taken to play the ith said PCM packet;
The device further comprises: a judging module for judging the system time P and the playing time Qnjudging whether to play the current PCM data packet;
if it is judged thatDiscarding the current PCM packet;
If it is judged thatplaying the current PCM data packet;
Updating the playing time to
where α is the audio sampling rate and β is the coding frame length.
The apparatus as described above, further comprising:
a sending module, configured to send a pause playing command to the first device;
the receiving module is further configured to receive a play pause response and silent audio data sent by the first device;
the audio playing module is further configured to play the silent audio data.
A system for synchronized audio image playback, the system comprising the apparatus for synchronized audio image playback across devices of any one of claims 18-20, and the apparatus for synchronized audio image playback across devices of any one of claims 21-24.
According to the method, the device and the system for synchronously playing the audio images of the cross-device, the first device obtains the audio data locally and sends the audio data to the second device, obtains the audio delay time, and plays the images after the audio delay time is reached, so that the audio data is obtained by recording through the system, the audio data of the television-end device is transmitted back to the mobile phone and other devices, the audio playing delay time is obtained at the same time, the images are played after the audio delay time is reached, and the synchronous playing of the audio images is realized. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
in order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
fig. 1 is a flowchart of a method for synchronously playing audio images across devices according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for synchronously playing audio images across devices according to a second embodiment of the present invention;
Fig. 3 is a schematic diagram illustrating that the first device obtains the average network delay time in the method for synchronously playing the audio image across devices according to the second embodiment of the present invention;
fig. 4 is a flowchart of a method for synchronously playing audio images across devices according to a third embodiment of the present invention;
fig. 5 is a flowchart of a method for synchronously playing audio images across devices according to a fourth embodiment of the present invention;
Fig. 6 is a schematic structural diagram of an apparatus for synchronously playing audio and video images across devices according to an embodiment of the present invention;
Fig. 7 is a schematic structural diagram of an apparatus for synchronously playing audio and video across devices according to a second embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
in each embodiment of the present invention, the first device may be a large screen device such as a smart television, the second device may be a mobile phone, and the like, and the system recording function of the smart television or the set top box is utilized to encode audio data obtained by recording, send the encoded audio data to the mobile phone through the wireless network, decode the encoded audio data by the mobile phone, and return the delay time of the whole audio return play to the smart television, so that the video player delays the play of the image, and thereby the synchronous play of the image at the television end and the audio at the mobile phone end is realized.
fig. 1 is a flowchart of a method for synchronously playing audio and video images across devices according to an embodiment of the present invention. As shown in fig. 1, the method provided in this embodiment may be executed by a first device, such as a large-screen device like a smart television, and specifically, the method provided in this embodiment includes:
Step 101, the first device locally obtains audio data and sends the audio data to the second device.
in this embodiment, the first device uses a system recording function to encode the recorded audio data and send the encoded audio data to the second device through the wireless network, so that the audio data is obtained through system recording, and the second device is very simple to adapt to a third-party player on the first device.
in practical application, when the first device locally obtains audio data, the first device closes local sound output and obtains the audio data through system recording; and encoding the audio data and then sending the encoded audio data to the second device.
and 102, the first device acquires audio delay time, and plays an image corresponding to the audio data after the audio delay time is reached.
In this step, the audio delay time includes an encoding delay time, a decoding delay time, a buffering delay time, a playing delay time, and an average network delay time.
It should be noted that the coding delay time and the average network delay time are calculated by the first device, and specifically, the coding delay time is a time taken by the first device to encode a first Pulse Code Modulation (PCM) packet of the audio data; the average network delay time is a network delay time for transmitting the audio data between the first device and the second device.
it should be further noted that the decoding delay time, the buffering delay time, and the playing delay time are obtained by the second device and sent to the first device; the decoding delay time is a time taken by the second device to decode the first PCM data packet of the audio data, the buffering delay time is preset, and the playing delay time is a time required by the second device to write the audio data to be played into the playing device.
In this step, after the audio delay time is obtained, the first device starts to play the image corresponding to the audio data after the audio delay time is reached, so as to realize synchronous playing of the image and the audio.
in the technical solution of this embodiment, the first device locally acquires the audio data and sends the audio data to the second device, acquires the audio delay time, and plays the image corresponding to the audio data after the audio delay time is reached. Therefore, the system is used for recording to obtain audio data, the audio data of the first device is transmitted back to the second device, meanwhile, the delay time of audio playing is obtained, the image is played after the audio delay time is reached, and synchronous playing of the audio image is achieved.
Fig. 2 is a flowchart of a cross-device audio image synchronous playing method according to a second embodiment of the present invention. As shown in fig. 2, on the basis of the foregoing embodiment, in this embodiment, the step 102 of acquiring, by the first device, the audio delay time specifically includes:
step 1021, the first device times to obtain coding delay time; the encoding delay time is a time taken by the first device to encode a first PCM packet of the audio data.
In practical applications, the first device times an encoding delay time T from the time when the first PCM packet is taken to the time when the first PCM packet is encoded to obtain the first frame of encoded audio data1。
step 1022, the first device obtains an average network delay time for transmitting the audio data with the second device by transmitting a network transmission delay request to the second device, and obtains a decoding delay time, a buffering delay time, and a playing delay time of the second device.
Specifically, the first device calculates the average network delay time by sending the network transmission delay request multiple times.
In this step, the first device records the sending time of the network sending delay request sent to the second device and the receiving time of the network sending delay response sent by the second device; wherein, the network sending delay response includes the decoding delay time, the caching delay time and the playing delay time of the second device; and the first equipment calculates the average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time.
Fig. 3 is a schematic diagram of obtaining an average network delay time by a first device in a method for synchronously playing audio images across devices according to a second embodiment of the present invention, as shown in fig. 3, where the first device obtains the average network delay time at t0sending the network-sent delay request to a second device, the second device being at t1receiving the network transmission delay request and at t2sending the network-sent delay response to the first device, the first device being at t3receiving the network transmission delay response; recording the network delay time as theta,
From this, the average network delay timeWherein n is the number of the sending time and the receiving time recorded by the first device, and i represents the ith network delay time acquired by the first device.
step 1023, the first device calculates the sum of the coding delay time, the average network delay time, the decoding delay time of the second device, the buffering delay time and the playing delay time to obtain the audio delay time.
According to the technical scheme of the embodiment, the first device sends the audio data to the second device, obtains the audio delay time, and starts to play the image after the audio delay time is reached, so that the audio image is synchronously played.
Fig. 4 is a flowchart of a method for synchronously playing audio and video images across devices according to a third embodiment of the present invention. As shown in fig. 4, on the basis of the foregoing embodiment, before the foregoing step 101, the method provided by this embodiment further includes:
step 201, the first device receives a synchronous connection establishment request sent by the second device; wherein the synchronous connection establishment request includes Internet Protocol (IP) address information and port information of the second device.
It should be noted that, when the second device sends the synchronous connection establishment request to the target first device, a data receiving and control service (server) is created, and meanwhile, the IP address information and the port information of the second device are sent to the first device.
Step 202, the first device establishes a synchronous connection with the second device according to the IP address information and the port information.
specifically, after receiving the synchronous connection establishment request, the first device creates a client (client) to connect to the data and control service (server) of the second device, thereby completing the synchronous connection between the first device and the second device.
Further, in practical applications, operation scenarios such as pause playing, resume playing, fast forward and fast backward may be encountered, which will be described below.
Specifically, for a scenario of playing pause (past) and a scenario of playing resume (resume), the first device receives a command of playing pause sent by the second device; and after receiving the pause playing command, pausing the playing of the image, and simultaneously sending a pause playing response to the second equipment by the first equipment. But the system recording is continued at this time, only the recorded audio is silent audio data, and the first device sends the silent audio data obtained by recording to the second device, so that the second device plays the silent audio data, that is, during the pause of playing, the second device plays silent audio data, thereby ensuring the synchronous playing of audio and images.
And the first equipment resumes playing the image after receiving a playback resuming command sent by the second equipment, sends a playback resuming response to the second equipment, and resumes sending normal audio data to the second equipment. That is, upon receiving the resume play command, the first device then plays the image before the pause and resumes transmitting normal audio data to the second device.
Specifically, for a fast-forward and fast-backward scene, the first device receives a drag command (seek) sent by the second device and sends a drag response to the second device after finishing the drag operation of the image playing. At this time, the system recording acquires new audio data after the dragging operation and sends the new audio data to the second device; it is understood that the drag command may be a fast forward command for fast forward playing the image or a fast backward command for fast backward playing the image.
for example, after the first device receives the fast forward command, the image playing skips to the time point of fast forward by the user, and sends a fast forward response to the second device. At this time, the system recording may obtain audio data played after the fast forward time point, and send the encoded audio data to the second device.
in this embodiment, after the synchronous connection is established between the first device and the second device, the first device sends the locally acquired audio data to the second device, acquires the audio delay time, and plays the image corresponding to the audio data after the audio delay time is reached, so that the synchronous playing of the audio image is realized, and the synchronous playing of the audio image can be maintained while operations such as pause and dragging are supported.
fig. 5 is a flowchart of a method for synchronously playing audio and video images across devices according to a fourth embodiment of the present invention. As shown in fig. 5, the method provided in this embodiment may specifically include:
step 301, the second device receives the audio data sent by the first device.
before this step, the second device sends a synchronous connection establishment request to the first device; wherein, the synchronous connection establishment request comprises the internet protocol IP address information and the port information of the second equipment; the synchronous connection establishment request is used for requesting the establishment of synchronous connection with the first equipment.
It should be noted that, when the second device sends the synchronous connection establishment request to the target first device, a data receiving and control service (server) is created, and meanwhile, the IP address information and the port information of the second device are sent to the first device.
after receiving the synchronous connection establishment request, the first device creates a client (client) to connect to a data and control service (server) of the second device, thereby completing the synchronous connection between the first device and the second device.
It will be appreciated that the transmission of audio data may only take place after a synchronous connection has been established between the second device and the first device.
step 302, the second device decodes and buffers the audio data.
In this step, the second device performs timing to obtain a decoding delay time T while decoding the audio data3(ii) a It is to be understood that the decoding delay time is the time taken by the second device to decode the first PCM packet of audio data.
And step 303, the second device plays the audio data after reaching a preset buffering delay time.
specifically, before this step, the second device further receives a network transmission delay request sent by the first device; and sending a network transmission delay response to the first device; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
It should be noted that the buffering delay time T4is preset, the playing delay time T5is a parameter inherent and known to the apparatus, according to said second settingthe devices are different.
In particular, the second device may be according to a formulacalculating to obtain the cache delay time T4The number N of the corresponding PCM data packets needing to be cached; wherein α is the sampling rate of the audio data, and β is the coding frame length of the audio data.
in the method for synchronously playing audio images across devices provided by this embodiment, after a synchronous connection is established between the first device and the second device, the second device receives audio data sent by the first device; the second device decodes and buffers the audio data; and playing the audio data after the preset cache delay time is reached, thereby realizing the synchronous playing of the audio images.
It can be understood that, sometimes, due to network instability or blocking, the transmitted audio data does not arrive at the second device in time, so that the audio data buffered by the second device is played completely, when the network recovers, the blocked audio data arrives later than the corresponding image, at this time, it needs to be determined whether to delay, and if so, the blocked audio data is discarded until the audio data is played in synchronization.
specifically, on the basis of the foregoing embodiment, after the foregoing step 303, the method provided in this embodiment further includes:
The second device records a system time P from the beginning of playing the audio data; the second device clocks each PCM packet of the audio datacounting playing timeThe second equipment is according to the system time P and the playing time QnJudging whether to play the current PCM data packet; where α is the sampling rate of the audio data and β is the encoded frame of the audio datalong, qiIndicating the time taken to play the ith said PCM data packet.
specifically, the second device is configured to play the audio stream according to the system time P and the play time Qnjudging whether to play the current PCM data packet or not, including: if the second device judgesdiscarding the current PCM packet; if the second device judgesplaying the current PCM data packet; the second device updates the playing time toWhere α is the audio sampling rate and β is the coding frame length.
Further, in practical applications, operation scenarios such as pause playing, resume playing, fast forward and fast backward may be encountered, which will be described below.
Specifically, for a scenario of pause playing (pause) and a scenario of resume playing (resume), the second device sends a pause playing command to the first device; after the pause playing command is sent, the second device receives a pause playing response and silent audio data sent by the first device; the second device plays the silent audio data; that is, during the pause of the playing, the second device inserts the received silent audio data into the beginning of the buffered audio data queue for playing, and plays the silent audio data, and at this time, the image is stopped and the sound is silent for the user, so as to ensure the synchronization of the audio and the image. Further, in order to avoid noise caused by a time difference between the control command and the network transmission of the audio data, the second device may further turn down the played audio to be silent.
The second equipment sends a play resuming command to the first equipment; and receiving a play resuming response sent by the first device, and resuming playing the audio data. In practical application, after the second device resumes playing, the second device inserts the received audio data sent by the first device into the end of the buffered audio data queue, and resumes playing volume.
specifically, for a fast-forward and fast-backward scene, the second device sends a dragging command to the first device, receives a dragging response sent by the first device, empties the cached audio data, and receives and caches the audio data sent by the first device again; wherein, the dragging command includes a fast forward command or a fast backward command.
Specifically, the second device clears the cached audio data, and resets the system time P and the play time Qnand if the number of the PCM data packets is zero, the audio data sent by the first equipment is received again and buffered, and the audio is played after the N PCM data packets are buffered again.
In the method for synchronously playing audio images across devices provided by this embodiment, after the synchronous connection is established between the first device and the second device, the first device sends the locally acquired audio data to the second device, acquires the audio delay time, and plays the images after the audio delay time is reached, so that the synchronous playing of the audio images among different devices is realized, and the synchronous playing of the audio images can be maintained while operations such as pausing and dragging are supported.
Fig. 6 is a schematic structural diagram of an apparatus for synchronously playing audio and video across devices according to an embodiment of the present invention. As shown in fig. 6, the apparatus 60 for synchronously playing audio images across devices provided by the present embodiment may include: a capture and transmission module 61 and an image playing module 62.
The acquiring and sending module 61 is configured to locally acquire audio data and send the audio data to the second device; acquiring audio delay time;
And an image playing module 62, configured to play an image corresponding to the audio data after the audio delay time is reached.
Specifically, when the acquiring and sending module 61 acquires the audio data locally and sends the audio data to the second device, the local sound output is closed, and the audio data is acquired through system recording; and encoding the audio data and then sending the encoded audio data to the second equipment.
when the obtaining and sending module 61 obtains the audio delay time, timing to obtain coding delay time; the encoding delay time is a time taken by the first device to encode a first PCM packet of the audio data; acquiring the average network delay time for transmitting the audio data with the second equipment, the decoding delay time, the caching delay time and the playing delay time of the second equipment by transmitting a network transmission delay request to the second equipment; and calculating the sum of the coding delay time, the average network delay time, the decoding delay time, the caching delay time and the playing delay time to obtain the audio delay time.
When the obtaining and sending module 61 obtains the average network delay time, specifically, by recording a sending time of sending the network sending delay request to the second device and a receiving time of receiving a network sending delay response sent by the second device; calculating to obtain the average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time; wherein the network transmission delay response includes the decoding delay time, the buffering delay time and the playing delay time of the second device.
Further, in consideration of the practical application that operation scenarios such as pause playing, resume playing, fast forward and fast backward are encountered, the apparatus 60 provided in this embodiment may further include: and the receiving module is used for receiving the pause playing command sent by the second equipment.
Correspondingly, the obtaining and sending module 61 is further configured to send a pause playing response and silent audio data to the second device; the image playing module 62 is further configured to pause playing the image.
When the playing is resumed, the receiving module is further configured to receive a resume playing command sent by the second device; the acquiring and sending module 61 is further configured to send a playback resuming response to the second device, and the image playing module 62 is further configured to resume playing the image.
when the second device is in fast forward or fast backward, the receiving module is used for receiving a dragging command sent by the second device; the dragging command comprises a fast forward command or a fast backward command; the fast forward command is used for fast forward playing the image, and the fast backward command is used for fast backward playing the image; the acquiring and sending module 61 is further configured to send a dragging response to the second device; and acquiring the audio data again through system recording according to the dragging command and sending the audio data to the second equipment.
it should be noted that, before locally acquiring the audio data, the receiving module is configured to receive a synchronous connection establishment request sent by the second device; wherein, the synchronous connection establishment request includes the IP address information and the port information of the second device; and the connection establishing module is used for establishing synchronous connection with the second equipment according to the IP address information and the port information.
the apparatus for synchronously playing audio and video across devices provided in this embodiment may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, and are not described herein again.
Fig. 7 is a schematic structural diagram of an apparatus for synchronously playing audio and video across devices according to a second embodiment of the present invention. As shown in fig. 7, the apparatus 70 for synchronously playing audio images across devices provided by the present embodiment may include: a receiving module 71, a buffer module 72 and an audio playing module 73.
the receiving module 71 is configured to receive audio data sent by a first device;
a buffer module 72, configured to decode and buffer the audio data;
And the audio playing module 73 is configured to play the audio data after the preset buffering delay time is reached.
In practical applications, the apparatus 70 provided in this embodiment further includes: a timing module, configured to time to obtain a decoding delay time while the buffer module 72 decodes the audio data; the decoding delay time is a time taken by the second device to decode a first PCM packet of the audio data.
further, the receiving module 71 is further configured to receive a network transmission delay request sent by the first device; the apparatus 70 further comprises: a sending module, configured to send a network sending delay response to the first device; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
further, sometimes, because the network is unstable or blocked, the sent audio data does not arrive at the second device in time, so that the audio data buffered by the second device is played completely, when the network is recovered, the blocked audio data arrives later than the corresponding image, at this time, whether the delay is needed to be judged, and if the delay is needed, the blocked audio data is discarded until the audio data is played in synchronization.
in this embodiment, the timing module is further configured to: recording a system time P from when the playing module 73 starts playing the audio data; timing of PCM packets for each play of said audio dataCounting playing timeWherein α is the sampling rate of the audio data, β is the coding frame length of the audio data, qiIndicating the time taken to play the ith said PCM packet; the apparatus 70 further comprises: a judging module for judging the system time P and the playing time QnAnd judging whether to play the current PCM data packet.
wherein, the judging module is specifically configured to: if it is judged thatDiscarding the current PCM packet; if it is judged thatplaying the current PCM data packet; updating the playing time toWhere α is the audio sampling rate and β is the coding frame length.
Further, in consideration of the practical application that operation scenarios such as pause playing, resume playing, fast forward and fast backward are encountered, the apparatus 70 provided in this embodiment may further include: a sending module, configured to send a pause playing command to the first device; the receiving module 71 is further configured to receive a pause playing response and silent audio data sent by the first device; the audio playing module 73 is also used for playing the silent audio data.
When the playing is resumed, the sending module is further configured to send a resume playing command to the first device; the receiving module 71 is further configured to receive a playback resuming response sent by the first device, and the audio playing module 73 is further configured to resume playing the audio data.
When fast forwarding or fast rewinding, the sending module is configured to send a dragging command to the first device, where the dragging command includes a fast forwarding command or a fast rewinding command; the receiving module 71 is further configured to receive a drag response sent by the first device; the buffer module 72 is further configured to empty the buffered audio data; the receiving module 71 is further configured to re-receive the audio data sent by the first device; the buffering module 72 is further configured to buffer the audio data re-received by the receiving module.
it should be noted that, before receiving the audio data sent by the first device, the sending module is configured to send a synchronous connection establishment request to the first device; wherein, the synchronous connection establishment request includes the IP address information and the port information of the second device; the synchronous connection establishment request is used for requesting the establishment of synchronous connection with the first equipment.
The apparatus for synchronously playing audio and video across devices provided in this embodiment may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, and are not described herein again.
The invention also provides a system for synchronously playing the audio images across the equipment. The system provided in this embodiment includes the device 60 for synchronously playing the audio images across devices provided in the first apparatus embodiment, and the device 70 for synchronously playing the audio images across devices provided in the second apparatus embodiment, which have similar implementation principles and technical effects, and are not described herein again.
it is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
all the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (22)
1. A method for synchronously playing audio images across devices is characterized by comprising the following steps:
The method comprises the steps that a first device obtains audio data locally and sends the audio data to a second device;
The first equipment acquires audio delay time, and plays an image corresponding to the audio data after the audio delay time is reached;
the first device obtaining an audio delay time comprises:
The first equipment is used for timing to obtain coding delay time; said encoding delay time is the time taken by said first device to encode a first pulse code modulation, PCM, packet of said audio data;
The first device sends a network sending delay request to the second device, obtains the average network delay time for sending the audio data between the first device and the second device, and obtains the decoding delay time, the caching delay time and the playing delay time of the second device;
and the first device calculates the sum of the coding delay time, the average network delay time, the decoding delay time of the second device, the caching delay time and the playing delay time to obtain the audio delay time.
2. The method of claim 1, wherein the first device locally obtains audio data and sends the audio data to the second device, comprising:
The first equipment closes local sound output and obtains the audio data through system recording;
And the first equipment encodes the audio data and then sends the encoded audio data to the second equipment.
3. the method of claim 1, wherein the obtaining, by the first device, an average network latency time for transmitting the audio data with the second device by transmitting a network transmission latency request to the second device, comprises:
The first device records the sending time of the network sending delay request sent to the second device and the receiving time of the network sending delay response sent by the second device; wherein, the network sending delay response includes the decoding delay time, the caching delay time and the playing delay time of the second device;
And the first equipment calculates the average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time.
4. The method of claim 1, wherein the first device obtains an audio delay time, and further comprises, after playing the image corresponding to the audio data after reaching the audio delay time:
the first equipment receives a playing pause command sent by the second equipment and pauses playing of the image;
and the first equipment sends a pause playing response to the second equipment and sends silent audio data to the second equipment.
5. The method of claim 4, wherein after the first device pauses playback of the image and sends a pause playback response to the second device, further comprising:
The first equipment receives a playing resuming command sent by the second equipment and resumes playing the image;
and the first equipment sends a play resuming response to the second equipment and resumes sending normal audio data to the second equipment.
6. The method of claim 1, wherein the first device obtains an audio delay time, and further comprises, after playing the image corresponding to the audio data after reaching the audio delay time:
The first equipment receives a dragging command sent by the second equipment; the dragging command comprises a fast forward command or a fast backward command; the fast forward command is used for fast forward playing the image, and the fast backward command is used for fast backward playing the image;
the first device sends a dragging response to the second device;
And the first equipment acquires the audio data again through system recording according to the dragging command and sends the audio data to the second equipment.
7. The method of claim 1, wherein prior to the first device locally obtaining audio data and transmitting the audio data to the second device, further comprising:
The first equipment receives a synchronous connection establishment request sent by the second equipment; wherein, the synchronous connection establishment request comprises the internet protocol IP address information and the port information of the second equipment;
And the first equipment establishes synchronous connection with the second equipment according to the IP address information and the port information.
8. A method for synchronously playing audio images across devices is characterized by comprising the following steps:
The second equipment receives the audio data sent by the first equipment;
The second device decodes and buffers the audio data;
the second device plays the audio data after reaching the preset cache delay time;
Before the second device plays the audio data after reaching the preset buffering delay time, the method further includes:
The second equipment receives a network transmission delay request sent by the first equipment;
The second equipment sends a network sending delay response to the first equipment; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
9. The method of claim 8, further comprising:
the second equipment decodes the audio data and counts time to obtain decoding delay time; the decoding delay time is the time taken by the second device to decode a first pulse code modulation, PCM, packet of the audio data.
10. the method according to claim 8 or 9, wherein after the second device plays the audio data after reaching the preset buffering delay time, the method further comprises:
The second device records a system time P from the beginning of playing the audio data;
The second device clocks each PCM packet of the audio dataCounting playing timeWherein α is the sampling rate of the audio data, β is the coding frame length of the audio data, qiIndicating the time taken to play the ith said PCM packet;
the second equipment is according to the system time P and the playing time Qnand judging whether to play the current PCM data packet.
11. The method according to claim 10, wherein the second device is configured to play the stream according to the system time P and the play time QnJudging whether to play the current PCM data packet or not, including:
If the second device judgesdiscarding the current PCM packet;
If the second device judgesPlaying the current PCM data packet;
The second device updates the playing time to
Where α is the audio sampling rate and β is the coding frame length.
12. The method of claim 8, wherein after the second device plays the audio data after reaching the preset buffering delay time, further comprising:
The second equipment sends a pause playing command to the first equipment;
The second equipment receives a pause playing response and silent audio data sent by the first equipment;
The second device plays the silent audio data.
13. The method of claim 12, wherein after the second device plays the silent audio data, further comprising:
The second equipment sends a play resuming command to the first equipment;
And the second equipment receives the play resuming response sent by the first equipment and resumes playing the audio data.
14. The method of claim 8, wherein after the second device plays the audio data after reaching the preset buffering delay time, further comprising:
The second equipment sends a dragging command to the first equipment, wherein the dragging command comprises a fast forward command or a fast backward command;
the second equipment receives the dragging response sent by the first equipment;
The second device empties the cached audio data;
and the second equipment receives the audio data sent by the first equipment again and buffers the audio data.
15. The method of claim 8, wherein before the second device receives the audio data transmitted by the first device, the method further comprises:
the second equipment sends a synchronous connection establishment request to the first equipment; wherein, the synchronous connection establishment request comprises the internet protocol IP address information and the port information of the second equipment; the synchronous connection establishment request is used for requesting the establishment of synchronous connection with the first equipment.
16. An apparatus for synchronized playback of audio images across devices, comprising:
the acquisition and sending module is used for acquiring the audio data locally and sending the audio data to the second equipment; acquiring audio delay time;
The image playing module is used for playing an image corresponding to the audio data after the audio delay time is reached;
The acquisition and sending module is specifically configured to:
Timing to obtain coding delay time; said encoding delay time is the time taken by the first device to encode a first pulse code modulation, PCM, data packet of said audio data;
Recording the sending time of sending a network sending delay request to the second equipment and the receiving time of receiving a network sending delay response sent by the second equipment; wherein, the network sending delay response includes the decoding delay time, the caching delay time and the playing delay time of the second device;
Calculating to obtain average network delay time according to the sending time, the receiving time and the recorded quantity of the sending time and the receiving time;
and calculating the sum of the coding delay time, the average network delay time, the decoding delay time, the caching delay time and the playing delay time to obtain the audio delay time.
17. the apparatus of claim 16, wherein the acquisition sending module is further configured to:
Closing local sound output, and acquiring the audio data through system recording; and encoding the audio data and then sending the encoded audio data to the second equipment.
18. the apparatus of claim 16, further comprising:
The receiving module is used for receiving a pause playing command sent by the second equipment;
The acquisition and sending module is further configured to send a play pause response and silent audio data to the second device;
The image playing module is also used for pausing the playing of the image.
19. An apparatus for synchronized playback of audio images across devices, comprising:
The receiving module is used for receiving audio data sent by first equipment;
the buffer module is used for decoding and buffering the audio data;
the audio playing module is used for playing the audio data after the preset cache delay time is reached;
the timing module is used for timing to obtain decoding delay time when the cache module decodes the audio data; said decoding delay time is the time taken by the second device to decode a first pulse code modulation, PCM, data packet of said audio data;
The receiving module is further configured to receive a network transmission delay request sent by the first device;
the device further comprises: a sending module, configured to send a network sending delay response to the first device; wherein the network transmission delay response includes the decoding delay time, the buffering delay time, and the playing delay time of the second device.
20. The apparatus of claim 19, wherein the timing module is further configured to:
Recording system time P when the audio playing module starts playing the audio data;
timing of PCM packets for each play of said audio dataCounting playing timeWherein α is the sampling rate of the audio data, β is the coding frame length of the audio data, qiindicating the time taken to play the ith said PCM packet;
The device further comprises: a judging module for judging the system time P and the playing time QnJudging whether to play the current PCM data packet;
if it is judged thatdiscarding the current PCM packet;
if it is judged thatPlaying the current PCM data packet;
Updating the playing time to
where α is the audio sampling rate and β is the coding frame length.
21. The apparatus of claim 19, further comprising:
a sending module, configured to send a pause playing command to the first device;
The receiving module is further configured to receive a play pause response and silent audio data sent by the first device;
the audio playing module is further configured to play the silent audio data.
22. a system for synchronized audio image playback, the system comprising the apparatus for synchronized audio image playback across devices of any one of claims 16-18, and the apparatus for synchronized audio image playback across devices of any one of claims 19-21.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611251322.9A CN106686438B (en) | 2016-12-29 | 2016-12-29 | method, device and system for synchronously playing audio images across equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611251322.9A CN106686438B (en) | 2016-12-29 | 2016-12-29 | method, device and system for synchronously playing audio images across equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106686438A CN106686438A (en) | 2017-05-17 |
| CN106686438B true CN106686438B (en) | 2019-12-13 |
Family
ID=58872637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611251322.9A Active CN106686438B (en) | 2016-12-29 | 2016-12-29 | method, device and system for synchronously playing audio images across equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106686438B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327724B (en) * | 2017-08-01 | 2021-08-31 | 成都鼎桥通信技术有限公司 | Audio and video synchronous playing method and device |
| TWI672956B (en) * | 2017-12-22 | 2019-09-21 | 宏正自動科技股份有限公司 | Apparatus and method for audio and video synchronization |
| CN109450490B (en) * | 2018-11-02 | 2019-11-19 | 南京中感微电子有限公司 | A kind of voice data communication equipment and system |
| CN110430457B (en) * | 2019-07-25 | 2021-09-10 | 北京奇艺世纪科技有限公司 | Method and device for playing different-end audio and video and audio playing system |
| CN110545454A (en) * | 2019-08-27 | 2019-12-06 | 北京奇艺世纪科技有限公司 | Data synchronous playing method and device |
| CN112105005A (en) * | 2019-08-30 | 2020-12-18 | 炬力(珠海)微电子有限公司 | Method and device for controlling Bluetooth equipment to play |
| CN112860211B (en) * | 2021-01-28 | 2022-12-27 | 成都极米科技股份有限公司 | Method, device, terminal and storage medium for determining time delay |
| CN114296666A (en) * | 2021-03-11 | 2022-04-08 | 海信视像科技股份有限公司 | Display device |
| CN114040237A (en) * | 2021-09-30 | 2022-02-11 | 茂佳科技(广东)有限公司 | Audio and video synchronous playing method, terminal, multimedia playing system and medium |
| CN116156233A (en) * | 2021-11-22 | 2023-05-23 | 华为终端有限公司 | Display picture synchronization method and system and electronic equipment |
| CN114554270A (en) * | 2022-02-28 | 2022-05-27 | 维沃移动通信有限公司 | Audio and video playing method and device |
| WO2024026662A1 (en) * | 2022-08-02 | 2024-02-08 | Qualcomm Incorporated | Hybrid codec present delay sync for asymmetric sound boxes |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101022561A (en) * | 2006-02-15 | 2007-08-22 | 中国科学院声学研究所 | Method for realizing MXF video file and PCM audio file synchronous broadcasting |
| CN101237586A (en) * | 2008-02-22 | 2008-08-06 | 上海华平信息技术股份有限公司 | Synchronous playing method for audio and video buffer |
| CN102638774A (en) * | 2012-03-31 | 2012-08-15 | 王方淇 | Method and system for synchronously sharing mobile terminal data |
| CN103905880A (en) * | 2014-03-13 | 2014-07-02 | 北京奇艺世纪科技有限公司 | Playing method of audio data and video data, smart television set and mobile equipment |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN104837046A (en) * | 2015-01-08 | 2015-08-12 | 腾讯科技（北京）有限公司 | Multi-media file processing method and device |
| CN105992025A (en) * | 2015-02-15 | 2016-10-05 | 深圳市民展科技开发有限公司 | Audio synchronous playing-based system time calibration method, audio synchronous playing method and devices |
-
2016
- 2016-12-29 CN CN201611251322.9A patent/CN106686438B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101022561A (en) * | 2006-02-15 | 2007-08-22 | 中国科学院声学研究所 | Method for realizing MXF video file and PCM audio file synchronous broadcasting |
| CN101237586A (en) * | 2008-02-22 | 2008-08-06 | 上海华平信息技术股份有限公司 | Synchronous playing method for audio and video buffer |
| CN102638774A (en) * | 2012-03-31 | 2012-08-15 | 王方淇 | Method and system for synchronously sharing mobile terminal data |
| CN103905880A (en) * | 2014-03-13 | 2014-07-02 | 北京奇艺世纪科技有限公司 | Playing method of audio data and video data, smart television set and mobile equipment |
| CN104618786A (en) * | 2014-12-22 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Audio/video synchronization method and device |
| CN104837046A (en) * | 2015-01-08 | 2015-08-12 | 腾讯科技（北京）有限公司 | Multi-media file processing method and device |
| CN105992025A (en) * | 2015-02-15 | 2016-10-05 | 深圳市民展科技开发有限公司 | Audio synchronous playing-based system time calibration method, audio synchronous playing method and devices |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106686438A (en) | 2017-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106686438B (en) | method, device and system for synchronously playing audio images across equipment | |
| US20230256345A1 (en) | Game data processing method and apparatus, and storage medium | |
| US7870281B2 (en) | Content playback device, content playback method, computer-readable storage medium, and content playback system | |
| CN109889543B (en) | Video transmission method, root node, child node, P2P server and system | |
| KR101704619B1 (en) | Determining available media data for network streaming | |
| US7779146B2 (en) | Methods and systems for HTTP streaming using server-side pacing | |
| CN107566918B (en) | A kind of low delay under video distribution scene takes the neutrel extraction of root | |
| US8711929B2 (en) | Network-based dynamic encoding | |
| CN107819809B (en) | Method and device for synchronizing content | |
| US20080148327A1 (en) | Method and Apparatus for Providing Adaptive Trick Play Control of Streaming Digital Video | |
| US8244897B2 (en) | Content reproduction apparatus, content reproduction method, and program | |
| JP5026167B2 (en) | Stream transmission server and stream transmission system | |
| US10177899B2 (en) | Adapting a jitter buffer | |
| KR20060114080A (en) | System and method of providing multimedia streaming service | |
| EP1855483A2 (en) | Apparatus and method for transmitting and receiving moving pictures using near field communication | |
| JP2004502359A (en) | Video error recovery method | |
| US8630178B2 (en) | Transmitting apparatus and transmission method | |
| CN113286184B (en) | Lip synchronization method for respectively playing audio and video on different devices | |
| CN113497792B (en) | Audio and video communication method, terminal, server, computer equipment and storage medium | |
| EP3560205A1 (en) | Synchronizing processing between streams | |
| JP2015520964A (en) | Integrated controller-based pacing for HTTP pseudo-streaming | |
| CN109862400B (en) | Streaming media transmission method, device and system | |
| JP2005322995A (en) | Buffer control method in real-time video image transfer, transmitting terminal, receiving terminal, video image distributing system and program | |
| CN106331847B (en) | Audio and video playing method and apparatus | |
| KR20140070896A (en) | Method for video streaming and an electronic device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |