WO2018170852A1 - 多设备间唇音同步方法及设备 - Google Patents

多设备间唇音同步方法及设备 Download PDF

Info

Publication number
WO2018170852A1
WO2018170852A1 PCT/CN2017/077925 CN2017077925W WO2018170852A1 WO 2018170852 A1 WO2018170852 A1 WO 2018170852A1 CN 2017077925 W CN2017077925 W CN 2017077925W WO 2018170852 A1 WO2018170852 A1 WO 2018170852A1
Authority
WO
WIPO (PCT)
Prior art keywords
rtcp
slave device
master device
port
service address
Prior art date
Application number
PCT/CN2017/077925
Other languages
English (en)
French (fr)
Inventor
孙涛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2017/077925 priority Critical patent/WO2018170852A1/zh
Priority to US16/496,306 priority patent/US11146611B2/en
Priority to CN201780049030.3A priority patent/CN109565466B/zh
Priority to EP17902312.2A priority patent/EP3591908B9/en
Publication of WO2018170852A1 publication Critical patent/WO2018170852A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04J—MULTIPLEX COMMUNICATION
    • H04J3/00—Time-division multiplex systems
    • H04J3/02—Details
    • H04J3/06—Synchronising arrangements
    • H04J3/0635—Clock or time synchronisation in a network
    • H04J3/0638—Clock or time synchronisation among nodes; Internode synchronisation
    • H04J3/0658—Clock or time synchronisation among packet nodes
    • H04J3/0673—Clock or time synchronisation among packet nodes using intermediate nodes, e.g. modification of a received timestamp before further transmission to the next packet node, e.g. including internal delay time or residence time into the packet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04J—MULTIPLEX COMMUNICATION
    • H04J3/00—Time-division multiplex systems
    • H04J3/02—Details
    • H04J3/06—Synchronising arrangements
    • H04J3/0635—Clock or time synchronisation in a network
    • H04J3/0685—Clock or time synchronisation in a node; Intranode synchronisation
    • H04J3/0697—Synchronisation in a packet node
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
    • H04L12/2854—Wide area networks, e.g. public data networks
    • H04L12/2856—Access arrangements, e.g. Internet access
    • H04L12/2869—Operational details of access network equipments
    • H04L12/287—Remote access server, e.g. BRAS
    • H04L12/2874—Processing of data for distribution to the subscribers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/28—Timers or timing mechanisms used in protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643—Communication protocols
    • H04N21/6437—Real-time Transport Protocol [RTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/91—Television signal processing therefor
    • H04N5/92—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
    • H04N5/926—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback by pulse code modulation
    • H04N5/9265—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback by pulse code modulation with processing of the sound signal
    • H04N5/9267—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback by pulse code modulation with processing of the sound signal using time division multiplex of the PCM audio and PCM video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Definitions

  • the embodiments of the present invention relate to the field of communications technologies, and in particular, to a lip tone synchronization method and device between multiple devices.
  • audio and video playback devices send audio data to WIFI speakers or Bluetooth speakers through wireless connection, video playback by audio and video playback devices, WIFI speakers or Bluetooth speakers for audio applications. More and more users are favored. The most important issue in this application is to ensure lip sync, in order to give users a good experience.
  • the Bluetooth driver module in the audio and video playback device reads the audio PCM data from its own Pulse Code Modulation (PCM) buffer and transmits it to the Bluetooth in real time via the Bluetooth protocol.
  • PCM Pulse Code Modulation
  • the Bluetooth audio driver module of the Bluetooth speaker receives the PCM data based on the Bluetooth protocol, and transmits the PCM data to the audio PCM buffer of the Bluetooth speaker in real time.
  • the PCM playback driver module of the Bluetooth speaker directly reads the audio PCM data from the audio PCM buffer of the Bluetooth speaker. Play the output.
  • the bluetooth speaker only solves the problem of audio cross-device output, and can not realize lip sync between devices.
  • the above method is also applied to WIFI speakers, and for WIFI speakers, audio and video are caused by WIFI router forwarding delay. The output synchronization is further aggravated. How to achieve lip-tone synchronization between multiple devices is a problem that needs to be solved.
  • the embodiment of the invention provides a lip sound synchronization method and device between multiple devices, which solves the problem of how to realize lip synchronization when playing video and audio synchronously between multiple devices.
  • an embodiment of the present invention provides a multi-device lip-synchronization method for synchronously outputting audio and video to at least one slave device by a master device, where the method includes: receiving, by the device, a real-time control protocol RTCP report sent by the master device.
  • the real-time stream packet RTP timestamp header field in the RTCP packet carries the program clock reference PCR periodically collected by the master device, and the network time protocol NTP domain in the RTCP packet carries the RTCP packet transmission time point; the slave device according to the PCR , the main device program clock frequency, the program clock frequency of the slave device, and the RTCP delay to correct its own system clock STC; the slave device splices the RTP into a complete RTP according to the multicast service address sent by the master device and the port receiving the RTP issued by the master device.
  • Audio data frame and obtain a display time stamp PTS corresponding to the audio data frame from the time stamp header field of the RTP packet, and put the audio data frame into the audio pulse code modulation PCM buffer of the slave device; the slave device according to its own STC and audio data frame
  • the display timestamp outputs the audio data frame in the audio PCM buffer. Therefore, the master device is driven to synchronously output audio and video data from the device, thereby realizing lip-synchronization between the master device and the slave device, thereby providing a good user experience.
  • the RTCP message carries the RTCP session identifier.
  • the slave device Before the slave device corrects its own system clock STC according to the PCR, the master device clock frequency, the slave program clock frequency, and the RTCP delay, the slave device further includes: The device sends a join RTCP session request to the master device, so that the master device sends the RTCP session flag to the slave device. knowledge.
  • the slave device corrects its own system clock STC according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay, including: the slave device calculates the slave device's STC correction according to the following formula: Value scr_correct:
  • scf_clt is the slave program clock frequency
  • ntp_rcv is the time when the RTCP message is received
  • scr_srv and ntp_snd are the transmission time points of the PCR and RTCP messages, respectively;
  • the slave device corrects its own STC based on the calculated STC correction value.
  • the slave device before receiving the real-time control protocol RTCP packet sent by the master device, the slave device further includes: receiving, by the device, media description information sent by the master device, where the media description information includes a simple network time protocol SNTP service address and port. , RTCP service address and port, multicast service address and port, main device program clock frequency, media format and time stamp frequency; the slave device performs system time synchronization according to the SNTP service address and port;
  • the slave device receives the real-time control protocol RTCP packet sent by the master device, including:
  • the slave device receives the RTCP packet sent by the master device according to the RTCP service address and port.
  • an embodiment of the present invention provides a multi-device lip-synchronization method for synchronously outputting audio and video to at least one slave device by a master device, where the method includes: the master device collects a program clock reference PCR according to a preset acquisition period. The master device sends a real-time control protocol RTCP packet to the slave device when it determines that the preset condition is met.
  • the real-time stream packet RTP timestamp header field in the RTCP packet carries the PCR, and the network time protocol NTP domain in the RTCP packet carries the RTCP packet.
  • the transmission time point of the text so that the slave device corrects the system clock STC of the slave device according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay; the master device decodes the PCM buffer from the audio pulse code of the master device.
  • the audio data frame is copied and packaged into a real-time stream packet RTP, and the RTP is advertised to the multicast service address and port.
  • the timestamp header field of the RTP carries the display timestamp corresponding to the audio data frame, and is used by the slave device according to the multicast service address. And the port receives RTP. Therefore, the master device is driven to synchronously output audio and video data from the device, thereby realizing lip-synchronization between the master device and the slave device, thereby providing a good user experience.
  • the RTCP message carries the RTCP session identifier
  • the master device sends the RTCP message sent by the slave device to the slave device before sending the real-time control protocol RTCP packet to the slave device when determining that the preset condition is met.
  • a session request and an RTCP session ID is sent to the slave.
  • the method further includes: the master device sends the media description information to the slave device, where the media description information includes a simple network time protocol SNTP service address and port, and RTCP.
  • the media description information includes a simple network time protocol SNTP service address and port, and RTCP.
  • Service address and port, multicast service address and port, master device program clock frequency, media format and timestamp frequency, SNTP service address and port are used for system time synchronization from the device, RTCP service address and port are used for slave devices according to RTCP
  • the service address and port receive RTCP packets sent by the master device.
  • the preset condition is that the deviation between the time interval at which the primary device actually collects the PCR and the preset collection period is greater than a preset threshold.
  • the preset threshold is 20ms
  • the preset conditions are:
  • scr_curr is the current acquisition clock value
  • scr_last is the last acquisition clock value
  • scf_srv is the main device program clock frequency
  • cycle_read_x is the preset acquisition period.
  • an embodiment of the present invention provides a slave device, including:
  • the first receiving module is configured to receive the real-time control protocol RTCP packet sent by the master device, and the real-time stream packet RTP timestamp header field in the RTCP packet carries the program clock reference PCR periodically collected by the master device, and the network in the RTCP packet
  • the time protocol NTP domain carries the transmission time point of the RTCP packet
  • the correction module is configured to correct its own system clock STC according to the PCR, the main device program clock frequency, the slave program clock frequency, and the RTCP delay
  • the second receiving module uses Receiving, by the multicast service address and port sent by the master device, the RTP issued by the master device;
  • the processing module configured to splicing the RTP into a complete audio data frame, and acquiring the display corresponding to the audio data frame from the timestamp header field of the RTP packet
  • the timestamp PTS, the audio data frame is put into the audio pulse code of the slave device to modulate the PCM buffer
  • the output module is configured to output the audio data frame in the audio
  • the RTCP message carries the RTCP session identifier
  • the slave device further includes: a sending module, configured, in the correction module, according to the PCR, the main device program clock frequency, the slave program clock frequency, and the RTCP delay correction
  • the RTCP session request is sent to the master device, so that the master device sends the RTCP session identifier to the slave device.
  • the correction module is specifically configured to: calculate the STC correction value scr_correct of the slave device according to the following formula:
  • scf_clt is the slave program clock frequency
  • ntp_rcv is the time when the RTCP message is received
  • scr_srv and ntp_snd are the transmission time points of the PCR and RTCP messages, respectively;
  • the STC of its own is corrected based on the calculated STC correction value.
  • the first receiving module is further configured to: before receiving the real-time control protocol RTCP packet sent by the master device, receive media description information sent by the master device, where the media description information includes a simple network time protocol SNTP service address. And port, RTCP service address and port, multicast service address and port, main device program clock frequency, media format and timestamp frequency; the processing module is further configured to perform system time synchronization according to the SNTP service address and port;
  • the first receiving module is specifically configured to: receive the RTCP packet sent by the master device according to the RTCP service address and the port.
  • the embodiment of the present invention provides a master device, including: an acquiring module, configured to collect a program clock reference PCR according to a preset collection period; and a sending module, configured to send a real-time control to the slave device when determining that the preset condition is met
  • the protocol RTCP packet, the RTP packet in the RTCP packet, the RTP timestamp header field carries the PCR, and the network time protocol NTP domain in the RTCP packet carries the transmission time point of the RTCP packet, so that the slave device according to the PCR, the main device program Clock frequency, program clock frequency of the slave device, and RTCP delay correction system clock STC of the slave device; processing module for copying the audio data frame from the audio pulse code modulation PCM buffer of the master device and packaging into a real-time stream packet RTP,
  • the RTP is advertised to the multicast service address and port.
  • the timestamp header field of the RTP carries the display timestamp corresponding to the audio data frame, and is used by the slave device to receive the RTP according to the multicast service address and the port. Therefore, the master device is driven to synchronously output audio and video data from the device, thereby realizing lip-synchronization between the master device and the slave device, thereby providing a good user experience.
  • the RTCP message carries the RTCP session identifier
  • the master device further includes: a receiving module, configured to receive, before the sending module sends the real-time control protocol RTCP packet to the slave device when determining that the preset condition is met
  • the RTCP session request sent from the device is added, and the RTCP session identifier is sent to the slave device.
  • the sending module is further configured to: before the collecting module collects the program clock reference PCR according to the preset collection period, send the media description information to the slave device, where the media description information includes a simple network time protocol SNTP service address and port. , RTCP service address and port, multicast service address and port, master device program clock frequency, media format and time stamp frequency, SNTP service address and port for system time synchronization from the device, RTCP service address and port for slave device Receive RTCP packets sent by the master device according to the RTCP service address and port.
  • the media description information includes a simple network time protocol SNTP service address and port. , RTCP service address and port, multicast service address and port, master device program clock frequency, media format and time stamp frequency, SNTP service address and port for system time synchronization from the device, RTCP service address and port for slave device Receive RTCP packets sent by the master device according to the RTCP service address and port.
  • the preset condition is that the deviation between the time interval at which the primary device actually collects the PCR and the preset collection period is greater than a preset threshold.
  • the preset threshold is 20ms
  • the preset conditions are:
  • scr_curr is the current acquisition clock value
  • scr_last is the last acquisition clock value
  • scf_srv is the main device program clock frequency
  • cycle_read_x is the preset acquisition period.
  • an embodiment of the present invention provides a terminal, including: a receiver, a processor, and a transmitter; a receiver for receiving data from outside the terminal, a transmitter for transmitting data to an external device; and a processor configured to execute the foregoing The method of any of the aspects or the second aspect.
  • the embodiment of the present invention further provides a computer storage medium for storing computer software instructions for a slave device or a master device in any of the above aspects, comprising a method or a program for performing the above aspects.
  • the embodiment of the invention further provides a data processing system, comprising a module for performing the methods provided in the first aspect or the second aspect.
  • the embodiment of the invention further provides a computer program for performing the methods provided in the first aspect or the second aspect.
  • FIG. 1 is a schematic flowchart of Embodiment 1 of a lip-to-synchronization method for multiple devices according to an embodiment of the present invention
  • FIG. 2 is a schematic flowchart of Embodiment 2 of a lip-to-synchronization method for multiple devices according to an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of a principle of a lip-to-synchronization method between multiple devices according to an embodiment of the present invention
  • FIG. 4 is a flow chart of interaction of each module when the media sharing service is started by the primary device
  • FIG. 5 is a flow chart of interaction of each module when querying a media sharing resource from a device
  • FIG. 6 is a flow chart of interaction of each module when a shared media resource is played synchronously from a device
  • FIG. 7 is a schematic structural diagram of Embodiment 1 of a slave device according to an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of Embodiment 2 of a slave device according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of Embodiment 1 of a main device according to an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of Embodiment 2 of a main device according to an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of Embodiment 3 of a slave device according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of Embodiment 3 of a main device according to an embodiment of the present invention.
  • FIG. 13 is a schematic structural diagram of Embodiment 4 of a main device according to an embodiment of the present invention.
  • the technical solution of the embodiment of the present invention can be applied to various communication systems of a wireless cellular network, for example, a Global System of Mobile communication (GSM) system, a Code Division Multiple Access (CDMA) system, Wideband Code Division Multiple Access Wireless (WCDMA) system, General Packet Radio Service (GPRS) system, LTE system, Universal Mobile Telecommunications System (UMTS), etc., the present invention
  • GSM Global System of Mobile communication
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access Wireless
  • GPRS General Packet Radio Service
  • LTE Long Term Evolution
  • UMTS Universal Mobile Telecommunications System
  • the technical solution of the embodiment of the present invention is mainly applied to a scenario in which a master device and one or more slave devices synchronously play video data and audio data to ensure lip synchronization.
  • Lip-synchronization means that the main device plays video data synchronously with the audio data played by the device. When the user watches the video, the playback effect of the lip shape and the sound can be synchronized.
  • the master device and the slave device are connected by WIFI, etc., and the master device plays video data, and the slave device plays audio data, and the master device supports audio and video synchronous output, such as a mobile phone, a set-top box (STB), a television box ( Over The Top, OTT) boxes, etc.
  • the slave device can be a device such as a WIFI speaker, and the slave device can play audio data.
  • the multi-device lip-synchronization method and device can realize that one master device drives multiple slave devices to synchronously output audio and video data, thereby realizing lip-synchronization between the master device and the slave device, and providing a good user experience. .
  • the technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
  • FIG. 1 is a schematic flowchart of Embodiment 1 of a method for synthesizing lip-synchronization between multiple devices according to an embodiment of the present invention, for a master device to synchronously output audio and video to at least one slave device, as shown in FIG. 1 , the method includes:
  • the master device collects a program clock reference (PCR) according to a preset collection period.
  • PCR program clock reference
  • the master device determines, according to the collected PCR, whether the preset condition is met, and sends a Real Time Control Protocol (RTCP) packet to the slave device when the preset condition is met, and the real-time stream packet RTP timestamp in the RTCP packet.
  • RTCP Real Time Control Protocol
  • the header field carries the PCR
  • NTP Network Time Protocol
  • the preset condition is that the deviation between the time interval for actually collecting the PCR of the primary device and the preset collection period is greater than a preset threshold.
  • the preset threshold is, for example, a value between -60 ms and 20 ms. Taking 20ms as an example, the preset conditions are:
  • scr_curr is the current acquisition clock value
  • scr_last is the last acquisition clock value
  • scf_srv is the main device program clock frequency
  • cycle_read_x is the preset acquisition period.
  • the slave device receives the RTCP packet sent by the master device, and the slave device corrects its own system clock (STC) according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay.
  • STC system clock
  • the RTP packet header field in the RTCP packet carries the PCR
  • the NTP domain in the RTCP packet carries the transmission time point of the RTCP packet, so the slave device can obtain the packet.
  • the RTCP delay is the RTCP transmission delay, which is the difference between the time when the RTCP packet is sent and the time when the RTCP packet is received.
  • the RTCP message carries the RTCP session identifier, and the RTCP session identifier is used by the master device to distinguish the identifiers assigned by the different slave devices. Specifically, the slave device sends an RTCP session request to the master device, and the master device receives the join RTCP session. After the request, the RTCP session identifier is sent to the slave device, and then the RTCP session identifier is carried in the RTCP packet sent by the master device to the slave device.
  • the slave device can calculate the STC correction value scr_correct of the slave device according to the following formula:
  • scf_clt is the slave program clock frequency
  • ntp_rcv is the time when the RTCP message is received
  • scr_srv and ntp_snd are the transmission time points of the PCR and RTCP messages, respectively.
  • the slave device then corrects the STC of the slave device based on the calculated STC correction value.
  • the master device copies the audio data frame from the audio pulse code modulation (PCM) buffer of the master device and packs it into a real time stream packet (RTP), and issues the RTP to the multicast service address and port.
  • the timestamp header field of the RTP carries the display timestamp corresponding to the audio data frame.
  • the slave device receives the RTP advertised by the master device according to the multicast service address and port sent by the master device, and splices the RTP into a complete audio data frame, and obtains a display time corresponding to the audio data frame from a timestamp header field of the RTP packet. Stamped, put the audio data frame into the PCM buffer of the slave device.
  • the audio data frame is, for example, an audio frame.
  • the slave device outputs the audio data frame in the audio PCM buffer according to the STC and the audio data frame display time stamp.
  • the slave device displays the timestamp according to the audio data frame, and plays the audio data frame in the audio PCM buffer synchronously with reference to its own STC.
  • the master device collects the PCR according to the preset period, and sends an RTCP packet to the slave device when the predetermined condition is met (that is, the time correction is required), and the RTCP packet carries the RTCP packet.
  • the slave device corrects its own STC according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay.
  • the slave device receives the RTP advertised by the master device according to the multicast service address and port sent by the master device, and splices the RTP into a complete audio data frame, and obtains a display timestamp corresponding to the audio data frame from the timestamp header field of the RTP packet.
  • the audio data frame is placed in the PCM buffer of the slave device.
  • the slave device references its own STC and audio data frame display timestamps to output audio data frames in the audio PCM buffer. Therefore, the master device drives the slave device to synchronously output audio and video data, and realizes lip synchronization between the master device and the slave device, thereby providing a good user experience.
  • FIG. 2 is a schematic flowchart of a second embodiment of a lip-to-synchronization method for multiple devices according to an embodiment of the present invention.
  • the method in this embodiment is based on the embodiment shown in FIG.
  • the master device is not the first time to synchronously output audio and video data, and the master device program clock frequency may be stored in the slave device after the first synchronous output of the audio and video data, and the corresponding master device identifier may be stored, and then It can be used directly when synchronizing output. If the slave device and the master device output audio and video data synchronously for the first time, the master device transmits the master device program clock frequency to the slave device before S103.
  • a simple Network Time Protocol (SNTP) service address and port for system time synchronization of the slave device is used for the slave device.
  • the RTCP service address and port, the multicast service address and port, the main device program clock frequency, the media format, and the timestamp frequency of the RTCP packet may be stored in the slave device after the first synchronization of the output audio and video data, and may be The identifier of the corresponding master device is stored and can be used directly in the secondary synchronization output. If the slave device and the master device output audio and video data in synchronization for the first time, before S101, the method may further include:
  • the master device sends media description information to the slave device, where the media description information includes an SNTP service address and port, an RTCP service address and port, a multicast service address and port, a master device program clock frequency, a media format, and a timestamp frequency.
  • the master device may publish media description information through a hypertext transfer protocol.
  • the slave device receives media description information sent by the master device. Obtain the SNTP service address and port, RTCP service address and port, multicast service address and port, main device program clock frequency, and media by parsing the media description information. The body format and the timestamp frequency are then systematically synchronized according to the SNTP service address and port, and then the RTCP message sent by the master device is received according to the RTCP service address and port. The multicast service address and port are used to receive the RTP issued by the master device from the device.
  • the media format is a format used by the main device to compress and encode the audio data frame, and is used to determine a corresponding decoding format from the device.
  • the timestamp frequency is used to adjust the timestamp of the interval between two frames of the slave device from the timestamp of the interval between two frames of the master device.
  • the master device sends the media description information to the slave device, and the slave device performs system time synchronization according to the SNTP service address and port in the media description information, and then collects the PCR according to the preset period by the master device.
  • Sending an RTCP packet to the slave device when it is determined that the preset condition is met (that is, the time correction is required) the RTCP packet carries the time point of sending the collected PCR and RTCP packet, and the slave device according to the PCR, the master device program
  • the clock frequency, the program clock frequency of the slave device, and the RTCP delay correct its own STC.
  • the slave device receives the RTP advertised by the master device according to the multicast service address and port sent by the master device, and splices the RTP into a complete audio data frame, and obtains a display timestamp corresponding to the audio data frame from the timestamp header field of the RTP packet.
  • the audio data frame is placed in the PCM buffer of the slave device.
  • the slave device references its own STC and audio data frame display timestamps to output audio data frames in the audio PCM buffer. Therefore, the master device drives the slave device to synchronously output audio and video data, and realizes lip synchronization between the master device and the slave device, thereby providing a good user experience.
  • FIG. 3 is a schematic diagram of a principle of a lip-to-synchronization method for multiple devices according to an embodiment of the present invention.
  • an embodiment of the present invention adds a new media based on an existing master device (media player).
  • the information distribution module, the system time server, the program clock server, and the RTP packet server add a media information download module, a system time client, a program clock client, an RTP packet client, and the existing slave device.
  • Program clock driver module The media information publishing module, the system time server, the program clock server, and the RTP packet server, the media information download module, the system time client, the program clock client RTP packet client, and the program clock driver module may all be software modules. .
  • the media information publishing module is configured to issue media description information, where the media description information includes an SNTP service address and port, an RTCP service address and port, a multicast service address and port, a main device program clock frequency, a media format, and a timestamp frequency.
  • the media information downloading module is configured to receive media description information.
  • the system time server and the system time client implement time synchronization between the slave device and the master device.
  • the program clock server and the program clock client enable high-precision synchronization of the program clock from the device to the master device, for example, ensuring that the program clock skew is less than 20 ms (approximately 1 frame of PCM output delay).
  • the RTP packet server establishes an RTP channel for the audio data frame to be transmitted from the master device to the slave device with the RTP packet client.
  • the slave program clock drive module is used to output the program clock client clock output when the PCM play driver module plays audio data.
  • the original PCM playback driver module of the master device can be disconnected and not used after the slave device is connected, and the synchronous output audio is played by the slave device, and the slave device is connected to the master device.
  • the original audio target decoder of the device is also disconnected from the audio PCM buffer (FIFO) to play the audio output by the master device.
  • the audio and video synchronization process of the master device is: the audio and video synchronization module of the master device is based on the synchronization strategy of the target program (PCR benchmark, video stream reference, audio stream reference or audio and video reference reference) and pure tone synchronization algorithm, according to the input. PCR, audio display timestamp (PTS) and video PTS when calculating the program Clock reference value, and then the audio and video synchronization module compares the program clock reference value with the current system clock sample (SCR). When the offset exceeds the threshold (usually 100ms to 200ms), the system clock reference value is used to correct the system clock. Corrected PCR. When audio, video, and other media streams are output, the PTS and SCR are arranged to display the output timing.
  • PCR synchronization strategy of the target program
  • PTS audio display timestamp
  • video PTS when calculating the program Clock reference value
  • SCR system clock sample
  • the system clock reference value is used to correct the system clock. Corrected PCR.
  • the current audio frame is output, otherwise it is not output.
  • the video is output normally, and the audio is not played by the PCM playback driver module, but is output to the slave device through the RTP packet server, and the audio is output synchronously by the slave device.
  • FIG. 4 is a flow chart of interaction of each module when the media sharing service is started by the primary device, as shown in FIG. 4, including:
  • the media sharing service process is started, including starting system time synchronization (system time server execution), starting program clock synchronization (program clock server execution), and starting group. Broadcast (RTP packet server execution), start media information release (media information release module execution).
  • the media information publishing module issues media description information, where the media description information includes an SNTP service address and port, an RTCP service address and port, a multicast service address and port, a main device program clock frequency, a media format, and a timestamp frequency.
  • the system time server performs system time synchronization according to the SNTP service address and the port
  • the program clock server receives the RTCP message sent by the master device according to the RTCP service address and the port, and performs program clock synchronization according to the RTCP message, and the RTP packet server end is based on The multicast service address and port issue RTP.
  • FIG. 5 is a flow chart of interaction of each module when the device shares the media sharing resource, as shown in FIG. 5, including:
  • the monitoring module of the slave device sends a request for querying media information of the server to the media information downloading module.
  • the media information downloading module sends a media description information request to the media information publishing module, and the media information publishing module returns a media description information request response, and carries the media description information, where the media description information includes an SNTP service address and port, an RTCP service address, a port, and a group. Broadcast service address and port, master device program clock frequency, media format, and timestamp frequency.
  • the media information downloading module parses the media description information to obtain all service addresses and ports, a main device program clock frequency, a media format, and a timestamp frequency.
  • S304 The SNTP service address and port are configured in the monitoring module of the slave device, and the startup time client performs SNTP time synchronization.
  • system time synchronization is performed between the time client and the system time server.
  • FIG. 6 is a flow chart of interaction of each module when the shared media resource is synchronously played by the device, as shown in FIG. 6, including:
  • the monitoring module of the slave device sends a synchronous program clock request to the program clock client.
  • the program clock client sends an RTCP session request to the program clock server, and the program clock server returns the RTCP session identifier.
  • the program clock server collects the PCR according to the preset collection period, and sends the PCR sample value to the program clock client when the preset condition is met.
  • the program clock client corrects its own STC according to the PCR, the main device program clock frequency, the slave program clock frequency, and the RTCP delay.
  • the monitoring module of the slave device sends a start audio data frame receiving request to the RTP packet client.
  • the RTP packet client sends a join multicast group request to the RTP packet server.
  • the RTP packet server sends an RTP to the RTP packet client.
  • the RTP packet client receives the RTP, and splices the RTP into a complete audio data frame, and groups the packets from the RTP.
  • the timestamp header field obtains the display timestamp corresponding to the audio data frame, and the audio data frame is placed in the PCM buffer of the slave device.
  • the PCM playback driver module outputs the audio data frame in the audio PCM buffer according to the STC and the audio data frame display time stamp of the slave device.
  • the embodiments of the present invention may divide the function modules of the master device and the slave device according to the foregoing method.
  • each function module may be divided according to each function, or two or more functions may be integrated into one processing module.
  • the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the modules in the embodiments of the embodiments of the present invention is schematic, and is only a logical function division. In actual implementation, there may be another division manner.
  • FIG. 7 is a schematic structural diagram of Embodiment 1 of a slave device according to an embodiment of the present invention.
  • the slave device in this embodiment may include: a first receiving module 11, a correcting module 12, a second receiving module 13, and a processing module 14. And the output module 15, wherein the first receiving module 11 is configured to receive the RTCP packet sent by the master device, and the RTP timestamp header field in the RTCP packet carries the program clock reference PCR periodically collected by the master device, and the RTCP packet
  • the NTP domain of the network time protocol carries the time of sending RTCP packets.
  • the correction module 12 is configured to correct its own system clock STC according to the PCR, the main device program clock frequency, the program clock frequency of the slave device, and the RTCP delay.
  • the second receiving module 13 is configured to receive the RTP issued by the master device according to the multicast service address and port sent by the master device.
  • the processing module 14 is configured to splicing the RTP into a complete audio data frame, and acquiring a display timestamp PTS corresponding to the audio data frame from the timestamp header field of the RTP packet, and placing the audio data frame into the audio pulse code modulation PCM buffer of the slave device.
  • the output module 15 is configured to output an audio data frame in the audio PCM buffer according to its own STC and audio data frame display time stamp.
  • FIG. 8 is a schematic structural diagram of a second embodiment of a slave device according to an embodiment of the present invention.
  • the slave device in this embodiment may further include: a sending module 16 .
  • the sending module 16 is configured to send a join RTCP session request to the master device to enable the master device before the correction module 12 corrects its own system clock STC according to the PCR, the master device clock frequency, the slave program clock frequency, and the RTCP delay. Send an RTCP session ID to the slave.
  • correction module 12 is specifically configured to: calculate the STC correction value scr_correct of the slave device according to the following formula:
  • scf_clt is the slave program clock frequency
  • ntp_rcv is the time when the RTCP message is received
  • scr_srv and ntp_snd are the transmission time points of the PCR and RTCP messages, respectively.
  • the STC of its own is corrected based on the calculated STC correction value.
  • the first receiving module 11 is further configured to: before receiving the real-time control protocol RTCP packet sent by the master device, receive media description information sent by the master device, where the media description information includes an SNTP service address and port, an RTCP service address, and a port.
  • the processing module 14 is further configured to perform system time synchronization according to the SNTP service address and the port; the first receiving module 11 is specifically configured to: according to the RTCP service, the multicast service address and port, the main device program clock frequency, the media format, and the time stamp frequency.
  • the address and port receive RTCP packets sent by the master device.
  • the slave device of the embodiment shown in FIG. 7 or FIG. 8 can be used to implement the technical solution of the method embodiment shown in FIG. 1 or FIG. 2, and the implementation principle and the technical effect are similar, and details are not described herein again.
  • FIG. 9 is a schematic structural diagram of Embodiment 1 of a main device according to an embodiment of the present invention.
  • the main device in this embodiment may include: an acquisition module 21, a sending module 22, and a processing module 23, where the acquisition module 21 is used.
  • the acquisition cycle collects the program clock reference PCR.
  • the sending module 22 is configured to send a real-time control protocol (RTCP) packet to the slave device when the predetermined condition is met.
  • RTCP real-time control protocol
  • the real-time stream packet RTP timestamp header field in the RTCP packet carries the PCR
  • the network time protocol NTP domain in the RTCP packet carries
  • the transmission time point of the RTCP message is such that the slave device corrects the system clock STC of the slave device according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay.
  • the processing module 23 is configured to copy the audio data frame from the audio pulse code modulation PCM buffer of the master device and package the data into a real-time stream packet RTP, and advertise the RTP to the multicast service address and port, and the time stamp header field of the RTP carries the audio data.
  • FIG. 10 is a schematic structural diagram of Embodiment 2 of a main device according to an embodiment of the present invention.
  • the main device in this embodiment may further include: a receiving module 24
  • the receiving module 24 is configured to receive the RTCP session request sent by the slave device and send the RTCP session identifier to the slave device before the sending module 22 sends the RTCP packet to the slave device when determining that the preset condition is met.
  • the sending module 22 is further configured to: before the collecting module 21 collects the program clock reference PCR according to the preset collection period, send the media description information to the slave device, where the media description information includes an SNTP service address and port, an RTCP service address and a port, Multicast service address and port, main device program clock frequency, media format and timestamp frequency, SNTP service address and port are used for system time synchronization from the device, RTCP service address and port are used for slave device to receive according to RTCP service address and port RTCP packet sent by the master device.
  • the media description information includes an SNTP service address and port, an RTCP service address and a port, Multicast service address and port, main device program clock frequency, media format and timestamp frequency, SNTP service address and port are used for system time synchronization from the device, RTCP service address and port are used for slave device to receive according to RTCP service address and port RTCP packet sent by the master device.
  • the preset condition is that the deviation between the time interval for actually collecting the PCR of the primary device and the preset collection period is greater than a preset threshold.
  • the preset threshold is 20ms
  • the preset condition is:
  • scr_curr is the current acquisition clock value
  • scr_last is the last acquisition clock value
  • scf_srv is the main device program clock frequency
  • cycle_read_x is the preset acquisition period.
  • the main device of the embodiment shown in FIG. 9 and FIG. 10 can be used to implement the technical solution of the method embodiment shown in FIG. 1 or FIG. 2, and the implementation principle and technical effects are similar, and details are not described herein again.
  • FIG. 11 is a schematic structural diagram of Embodiment 3 of a slave device according to an embodiment of the present invention.
  • the slave device in this embodiment may include: a receiver 31, a processor 32, and a transmitter 33, where the receiver 31 is configured to Receiving the RTCP packet sent by the master device, the RTP timestamp header field in the RTCP packet carries the program clock reference PCR periodically collected by the master device, and the network time protocol NTP domain in the RTCP packet carries the transmission time point of the RTCP packet.
  • the processor 32 is operative to correct its own system clock STC based on the PCR, the master device clock frequency, the slave program clock frequency, and the RTCP delay.
  • the receiver 31 is further configured to receive the RTP issued by the master device according to the multicast service address and port sent by the master device.
  • the processor 32 is further configured to splicing the RTP into a complete audio data frame, and acquiring a display time stamp PTS corresponding to the audio data frame from the time stamp header field of the RTP packet, and placing the audio data frame into the audio pulse code modulation PCM of the slave device. Cache.
  • the transmitter 33 is configured to output an audio data frame in the audio PCM buffer according to its own STC and audio data frame display time stamp.
  • the transmitter 33 is further configured to send a join RTCP session request to the master device before the correction module corrects its own system clock STC according to the PCR, the master device clock frequency, the slave program clock frequency, and the RTCP delay.
  • the master device sends an RTCP session identifier to the slave device.
  • processor 32 is specifically configured to: calculate the STC correction value scr_correct of the slave device according to the following formula:
  • scf_clt is the slave program clock frequency
  • ntp_rcv is the time when the RTCP message is received
  • scr_srv and ntp_snd are the transmission time points of the PCR and RTCP messages, respectively.
  • the STC of its own is corrected based on the calculated STC correction value.
  • the receiver 31 is further configured to: before receiving the real-time control protocol RTCP packet sent by the master device, receive media description information sent by the master device, where the media description information includes an SNTP service address and port, an RTCP service address, a port, and a group.
  • the processor 32 is further configured to perform system time synchronization according to the SNTP service address and the port;
  • the receiver 31 is specifically configured to: receive according to the RTCP service address and port RTCP packet sent by the master device.
  • the slave device of this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 1 or FIG. 2, and the implementation principle and the technical effect are similar, and details are not described herein again.
  • FIG. 12 is a schematic structural diagram of Embodiment 3 of a main device according to an embodiment of the present invention.
  • the main device in this embodiment may include: a processor 41 and a transmitter 42, where the processor 41 is configured to collect according to a preset. Periodically collect program clock reference PCR.
  • the transmitter 42 is configured to send a real-time control protocol (RTCP) packet to the slave device when the preset condition is met.
  • RTCP real-time control protocol
  • the real-time stream packet in the RTCP packet carries the PCR in the RTP timestamp header field, and the network time protocol NTP domain in the RTCP packet carries The transmission time point of the RTCP message is such that the slave device corrects the system clock STC of the slave device according to the PCR, the master device program clock frequency, the slave program clock frequency, and the RTCP delay.
  • the processor 41 is further configured to copy the audio data frame from the audio pulse code modulation PCM buffer of the master device and package the data into a real-time stream packet RTP, and publish the RTP to the multicast service address and port, and the time stamp header field of the RTP carries the audio.
  • the display timestamp corresponding to the data frame is used by the slave device to receive the RTP according to the multicast service address and port.
  • FIG. 13 is a schematic structural diagram of Embodiment 4 of a main device according to an embodiment of the present invention.
  • the main device in this embodiment may further include: a receiver 43.
  • the receiver 43 is configured to receive the join RTCP session request sent by the slave device and send the RTCP session identifier to the slave device before the transmitter 42 sends the RTCP message to the slave device when determining that the preset condition is met.
  • the transmitter 42 is further configured to: before the processor 41 collects the program clock reference PCR according to the preset collection period, send the media description information to the slave device, where the media description information includes an SNTP service address and port, an RTCP service address and a port, Multicast service address and port, main device program clock frequency, media format and timestamp frequency, SNTP service address and port are used for system time synchronization from the device, RTCP service address and port are used for slave device to receive according to RTCP service address and port RTCP packet sent by the master device.
  • the media description information includes an SNTP service address and port, an RTCP service address and a port, Multicast service address and port, main device program clock frequency, media format and timestamp frequency, SNTP service address and port are used for system time synchronization from the device, RTCP service address and port are used for slave device to receive according to RTCP service address and port RTCP packet sent by the master device.
  • the preset condition is that the deviation between the time interval for actually collecting the PCR of the primary device and the preset collection period is greater than a preset threshold.
  • the preset threshold is 20ms
  • the preset condition is:
  • scr_curr is the current acquisition clock value
  • scr_last is the last acquisition clock value
  • scf_srv is the main device program clock frequency
  • cycle_read_x is the preset acquisition period.
  • the main device of the embodiment shown in FIG. 12 and FIG. 13 can be used to implement the technical solution of the method embodiment shown in FIG. 1 or FIG. 2, and the implementation principle and technical effects are similar, and details are not described herein again.
  • aspects of the embodiments of the invention, or possible implementations of various aspects may be embodied as a system, method, or computer program product.
  • aspects of the embodiments of the invention, or possible implementations of various aspects may be in the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, etc.), or a combination of software and hardware aspects. They are collectively referred to herein as "circuits," “modules,” or “systems.”
  • aspects of the embodiments of the invention, or possible implementations of various aspects may take the form of a computer program product, which is a computer readable program code stored in a computer readable medium.
  • the computer readable medium can be a computer readable signal medium or a computer readable storage medium.
  • the computer readable storage medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing, such as random access memory (RAM), read only memory (ROM), Erase programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).
  • the processor in the computer reads the computer readable program code stored in the computer readable medium such that the processor is capable of performing the various functional steps specified in each step of the flowchart, or a combination of steps; A device that functions as specified in each block, or combination of blocks.
  • the computer readable program code can execute entirely on the user's local computer, partly on the user's local computer, as a separate software package, partly on the user's local computer and partly on the remote computer, or entirely on the remote computer or Executed on the server. It should also be noted that in some alternative implementations, the functions noted in the various steps in the flowcharts or in the blocks in the block diagrams may not occur in the order noted. For example, two steps, or two blocks, shown in succession may be executed substantially concurrently or the blocks may be executed in the reverse order.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Security & Cryptography (AREA)
  • Synchronisation In Digital Transmission Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本发明实施例提供一种多设备间唇音同步方法及设备，该方法包括：从设备接收主设备发送的RTCP报文，根据RTCP报文中的PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的STC，接着接收主设备发布的RTP，将RTP拼接成完整的音频数据帧放入从设备的PCM缓存，输出PCM缓存中的音频数据帧。

Description

多设备间唇音同步方法及设备 技术领域
本发明实施例涉及通信技术领域，尤其涉及一种多设备间唇音同步方法及设备。
背景技术
随着WIFI音箱和蓝牙音箱的广泛应用，音视频播放设备通过无线连接的方式将音频数据发送到WIFI音箱或蓝牙音箱，由音视频播放设备播放视频，WIFI音箱或蓝牙音箱播放音频的这一应用得到越来越多的用户青睐。该应用中，最重要的问题是要保证唇音同步，才能给用户带来良好的使用体验。
在现有的蓝牙音箱的使用过程中，音视频播放设备中的蓝牙驱动模块从自身的音频脉冲编码调制(Pulse Code Modulation，PCM)缓存中读取音频PCM数据，并通过蓝牙协议实时发送到蓝牙音箱，蓝牙音箱的蓝牙音频驱动模块基于蓝牙协议接收PCM数据，并将PCM数据实时发送到蓝牙音箱的音频PCM缓存，蓝牙音箱的PCM播放驱动模块直接从蓝牙音箱的音频PCM缓存读取音频PCM数据播放输出。
可以看出，蓝牙音箱只解决了音频跨设备输出的问题，不能实现设备间的唇音同步，上述方法应用到WIFI音箱亦是如此，且对于WIFI音箱而言，由于WIFI路由器转发时延导致音视频输出不同步进一步加剧，如何实现多设备间唇音同步，是一个需要解决的问题。
发明内容
本发明实施例提供一种多设备间唇音同步方法及设备，解决多设备间同步播放视频和音频时如何实现唇音同步的问题。
第一方面，本发明实施例提供一种多设备间唇音同步方法，用于一个主设备同步输出音频和视频到至少一个从设备中，方法包括：从设备接收主设备发送的实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带主设备周期性采集的节目时钟参考PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点；从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC；从设备根据主设备发送的组播服务地址和端口接收主设备发布的RTP，将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取音频数据帧对应的显示时间戳PTS，将音频数据帧放入从设备的音频脉冲编码调制PCM缓存；从设备根据自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。从而，实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
在一种可能的设计中，RTCP报文中携带RTCP会话标识，从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，还包括：从设备向主设备发送加入RTCP会话请求，以使主设备向从设备发送RTCP会话标 识。
在一种可能的设计中，从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC，包括：从设备根据如下公式计算出从设备的STC校正值scr_correct：
scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
其中，scf_clt为从设备节目时钟频率，scf_srv主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为PCR和RTCP报文的发送时间点；
从设备根据计算出的STC校正值校正自身的STC。
在一种可能的设计中，从设备接收主设备发送的实时控制协议RTCP报文之前，还包括：从设备接收主设备发送的媒体描述信息，媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率；从设备根据SNTP服务地址和端口进行系统时间同步；
从设备接收主设备发送的实时控制协议RTCP报文，包括：
从设备根据RTCP服务地址和端口接收主设备发送的RTCP报文。
第二方面，本发明实施例提供一种多设备间唇音同步方法，用于一个主设备同步输出音频和视频到至少一个从设备中，方法包括：主设备根据预设采集周期采集节目时钟参考PCR；主设备在确定满足预设条件时向从设备发送实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点，以使从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正从设备的系统时钟STC；主设备从主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将RTP发布到组播服务地址和端口，RTP的时间戳头域携带了音频数据帧对应的显示时间戳，用于从设备根据组播服务地址和端口接收RTP。从而，实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
在一种可能的设计中，RTCP报文中携带RTCP会话标识，主设备在确定满足预设条件时向从设备发送实时控制协议RTCP报文之前，还包括：主设备接收从设备发送的加入RTCP会话请求，并向从设备发送RTCP会话标识。
在一种可能的设计中，主设备根据预设采集周期采集节目时钟参考PCR之前，还包括：主设备向从设备发送媒体描述信息，媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率，SNTP服务地址和端口用于从设备进行系统时间同步，RTCP服务地址和端口用于从设备根据RTCP服务地址和端口接收主设备发送的RTCP报文。
在一种可能的设计中，预设条件为：主设备实际采集PCR的时间间隔与预设采集周期之间的偏差大于预设阈值。
在一种可能的设计中，预设阈值为20ms，预设条件为：
(scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
(scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为主设备节目时钟频率，cycle_read_x为预设采集周期。
第三方面，本发明实施例提供一种从设备，包括：
第一接收模块，用于接收主设备发送的实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带主设备周期性采集的节目时钟参考PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点；校正模块，用于根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC；第二接收模块，用于根据主设备发送的组播服务地址和端口接收主设备发布的RTP；处理模块，用于将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取音频数据帧对应的显示时间戳PTS，将音频数据帧放入从设备的音频脉冲编码调制PCM缓存；输出模块，用于根据自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。从而，实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
在一种可能的设计中，RTCP报文中携带RTCP会话标识，从设备还包括：发送模块，用于在校正模块根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，向主设备发送加入RTCP会话请求，以使主设备向从设备发送RTCP会话标识。
在一种可能的设计中，校正模块具体用于：根据如下公式计算出从设备的STC校正值scr_correct：
scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
其中，scf_clt为从设备节目时钟频率，scf_srv主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为PCR和RTCP报文的发送时间点；
根据计算出的STC校正值校正自身的STC。
在一种可能的设计中，第一接收模块还用于：在接收主设备发送的实时控制协议RTCP报文之前，接收主设备发送的媒体描述信息，媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率；处理模块还用于根据SNTP服务地址和端口进行系统时间同步；
第一接收模块具体用于：根据RTCP服务地址和端口接收主设备发送的RTCP报文。
第四方面，本发明实施例提供一种主设备，包括：采集模块，用于根据预设采集周期采集节目时钟参考PCR；发送模块，用于在确定满足预设条件时向从设备发送实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点，以使从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正从设备的系统时钟STC；处理模块，用于从主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将RTP发布到组播服务地址和端口，RTP的时间戳头域携带了音频数据帧对应的显示时间戳，用于从设备根据组播服务地址和端口接收RTP。从而，实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
在一种可能的设计中，RTCP报文中携带RTCP会话标识，主设备还包括：接收模块，用于在发送模块在确定满足预设条件时向从设备发送实时控制协议RTCP报文之前，接收从设备发送的加入RTCP会话请求，并向从设备发送RTCP会话标识。
在一种可能的设计中，发送模块还用于：在采集模块根据预设采集周期采集节目时钟参考PCR之前，向从设备发送媒体描述信息，媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率，SNTP服务地址和端口用于从设备进行系统时间同步，RTCP服务地址和端口用于从设备根据RTCP服务地址和端口接收主设备发送的RTCP报文。
在一种可能的设计中，预设条件为：主设备实际采集PCR的时间间隔与预设采集周期之间的偏差大于预设阈值。
在一种可能的设计中，预设阈值为20ms，预设条件为：
(scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
(scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为主设备节目时钟频率，cycle_read_x为预设采集周期。
第五方面，本发明实施例提供一种终端，包括：接收器、处理器和发送器；接收器用于从所述终端外部接收数据，发送器用于向外部设备发送数据；处理器用于执行上述第一方面或第二方面任一所述的方法。
本发明实施例还提供了一种计算机存储介质，用于储存为上述任一方面中从设备或主设备所用的计算机软件指令，其包含用于执行上述各方面所设计的方法或程序。
本发明实施例还提供了一种数据处理系统，包括用于执行上述第一方面或第二方面提供的各方法的模块。
本发明实施例还提供了一种计算机程序，用于执行上述第一方面或第二方面提供的各方法。
附图说明
图1为本发明实施例提供的多设备间唇音同步方法实施例一的流程示意图；
图2为本发明实施例提供的多设备间唇音同步方法实施例二的流程示意图；
图3为本发明实施例提供的多设备间唇音同步方法的原理示意图；
图4为主设备启动媒体共享服务时各个模块的交互流程图；
图5为从设备查询媒体共享资源时各个模块的交互流程图；
图6为从设备同步播放共享媒体资源时各个模块的交互流程图；
图7为本发明实施例从设备实施例一的结构示意图；
图8为本发明实施例从设备实施例二的结构示意图；
图9为本发明实施例主设备实施例一的结构示意图；
图10为本发明实施例主设备实施例二的结构示意图；
图11为本发明实施例从设备实施例三的结构示意图；
图12为本发明实施例主设备实施例三的结构示意图；
图13为本发明实施例主设备实施例四的结构示意图。
具体实施方式
本发明实施例的技术方案，可以应用于无线蜂窝网络的各种通信系统，例如：全球移动通信(Global System of Mobile communication，GSM)系统，码分多址(Code Division Multiple Access，CDMA)系统，宽带码分多址(Wideband Code Division Multiple Access Wireless，WCDMA)系统，通用分组无线业务(General Packet Radio Service，GPRS)系统，LTE系统，通用移动通信系统(Universal Mobile Telecommunications System，UMTS)等，本发明实施例并不限定。
本发明实施例的技术方案主要应用于一个主设备和一个或多个从设备同步播放视频数据和音频数据时如何保证唇音同步的场景。唇音同步是指主设备播放视频数据与从设备播放音频数据同步，用户观看视频时可以达到口型与声音同步的播放效果。主设备与从设备之间通过WIFI等方式连接，由主设备播放视频数据，从设备播放音频数据，主设备支持音视频同步输出，如手机、机顶盒(Set-Top Box，STB)、电视盒子(Over The Top，OTT)盒子等。从设备可以为WIFI音箱等设备，从设备可以播放音频数据。
本发明实施例提出的多设备间唇音同步方法及设备，可实现一个主设备带动多个从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。用于下面结合附图详细说明本发明实施例提供的技术方案。
图1为本发明实施例提供的多设备间唇音同步方法实施例一的流程示意图，用于一个主设备同步输出音频和视频到至少一个从设备中，如图1所示，该方法包括：
S101、主设备根据预设采集周期采集节目时钟参考(Program Clock Reference，PCR)。
S102、主设备根据采集的PCR确定是否满足预设条件，在满足预设条件时向从设备发送实时控制协议(Real Time Control Protocol，RTCP)报文，RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的网络时间协议(Network Time protocol，NTP)域携带RTCP报文的发送时间点。
其中，预设条件为：主设备实际采集PCR的时间间隔与预设采集周期之间的偏差大于预设阈值。预设阈值例如为-60ms-20ms之间的值。以20ms为例，预设条件为：
(scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
(scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为主设备节目时钟频率，cycle_read_x为预设采集周期。
S103、从设备接收主设备发送的RTCP报文，从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟(System Time Clock，STC)。
其中，从设备接收到RTCP报文后，由于RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的NTP域携带RTCP报文的发送时间点，因此从设备可获取到PCR、RTCP报文的发送时间点以及接收到RTCP报文的时刻。RTCP时延就是RTCP传输时延，为RTCP报文的发送时间点与接收到RTCP报文的时刻之差。
其中，RTCP报文中携带RTCP会话标识，RTCP会话标识是主设备用于区分不同的从设备所分配的标识，具体地，从设备向主设备发送加入RTCP会话请求，主设备接收到加入RTCP会话请求后向该从设备发送RTCP会话标识，之后在主设备向该从设备发送的RTCP报文中携带RTCP会话标识。
具体地，从设备可以根据如下公式计算出从设备的STC校正值scr_correct：
scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
其中，scf_clt为从设备节目时钟频率，scf_srv主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为PCR和RTCP报文的发送时间点。
接着从设备根据计算出的STC校正值校正从设备的STC。
S104、主设备从主设备的音频脉冲编码调制(Pulse Code Modulation，PCM)缓存中复制出音频数据帧并打包成实时流分组(Real Time Packet，RTP)，将RTP发布到组播服务地址和端口，RTP的时间戳头域携带了该音频数据帧对应的显示时间戳。
S105、从设备根据主设备发送的组播服务地址和端口接收主设备发布的RTP，将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取该音频数据帧对应的显示时间戳，将该音频数据帧放入从设备的PCM缓存。音频数据帧例如为音频帧。
S106、从设备根据自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。
具体地，从设备根据音频数据帧显示时间戳，参考自身的STC同步播放音频PCM缓存中的音频数据帧。
本实施例提供的多设备间唇音同步方法，通过主设备根据预设周期采集PCR，在确定满足预设条件(也即需要进行时间校正)时向从设备发送RTCP报文，RTCP报文携带所采集的PCR和RTCP报文的发送时间点，从设备根据该PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的STC。然后从设备根据主设备发送的组播服务地址和端口接收主设备发布的RTP，将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取该音频数据帧对应的显示时间戳，将该音频数据帧放入从设备的PCM缓存。最后从设备参考自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。从而实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
图2为本发明实施例提供的多设备间唇音同步方法实施例二的流程示意图，如图2所示，本实施例的方法在图1所示实施例的基础上，其中，若从设备与主设备是非第一次同步输出音视频数据，则主设备节目时钟频率可以是在第一次同步输出音视频数据后存储在从设备中，同时可以存储相应的主设备的标识，则在二次同步输出时可直接使用。若从设备与主设备是第一次同步输出音视频数据，则在S103之前，主设备要向从设备发送主设备节目时钟频率。同样的，若从设备与主设备是非第一次同步输出音视频数据，则用于从设备进行系统时间同步的简单网络时间协议(Simple Network Time protocol，SNTP)服务地址和端口、用于从设备接收RTCP报文的RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率可以是在第一次同步输出音视频数据后存储在从设备中，同时可以存储相应的主设备的标识，则在二次同步输出时可直接使用。若从设备与主设备是第一次同步输出音视频数据，则在S101之前，还可以包括：
S107、主设备向从设备发送媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率。具体地，主设备可以通过超文本传输协议发布媒体描述信息，
S108、从设备接收主设备发送的媒体描述信息。通过解析媒体描述信息获取SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒 体格式和时戳频率，然后根据SNTP服务地址和端口进行系统时间同步，之后根据RTCP服务地址和端口接收主设备发送的RTCP报文。组播服务地址和端口用于从设备接收主设备发布的RTP。媒体格式为主设备对音频数据帧压缩编码索使用的格式，用于从设备确定相应的解码格式。时戳频率用于从设备根据主设备的两个帧之间间隔的时戳调整从设备的两个帧之间间隔的时戳。
后面的流程与图1所示的相同，此处不再赘述。
本实施例提供的多设备间唇音同步方法，主设备向从设备发送媒体描述信息，从设备根据媒体描述信息中的SNTP服务地址和端口进行系统时间同步，接着通过主设备根据预设周期采集PCR，在确定满足预设条件(也即需要进行时间校正)时向从设备发送RTCP报文，RTCP报文携带所采集的PCR和RTCP报文的发送时间点，从设备根据该PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的STC。然后从设备根据主设备发送的组播服务地址和端口接收主设备发布的RTP，将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取该音频数据帧对应的显示时间戳，将该音频数据帧放入从设备的PCM缓存。最后从设备参考自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。从而实现了主设备带动从设备同步输出音视频数据，实现主设备和从设备间的唇音同步，给用户带来良好的使用体验。
下面结合附图，结合主设备以及从设备内部的模块构成以及模块之间的交互过程详细说明主设备音频输出至从设备的原理示意图。
图3为本发明实施例提供的多设备间唇音同步方法的原理示意图，如图3所示，对于主设备，本发明实施例在现有主设备(媒体播放器)的基础上，新增加媒体信息发布模块、系统时间服务端、节目时钟服务端和RTP分组服务端，在现有从设备的基础上，新增加媒体信息下载模块、系统时间客户端、节目时钟客户端、RTP分组客户端和节目时钟驱动模块。其中，媒体信息发布模块、系统时间服务端、节目时钟服务端和RTP分组服务端以及媒体信息下载模块、系统时间客户端、节目时钟客户端RTP分组客户端和节目时钟驱动模块都可以是软件模块。媒体信息发布模块用于发布媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率。媒体信息下载模块用于接收媒体描述信息。系统时间服务端和系统时间客户端实现从设备与主设备之间的时间同步。节目时钟服务端和节目时钟客户端实现从设备与主设备的节目时钟的高精度同步，例如确保节目时钟偏差低于20ms(约1帧PCM输出时延)。如果节目时钟客户端没法将时钟频率调制到与节目时钟服务端一致时，应对SCR和PTS进行相应的时钟频率转换。RTP分组服务端与RTP分组客户端建立了音频数据帧从主设备传输到从设备的RTP通道。从设备节目时钟驱动模块用于在PCM播放驱动模块播放音频数据时参靠节目时钟客户端时钟输出。
需要说明的是，如图3所示，主设备原有的PCM播放驱动模块在连接了从设备后可以断开不使用，由从设备播放同步输出的音频，从设备连接了主设备后，从设备原有的音频目标解码器与音频PCM缓存(FIFO)也断开，播放主设备输出的音频。
具体来说，主设备的音视频同步过程为：主设备的音视频同步模块基于目标节目的同步策略(PCR基准、视频流基准、音频流基准或音视频参考基准)和纯音同步算法，根据输入的PCR、音频显示时间戳(Presentation Timestamp，PTS)和视频PTS计算出节目时 钟参考值，接着音视频同步模块将节目时钟参考值和当前系统时钟采样(SCR)比较，当偏移超过门限值(通常100ms～200ms)时，则使用节目时钟参考值校正系统时钟，得到经过校正的PCR。音频、视频等媒体流输出时，对比PTS和SCR安排显示输出时序。如，当PTS小于或等于当前采样SCR时，则输出当前音频帧，否则不输出。主设备与从设备连接后，视频正常输出，音频不通过PCM播放驱动模块播放，而是通过RTP分组服务端输出至从设备，由从设备同步输出音频。
图4为主设备启动媒体共享服务时各个模块的交互流程图，如图4所示，包括：
S201、主设备的监控模块连接上从设备或者发现从设备后，启动媒体共享服务流程，包括启动系统时间同步(系统时间服务端执行)、启动节目时钟同步(节目时钟服务端执行)、启动组播(RTP分组服务端执行)、启动媒体信息发布(媒体信息发布模块执行)。
S202、媒体信息发布模块发布媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率。
接着，系统时间服务端根据SNTP服务地址和端口进行系统时间同步，节目时钟服务端根据RTCP服务地址和端口接收主设备发送的RTCP报文，根据RTCP报文进行节目时钟同步，RTP分组服务端根据组播服务地址和端口发布RTP。
图5为从设备查询媒体共享资源时各个模块的交互流程图，如图5所示，包括：
S301、从设备的监控模块向媒体信息下载模块发送查询服务端媒体信息的请求。
S302、媒体信息下载模块向媒体信息发布模块发送媒体描述信息请求，媒体信息发布模块返回媒体描述信息请求响应，携带媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率。
S303、媒体信息下载模块解析媒体描述信息获得所有服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率。
S304、从设备的监控模块配置SNTP服务地址和端口，启动时间客户端进行SNTP时间同步。
最后，时间客户端与系统时间服务端之间进行系统时间同步。
图6为从设备同步播放共享媒体资源时各个模块的交互流程图，如图6所示，包括：
S401、从设备的监控模块向节目时钟客户端发送同步节目时钟请求。
S402、节目时钟客户端向节目时钟服务端发送加入RTCP会话请求，节目时钟服务端返回RTCP会话标识。
S403、节目时钟服务端根据预设采集周期采集PCR，并在满足预设条件时向节目时钟客户端发送PCR采样值。
S404、节目时钟客户端根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的STC。
S405、从设备的监控模块向RTP分组客户端发送启动音频数据帧接收请求。
S406、RTP分组客户端向RTP分组服务端发送加入组播组请求。
S407、RTP分组服务端向RTP分组客户端发送RTP。
S408、RTP分组客户端接收RTP，将RTP拼接成完整的音频数据帧，并从RTP分组 的时间戳头域获取该音频数据帧对应的显示时间戳，将该音频数据帧放入从设备的PCM缓存。
S409、PCM播放驱动模块根据从设备的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。
本发明实施例可以根据上述方法示例对主设备和从设备进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本发明实施例各实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
图7为本发明实施例从设备实施例一的结构示意图，如图7所示，本实施例的从设备可以包括：第一接收模块11、校正模块12、第二接收模块13、处理模块14和输出模块15，其中，第一接收模块11用于接收主设备发送的RTCP报文，RTCP报文中的RTP时间戳头域携带主设备周期性采集的节目时钟参考PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点。校正模块12用于根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC。第二接收模块13用于根据主设备发送的组播服务地址和端口接收主设备发布的RTP。处理模块14用于将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取音频数据帧对应的显示时间戳PTS，将音频数据帧放入从设备的音频脉冲编码调制PCM缓存。输出模块15用于根据自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。
图8为本发明实施例从设备实施例二的结构示意图，如图8所示，在图7所示的从设备的基础上，进一步地，本实施例的从设备还可以包括：发送模块16，发送模块16用于在校正模块12根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，向主设备发送加入RTCP会话请求，以使主设备向从设备发送RTCP会话标识。
进一步地，校正模块12具体用于：根据如下公式计算出从设备的STC校正值scr_correct：
scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
其中，scf_clt为从设备节目时钟频率，scf_srv主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为PCR和RTCP报文的发送时间点。根据计算出的STC校正值校正自身的STC。
进一步地，第一接收模块11还用于：在接收主设备发送的实时控制协议RTCP报文之前，接收主设备发送的媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率；处理模块14还用于根据SNTP服务地址和端口进行系统时间同步；第一接收模块11具体用于：根据RTCP服务地址和端口接收主设备发送的RTCP报文。
图7或图8所示实施例的从设备，可以用于执行图1或图2所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
图9为本发明实施例主设备实施例一的结构示意图，如图9所示，本实施例的主设备可以包括：采集模块21、发送模块22和处理模块23，其中，采集模块21用于根据预设 采集周期采集节目时钟参考PCR。发送模块22用于在确定满足预设条件时向从设备发送实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点，以使从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正从设备的系统时钟STC。处理模块23用于从主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将RTP发布到组播服务地址和端口，RTP的时间戳头域携带了音频数据帧对应的显示时间戳，用于从设备根据组播服务地址和端口接收RTP。
图10为本发明实施例主设备实施例二的结构示意图，如图10所示，在图9所示的主设备的基础上，进一步地，本实施例的主设备还可以包括：接收模块24，该接收模块24用于在发送模块22在确定满足预设条件时向从设备发送RTCP报文之前，接收从设备发送的加入RTCP会话请求，并向从设备发送RTCP会话标识。
进一步地，发送模块22还用于：在采集模块21根据预设采集周期采集节目时钟参考PCR之前，向从设备发送媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率，SNTP服务地址和端口用于从设备进行系统时间同步，RTCP服务地址和端口用于从设备根据RTCP服务地址和端口接收主设备发送的RTCP报文。
其中，预设条件为主设备实际采集PCR的时间间隔与预设采集周期之间的偏差大于预设阈值。
可选的，预设阈值为20ms，预设条件为：
(scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
(scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为主设备节目时钟频率，cycle_read_x为预设采集周期。
图9和图10所示实施例的主设备，可以用于执行图1或图2所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
图11为本发明实施例从设备实施例三的结构示意图，如图11所示，本实施例的从设备可以包括：接收器31、处理器32和发送器33，其中，接收器31用于接收主设备发送的RTCP报文，RTCP报文中的RTP时间戳头域携带主设备周期性采集的节目时钟参考PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点。处理器32用于根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC。接收器31还用于根据主设备发送的组播服务地址和端口接收主设备发布的RTP。处理器32还用于将RTP拼接成完整的音频数据帧，并从RTP分组的时间戳头域获取音频数据帧对应的显示时间戳PTS，将音频数据帧放入从设备的音频脉冲编码调制PCM缓存。发送器33用于根据自身的STC和音频数据帧显示时间戳输出音频PCM缓存中的音频数据帧。
进一步地，发送器33还用于在校正模块根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，向主设备发送加入RTCP会话请求，以使主设备向从设备发送RTCP会话标识。
进一步地，处理器32具体用于：根据如下公式计算出从设备的STC校正值scr_correct：
scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
其中，scf_clt为从设备节目时钟频率，scf_srv主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为PCR和RTCP报文的发送时间点。根据计算出的STC校正值校正自身的STC。
进一步地，接收器31还用于：在接收主设备发送的实时控制协议RTCP报文之前，接收主设备发送的媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率；处理器32还用于根据SNTP服务地址和端口进行系统时间同步；接收器31具体用于：根据RTCP服务地址和端口接收主设备发送的RTCP报文。
本实施例的从设备，可以用于执行图1或图2所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
图12为本发明实施例主设备实施例三的结构示意图，如图12所示，本实施例的主设备可以包括：处理器41和发送器42，其中，处理器41用于根据预设采集周期采集节目时钟参考PCR。发送器42用于在确定满足预设条件时向从设备发送实时控制协议RTCP报文，RTCP报文中的实时流分组RTP时间戳头域携带PCR，RTCP报文中的网络时间协议NTP域携带RTCP报文的发送时间点，以使从设备根据PCR、主设备节目时钟频率、从设备的节目时钟频率和RTCP时延校正从设备的系统时钟STC。处理器41还用于从主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将RTP发布到组播服务地址和端口，RTP的时间戳头域携带了音频数据帧对应的显示时间戳，用于从设备根据组播服务地址和端口接收RTP。
图13为本发明实施例主设备实施例四的结构示意图，如图13所示，在图12所示的主设备的基础上，进一步地，本实施例的主设备还可以包括：接收器43，该接收器43用于在发送器42在确定满足预设条件时向从设备发送RTCP报文之前，接收从设备发送的加入RTCP会话请求，并向从设备发送RTCP会话标识。
进一步地，发送器42还用于：在处理器41根据预设采集周期采集节目时钟参考PCR之前，向从设备发送媒体描述信息，媒体描述信息包括SNTP服务地址和端口、RTCP服务地址和端口、组播服务地址和端口、主设备节目时钟频率、媒体格式和时戳频率，SNTP服务地址和端口用于从设备进行系统时间同步，RTCP服务地址和端口用于从设备根据RTCP服务地址和端口接收主设备发送的RTCP报文。
其中，预设条件为主设备实际采集PCR的时间间隔与预设采集周期之间的偏差大于预设阈值。
可选的，预设阈值为20ms，预设条件为：
(scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
(scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为主设备节目时钟频率，cycle_read_x为预设采集周期。
图12和图13所示实施例的主设备，可以用于执行图1或图2所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部 分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统或装置实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
本领域普通技术人员将会理解，本发明实施例的各个方面、或各个方面的可能实现方式可以被具体实施为系统、方法或者计算机程序产品。因此，本发明实施例的各方面、或各个方面的可能实现方式可以采用完全硬件实施例、完全软件实施例(包括固件、驻留软件等等)，或者组合软件和硬件方面的实施例的形式，在这里都统称为“电路”、“模块”或者“系统”。此外，本发明实施例的各方面、或各个方面的可能实现方式可以采用计算机程序产品的形式，计算机程序产品是指存储在计算机可读介质中的计算机可读程序代码。
计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质包含但不限于电子、磁性、光学、电磁、红外或半导体系统、设备或者装置，或者前述的任意适当组合，如随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或者快闪存储器)、光纤、便携式只读存储器(CD-ROM)。
计算机中的处理器读取存储在计算机可读介质中的计算机可读程序代码，使得处理器能够执行在流程图中每个步骤、或各步骤的组合中规定的功能动作；生成实施在框图的每一块、或各块的组合中规定的功能动作的装置。
计算机可读程序代码可以完全在用户的本地计算机上执行、部分在用户的本地计算机上执行、作为单独的软件包、部分在用户的本地计算机上并且部分在远程计算机上，或者完全在远程计算机或者服务器上执行。也应该注意，在某些替代实施方案中，在流程图中各步骤、或框图中各块所注明的功能可能不按图中注明的顺序发生。例如，依赖于所涉及的功能，接连示出的两个步骤、或两个块实际上可能被大致同时执行，或者这些块有时候可能被以相反顺序执行。

Claims (18)

  1. 一种多设备间唇音同步方法，用于一个主设备同步输出音频和视频到至少一个从设备中，其特征在于，所述方法包括：
    所述从设备接收所述主设备发送的实时控制协议RTCP报文，所述RTCP报文中的实时流分组RTP时间戳头域携带所述主设备周期性采集的节目时钟参考PCR，所述RTCP报文中的网络时间协议NTP域携带所述RTCP报文的发送时间点；
    所述从设备根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC；
    所述从设备根据所述主设备发送的组播服务地址和端口接收所述主设备发布的RTP，将所述RTP拼接成完整的音频数据帧，并从所述RTP分组的时间戳头域获取所述音频数据帧对应的显示时间戳PTS，将所述音频数据帧放入所述从设备的音频脉冲编码调制PCM缓存；
    所述从设备根据自身的STC和音频数据帧显示时间戳输出所述音频PCM缓存中的音频数据帧。
  2. 根据权利要求1所述的方法，其特征在于，所述RTCP报文中携带RTCP会话标识，所述从设备根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，还包括：
    所述从设备向所述主设备发送加入RTCP会话请求，以使所述主设备向所述从设备发送RTCP会话标识。
  3. 根据权利要求1或2所述的方法，其特征在于，所述从设备根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC，包括：
    所述从设备根据如下公式计算出所述从设备的STC校正值scr_correct：
    scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
    其中，scf_clt为所述从设备节目时钟频率，scf_srv所述主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为所述PCR和所述RTCP报文的发送时间点；
    所述从设备根据计算出的STC校正值校正自身的STC。
  4. 根据权利要求1～3任一项所述的方法，其特征在于，所述从设备接收所述主设备发送的实时控制协议RTCP报文之前，还包括：
    所述从设备接收所述主设备发送的媒体描述信息，所述媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、所述组播服务地址和端口、所述主设备节目时钟频率、媒体格式和时戳频率；
    所述从设备根据所述SNTP服务地址和端口进行系统时间同步；
    所述从设备接收所述主设备发送的实时控制协议RTCP报文，包括：
    所述从设备根据所述RTCP服务地址和端口接收所述主设备发送的RTCP报文。
  5. 一种多设备间唇音同步方法，用于一个主设备同步输出音频和视频到至少一个从设备中，其特征在于，所述方法包括：
    所述主设备根据预设采集周期采集节目时钟参考PCR；
    所述主设备在确定满足预设条件时向所述从设备发送实时控制协议RTCP报文，所述RTCP报文中的实时流分组RTP时间戳头域携带所述PCR，所述RTCP报文中的网络时间协议NTP域携带所述RTCP报文的发送时间点，以使所述从设备根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正所述从设备的系统时钟STC；
    所述主设备从所述主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将所述RTP发布到组播服务地址和端口，所述RTP的时间戳头域携带了所述音频数据帧对应的显示时间戳，用于所述从设备根据所述组播服务地址和端口接收所述RTP。
  6. 根据权利要求5所述的方法，其特征在于，所述RTCP报文中携带RTCP会话标识，所述主设备在确定满足预设条件时向所述从设备发送实时控制协议RTCP报文之前，还包括：
    所述主设备接收所述从设备发送的加入RTCP会话请求，并向所述从设备发送RTCP会话标识。
  7. 根据权利要求5或6所述的方法，其特征在于，所述主设备根据预设采集周期采集节目时钟参考PCR之前，还包括：
    所述主设备向所述从设备发送媒体描述信息，所述媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、所述组播服务地址和端口、所述主设备节目时钟频率、媒体格式和时戳频率，所述SNTP服务地址和端口用于所述从设备进行系统时间同步，所述RTCP服务地址和端口用于所述从设备根据所述RTCP服务地址和端口接收所述主设备发送的RTCP报文。
  8. 根据权利要求5～7任一项所述的方法，其特征在于，所述预设条件为：
    所述主设备实际采集PCR的时间间隔与所述预设采集周期之间的偏差大于预设阈值。
  9. 根据权利要求8所述的方法，其特征在于，所述预设阈值为20ms，所述预设条件为：
    (scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
    (scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
    其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为所述主设备节目时钟频率，cycle_read_x为所述预设采集周期。
  10. 一种从设备，其特征在于，包括：
    第一接收模块，用于接收所述主设备发送的实时控制协议RTCP报文，所述RTCP报文中的实时流分组RTP时间戳头域携带所述主设备周期性采集的节目时钟参考PCR，所述RTCP报文中的网络时间协议NTP域携带所述RTCP报文的发送时间点；
    校正模块，用于根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC；
    第二接收模块，用于根据所述主设备发送的组播服务地址和端口接收所述主设备发布的RTP；
    处理模块，用于将所述RTP拼接成完整的音频数据帧，并从所述RTP分组的时间戳 头域获取所述音频数据帧对应的显示时间戳PTS，将所述音频数据帧放入所述从设备的音频脉冲编码调制PCM缓存；
    输出模块，用于根据自身的STC和音频数据帧显示时间戳输出所述音频PCM缓存中的音频数据帧。
  11. 根据权利要求10所述的从设备，其特征在于，所述RTCP报文中携带RTCP会话标识，所述从设备还包括：
    发送模块，用于在所述校正模块根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正自身的系统时钟STC之前，向所述主设备发送加入RTCP会话请求，以使所述主设备向所述从设备发送RTCP会话标识。
  12. 根据权利要求10或11所述的从设备，其特征在于，所述校正模块具体用于：
    根据如下公式计算出所述从设备的STC校正值scr_correct：
    scr_correct＝scr_srv*scf_clt/scf_srv+(ntp_rcv–ntp_snd)*scf_clt/1000
    其中，scf_clt为所述从设备节目时钟频率，scf_srv所述主设备节目时钟频率，ntp_rcv为接收到RTCP报文的时刻，scr_srv和ntp_snd分别为所述PCR和所述RTCP报文的发送时间点；
    根据计算出的STC校正值校正自身的STC。
  13. 根据权利要求10～12任一项所述的从设备，其特征在于，所述第一接收模块还用于：
    在接收所述主设备发送的实时控制协议RTCP报文之前，接收所述主设备发送的媒体描述信息，所述媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、所述组播服务地址和端口、所述主设备节目时钟频率、媒体格式和时戳频率；
    所述处理模块还用于根据所述SNTP服务地址和端口进行系统时间同步；
    所述第一接收模块具体用于：根据所述RTCP服务地址和端口接收所述主设备发送的RTCP报文。
  14. 一种主设备，其特征在于，包括：
    采集模块，用于根据预设采集周期采集节目时钟参考PCR；
    发送模块，用于在确定满足预设条件时向所述从设备发送实时控制协议RTCP报文，所述RTCP报文中的实时流分组RTP时间戳头域携带所述PCR，所述RTCP报文中的网络时间协议NTP域携带所述RTCP报文的发送时间点，以使所述从设备根据所述PCR、所述主设备节目时钟频率、所述从设备的节目时钟频率和RTCP时延校正所述从设备的系统时钟STC；
    处理模块，用于从所述主设备的音频脉冲编码调制PCM缓存中复制出音频数据帧并打包成实时流分组RTP，将所述RTP发布到组播服务地址和端口，所述RTP的时间戳头域携带了所述音频数据帧对应的显示时间戳，用于所述从设备根据所述组播服务地址和端口接收所述RTP。
  15. 根据权利要求14所述的主设备，其特征在于，所述RTCP报文中携带RTCP会话标识，所述主设备还包括：
    接收模块，用于在所述发送模块在确定满足预设条件时向所述从设备发送实时控制协议RTCP报文之前，接收所述从设备发送的加入RTCP会话请求，并向所述从设备发送 RTCP会话标识。
  16. 根据权利要求14或15所述的主设备，其特征在于，所述发送模块还用于：
    在所述采集模块根据预设采集周期采集节目时钟参考PCR之前，向所述从设备发送媒体描述信息，所述媒体描述信息包括简单网络时间协议SNTP服务地址和端口、RTCP服务地址和端口、所述组播服务地址和端口、所述主设备节目时钟频率、媒体格式和时戳频率，所述SNTP服务地址和端口用于所述从设备进行系统时间同步，所述RTCP服务地址和端口用于所述从设备根据所述RTCP服务地址和端口接收所述主设备发送的RTCP报文。
  17. 根据权利要求14～16任一项所述的主设备，其特征在于，所述预设条件为：
    所述主设备实际采集PCR的时间间隔与所述预设采集周期之间的偏差大于预设阈值。
  18. 根据权利要求17所述的主设备，其特征在于，所述预设阈值为20ms，所述预设条件为：
    (scr_curr–scr_last)*1000/scf_srv<(cycle_read_x–20)或
    (scr_curr–scr_last)*1000/scf_srv>(cycle_read_x+20)；
    其中，scr_curr为当前采集时钟值，scr_last为上一个采集时钟值，scf_srv为所述主设备节目时钟频率，cycle_read_x为所述预设采集周期。
PCT/CN2017/077925 2017-03-23 2017-03-23 多设备间唇音同步方法及设备 WO2018170852A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
PCT/CN2017/077925 WO2018170852A1 (zh) 2017-03-23 2017-03-23 多设备间唇音同步方法及设备
US16/496,306 US11146611B2 (en) 2017-03-23 2017-03-23 Lip synchronization of audio and video signals for broadcast transmission
CN201780049030.3A CN109565466B (zh) 2017-03-23 2017-03-23 多设备间唇音同步方法及设备
EP17902312.2A EP3591908B9 (en) 2017-03-23 2017-03-23 Method and device for lip-speech synchronization among multiple devices

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/077925 WO2018170852A1 (zh) 2017-03-23 2017-03-23 多设备间唇音同步方法及设备

Publications (1)

Publication Number Publication Date
WO2018170852A1 true WO2018170852A1 (zh) 2018-09-27

Family

ID=63584723

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/077925 WO2018170852A1 (zh) 2017-03-23 2017-03-23 多设备间唇音同步方法及设备

Country Status (4)

Country Link
US (1) US11146611B2 (zh)
EP (1) EP3591908B9 (zh)
CN (1) CN109565466B (zh)
WO (1) WO2018170852A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111954050A (zh) * 2019-05-14 2020-11-17 福州瑞芯微电子股份有限公司 一种多设备间视频同步的方法及系统
CN112130616A (zh) * 2020-09-28 2020-12-25 北京小米松果电子有限公司 时钟同步方法、装置及存储介质

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE112019006635T5 (de) * 2019-01-08 2021-09-30 Sony Group Corporation Empfangsvorrichtung und verfahren, und übertragungsvorrichtung und verfahren
US11394480B2 (en) * 2019-08-23 2022-07-19 Bose Corporation Systems and methods for synchronizing device clocks
US11005585B1 (en) 2019-12-31 2021-05-11 Juniper Networks, Inc. Transporting client timing information across a network
CN113519146B (zh) * 2020-02-12 2023-06-23 深圳元戎启行科技有限公司 流媒体网络时延确定方法、装置、计算机设备、可读存储介质和远程驾驶系统
CN112887773A (zh) * 2021-01-22 2021-06-01 昆腾微电子股份有限公司 一种音频设备的同步方法及装置
CN114827696B (zh) * 2021-01-29 2023-06-27 华为技术有限公司 一种跨设备的音视频数据同步播放的方法和电子设备
CN113613125B (zh) * 2021-04-26 2024-07-09 珠海市杰理科技股份有限公司 音频同步控制方法、装置及音频设备、系统
CN113676470B (zh) * 2021-08-17 2023-02-07 维沃移动通信有限公司 获取rtcp报告信息的方法和装置
CN115801859B (zh) * 2022-11-10 2024-06-04 广东美的智能科技有限公司 组态设备之间的连接方法、工业控制装置和系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050177643A1 (en) * 2004-02-05 2005-08-11 Shugong Xu System and method for transporting MPEG2TS in RTP/UDP/IP
CN101123611A (zh) * 2007-09-25 2008-02-13 中兴通讯股份有限公司 一种流媒体数据的发送方法
CN101179484A (zh) * 2006-11-09 2008-05-14 华为技术有限公司 一种不同媒体流间的同步方法及系统
CN101237586A (zh) * 2008-02-22 2008-08-06 上海华平信息技术股份有限公司 音视频缓存同步播放的方法
CN101288257A (zh) * 2005-08-26 2008-10-15 诺基亚公司 向设备发送信令以不执行同步或在多媒体流上包括同步延迟的方法

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3255308B2 (ja) 1992-12-18 2002-02-12 ソニー株式会社 データ再生装置
US5486864A (en) 1993-05-13 1996-01-23 Rca Thomson Licensing Corporation Differential time code method and apparatus as for a compressed video signal
JPH11261958A (ja) 1998-03-09 1999-09-24 Sony Corp 映像編集装置及び映像編集方法
US20030066094A1 (en) * 2001-09-29 2003-04-03 Koninklijke Philips Electronics N.V. Robust method for recovering a program time base in MPEG-2 transport streams and achieving audio/video sychronization
CN1286314C (zh) 2004-08-05 2006-11-22 联合信源数字音视频技术（北京）有限公司 视频解码系统中保持显示同步的方法及其装置
KR101263522B1 (ko) * 2004-09-02 2013-05-13 소니 주식회사 콘텐츠 수신 장치, 비디오 오디오 출력 타이밍 제어 방법및 콘텐츠 제공 시스템
US8068515B2 (en) * 2005-06-22 2011-11-29 Cisco Technology, Inc. Faster multimedia synchronization of broadcast streams using router caching of RTCP packets
CN100442858C (zh) 2005-10-11 2008-12-10 华为技术有限公司 分组网络中多媒体实时传输的唇同步方法及其装置
CN101438256B (zh) * 2006-03-07 2011-12-21 索尼株式会社 信息处理设备、信息通信系统、信息处理方法
EP2254342A1 (de) 2009-05-18 2010-11-24 Albis Technologies AG Verfahren zur Audio-/Videosynchronisation
CN101710997A (zh) * 2009-11-04 2010-05-19 中兴通讯股份有限公司 基于mpeg-2系统实现视、音频同步的方法及系统
CN101848396B (zh) 2009-11-30 2012-10-17 深圳市华曦达科技股份有限公司 传输流音视频同步及防抖动方法
CN102547299A (zh) 2010-12-30 2012-07-04 福建星网视易信息系统有限公司 基于mpeg-2视频流的音视频同步控制方法
CN102421035A (zh) * 2011-12-31 2012-04-18 青岛海信宽带多媒体技术有限公司 数字电视音视频同步的实现方法及装置
CN102665141B (zh) 2012-05-16 2014-04-09 哈尔滨工业大学深圳研究生院 一种基于rtp封装的avs音视频预同步方法
CN103888813A (zh) 2012-12-21 2014-06-25 北京计算机技术及应用研究所 一种音视频同步的实现方法及系统
US9978395B2 (en) * 2013-03-15 2018-05-22 Vocollect, Inc. Method and system for mitigating delay in receiving audio stream during production of sound from audio stream
US9800908B2 (en) * 2013-07-09 2017-10-24 Koninklijke Kpn N.V. Synchronized data processing of broadcast streams between receivers, including synchronized data processing between a receiver that is in the process of processing a stream and a receiver that wants to join the stream
CN103428584A (zh) 2013-08-01 2013-12-04 珠海全志科技股份有限公司 多媒体播放平台上保持音视频同步的方法及设备
WO2015105391A1 (en) * 2014-01-13 2015-07-16 Lg Electronics Inc. Apparatuses and methods for transmitting or receiving a broadcast content via one or more networks
EP3226570A1 (en) * 2016-03-31 2017-10-04 Thomson Licensing Synchronizing audio and video signals rendered on different devices

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050177643A1 (en) * 2004-02-05 2005-08-11 Shugong Xu System and method for transporting MPEG2TS in RTP/UDP/IP
CN101288257A (zh) * 2005-08-26 2008-10-15 诺基亚公司 向设备发送信令以不执行同步或在多媒体流上包括同步延迟的方法
CN101179484A (zh) * 2006-11-09 2008-05-14 华为技术有限公司 一种不同媒体流间的同步方法及系统
CN101123611A (zh) * 2007-09-25 2008-02-13 中兴通讯股份有限公司 一种流媒体数据的发送方法
CN101237586A (zh) * 2008-02-22 2008-08-06 上海华平信息技术股份有限公司 音视频缓存同步播放的方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111954050A (zh) * 2019-05-14 2020-11-17 福州瑞芯微电子股份有限公司 一种多设备间视频同步的方法及系统
CN112130616A (zh) * 2020-09-28 2020-12-25 北京小米松果电子有限公司 时钟同步方法、装置及存储介质

Also Published As

Publication number Publication date
EP3591908B9 (en) 2022-04-06
EP3591908A1 (en) 2020-01-08
EP3591908A4 (en) 2020-01-08
EP3591908B1 (en) 2022-02-09
US20200099734A1 (en) 2020-03-26
CN109565466A (zh) 2019-04-02
US11146611B2 (en) 2021-10-12
CN109565466B (zh) 2020-11-06

Similar Documents

Publication Publication Date Title
WO2018170852A1 (zh) 多设备间唇音同步方法及设备
US9479584B2 (en) Synchronous media rendering of demuxed media components across multiple devices
EP3118855B1 (en) Audio synchronous playing method, device and system
US10652849B2 (en) Using broadcast physical layer for one-way time transfer of universal coordinated time to receivers
JP2019532576A (ja) オーディオとビデオのマルチモード同期レンダリング
JP5086285B2 (ja) 映像配信システム，映像配信装置，及び同期補正処理装置
JP2014528217A (ja) 無線ネットワーク時刻同期のための方法及び装置
JP2019024214A (ja) 再生同期
JP2023106456A (ja) 受信装置および受信方法
WO2016090348A1 (en) Techniques for synchronizing timing of wireless streaming transmissions to multiple sink devices
JP5841715B2 (ja) 映像音声出力装置、および映像音声出力システム、およびマスタ装置
JP6789761B2 (ja) 受信端末及びプログラム
KR20150146116A (ko) 이종망 기반 방송 서비스를 제공하는 방법 및 장치
WO2015098285A1 (ja) デジタル放送受信機、及び、外部端末
US20210311692A1 (en) Method of Managing an Audio Stream Read in a Manner That is Synchronized on a Reference Clock
KR102271686B1 (ko) 이종 네트워크 기반의 멀티미디어 자원 동기화 푸시 방법
US12028691B2 (en) Method of applying a set of equalization parameters
WO2023273601A1 (zh) 一种音频同步方法及音频播放设备、音频源、存储介质
CN114554242B (zh) 直播方法和可读存储介质
WO2024046071A1 (zh) 一种数据传输方法及装置
US20240040065A1 (en) Video device, wireless audio device and method for performing audio and video synchronization between video device and wireless audio device
JP2010183237A (ja) コンテンツ同期再生システム、コンテンツ同期再生方法、再生端末、再生端末の制御方法、及び制御プログラム
TWI660628B (zh) 多設備間的媒體同步播放方法
KR20180050983A (ko) Ｒｔｐ 패킷 전송 방법 및 장치
JP2013065958A (ja) パケット伝送システムおよび方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17902312

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017902312

Country of ref document: EP

Effective date: 20191002