JP6290915B2 - Common event-based multi-device media playback - Google Patents

Description

  The present disclosure relates to the field of digital media, and in particular to the field of digital multimedia synchronized playback.

  Today there are many forms of digital media, many types of digital media sources, many types of digital media playback (rendering) systems, and many ways of connecting media sources to media playback systems.

  Digital media, hereinafter referred to as media, is provided in many forms, formats, and containers including digital video discs, media files, and media streams. Media content can be audio, video, image, or metadata media components, and various combinations of each. For example, a popular audio format is known as MP3, and a popular video format is H264. MP3 is an audio-specific media format that was designed as part of the MPEG-1 standard by the Moving Picture Experts Group (MPEG) and later extended with the MPEG-2 standard. H264 is a standard developed by a group of motion picture professionals (MPEG), a joint working group of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). Movie is usually a multimedia format with a video channel and multiple audio channels inside. For example, a 5.1 moving image includes one video channel (media component) and six audio channels (audio component). 5.1 is a generic name for a 6-channel surround sound multi-channel audio system.

  Digital media sources include digital video disc players, Blu-ray players, media devices such as computers and mobile devices, and Internet-based “cloud” media services. Blu-ray Disc (BD) is an optical disc storage medium developed by the Blu-ray Disc Association. Internet-based media services include services such as Netflix (registered trademark) and Spotify (registered trademark). Netflix is a media service and trademark of Netflix Inc. Spotify is a media service and trademark of Spotify Ltd. Digital media playback (media rendering destination) systems include computer-based devices, laptops and smartphones, and network audio and video devices. Smart TV is an example of a digital media rendering device that can play media from Internet (cloud) based media services such as net flicks. Smart TV, sometimes referred to as “Connected TV” or “Hybrid TV”, integrates Internet and web features into modern TV sets and set-top boxes, computers and these Used to describe the technical convergence between the television set / set top box. An Internet radio device is another example of a digital media rendering device.

  The connection between these media sources and devices varies, but has evolved over time to network-based connections using the Internet Protocol (IP) protocol. This is because IP connection is convenient, universal and inexpensive.

  IP networks are provided in many forms. The most popular is Ethernet-based wired IP networking. Ethernet is a family of local area network (LAN) computer networking technologies standardized as IEEE (Institute of Electrical and Electronics Engineers) standard 802.3. In recent years, with the spread of mobile computing devices, Wi-Fi (registered trademark) (a type of IP network) has become the most popular means of wirelessly connecting network devices. Wi-Fi is a trademark of the Wi-Fi Alliance and is the brand name for products that use the standard IEEE 802.11 family. An IP network can use several different types of messaging, including unicast, multicast, and broadcast messaging, such messaging being the transmission of IP packets.

  The term “unicast” may be used to refer to a type of Internet protocol transmission in which information is transmitted from only one sender to only one recipient. In other words, unicast transmission is a one-to-one node transmission between only two nodes. In unicast, each outgoing packet has a unicast destination address, which means that the packet is directed to a specific destination with that address. All other destinations that can listen to the packet ignore the packet if the destination address of the packet is not the same as the destination address.

  Many IP protocols are accessed from software programs through a socket application programming interface. This socket interface is defined as part of the POSIX standard. POSIX is an acronym for “Portable Operating System Interface,” a family of standards specified by the IEEE to maintain operating system compatibility.

  The convenience and advantage of IP networking means that all of these media sources and playback systems are becoming network capable if they are not yet network capable. Many Blu-ray players now have Ethernet and Wi-Fi network connections. Today, most higher-end TVs are smart TVs with network capabilities. Similarly, audio playback devices and radios are compatible with the network and the Internet.

  Mobile devices such as mobile phones, tablets, document readers, or notebooks can receive and store media, have powerful multimedia (audio and video) capabilities, high bandwidth, wide and deep It may be connected to the Internet via a cell phone data service or a broadband link (such as Wi-Fi) that can access an online media service with content.

  Use cases or applications of these various forms of digital media, media services and media sources, and playback systems have evolved. Initially, it was sufficient to connect the media source to the media destination via the IP network. This is widely used today with computers such as Internet-based media source services such as Netflix and media destinations. A user views a Netflix video streamed to a computer via a wired IP network (Internet). This is the case for a single point (one IP source) -single point (one IP destination) connection over a wired IP network. Netflix media services may send the same media to multiple assumptions, each of which is a single point-single point TCP / IP connection. This further evolution is to use a wireless Wi-Fi connection instead of a wired Ethernet connection. Nevertheless, this is still a single point-single point connection.

  There is a further extension of the use case above where the media source connects to multiple destinations instead of a single destination. These are single point (one IP source) -multipoint (multiple IP destinations) applications. An example is when a user is playing a 5.1 video media file to a wireless video playback device, and six independent wireless audio destinations constitute a complete 5.1 surround sound system. In this case, media is being sent simultaneously from one media source to seven media destinations. In another example, a user is playing music from one media source to six playback systems located around six different rooms in the house.

  In both of these cases, it is necessary to play (render) the media in time synchronization at all destinations. In addition, resource usage at the media source needs to be limited, such as keeping memory usage to a minimum. In addition, multiple devices that receive media need to manage network bandwidth efficiently.

  If currently the same media data needs to be sent to multiple network destinations, a common technique to do so is to use data multicast to multiple destinations that need to receive the data. In such a system, the media is multicast to all destinations, and it is up to each destination to attempt to render the media accordingly. During rendering, if there is an error that the renderer does not receive new media data or does not receive it correctly, the renderer renders the incorrect data and then receives the correct data from the point after the error when it receives the correct data. The media can be restored to continue accurate media rendering.

  In the application considered here, it is necessary to send media from a source to multiple media devices such as TVs and speakers in the same listening and viewing space. Furthermore, it is necessary to transmit this media via a wireless network such as Wi-Fi.

  In these applications, this precisely synchronizes all media rendering devices such as speakers within the same listening or viewing zone with each other so that the listener and / or viewer is unaware of any unintended media experience. It means you need to.

  Second, because the media is transported over the air, there is a high probability of media errors where the media is not reliably received at each destination or is not received uniformly. When using broadcast or multicast for packet transmission, the same broadcast or multicast packet may be received at one destination but not received / listened at another destination.

An event-based synchronized multimedia playback method is disclosed that solves the above problems of media errors and loss of synchronization when transmitted to multiple media receiving devices.
According to a preferred embodiment, an event-based synchronized multimedia playback system is disclosed comprising a plurality of media playback devices each maintaining audio phase and timing through synchronization based on crystal clock cycles. According to this embodiment, the plurality of media playback devices may be connected to the plurality of media source devices by an IP-enabled network such as the Internet or any other data network that utilizes the IP protocol. Media can be broadcast to playback devices over such networks, and each playback device can play similar or separate media elements that may be desirable during playback (eg, common in acoustic systems in the art). As can be seen, one speaker can play the left audio channel, while the second speaker can play the right audio channel).

According to a preferred embodiment, an event-based synchronized multimedia playback system is disclosed comprising a media source device and a plurality of destination devices each comprising a local clock. Synchronization the media source device module sends a common event E _n that contains a unique event number n to each of the plurality of destination devices in the media sources. Each destination device records the time Dx _n that received the event E _n, and returns an acknowledgment message containing the time Dx _n and unique event number n in synchronization module. The synchronization module identifies, for each of the plurality of receiving devices, a phase difference and a frequency difference between the clock signals of each receiving device, calculates a phase adjustment value and a frequency adjustment value to compensate for the difference, and accordingly the clock phase and frequency Instruct each destination device to adjust. Each receiving device adjusts the local clock to the desired value, performs sample rate conversion, and synchronizes media playback.

According to another preferred embodiment, an event-based synchronized multimedia playback method is disclosed, the method comprising (a) a synchronization module, connected to a network and configured to stream media over the network. from a media source device has, for receiving and transmitting a common event E _n that contains a unique event number n regularly, the event E _n via a network at each of a plurality of receiving device with (b) a local clock step a, step a, (d) media source an acknowledgment message from the receiving device comprising at least time Dx _n and unique event number n for recording by the receiving device time Dx _n that received the (c) events E _n And (e) a media source device and a plurality Identifying a phase difference and a frequency difference of the clock signal of each destination device using a synchronization module stored in and operating on one of the destination devices; and (f) identified in step (e) Calculating a frequency adjustment value that compensates for the phase difference and the frequency difference; and (g) instructing each receiving device to adjust the phase and frequency of the clock by an amount equal to the frequency adjustment value calculated in step (f). And (h) adjusting the local clock of each receiving device as instructed in step (c) to synchronize media playback.

  The accompanying drawings illustrate several embodiments of the disclosed configuration and together with the description serve to explain the principles of the invention according to the embodiments. One skilled in the art can appreciate that the specific embodiments illustrated in the drawings are merely exemplary and are not intended to limit the scope of the disclosed embodiments.

1 is an exemplary multimedia system comprising two audio subsystems according to an embodiment of the invention. FIG. 1 is a detailed view of an exemplary audio playback system, according to one embodiment of the invention. FIG. FIG. 3 is an exemplary clock-based system for time reference according to one embodiment of the invention. FIG. 4 is an exemplary message timeline diagram according to one embodiment of the invention. 1 is a diagram of an exemplary common event-based system according to one embodiment of the invention. FIG. FIG. 4 is an exemplary event timeline diagram according to one embodiment of the invention. FIG. 6 details a difference between a common clock method and a common event method according to an embodiment of the present invention. FIG. 4 is a diagram of the overall effect of a common event-based system according to an embodiment of the invention. FIG. 4 is a diagram of the overall effect of using a common event-based algorithm according to one embodiment of the invention. FIG. 2 is a block diagram illustrating an exemplary hardware architecture of a computing device used in one embodiment of the present invention. 2 is a block diagram illustrating an exemplary logical architecture of a client device, according to one embodiment of the invention. FIG. FIG. 3 is a block diagram illustrating exemplary structural configurations of a client, a server, and an external service according to an embodiment of the present invention.

  The term “unicast” may be used to refer to a type of Internet protocol transmission in which information is transmitted from only one sender to only one recipient. In other words, unicast transmission is a one-to-one node transmission between only two nodes. In unicast, each outgoing packet has a unicast destination address, which means that the packet is directed to a specific destination with that address. All other destinations that can listen to the packet ignore the packet if the destination address of the packet is not the same as the destination address.

  As used herein, “broadcast messaging” or “broadcast” refers to a type of Internet protocol transmission in which information is transmitted from only one computer but received by all computers connected to the network. Point to. This means that every time a computer or node sends a “broadcast” packet, all other computers receive that information packet.

  As used herein, “multicast messaging” or “multicast” refers to a set of recipients in which there may be more than one sender, and the transmitted information is grouped as a multicast group, A type of Internet protocol transmission or communication in which a set of recipients may be a subset of all recipients. In multicast, each multicast packet is addressed to a multicast address. This address is a group address. Any destination can subscribe to the address and thus listen and receive packets sent to the subscribed multicast address. The advantage of multicast is that multiple destinations can receive a single transmitted multicast packet. This saves network traffic when the same packet needs to be sent to multiple destinations. In general, if the same data needs to be sent to multiple IP destinations, broadcast or multicast rather than unicast provides the most efficient use of the network.

  In this description, the terms broadcast and multicast may be used. In both broadcast and multicast, when a message is sent, the message is received by multiple destinations. Thus, as used herein, the terms broadcast and multicast may be used interchangeably to refer to a single packet received by multiple destinations. In some cases, this description refers only to media that is transmitted or transmitted without specifying whether it is broadcast, multicast, or unicast. In such cases, it is meant that any one of these methods can be used for transmission or transmission of media.

  As used herein, the terms “message” and “packet” are often used and may be used interchangeably. A packet is data that is configured to be transmitted or received over an Internet Protocol (“IP”) network. The packet may or may not be the same as the “IP packet”. The term “message” as used herein refers to logical information contained in such a packet.

  As used herein, “segment” may also be used to refer to a data set. The data set is a data byte set. The data can be any type of data, including media data, control data, or information data. In this description, the terms data and packet may be used interchangeably depending on the context. A “packet” may refer to a data set, and data refers to data in general.

  It should be understood that many alternative embodiments are disclosed herein and these embodiments are presented by way of example only. The described embodiments are in no way intended to be limiting. In general, embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the invention, and other embodiments may be utilized without departing from the scope disclosed. It should be understood that structural, logical, software, electrical, and other changes may be made.

  According to one embodiment of the present invention, in order to broadcast media over a Wi-Fi network, it must first be recognized that broadcast or multicast media is not received uniformly at all destinations. . Depending on the destination, it refers to a data set transmitted or received over a multicast “packet” (Internet Protocol (“IP”) network. A packet may or may not be the same as an “IP packet”. The term “message”, as used herein, refers to logical information contained in such a packet and may be synonymously referred to as “message” or “segment”) Some do not receive.

  IP networks were originally designed to operate over wired networks. By design, packet communication on these networks was “best effort”. This means that any packet sent on the network may not be received by the intended destination. This is most often due to a collision, where another device starts communicating at the same instant as the target device, thereby causing a collision. Another reason for loss is that the device in the network path with the router simply drops the packet, for example due to lack of buffer space. Another reason for the loss may simply be that the wired line is noisy and the packet transmission is corrupted, but this is rare in the wired case compared to the wireless case.

  In all these wired situations, if a transmission (eg, a multicast message) is received by one device on a subnet or wire, all other devices on the same wire or subset may receive the transmission correctly. Generally true. This is because in the case of wired, the noise or interference situation of the device on one part of the wire is not much different from the noise situation in another part of the wire. If wired devices are connected via switches rather than hubs, the same is true and the amount of noise or interference is minimal.

In Wi-Fi, the difference in receiving Wi-Fi traffic at each Wi-Fi device in the subset is large. It is therefore necessary to explain this.
According to one embodiment of the present invention, all devices (ie, source device 104 and various destination devices 106 and 106 ′) (see FIGS. 1 and 3) may be networked together on local network 120. . This is typically an IP network and can be a wired or wireless (eg, Wi-Fi) network. This local network may further connect to the Internet.

  In many media systems, it is desirable to send media to multiple playback devices, causing each playback device to render the media in phase. For example, it is desirable to send the left channel of stereo audio media to the left audio playback device, send the right channel of stereo media to the right audio playback device, and have both devices play the media accurately in phase.

  FIG. 1 shows such a system having two digital audio subsystems, a first subsystem 106 and a second subsystem 106 '. The first subsystem 106 and the second subsystem 106 'each render the same or related audio data. For example, each audio subsystem may be receiving media from a media file 222 on the network. Each system (ie, audio system 106 and 106 ') includes a central processing unit (CPU) and a digital / analog converter. Each system 106, 106 'renders audio output 112 and 112' via a DAC. In this case, the rendered output waves 220 and 220 'need to be in-phase with the audio, as shown in FIG. In order to be in-phase with the audio, the rendering waves 220 and 220 'need to have the correct sound (frequency spectrum) reproduced at the correct time (with phase offset). For example, if the media includes a left channel that rings the bell for 1 minute on that media, the right channel plays a drumbeat at the same time, device 106 plays the left channel, and device 106 'plays the right channel, the media In one minute of playback, if device 106 renders a bell sound and device 106 ′ renders a drum beat, the devices are in phase.

  If the rendering waves 220 and 220 'are not in phase, the user may hear a beat frequency associated with the frequency difference between the two waves 220 and 220'. Furthermore, over time, the difference between the two audio outputs may continue to increase. For example, in the example used previously, if the second subsystem 106 ′ is shifted by +50 ppm and the three-minute song ends with a drum beat, the second subsystem 106 ′ will replace the last drum beat with the first subsystem. It will play back 9 milliseconds after the system 106. After 10 such songs, the difference is 90 milliseconds, which is very noticeable.

  Thus, if multiple audio devices 106 and 106 'are playing the same media, it is necessary to adjust the rendering clock of each system to ensure that it has the same phase offset and frequency.

  FIG. 2 shows in greater detail an exemplary digital system 106 for playing audio. The CPU 114 sends audio samples to the DAC 108, which is rendered by the DAC 108 converting the digital samples into the analog signal 112. The rate at which the DAC converts digital samples to analog signal levels is referred to as the output sample rate, which typically matches the rate at which audio samples being sent from the CPU 114 were sampled. The exact output sample rate depends on the DAC clock subsystem that is driving the DAC. The accuracy of this clock subsystem affects the accuracy of the output sample rate and thus the accuracy of audio rendering by this system. This clock subsystem may use a clock emanating from the CPU 114 or from an external source 105.

  A CPU clock crystal 102 of a typical CPU 114 is a base of a CPU clock generated internally by the CPU 114. This CPU clock is then used for all CPU timing activities. Some CPUs generate many different clock signals inside the CPU 114 based on the CPU clock. The CPU 114 may have many clock peripherals and clock registers 136 based on the CPU clock that can be used for various timing related activities. For example, the clock peripheral can be configured to periodically interrupt the CPU every 100 milliseconds. Since this clock is originally based on the CPU crystal 102, the accuracy of this period depends on the accuracy of the CPU crystal 102. Normally, a program executed by the CPU 114 can read the clock register 136 indicating the clock count number counted by the CPU 114 after the power is turned on and reset by the CPU 114. These clock counts are incremented at a rate associated with the CPU crystal 102. This clock may be used to drive the DAC output sample rate and the rate at which audio samples are provided to the DAC.

  In some embodiments, the system 106 sends an audio clock signal 116 to the CPU 114 and the DAC 108 to manage the timing of playback of the audio signal sent from the CPU 114 to the DAC 108 and to send such audio via the DAC 108. In order to generate an audio output 112 from the signal, a separate audio clock 105 having its own independent crystal 103 can be further provided.

Alternate embodiments may use various combinations of external and internal clock sources to drive the DAC sample rate.
In a media system as shown in FIG. 1 where media needs to be sent to multiple playback devices and each playback device needs to render the media in phase, the media needs to be marked with some time reference information, Needs to play this media according to this time information. This means that each device needs to use the same time reference in order to time the rendering activity.

FIG. 3 illustrates a common clock-based approach for time reference as described above. As shown, source device 104 and one or more destination devices 106, 106 ′ may be connected by network 120. The source device 106 is a “common” source clock and has a local clock C135, and each destination device 106, 106 ′ is also shown as D1 — 136 for the first device 106 and Dx136 ′ for the xth device 106 ′. Including a local clock. The source clock 135 is called a common source clock because its clock value is transmitted to all destinations and used as a standard clock common to all devices. The source device 104 periodically sends a “clock” message to each destination device 106, 106 ′ that includes the latest common source clock value C that was read before sending the message. A clock message transmitted to the first device with period n having a clock value is _denoted as C1 _n 250, and a clock message transmitted to the xth device with period n having a clock value is _denoted as Cx _n 252. Is done. Upon receiving the clock message, each destination device 106, 106 ′ records the latest received clock value C1 _n or Cx _n as 254 and 254 ′. Each destination 106, 106 'receives a clock message, with C1 _n or Cx _n common clock value received, the local clock value D1 _n or Dx _n corresponding also recorded. There is a delay from the moment a clock message is received to the time that the local clock associated with it is recorded. This “recording” delay may vary from clock message to clock message and is defined as recording jitter R. This change in recording delay may be due to the network adapter used for messaging and the change in software response time at the destination until packet reception.

FIG. 4 shows a time series of message transactions. The source device 104 transmits an n th clock message having a clock value C 1 _n 250 to the first device 106 and a clock value C x _n 252 to the x th device 106 ′. The same applies to the (n + 1) th and subsequent messages. Each clock message takes a specific amount of time (transport time) from the source 104 to the destinations 106-106 ′.

n-th message is time to reach the device of the x is Tx _n. The transport time to each destination may vary due to network packet transmission problems. This difference changes from device to device over time. These changes in Tx are called transport jitter Jx at device x.

And information in each message Cx _n, based on the local clock value Dx _n corresponding recorded, 'as is consistent with the common source clock C135, local clock Dx136,136' each destination x106,106 adjusted The information to do can be calculated. The destination 106, 106 'may adjust the local clock D1_136, Dx136' phase (count) Px to match the best estimate of the current phase (count) Pc of the source clock C135. Similarly, destinations 106, 106 'may adjust local clocks D1_136, Dx136' frequency Fx to match the current frequency Fc best estimate of common source clock C135.

An exemplary calculation for performing these steps may be:
Cx _n : n-th common source clock value transmitted to device X TA _x : average estimated transit time to device X RA _x : average estimated recording time at device X Jx: maximum from average TA _x transit time difference · Rx: maximum recording time deviation · Tx n from the _mean: device transport time of the n-th message sent to the X · Dx _n: the device x in the _n-th clock message clock value · Fx _n : New local clock frequency of device x Px _n : New local clock phase / count of device x Cx ′ _n = Cx _n + Tx _n + Rx _n : Estimated common clock n
Cx′n _{+ 1} = Cxn _{+ 1} + Txn _{+ 1} + Rxn _{+ 1} : Estimated common clock n + 1
Tx _n = Ta _x +/− Jx: transport time n
Tx _{n + 1} = Ta _x +/− Jx: transport time n + 1
Rx _n = Ra _x +/− Rx: recording time n
Rx _{n + 1} = Ra _x +/− Rx: recording time n + 1
_{· Ax n = (Dx n +} 1 -Dx n) - (Cx 'n + 1 -Cx' n): = Frequency adjustment value _{· Ax n (Dx n + 1} -Dx n) - (Cx n + 1 -Cx n) +/- 2 * Jx + / -2 * Rx
_{· Fx n = Fx n-1} -Ax n: new frequency _{· Fx n = Fx n-1} + (Cx n + 1 -Cx n) - (Dx n + 1 -Dx n) +/- 2 * Jx +/- 2 * Rx
Px _n = Cx ′ _n = Cx _n + Ta _x +/− Jx +/− Rx: phase offset.

The estimation of the transport time Tx is extremely important for this system. One common way to do this is to send a message from the source 104 to the destination Dx 106, 106 'and have the destination Dx 106, 106' send an acknowledgment to the message, so that the round trip from the source to the destination. Is to measure time. The total time from sending the output message to receiving the input acknowledgment message is the round trip time. Halving this time provides an estimate of the output transit time. This is done many times, when to be filtered, the filtering result Ta _x, can be used as the average transit time from the source to the destination Dx. The actual transit time Tx _n of n-th message _is a = Ta x +/- Jx _n, where, Jx _n is jitter, which is the deviation from the average of the n-th messages. Jx _n can be estimated using the standard deviation of the measured Tx time.

  In this scheme, each destination device 106, 106 ′ uses the common source clock 135 information to set the local clock 136, 136 ′ value and clock rate to match the clock value and clock rate of the common source clock 135. adjust. That is, the common source clock 135 is a “global” clock used in the system, and all destination clocks are matched to this. The media is then rendered at each device using local clocks 136, 136 '.

  The accuracy of this system can be limited because transport jitter can affect both frequency and phase calculations. This can be mitigated somewhat by filtering the parameters used in the calculation over a period of time, as described above. Since media rendering is performed by the destination devices 106, 106 'and the clocks associated with these devices are used as the basis for media rendering, this can affect the phase accuracy of media rendering due to transport time jitter. Means receiving. If the transport time jitter Jx is +/− 1 to 2 milliseconds, as can be a Wi-Fi network, the maximum phase accuracy that can be achieved is limited to this range.

  A further problem with this system is that the virtual clock time only increments with the interval of the received message. This can be addressed by interpolating the time during these message intervals using a local clock. The problem is that each device's local clock may be slightly different from the other local clocks, so the interpolated virtual clock may jump forward or move backward with each adjustment made as a result of the received message. That is. This can sometimes cause media rendering to jump forward or backward by a small amount.

  A further problem is that in many systems, a common global clock is not available, or even if it is available, it is not accurate enough for synchronization purposes. For example, the local clock of the source device may be accessed through a programming API, but the calling program is typically running on a multitasking operating system that may have a delay during the API call or may have a task swap. Therefore, the local clock time is inaccurate by that time. For example, the calling program can call the local clock API, call the hardware clock value, and then the current program thread is swapped before the function returns, and the next program thread only takes a time slice. return. This means that the returned hardware clock value is now delayed by the time slice period. Most common operating systems can have a time slice period of up to 25 milliseconds to 100 milliseconds, so the hardware clock value can be greatly delayed by milliseconds. This is true for many general purpose source devices such as smartphones or notebooks. Conversely, the destination device used in the present invention is a dedicated device, so the software operates at a lower level with much better local clock read accuracy.

Thus, various embodiments of the present invention may be designed with the goal of not using or requiring the ability to accurately read the local clock of the source device.
The disclosed arrangement renders media using each device's local rendering clock only. The destination device local clock referred to here serves as a base for the rate (sample rate) at which media samples are fed to a DAC (digital / analog converter) to convert (render) the digital media samples into an audio or video signal. This is the clock used. This local clock can be a hardware clock or a software object such as a virtual clock, interrupt service routine, etc. that can be supported by a hardware clock that supplies samples to the DAC for conversion to an audio or video signal. Sometimes there are.

  These destination clocks are continuous and smooth in increments, and thus smooth in how the media is rendered. However, it may be necessary to adjust for potential differences in the device clock. One embodiment addresses this adjustment by dividing it into two problems. The first problem is detecting the frequency difference. The second problem is adjusting for the difference. In order to detect the difference, one embodiment uses a common event-based approach rather than a common clock.

FIG. 5 is a diagram of the basic elements of an example of a common event-based system. The system consists of a source device 104 and one or more destination devices 106, 106 ′ connected by a network 120. The source device has two elements: a common event source E350 and a synchronization calculation manager (synchronization manager) S352. The common event source E350 generates an event message received on the network at the same time on all the destination devices at the same time. An example of a common event message E _n 358 is a multicast or broadcast message that is received simultaneously by all destination devices 106, 106 ′. A multicast or broadcast message is sent to the communication channel by a source device, which is one device, and is received by all destination devices. Messages are actually communicated over the communication channel, usually by electrical means in which electrical or RF signals are carried at a speed close to the speed of light. Thus, for example, the last bit of the message is received at all destination devices at a time that is the difference of the distance multiplied by the speed of light, which is almost instantaneous. Another example of a common event message is a beacon message in a network using, for example, the 802.11 (Wi-Fi) network protocol.

  In one embodiment, any event sequence received by all destination devices, each uniquely identifiable, and each received simultaneously by all destination devices, is used as a common event as described herein. Can do. In one embodiment, synchronization manager S352 is the source of such common events.

Synchronous calculation manager S352 may reside on any computing device on network 120, including destination devices 106, 106 '.
Each destination device x (106, 106 ') has a local clock Dx (136, 136') that is used as the basis for media rendering by the device.

FIG. 6 shows the time series of activities and messages in the system. Event source E350 generates an event message, and the _nth event generated at time T _n is noted as E _n 358. Event E _n 358 is identified by event number n, which is unique to each event. The event number n may simply be a sequentially incremented number generated by the sync manager S352 since the system was turned on, or it may be a timestamp value. In the latter case, this is only used as a unique number and the time value is not used as a time indicator. Any other form of generating a unique event number may be used. Upon receiving each event message _{E n,} each destination x records the current value of the local clock Dx136 as Dx _n. As with the common clock, there is a delay from the moment the event message is received until the associated local clock is recorded. This “recording” delay may vary from event message to message message and is defined as recording jitter R.

When each destination device x receives the event message n, each destination device x has a unique event acknowledgment message AK (Dx _n ) 354, 354 ′ having a value Dx _n stored when the event n is received. Event number n is sent to the synchronous calculation manager S352. Synchronization computation manager, Dx receives all events acknowledgment message _AK (Dx n) 354,354 'having _n values, perform calculations, then the clock adjustment message AJ (Ax _n with adjustment information Ax _n ) 356 and 356 ′ are returned to each device x.

The calculations performed are shown below.
Ex _n : n-th common event message transmitted to device X Tx _n : transport time of n-th event transmitted to device X Dx _n : clock value of device x in n-th event message Fx _n : New local clock frequency of device x Px _n : New local clock phase / count of device x Qx0 = Dx ₀ −D1 ₀ +/− 2 * Rx: Reference (initial) clock phase for device 1 Qx _n = Dx _n −D1 _n +/− 2 * Rx: nth clock phase Qf _n = F (Qx _n ): nth clock phase filtering function ex _n = (Qf _n −Qf ₀ ) + / -4 * Rx: Phase error (change)
_{· Kx n = T * ex n} : error * transfer function _{· Fx n = Fx 0 + kx} n +/- T * 4 * Rx: = new frequency _{· PFx n ((Fx n /} Fx n-1) -1) * 100: new frequency percent _{· Px n = Dx n -D1 n} +/- 2 * Rx: = phase offset _{· Ax n (PFx n, Px} n): adjustment information.

  In this common event system, the clock of one device, that is, the device 1 that is the first destination device, is used as the phase reference clock. Using these calculations, the synchronization module calculates a phase offset (local clock value) and frequency ratio (local clock rate) adjustment for each destination device x, sends this adjustment information to the destination x, and each destination device x It is requested to make an adjustment so that the local clock at the time coincides with the local clock of the destination device 1.

  For example, if the local clock of the first destination device is incremented by 100 counts and the destination device x is incremented by 101 counts between E1 being common event 1 and E2 being event 2 ignoring jitter, the device x is running faster than the destination device 1, and its clock rate needs to be decremented from 101 by one count. Further, when the local clock of the first destination device is 2500 at E1 which is event 1, and the local clock of destination x is 3000 at E1 which is event 1, the local clock of destination x is adjusted to 2500. Thus, it is necessary to match the phase of the first destination device.

  While this embodiment defines the frequency adjustment as a percentage of frequency, other embodiments may define the frequency adjustment in other forms including an absolute new frequency or the number of samples to be added or reduced. All these forms are referred to as frequency adjustment for purposes of this specification. Also, the use of the term frequency refers to the clock rate, and relates to both the audio sample crystal rate and the sample output rate, and may refer to both. In general, the sample output rate is a multiple of the audio sample crystal rate.

  Phase adjustment refers to adjusting the clock value, which can be the clock value for a clock that increments with the audio crystal rate, the sample output clock output rate, or some other counter value associated with the sample output clock.

  The main thing in this common event system is that acknowledgment messages 354, 354 ′ and coordination messages 356, 356 ′ may be sent over the IP network and therefore do not arrive at the receiver at a predictable time. That is. Each acknowledgment message includes a destination device number.

  Thus, the synchronization module 352 cannot group acknowledgment messages 354, 354 'to the same event by the time of arrival. For example, some acknowledgment messages for event 1 may arrive at synchronization module 352 after the arrival message for event 2 arrives. Thus, acknowledgment messages are grouped into event acknowledgment message sets based on unique event numbers.

  To this end, the synchronization module 352 can associate all acknowledgment messages sent from a set of destination devices for each unique event. Thus, the synchronization module marks each event message with a number n that is unique over the life of the system. This can simply be an event number that increments with each event. All destination devices include this unique event number in the acknowledgment messages AK354, 354 '. Whenever an acknowledgment message AK354, 354 'is received at the synchronization module 352, the acknowledgment message is a set of acknowledgment messages {AK (Dab), ..., AK (D1n), ..., AK (Dxn ), AK (D1n + 1),. This set is then searched to find a subset {AK (D1n),..., AK (Dxn)} of all acknowledgment messages for a particular event number n. Next, the phase change Qxn for device 1 is calculated using values from this subset, as shown in the calculation above. The values in this subset can be further filtered before use.

  In an alternative embodiment, the event message may be provided from an external source, such as using a Wi-Fi system beacon message. In this case, the time value in the beacon message can be used as a unique event number. The time value is not used as a time value, but only as a unique event number.

  FIG. 7 identifies some important and significant differences between the common clock method and the common event method described herein. The upper plot 450 shows the net effect of the common clock method. In this way, the common clock, which is the clock of source 104, is a “global” clock 454 used by the system. This clock is used by each destination D1_106-Dx106 'that estimates the common source clock time to adjust the local clock. As described above, the accuracy of this setting is affected by the transport latency L1_458 and Lx460. Since each destination device D1_106 and Dx 106 'independently estimate a common clock time, the phase difference P462 of these clocks will be highly dependent on their latency. Since transport latency has a message-by-message difference called transport jitter, this means that the clock settings of the destination device clocks (D1_136 and Dx136 ') are accurate to receive transport jitter. In particular, this means that the phase difference, which is the clock difference, receives the maximum message transport jitter Jx. It is desirable to maintain media rendering phase accuracy on both devices D1_106 and Dx 106 '. The best proposal is to prevent transport jitter from affecting phase accuracy.

  Overall, the common clock method aligns all clocks to the source common clock, the adjustment calculation is done independently at each destination, and the accuracy depends on the recording accuracy Rx and the transport jitter Jx.

  The bottom plot 452 illustrates a common event approach used in various embodiments of the present invention. In this approach, the clock D1 of the first destination device 106 becomes the reference “global” clock 456 used in the system. The additional destination device 106 'adjusts its own clock Dx to the global clock of D1_106. These clock phase differences P466 are not affected by transport latency, and are only related by how accurately common events are recorded upon reception. The transport latency L1_464 is only a factor in projecting the global clock 456 to the original source 104. In one embodiment of the present invention, the source 104 adjusts the clock to match the destination clock D1_106, if necessary. In this embodiment, no source clock or such adjustment is required.

  Overall, the common event approach matches all clocks to one of the destination clocks, and the accuracy exists only in the recording accuracy Rx. The transport jitter Jx is affected by all devices and processes involved in message transport, and therefore tends to be many times larger than the recording accuracy Rx (while the recording accuracy depends only on the destination hardware and software, Local to the destination hardware and software). Transport jitter can be in the order of milliseconds, while recording jitter is typically less than a few hundred microseconds. Therefore, the common event technique is a better technique that relies only on recording accuracy and provides complex and high level phase synchronization accuracy with less effort.

FIG. 8 shows an exaggerated view of the overall effect of the common event technique. The upper waveform 220 is the output of the DAC from the device D1_106, and the lower waveform 220 ′ is the output of the DAC from the device Dx 106 ′. The vertical lines in both plots show the clock ticks (DAC sample rate output clock) for each rendering clock. The vertical line also plots sample data rendered at each clock step. In FIG. 8, the lower waveform Dx has a much slower clock step than the upper waveform, so audio samples are rendered over a longer period of time in device D1. The figure also shows two common event _{E n} and _{E n + 1.} As can be seen, the clock increment incremented by the lower waveform from D 1 is less than the clock increment incremented by the upper waveform 220. Phase at time E _n is the clock count difference between the two devices at time E _n. The phase at time En _{+ 1} is also this difference. Clock rate for different, it can be seen that will differ from the phase of the phase of the event E _n is the event E _{n + 1.} The common event calculation and adjustment described above corrects this and increases the clock rate of the lower waveform 220 ′ until the clock rate matches the clock rate of the upper waveform 220. When the clock rates match, the two waveforms are similarly rendered and rendered in phase.

  The local clock rate can be adjusted in several ways. A common approach is literally to adjust the clock rate of the rendering clock. This typically requires some kind of hardware subsystem that can generate an adjustable clock. An alternative approach is to leave the rendering clock and adjust the media data instead. In this method, the sample rate is converted, the sample rate is converted, and the clock rate difference is compensated. When a 1 kHz audio signal is sampled at 44.1 kHz and then rendered with a 44.1 kHz clock, the rendered audio signal is an accurate representation of the original audio signal. Instead, if the rendering clock is 44 kHz, the 1 kHz audio signal is rendered with a slightly lower tone and is not an accurate representation of the original audio signal. If the audio data is now sample rate converted from 44.1 kHz to 44 kHz and rendered using a 44 kHz rendering clock, the rendered 1 kHz audio signal is again an accurate representation of the original audio signal. .

  The preferred embodiment of the present invention uses the latter sample rate conversion approach. As frequency adjustment messages are received at each destination, instead of adjusting the rendering clock, each destination performs a sample rate conversion on the data stream to increase or decrease the sample rate of the data. This has the same net effect as adjusting the rendering clock.

  The frequency difference is usually related to the accuracy of the clock crystal specified in PPM parts per million. A 100 PPM crystal is <0.01%, ie, an accuracy of 1 out of 10,000. Therefore, the sample rate conversion required here is usually a small adjustment of one sample out of 10,000 samples. Thus, adjustment can be accomplished by occasionally adding or dropping a single sample.

FIG. 9 shows the algorithm described above in the form of a control system. This shows a number of devices marked D1_106 to Dx106 ′. Each destination device, except for the first device D1_106, implements a software phase lock loop algorithm to lock the phase to the clock phase of device D1_106. Associated with each destination device Dx 106 ′ is a phase compensator 650 ′ that compares the phase of the output clock Dx _n 654 ′ with the reference clock D1_652. Next, the phase difference from the reference clock e _x-1 656 is scaled to produce a clock frequency adjustment Ax 700 ′. This adjustment then causes the clock generator 702 'at destination 106' to adjust the clock frequency. In this embodiment of the system, phase detector 650 ′ and gain / scaling 658 ′ are present in synchronization calculation manager 352. Clock generator 702 'resides at destination device 106'.

Additional considerations
The section headings and titles of this patent application provided in this patent application are for convenience only and should in no way be construed as limiting the present disclosure.

  Devices that communicate with each other need not communicate sequentially with each other, unless explicitly stated otherwise. Further, devices that communicate with each other may communicate directly or indirectly through one or more logical or physical intermediates.

  The description of an embodiment having several components that communicate with each other does not imply that all such components are required. Conversely, various optional components may be used to describe one or more of the various possible embodiments of the present invention and to more fully describe one or more aspects of the present invention. Can be described. Similarly, process steps, method steps, algorithms, etc. may be described in a sequential order, but such processes, methods, and algorithms are generally in an alternate order, unless stated to the contrary. It can also be configured to function. In other words, any order or sequence of steps that may be described in this patent application does not in itself indicate the requirement that the steps be performed in that order. The process steps described may actually be performed in any order. Furthermore, some steps may be performed simultaneously, even though they are not described or implied as occurring simultaneously (eg, because one step is described after another). Further, the description of the process by way of illustration in the drawings does not imply that the process shown excludes other variations and modifications, and the process shown or any step thereof is required for one or more of the invention. It does not imply that it is, nor does it imply that the process shown is preferred. Also, although steps are generally described once for each embodiment, this must be done only once, or can be done only once each time a process, method, or algorithm is implemented or executed. It doesn't mean that. Some steps may be omitted in some embodiments or in some cases, or some steps may be performed more than once in a given embodiment or case.

  Where a single device or article is described, it can be readily appreciated that more than one device or article can be used in place of a single device or article. Similarly, where more than one device or article is described, it can be readily appreciated that a single device or article can be used in place of more than one device or article.

  Device functions or features may alternatively be implemented by one or more other devices not explicitly described as having such functions or features. Thus, one or more other embodiments of the present invention need not include the device itself.

  The techniques and mechanisms described or referenced herein may be described in the singular for the sake of clarity. However, it should be noted that certain embodiments include multiple iterations of the technique or multiple iterations of the mechanism unless otherwise noted. Process descriptions or blocks in the figures should be understood as representing modules, segments, or portions of code that contain one or more executable instructions that perform a particular logical function or step in the process. As will be appreciated by those skilled in the art, alternative embodiments that may perform functions other than in the order shown or discussed, including substantially simultaneous or reverse order, depending on the function involved, are examples of implementations of the invention. Included within the scope of the form.

  In general, the techniques disclosed herein may be implemented in hardware or a combination of software and hardware. For example, it may be implemented in an operating system kernel, a separate user process, a library package bound to a network application, in particular a built machine, an application specific integrated circuit (ASIC), or a network interface card.

  A software / hardware hybrid implementation of at least some of the embodiments disclosed herein is a programmable network resident machine (intermittently) that is selectively activated or reconfigured by a computer program stored in memory. Can be understood as including network-aware machines connected to the network). Such a network device may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols. Some general structures of these machines may be disclosed herein to illustrate one or more exemplary means that can implement a given functional unit. According to certain embodiments, at least some of the features or functions of the various embodiments disclosed herein can be, for example, an end-user computer system, a client computer, a network server or other server system, a mobile computing device One or more of (eg, tablet computing device, mobile phone, smartphone, laptop, etc.), consumer electronic device, music player, any other suitable electronic device, router, switch, etc., or any combination thereof One or more general purpose computers associated with a network of In at least some embodiments, at least some of the features or functions of the various embodiments disclosed herein may include one or more virtualized computing environments (eg, a network computing cloud, one or more Or a virtual machine hosted on a physical computing machine). Further, in some embodiments, one or more aspects or all aspects of the present invention optionally include a specifically programmed chip (eg, application specific integrated circuit, ie, ASIC, or erasable programmable). Read-only memory (or EPROM) or any other hardware-only technique known in the art.

[Description of Exemplary Hardware Embodiment]
With reference now to FIG. 10, a block diagram illustrating an exemplary computing device 1200 suitable for implementing at least some of the features or functions disclosed herein is shown. The computing device 1200 can be, for example, any one of the computing machines listed in the previous paragraph, or in fact, software or hardware-based instructions according to one or more programs stored in memory. It can be any other electronic device that can be implemented. The computing device 1200 can communicate over a communication network such as a wide area network, a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, whether wired or wireless. It may be configured to communicate with multiple other computing devices such as clients or servers using known protocols.

  In one embodiment, the computing device 1200 includes one or more central processing units (CPUs) 1202, one or more interfaces 1210, and one or more buses 1206 (such as a peripheral component interconnect (PCI) bus). )including. When operating under the control of appropriate software or firmware, the CPU 1202 may be responsible for performing specific functions associated with functions of a specifically configured computing device or machine. For example, in at least one embodiment, computing device 1200 may be configured or designed to function as a server system that utilizes CPU 1202, local memory 1201 and / or remote memory 1220, and interface 1210. In at least one embodiment, the CPU 1202 is one of different types of functions and / or operations under the control of a software module or component that may include, for example, an operating system and any suitable application software, drivers, etc. Or more than one may be executed.

  The CPU 1202 may include one or more processors 1203, such as, for example, a processor from one of Intel, ARM, Qualcomm, and AMD family of microprocessors. In some embodiments, the processor 1203 controls the operation of the computing device 1200, such as an application specific integrated circuit (ASIC), an electrically erasable programmable read only memory (EEPROM), a field programmable gate array (FPGA). And specifically designed hardware. In certain embodiments, local memory 1201 (eg, non-volatile random access memory (RAM) and / or read only memory (ROM) including one or more levels of cache memory) is part of CPU 1202. May be. However, there are many different ways in which memory can be coupled to system 1200. The memory 1201 may be used for various purposes such as caching and / or storing data, programming instructions, etc., for example.

  As used herein, the term “processor” is not limited to simply an integrated circuit referred to in the art as a processor, mobile processor, or microprocessor, but includes a microcontroller, microcomputer, programmable logic controller, and specific application. Directed integrated circuit, and any other programmable circuit.

  In one embodiment, interface 1210 is provided as a network interface card (NIC). In general, the NIC controls the transmission and reception of data packets over the computer network, and other types of interfaces 1210 may support other peripherals used in conjunction with the computing device 1200, for example. Among the interfaces that can be provided, there are an Ethernet interface, a frame relay interface, a cable interface, a DSL interface, a token ring interface, a graphics interface, and the like. Further, for example, universal serial bus (USB), serial, Ethernet, Fire wire (registered trademark), PCI, parallel, radio frequency (RF), Bluetooth (registered trademark), near field communication (for example, , 802.11 (WiFi), frame relay, TCP / IP, ISDN, high-speed Ethernet interface, Gigabit Ethernet interface, asynchronous transfer mode (ATM) interface, high-speed serial interface (HSSI) interface, POS Various types of interfaces may be provided, such as interfaces, fiber data distribution interfaces (FDDI). In general, such an interface 1210 may include an appropriate port to communicate with the appropriate media. In some cases, an independent processor and, in some cases, volatile and / or non-volatile memory (eg, RAM) may be included.

  The system shown in FIG. 10 illustrates one particular architecture of computing device 1200 that implements one or more of the inventions described herein, but in no way is the features and techniques described herein. Is not the only device architecture that can implement at least part of For example, an architecture with one or any number of processors 1203 may be used, and such processors 1203 may reside on a single device or distributed across any number of devices. In one embodiment, a single processor 1203 handles communication and computational routing, but in other embodiments, a separate dedicated communication processor may be provided. In various embodiments, different types of features or functions are implemented in a system according to the present invention including a client device (such as a tablet device or a smartphone that executes client software) and a server system (such as a server system described in more detail below). Can do.

  Regardless of the configuration of the network device, the system of the present invention may be used for general network operation data, program instructions, or other information related to the functionality of the embodiments described herein (or any combination of the above). One or more memories or memory modules (eg, remote memory block 1220 and local memory 1201 etc.) configured to store The program instructions may, for example, control or include the execution of an operating system and / or one or more applications. Memory 1220 or memory 1201, 1220 is configured to store data structure, configuration data, encrypted data, past system operational information, or any other specific or general purpose non-program information described herein. You can also.

  Since such information and program instructions may be utilized to implement one or more systems or methods described herein, at least some network device embodiments are described herein, for example. Non-transitory machine readable storage media that may be configured or designed to store program instructions, state information for performing various operations performed. Examples of such non-transitory machine-readable storage media include magnetic media such as hard disks, floppy disks (registered trademark) and magnetic tapes, optical media such as CD-ROM disks, magneto-optical media such as optical disks, and reading Hardware devices that are specifically configured to store and execute program instructions, such as, but not limited to, dedicated memory devices (ROM), flash memory, solid state drives, memristor memory, random access memory (RAM), etc. Not. Examples of program instructions include object code that can be generated by a compiler, machine code that can be generated by an assembler or linker, for example, generated by a Java (registered trademark) compiler, and executed using a Java virtual machine or the like. A computer using bytecode, or an interpreter, such as a script written in Python, Perl, Ruby, Groovy, or any other scripting language File containing higher level code that can be executed by.

  In some embodiments, the system according to the present invention may be implemented in a standalone computing system. Referring now to FIG. 11, a block diagram illustrating an exemplary architecture of one or more embodiments or components thereof in a stand-alone computing system is shown. The computing device 1300 includes a processor 1310, which may execute software that implements one or more functions or applications of an embodiment of the invention, such as, for example, a client application 1330. The processor 1310 may be, for example, some versions of Microsoft Windows® operating system, Apple's MacOS® / X or iOS operating system, or Linux® operating system. Calculation instructions may be executed under the control of an operating system 1320, such as Google's Android ™ operating system. In many cases, one or more shared services 1325 may be operable in system 1300 and may be useful in providing common services to client applications 1330. Service 1325 can be, for example, a Windows® service, a user space common service in a Linux environment, or any other type of common service architecture used in conjunction with operating system 1310. Input device 1370 may be of any type suitable for receiving user input, including, for example, a keyboard, touch screen, microphone (eg, for voice input), mouse, touch pad, trackball, or any combination thereof. possible. Output device 1360 may be of any type suitable for providing output to one or more users, whether remote from system 1300 or local, eg, visual output One or more screens, speakers, printers, or any combination thereof. Memory 1340 may be random access memory used by processor 1310 and having any structure and architecture known in the art to execute software, for example. Storage device 1350 may be any magnetic, optical, mechanical, memristor, or electrical storage device that stores data in digital form. Examples of the storage device 1350 include a flash memory, a magnetic hard disk, a CD-ROM, and the like.

  In some embodiments, the system of the present invention may be implemented in a distributed computing network having any number of clients and / or servers. Referring now to FIG. 12, a block diagram illustrating an exemplary architecture for implementing at least a portion of a system according to an embodiment of the present invention in a distributed computing network is shown. According to this embodiment, any number of clients 1430 may be provided. Each client 1430 may execute software that implements the client-side portion of the present invention, and the client may comprise a system 1300 as shown in FIG. In addition, any number of servers 1420 may be provided to process requests received from one or more clients 1430. Client 1430 and server 1420 may communicate with each other via one or more electronic networks 1410, which in various embodiments may include the Internet, wide area networks, mobile telephone networks, wireless networks (WiFi (registration)). Trademark), WiMax®, etc.), or a local area network (or indeed any network topology known in the art), the present invention prefers any network topology over any other network topology Can be any). The network 1410 may be implemented using any known network protocol including, for example, a wired protocol and / or a wireless protocol.

  In addition, in some embodiments, the server 1420 may call an external service 1470 to obtain additional information, if necessary, or reference additional data regarding a particular call. Communication with external service 1470 may be performed via one or more networks 1410, for example. In various embodiments, the external service 1470 may include a web-enabled service or function that is associated with or installed on the hardware device itself. For example, in embodiments where the client application 1330 is implemented on a smartphone or other electronic device, the client application 1330 is deployed on one or more of a server system 1420 in the cloud or a specific company or user premises. Information stored in the external service 1470 may be obtained.

  In some embodiments of the present invention, client 1430 or server 1420 (or both) utilize one or more dedicated servers or applications that can be deployed locally or remotely over one or more networks 1410. obtain. For example, one or more databases 1440 may be used or referenced by one or more embodiments of the present invention. Those skilled in the art will appreciate that the database 1440 can be configured in a wide variety of architectures and using a wide variety of data access and processing means. For example, in various embodiments, one or more databases 1440 may include a relational database system that uses a structured query language (SQL), while other databases 1440 are “NoSQL” ( For example, it may include alternative data storage technologies such as those referred to as Hadoop®, Cassandra®, Google BigTable®, etc. In some embodiments, a modified database architecture such as a column-oriented database, an in-memory database, a cluster database, a distributed database, or a flat file data repository may be used with the present invention. Those skilled in the art will understand that any combination of known or future database technologies may be used as appropriate unless specific database technologies or components of a particular configuration are specified herein for a particular embodiment. . Further, it is understood that the term “database” as used herein may refer to a physical database machine, a cluster of machines functioning as a single database system, or a logical database within an overall database management system. I want. Unless a specific meaning is specified for a given use of the term “database”, it is to be construed to mean any of these meanings of this term, and all of these meanings It is understood as the ordinary meaning of the term “database” by a vendor.

  Similarly, most embodiments of the present invention may utilize one or more security systems 1460 and configuration systems 1450. Security and configuration management is a general information technology (IT) and web function, with some amount of each generally associated with any IT or web system. Any configuration or security subsystem currently known in the art or known in the future, without limitation, unless a particular security 1460 or configuration system 1450 or technique is specifically required by the description of any particular embodiment Can be used in conjunction with embodiments of the present invention.

  In various embodiments, the functionality to implement the system or method of the present invention may be distributed over any number of client and / or server components. For example, various software modules may be implemented that perform various functions in connection with the present invention, and such modules may be variously implemented and executed on server and / or client components.

  Accordingly, the various embodiments disclosed herein allow transmission of media from a source to multiple media devices such as TVs and speakers in the same audiovisual space. According to embodiments, this may be done via a wireless network such as Wi-Fi. Various embodiments allow all media rendering devices such as speakers within the same listening zone or viewing zone to be precisely synchronized with each other so that the listener and / or viewer can use any unintentional media. Does not recognize experience.

  Those skilled in the art will recognize the various possible variations of the various embodiments described above. Therefore, the present invention is defined by the claims and their equivalents.

Claims (15)

An event-based synchronized multimedia playback system,
A plurality of destination devices each comprising a local clock and connected to a network and configured to render media received over the network;
A synchronization module operating on a device connected to the network and configured to communicate with the plurality of destination devices, wherein a common event En having a unique event identifier n is assigned to each of the plurality of destination devices. Said synchronization module to periodically transmit to,
With
Each destination device records a time Dxn at which the event En is received and sends an acknowledgment message including at least the time Dxn and the unique event identifier n,
The synchronization module is further configured for each of the plurality of destination devices.
(A) identifying a frequency difference between the local clock of each destination device and the local clock of the first destination device;
(B) calculating a frequency adjustment value that compensates for the difference identified in (a) between the first destination device and each destination device;
(C) receiving from the media source device to perform the process of adjusting the clock frequency by an amount related to the frequency adjustment value determined in (b), and the frequency adjustment related to the frequency adjustment value determined in (b). Instructing each destination device to perform at least one of the processes for performing the sample rate conversion of the media,
Each destination device synchronously renders media received over the network from a media source device configured to stream audio or video media, with the synchronization module performing the steps. system. The system of claim 1, wherein the destination device uses the frequency adjustment value to adjust a local clock used for rendering the media. The system of claim 1, wherein the destination device uses the frequency adjustment value to perform a sample rate conversion of the media prior to rendering the media instead of adjusting a clock frequency. A common event-based synchronized multimedia playback method,
(A) periodically transmitting a common event En having a unique event identifier n from a synchronization module operating on a device connected to the network;
(B) receiving the event En at each of a plurality of destination devices having a local clock via the network;
(C) recording the time Dxn at which the event En is received at each destination device;
(D) sending an acknowledgment message including at least the time Dxn and the unique event identifier n from each destination device to the synchronization module;
(E) identifying a frequency difference between the local clocks of each destination device with the synchronization module;
(F) calculating a frequency adjustment value that compensates for the difference identified in step (e);
(G) a media source device to perform the process of adjusting the clock frequency by an amount related to the frequency adjustment value determined in step (f), and the frequency adjustment related to the frequency adjustment value determined in step (f). Instructing each destination device to perform at least one of processing to perform sample rate conversion of the media received from;
(H) each destination device uses this frequency adjustment value to synchronously render media received over the network from a media source device streaming audio or video media;
A method comprising: The method of claim 4, wherein the destination device uses the frequency adjustment value to adjust a local clock used for rendering the media. 5. The method of claim 4, wherein the destination device uses the frequency adjustment value to perform a sample rate conversion of the media prior to rendering the media instead of adjusting a clock frequency. An event-based synchronized multimedia playback system,
A plurality of destination devices each comprising a local clock and connected to a network and configured to render media received over the network;
A synchronization module operating on a device connected to the network and configured to communicate with the plurality of destination devices;
With
The synchronization module collects a plurality of acknowledgment messages as collection information for each event En, obtains collection information of the acknowledgment messages having the received event identifier n, and creates a subset of the acknowledgment messages for the event En Calculating a frequency adjustment value for a first destination device by using a subset of the acknowledgment messages for event En, and for the first destination device,
Adjusting the clock frequency by an amount related to the calculated frequency adjustment value, or
A system instructing to perform a sample rate conversion process for media received from a media source device to perform a frequency adjustment associated with the calculated frequency adjustment value.   The system of claim 7, wherein an event message received by each destination device originates from the synchronization module.   The system of claim 7, wherein an event message received by each destination device originates from an external source other than the plurality of destination devices, the media source device, and the synchronization module. The synchronization module is further configured for each of the plurality of destination devices.
(A) identifying a clock offset between the local clock of each destination device and the local clock of the first destination device;
And (b) instructing each destination device to adjust a clock count or phase by the clock offset. The synchronization module is further configured for each of the plurality of destination devices.
(A) identifying a clock offset between the local clock of each destination device and the local clock of the first destination device;
And (b) instructing each destination device to adjust a clock count or phase by the clock offset. The synchronization module is further configured for each of the plurality of destination devices.
(A) identifying a clock offset between the local clock of each destination device and the local clock of the first destination device;
And (b) instructing each destination device to adjust a clock count or phase by the clock offset. An event-based synchronized multimedia playback system,
A plurality of destination devices each comprising a local clock and connected to a network and configured to render media received over the network;
A synchronization module operating on one of the plurality of destination devices;
With
The synchronization module periodically transmits a common event En having a unique event identifier n to each of the plurality of destination devices;
Each destination device records a time Dxn at which the event En is received, and returns an acknowledgment message including at least the time Dxn and the unique event identifier n to the synchronization module;
The synchronization module for each of the plurality of destination devices;
(A) identifying a frequency difference between the local clock of each destination device and the local clock of the first destination device;
(B) calculating a frequency adjustment value that compensates for the difference identified in (a) between the first destination device and each destination device;
(C) receiving from the media source device to perform the process of adjusting the clock frequency by an amount related to the frequency adjustment value determined in (b), and the frequency adjustment related to the frequency adjustment value determined in (b). And a step of instructing each destination device to perform at least one of the processes of performing the sample rate conversion of the media. An event-based synchronized multimedia playback system,
Operating on a device comprising at least a memory and a processor, comprising a synchronization module connected to a network;
The synchronization module periodically transmits a common event En having a unique event identifier n to each of a plurality of destination devices via the network;
The synchronization module receives a plurality of acknowledgment messages over the network, each acknowledgment message including at least a time Dxn when a corresponding event En was received at a particular destination device and the unique event identifier n. Is,
The synchronization module for each of the plurality of destination devices;
(A) identifying a frequency difference between the local clock of each destination device and the local clock of the first destination device;
(B) calculating a frequency adjustment value that compensates for the difference identified in (a) between the first destination device and each destination device;
(C) receiving from the media source device to perform the process of adjusting the clock frequency by an amount related to the frequency adjustment value determined in (b), and the frequency adjustment related to the frequency adjustment value determined in (b). Transmitting an instruction message to each destination device to instruct each destination device to perform at least one of the processes for performing the sample rate conversion of the media. A common event-based synchronized multimedia playback method,
(A) transmitting a plurality of common events En each having a unique event identifier n from a synchronization device comprising at least a memory and a processor and connected to a network;
(B) receiving a plurality of acknowledgment messages from a plurality of destination devices at the synchronization device, each including at least a time Dxn at which a common event En is received at a specific destination device and the unique event identifier n Receiving a plurality of acknowledgment messages including:
(C) identifying a frequency difference between clocks of each destination device with the synchronization device based on the plurality of acknowledgment messages;
(D) calculating at the synchronization device a frequency adjustment value that compensates for the difference identified in step (c);
(E) a media source device to perform the process of adjusting the clock frequency by an amount related to the frequency adjustment value determined in step (d), and the frequency adjustment related to the frequency adjustment value determined in step (d). Sending an instruction message to each destination device to perform at least one of the processes of converting the sample rate of the media received from;
A method comprising: