RU2825341C1 - Automatic localization of audio devices - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to systems and methods for automatic location of audio devices. Disclosed method may include receiving direction of arrival (DOA) data corresponding to sound emitted by a first intelligent audio device in an audio environment, which includes a first audio transmitter and a first audio receiver, wherein the DOA data correspond to sound received by a second intelligent audio device in an audio environment, which includes a second audio transmitter and a second audio receiver, wherein the DOA data correspond to sound emitted by the second intelligent audio device and received by the first intelligent audio device, receiving configuration parameters corresponding to an audio environment, audio devices or both, and minimizing a cost function based on the DOA data and configuration parameters for estimating the position and orientation of the first and second intelligent audio devices. Also disclosed are a device and a system for localizing audio devices in an audio environment and a persistent medium.

EFFECT: enabling localization of a set of intelligent audio devices based on direction of arrival (DOA) data between each pair of audio devices in an audio environment and minimizing the problem of nonlinear optimization without the need for synchronism between devices.

14 cl, 15 dwg

Description

Cross-references to related applications

This application claims priority based on Spanish Patent Application Nos. P202031212, filed December 3, 2020, and P202130458, filed May 20, 2021, and U.S. Provisional Patent Application Nos. 63/155,369, filed March 2, 2021, 63/203,403, filed July 21, 2021, and 63/224,778, filed July 22, 2021, all of which are incorporated herein by reference in their entirety.

Field of technology to which the invention relates

This invention relates to systems and methods for automatically determining the location of audio devices.

State of the art

Audio devices, including but not limited to smart audio devices, are widely used and are becoming common elements in many homes. While existing systems and methods for locating audio devices provide benefits, improved systems and methods are needed.

Designations and terms

Throughout the present description, including in the claims, the terms "speaker", "loudspeaker" and "transducer for reproducing audio data" are used synonymously to refer to any sound-emitting transducer (or set of transducers). A typical pair of headphones includes two speakers. A speaker may be implemented in such a way that it includes a plurality of transducers (for example, a low-frequency driver and a high-frequency driver), which may be excited by one common sound feed to the speakers or by a plurality of sound feeds to the speakers. In some examples, the sound feed(s) to the speakers may be subject to different processing in different branches of the circuit, connected to different transducers.

Throughout this description, including in the claims, the expression "performing an operation "on" a signal or data" (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., on a version of the signal that has been pre-filtered or pre-processed prior to performing the operation).

Throughout the present description, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be called a "decoder system," and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M inputs, and the other X-M inputs are received from an external source) may also be called a "decoder system."

Throughout the present description, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., by software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chipset), a digital signal processor programmed and/or otherwise configured to perform pipeline processing on audio data or other sound data, a programmable general-purpose processor or computer, and a programmable microprocessor integrated circuit or chipset.

Throughout the present description, including in the claims, the term "connected" or "connected" is used to denote a direct or indirect connection. Thus, if a first device is connected to a second device, the connection may be via a direct connection or via an indirect connection through other devices and connections.

As used herein, a "smart device" is an electronic device generally capable of communicating with one or more other devices (or networks) via various wireless protocols such as Bluetooth, ZigBee, near field communication, Wi-Fi, Li-Fi, 3G, 4G, 5G, etc., which may operate interactively and/or autonomously to some degree. Several well-known types of smart devices include smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, tablet phones and tablet computers, smart watches, smart bracelets, smart keychains, and smart audio devices. The term "smart device" may also refer to a device that exhibits some properties of widespread computing, such as artificial intelligence.

In this document, the expression "intelligent audio device" is used to refer to an intelligent device that is a single-task audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of the functionality of a virtual assistant). A single-task audio device is a device (e.g., a television receiver (TV)) that includes or is connected to at least one microphone (and optionally also includes or is connected to at least one speaker and/or at least one camera) that is designed primarily or mainly to perform a single task. For example, although a TV can typically play (and is considered to be capable of playing) audio data from program material, in most cases a modern TV runs some operating system in which applications, including a TV viewing application, are locally executed. In this sense, a single-task audio device having speaker(s) and microphone(s) is often configured to execute a local application and/or service for directly using the speaker(s) and microphone(s). Some single-task audio devices may be configured to be grouped to provide playback of audio data in a certain zone or in a user-configured area.

One general type of multi-purpose audio device is an audio device that implements at least some aspects of the functionality of a virtual assistant, although other aspects of the functionality of the virtual assistant may be implemented by one or more other devices, such as one or more servers, with which the multi-purpose audio device is configured to communicate. Such a multi-purpose audio device may be referred to herein as a "virtual assistant". The virtual assistant is a device (e.g., a smart speaker or an integrated voice assistant device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and/or at least one camera). In some examples, the virtual assistant may enable the use of multiple devices (other than the virtual assistant) for applications that are in some sense cloud-enabled or otherwise not implemented entirely directly in/on the virtual assistant. In other words, at least some aspects of the functionality of the virtual assistant, such as the speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant may communicate via a network, such as the Internet. The virtual assistants may sometimes work together, such as in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them, such as the assistant that is most confident that it has heard the activation word, responds to the activation word. The connected virtual assistants may, in some implementations, form a kind of constellation that may be controlled by a single main application, which may be (or implement) the virtual assistant.

In this document, an "activation word" is used in a broad sense to refer to any sound (e.g., a word spoken by a person or some other sound), wherein the intelligent audio device is configured to activate in response to detecting ("listening") the sound (using at least one microphone enabled or connected to the intelligent audio device or at least one other microphone). In this context, "activation" means that the device enters a state in which it waits for (in other words, listens for) an audio command. In some cases, what may be referred to herein as an "activation word" may include more than one word, such as a phrase.

In this document, the expression "activation word detector" means a device (or software that includes instructions for configuring the device) configured to continuously perform a search for a match between audio (e.g., speech) features in real time and a trained model. Typically, an activation word utterance event is initiated whenever the activation word detector determines that the probability that an activation word has been detected exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold that is adjusted to provide a reasonable compromise between false acceptance and false rejection rates. After the activation word utterance event, the device may enter a state (which may be referred to as an "activated state" or an "attentional state") in which it listens for a command and forwards the received command to a larger, more computationally intensive recognition module.

As used herein, the terms "program stream" and "content stream" mean a collection of one or more audio signals and, in some cases, video signals, at least portions of which are intended to be heard together. Examples include a selection of music, a movie soundtrack, a movie, a television program, the audio portion of a television program, a podcast, a live voice call, a synthesized voice response from an intelligent assistant, etc. In some cases, a content stream may include multiple versions of at least a portion of the audio signals, such as identical dialogue in more than one language. In such cases, only one version of the audio data, or a portion thereof (e.g., a version corresponding to one language), is intended to be played at any one time.

Disclosure of invention

At least some aspects of the present invention may be implemented via methods. Some such methods may include audio device location. For example, some methods may include localizing audio devices in an audio environment. Some such methods may include obtaining, by a control system, direction of arrival (DOA) data corresponding to sound emitted by at least a first intelligent audio device in the audio environment. In some implementations, the first intelligent audio device may include a first audio transmitter and a first audio receiver. In some examples, the DOA data may correspond to sound received by at least a second intelligent audio device in the audio environment. In some cases, the second intelligent audio device may include a second audio transmitter and a second audio receiver. In some examples, the DOA data may also correspond to sound emitted by at least a second intelligent audio device and received by at least the first intelligent audio device.

Some such methods may include receiving, by a control system, configuration parameters. In some examples, the configuration parameters may correspond to an audio environment and/or may correspond to one or more audio devices of the audio environment. Some such methods may include minimizing, by a control system, a cost function based at least in part on DOA data and the configuration parameters for estimating the position and/or orientation of at least a first intelligent audio device and a second intelligent audio device.

According to some examples, the DOA data may also correspond to sound received by one or more passive audio receivers in an audio environment. In some examples, each of the one or more passive audio receivers may include a microphone array, but in some cases, it may lack an audio signal emitter. In some such examples, minimizing the cost function may also provide an estimated location and orientation of each of the one or more passive audio receivers.

In some examples, the DOA data may also correspond to sound emitted by one or more audio signal emitters in an audio environment. In some cases, each of the one or more audio signal emitters may include at least one sound-emitting transducer, but in some cases, it may lack a microphone array. In some such examples, minimizing the cost function may also provide an estimated location of each of the one or more audio signal emitters.

In some implementations, the DOA data may also correspond to sound emitted by a third-Nth intelligent audio device in the audio environment, where N corresponds to the total number of intelligent audio devices in the audio environment. In some examples, the DOA data may also correspond to sound received by each of the first-Nth intelligent audio devices from all other intelligent audio devices in the audio environment. In some such examples, minimizing the cost function may include estimating the position and/or orientation of the third-Nth intelligent audio devices.

According to some examples, the configuration parameters may include a number of audio devices in the audio environment, one or more dimensions of the audio environment, and/or one or more constraints on the location and/or orientation of the audio device. In some cases, the configuration parameters may include ambiguity resolution data for rotation, spatial translation, and/or scaling.

Some methods may include receiving, via a control system, a configuration of seed points for a cost function. In some examples, the configuration of seed points may specify the correct number of audio transmitters and receivers in the audio environment and an arbitrary location and orientation for each of the audio transmitters and receivers in the audio environment.

Some methods may include receiving, by the control system, a weighting factor associated with one or more DOA data elements. The weighting factor, for example, may indicate the availability and/or reliability of one or more DOA data elements.

Some methods may include obtaining, by the control system, one or more DOA data elements using a beamforming-based method, a directional power response-based method, a time-difference-based method, a structured signal-based method, or combinations thereof.

Some methods may include receiving, by a control system, time of arrival (TOA) data corresponding to sound emitted by at least one audio device of an audio environment and received by at least one other audio device in the audio environment. In some such examples, a cost function may be based at least in part on the TOA data. Some such methods may include estimating at least one playback delay time and/or estimating at least one recording delay time. In some examples, the cost function may operate with a rescaled position, a rescaled delay time, and/or a rescaled signal arrival time.

According to some examples, the cost function may include a first term depending only on the DOA data. In some such examples, the cost function may include a second term depending only on the TOA data. In some such examples, the first term may include a first weighting factor, and the second term may include a second weighting factor. In some cases, one or more TOA elements of the second term may have a TOA element weighting factor indicating the availability and/or reliability of each of the one or more TOA elements.

In some examples, the configuration parameters may include playback delay time data, recording delay time data, delay time symmetry ambiguity resolution data, rotation ambiguity resolution data, spatial translation ambiguity resolution data, scaling ambiguity resolution data, and/or one or more combinations of the foregoing.

Some other aspects of the present invention may be implemented through methods. Some such methods may include device location. For example, some methods may include localizing devices in an audio environment. Some such methods may include obtaining, by a control system, direction of arrival (DOA) data corresponding to transmissions of at least a first transceiver of a first device in the environment. The first transceiver, in some examples, may include a first transmitter and a first receiver. In some cases, the DOA data may correspond to transmissions received by at least a second transceiver of a second device in the environment. In some examples, the second transceiver may include a second transmitter and a second receiver. In some cases, the DOA data may correspond to transmissions from at least the second transceiver received by at least the first transceiver.

In some examples, the first device and the second device may be audio devices, and the environment may be an audio environment. According to some such examples, the first transmitter and the second transmitter may be audio transmitters. In some such examples, the first receiver and the second receiver may be audio receivers. In some implementations, the first transmitter and receiver and the second transmitter and receiver may be configured to transmit and receive electromagnetic waves.

Some such methods may include receiving, by a control system, configuration parameters. In some cases, the configuration parameters may correspond to an environment and/or may correspond to one or more devices in the environment. Some such methods may include minimizing, by a control system, a cost function based at least in part on DOA data and the configuration parameters for estimating the position and/or orientation of at least a first device and a second device.

In some examples, the DOA data may also correspond to transmissions received by one or more passive receivers in the environment. Each of the one or more passive receivers, for example, may include an array of receivers, but may lack a transmitter. In some such examples, minimizing the cost function may also provide an estimated location and/or orientation of each of the one or more passive receivers.

According to some examples, DOA data may also correspond to transmissions from one or more transmitters in the environment. In some cases, each of the one or more transmitters may lack an array of receivers. In some such examples, minimizing the cost function may also provide an estimated location for each of the one or more transmitters.

In some examples, the DOA data may also correspond to transmissions emitted by a third-Nth transceiver of a third-Nth device in the environment, where N corresponds to the total number of transceivers in the environment. In some such examples, the DOA data may also correspond to transmissions received by each of the first-Nth transceiver of all other transceivers in the environment. In some such examples, minimizing the cost function may include estimating the position and/or orientation of the third-Nth transceiver.

Some or all of the operations, functions, and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more persistent media. Such persistent media may include memory devices such as the memory devices described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. Accordingly, some innovative aspects of the subject matter described herein may be implemented on a persistent medium having software stored thereon.

At least some aspects of the present invention may be implemented through a device. For example, one or more devices may be capable of implementing at least part of the methods disclosed herein. In some implementations, the device may include an interface system and a control system. The control system may include one or more single- or multi-chip general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. In some examples, the device may be one of the above audio devices. However, in some implementations, the device may be another type of device, such as a mobile device, a laptop, a server, etc.

The details of one or more embodiments of the subject matter of the invention described in this detailed description are set forth in the accompanying drawings and in the description below. Other features, aspects and advantages will be apparent from the description, drawings and claims. It should be noted that the relative sizes of the following drawings may not be shown to scale.

Brief description of the drawings

Fig. 1 shows an example of geometric relationships between four audio devices in an environment.

Fig. 2 shows an audio signal emitter located in the audio environment of Fig. 1.

Fig. 3 shows an audio receiving device located in the audio environment of Fig. 1.

Fig. 4 is a block diagram that provides one example of a method that may be performed by a control system of a device such as the device shown in Fig. 10.

Fig. 5 is a block diagram that provides another example of a method for automatically estimating device locations and orientations based on DOA data.

Fig. 6 is a block diagram that provides one example of a method for automatically estimating device locations and orientations based on DOA data and TOA data.

Fig. 7 is a block diagram that provides another example of a method for automatically estimating device locations and orientations based on DOA data and TOA data.

Fig. 8A shows an example of an audio environment.

Fig. 8B shows a further example of determining listener angular orientation data.

Fig. 8C shows an additional example of determining listener angular orientation data.

Fig. 8D shows one example of determining an appropriate rotation for the coordinates of an audio device in accordance with the method described with reference to Fig. 8C.

Fig. 9A is a block diagram that shows one example of a localization method.

Fig. 9B is a block diagram that shows another example of a localization method.

Fig. 10 is a block diagram showing examples of components of a device capable of implementing various aspects of the present invention.

Fig. 11 shows an example of a room plan for an audio environment, which in this example is a living space.

Like reference numerals and symbols in different drawings indicate like elements.

Implementation of the invention

The advent of intelligent speakers that include multiple excitation units and microphone arrays, in addition to existing audio devices that include television receivers and sound bars, and new microphone- and speaker-equipped connected devices such as light bulbs and microwave ovens, creates the problem that dozens of microphones and speakers must be located relative to one another to provide orchestration. Audio devices cannot be assumed to be in canonical placement patterns (e.g., a discrete Dolby 5.1 speaker layout). In some cases, audio devices may be randomly located in an environment, or at least may be distributed in an irregular and/or asymmetric manner in the environment.

In addition, audio devices may not be assumed to be homogeneous or synchronous. As used herein, audio devices may be referred to as "synchronous" or "synchronized" if sounds are detected or emitted by the audio devices according to the same sampled clock signal or synchronized sampled clock signals. For example, a first synchronized microphone of a first audio device in the environment may digitally sample audio data according to a first sampled clock signal, and a second microphone of a second synchronized audio device in the environment may digitally sample audio data according to the first sampled clock signal. Alternatively or in addition, a first synchronized speaker of the first audio device in the environment may emit sound according to a clock signal of a speaker arrangement, and a second synchronized speaker of the second audio device in the environment may emit sound according to a clock signal of a speaker arrangement.

Some previously disclosed methods for automatic speaker location require synchronized microphones and/or speakers. For example, some previously existing device localization tools rely on sample synchrony between all microphones in the system, requiring known test drives and full-bandwidth audio data transfer between sensors.

The assignee of the present invention has created several speaker localization technologies for cinema and home environments that are excellent solutions for the applications for which they are designed. Some such methods are based on the time of flight of signals extracted from impulse responses between a sound source and microphone(s) that are placed approximately co-located with each loudspeaker. Although system delay times in the recording and playback chains can also be estimated, synchronicity of samples between clock signals is required along with the need for a known test control action from which impulse responses can be estimated.

Recent examples of source localization in this context relax the constraints of requiring microphone synchrony within a device, but do not require synchrony between devices. In addition, some such methods remove the need to send audio data between sensors using narrowband messaging, such as by detecting the time of arrival (TOA, also called time of flight) of direct (non-reflected) sound or by detecting the dominant direction of arrival (DOA) of direct sound. Each approach has some potential advantages and potential disadvantages. For example, some previously deployed TOA methods can determine the geometry of a device down to unknown spatial translation, rotation, and reflection about one of three axes. The rotations of individual devices are also unknown if there is only one microphone per device. Some previously deployed DOA methods can determine the geometry of a device down to unknown spatial translation, rotation, and scale. Although some such methods can produce satisfactory results under ideal conditions, the robustness of such methods with respect to measurement error has not been demonstrated.

Some embodiments disclosed in this application provide the ability to localize a plurality of intelligent audio devices based on 1) the DOA between each pair of audio devices in an audio environment and 2) minimizing a nonlinear optimization problem designed to accept input of type 1) of data. Other embodiments disclosed in the application provide the ability to localize a plurality of intelligent audio devices based on 1) the DOA between each pair of audio devices in a system, 2) the TOA between each pair of devices, and 3) minimizing a nonlinear optimization problem designed to accept input of types 1) and 2) of data.

Fig. 1 shows an example of geometric relationships between four audio devices in an environment. In this example, audio environment 100 is a room that includes a television receiver 101 and audio devices 105a, 105b, 105c and 105d. According to this example, audio devices 105a-105d are located at locations 1-4, respectively, of audio environment 100. Like other examples disclosed in this document, the types, numbers, locations and orientations of the elements shown in Fig. 1 are given only as an example. Other implementations may have different types, numbers and arrangements of elements, such as a greater or lesser number of audio devices, audio devices in different locations, audio devices having different characteristics, etc.

In this implementation, each of the audio devices 105a-105d is an intelligent speaker that includes a microphone system and a speaker system that includes at least one speaker. In some implementations, each microphone system includes an array of at least three microphones. According to some implementations, the television receiver 101 may include a speaker system and/or a microphone system. In some such implementations, the automatic localization method may be used to automatically localize the television receiver 101 or a part of the television receiver 101 (e.g., a television speaker, a television transceiver, etc.), for example, as described below with reference to the audio devices 105a-105d.

Some embodiments described in the present specification provide the ability to automatically localize a set of audio devices, such as audio devices 105a-105d shown in Fig. 1, based on the direction of arrival (DOA) of the signals between each pair of audio devices, based on the time of arrival (TOA) of the audio signals between each pair of devices, or based on both the DOA and the TOA of the audio signals between each pair of devices. In some cases, similar to the example shown in Fig. 1, each of the audio devices is equipped with at least one driver unit and one microphone array, wherein the microphone array is capable of providing a direction of arrival of incoming audio signals. According to this example, double-headed arrow 110ab represents sound transmitted by audio device 105a and received by audio device 105b, as well as sound transmitted by audio device 105b and received by audio device 105a. Similarly, the double-headed arrows 110ac, 110ad, 110bc, 110bd and 110cd represent sounds transmitted and received by the audio devices 105a and the audio device 105c, sounds transmitted and received by the audio devices 105a and the audio device 105d, sounds transmitted and received by the audio devices 105b and the audio device 105c, sounds transmitted and received by the audio devices 105b and the audio device 105d, and sounds transmitted and received by the audio devices 105c and the audio device 105d, respectively.

In this example, each of the audio devices 105a-105d has an orientation represented by arrows 115a-115d, which can be specified in various ways. For example, the orientation of an audio device having one speaker can correspond to the direction in which one speaker faces. In some examples, the orientation of an audio device having a plurality of speakers facing in different directions can be indicated by the direction in which one of the speakers faces. In other examples, the orientation of an audio device having a plurality of speakers facing in different directions can be indicated by the direction of a vector corresponding to the sum of the audio output in the different directions in which each of the plurality of speakers faces. In the example shown in Fig. 1, the orientations of the arrows 115a-115d are determined with reference to a Cartesian coordinate system. In other examples, the orientations of the arrows 115a-115d may be determined by reference to another type of coordinate system, such as a spherical or cylindrical coordinate system.

In this example, the television receiver 101 includes an electromagnetic interface 103 that is configured to receive electromagnetic waves. In some examples, the electromagnetic interface 103 may be configured to transmit and receive electromagnetic waves. According to some implementations, at least two of the audio devices 105a-105d may include an antenna system configured as a transceiver. The antenna system may be configured to transmit and receive electromagnetic waves. In some examples, the antenna system includes an antenna array having at least three antennas. Some embodiments described in this specification provide the ability to automatically localize a set of devices, such as the audio devices 105a-105d and/or the television receiver 101 shown in Fig. 1 at least in part based on the DOA of the electromagnetic waves transmitted between the devices. Accordingly, the double-headed arrows 110ab, 110ac, 110ad, 110bc, 110bd, and 110cd may also represent electromagnetic waves transmitted between the audio devices 105a-105d.

According to some examples, the antenna system of the device (e.g., the audio device) may be co-located with the loudspeaker of the device, such as near the loudspeaker. In some such examples, the orientation of the antenna system may correspond to the orientation of the loudspeakers. Alternatively or additionally, the antenna system of the device may have a known or specified orientation relative to one or more loudspeakers of the device.

In this example, the audio devices 105a-105d are configured to wirelessly communicate with each other and with other devices. In some examples, the audio devices 105a-105d may include network interfaces that are configured to communicate between the audio devices 105a-105d and other devices via the Internet. In some implementations, the automatic localization processes disclosed in this document may be performed by a control system of one of the audio devices 105a-105d. In other examples, the automatic localization processes may be performed by another device of the audio environment 100, for example, which may sometimes be referred to as a "smart home hub", which is configured to wirelessly communicate with the audio devices 105a-105d. In other examples, the automatic localization processes may be performed at least in part by a device outside of the audio environment 100, such as a server, such as based on information received from one or more audio devices 105a-105d and/or a smart home hub.

Fig. 2 shows an audio signal emitter located in the audio environment of Fig. 1. Some implementations provide for automatic localization of one or more audio signal emitters, such as the person 205 of Fig. 2. In this example, the person 205 is located at location 5. Here, the sound emitted by the person 205 and received by the audio device 105a is represented by a single-headed arrow 210a. Similarly, the sounds emitted by the person 205 and received by the audio devices 105b, 105c and 105d are represented by single-headed arrows 210b, 210c and 210d. The audio signal emitters can be localized either based on the DOA of the audio signal emitter's sound captured by the audio devices 105a-105d and/or the television receiver 101, or based on the differences in TOA of the audio signal emitter's sound measured by the audio devices 105a-105d and/or the television receiver 101, or based on both the DOA and the differences in TOA.

Alternatively or additionally, some implementations may provide automatic localization of one or more electromagnetic wave emitters. Some embodiments described in the present specification provide the ability to automatically localize one or more electromagnetic wave emitters at least in part based on the DOA of the electromagnetic waves transmitted by the one or more electromagnetic wave emitters. If the electromagnetic wave emitter is located at location 5, the electromagnetic waves emitted by the electromagnetic wave emitter and received by the audio devices 105a, 105b, 105c and 105d may also be represented by single-headed arrows 210a, 210b, 210c and 210c.

Fig. 3 shows an audio receiver located in the audio environment of Fig. 1. In this example, the microphones of the smartphone 305 are activated, but the speakers of the smartphone 305 are not currently emitting sound. Some embodiments provide for automatic localization of one or more passive audio receiver devices, such as the smartphone 305 of Fig. 3, when the smartphone 305 is not emitting sound. Here, the sound emitted by the audio device 105a and received by the smartphone 305 is represented by a single-headed arrow 310a. Similarly, the sounds emitted by the audio devices 105b, 105c and 105d and received by the smartphone 305 are represented by single-headed arrows 310b, 310c and 310d.

If the audio receiver is equipped with a microphone array and is configured to determine the DOA of the received sound, the audio receiver can be localized at least in part based on the DOA of the sounds emitted by the audio devices 105a-105d and captured by the audio receiver. In some examples, the audio receiver can be localized at least in part based on the difference in TOA of the intelligent audio devices captured by the audio receiver, regardless of whether the audio receiver is equipped with a microphone array. Yet another embodiment can provide the ability to automatically localize a set of intelligent audio devices, one or more audio signal emitters and one or more receivers, based only on the DOA or based on the DOA and TOA, by combining the methods described above.

Localization based on the direction of signal arrival

Fig. 4 is a block diagram that provides one example of a method that can be performed by a control system of a device such as the device shown in Fig. 10. The blocks of the method 400, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods may include more or fewer blocks than shown and/or described.

Method 400 is an example of a process for localizing audio devices. In this example, method 400 includes determining the location and orientation of two or more intelligent audio devices, each of which includes a speaker system and a microphone array. According to this example, method 400 includes determining the location and orientation of the intelligent audio devices at least in part based on an audio signal emitted by each intelligent audio device and captured by each other intelligent audio device, according to a DOA estimate. In this example, initial blocks of method 400 are based on a control system of each intelligent audio device to be able to extract DOA from input audio data received by means of an array of microphones of this intelligent audio device, for example, by using differences in the arrival times of signals between individual microphone capsules of the microphone array.

In this example, block 405 includes receiving an audio signal emitted by each intelligent audio device of the audio environment and captured by each other intelligent audio device of the audio environment. In some such examples, block 405 may include causing each intelligent audio device to emit a sound, which in some cases may be a sound having a specified duration, frequency spectrum, etc. This specified type of sound may be referred to herein as a "structured source signal". In some implementations, the intelligent audio devices may be or may include audio devices 105a-105d of FIG. 1.

In some such examples, block 405 may include a sequential process of causing one intelligent audio device to emit a sound while other intelligent audio devices are "listening" for the sound. For example, referring to Fig. 1, block 405 may include: (a) causing audio device 105a to emit a sound and receiving microphone data corresponding to the emitted sound from microphone arrays of audio devices 105b 105d; then (b) causing audio device 105b to emit a sound and receiving microphone data corresponding to the emitted sound from microphone arrays of audio devices 105a, 105c and 105d; then (c) causing audio device 105c to emit a sound and receiving microphone data corresponding to the emitted sound from microphone arrays of audio devices 105a, 105b and 105d; then (d) causing the audio device 105d to emit sound and receiving microphone data corresponding to the emitted sound from the microphone arrays of the audio devices 105a, 105b and 105c. The emitted sounds may or may not be identical, depending on the particular implementation.

In other examples, block 405 may include a simultaneous process of causing all intelligent audio devices to emit a sound while other intelligent audio devices are "listening" for the sound. For example, block 405 may include performing the following steps simultaneously: (1) causing audio device 105a to emit a first sound and receiving microphone data corresponding to the emitted first sound from microphone arrays of audio devices 105b 105d; (2) causing audio device 105b to emit a second sound different from the first sound and receiving microphone data corresponding to the emitted second sound from microphone arrays of audio devices 105a, 105c and 105d; (3) causing the audio device 105c to emit a third sound different from the first sound and the second sound, and receiving microphone data corresponding to the emitted third sound from the microphone arrays of the audio devices 105a, 105b and 105d; (4) causing the audio device 105d to emit a fourth sound different from the first sound, the second sound and the third sound, and receiving microphone data corresponding to the emitted fourth sound from the microphone arrays of the audio devices 105a, 105b and 105c.

In this example, block 410 includes a process for pre-processing audio signals received via microphones. Block 410, for example, may include applying one or more filters, a noise or echo cancellation process, etc. Some additional examples of pre-processing are described below.

According to this example, block 415 includes determining possible DOA variants from the pre-processed audio signals resulting from block 410. For example, if block 405 includes emission and reception of structured source signals, block 415 may include one or more deconvolution methods, resulting in impulse responses and/or "pseudo-ranges" from which the difference in arrival times of dominant peaks can be used, in combination with the known geometry of the microphone array of intelligent audio devices, to estimate possible DOA variants.

However, not all implementations of the method 400 include obtaining microphone signals based on the emission of specified sounds. Accordingly, some examples of block 415 include "blind" methods that are applied to arbitrary audio signals, such as directional response power, beamforming on the receiver side, or other similar methods, from which one or more DOAs can be extracted by peak matching. Some examples are described below. It should be taken into account that although DOA data can be determined via blind methods or using structured source signals, in most cases, TOA data can only be determined using structured source signals. In addition, more accurate DOA information can generally be obtained using structured source signals.

According to this example, block 420 includes selecting one DOA corresponding to the sound emitted by each of the other intelligent audio devices. In many cases, the microphone array can detect direct inputs and reflected sound that is transmitted by an identical audio device. Block 420 can include selecting audio signals that are most likely to correspond to the directly transmitted sound. Some additional examples of determining possible DOA options and selecting a DOA from two or more possible DOA options are described below.

In this example, block 425 includes receiving DOA information resulting from the implementation of each intelligent audio device of block 420 (in other words, receiving a set of DOAs corresponding to sound transmitted from each intelligent audio device to each other intelligent audio device in the audio environment), and performing a localization method (for example, implementing a localization algorithm via a control system) based on the DOA information. In some disclosed implementations, block 425 includes minimizing a cost function, possibly subject to some constraints and/or weighting factors, for example, as described below with reference to Fig. 5. In some such examples, the cost function receives as input DOA values from each intelligent audio device to each other intelligent device and returns as outputs an estimated location and an estimated orientation of each of the intelligent audio devices. In the example shown in Fig. 4, block 430 represents the estimated locations of the intelligent audio device and the estimated orientations of the intelligent audio device generated in block 425.

Fig. 5 is a block diagram that provides another example of a method for automatically estimating locations and orientations of a device based on DOA data. The method 500, for example, can be implemented by implementing a localization algorithm through a control system of a device, such as the device shown in Fig. 10. The blocks of the method 500, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods can include more or fewer blocks than shown and/or described.

According to this example, DOA data is obtained in block 505. According to some implementations, block 505 may include obtaining acoustic DOA data, such as described above with reference to blocks 405-420 of Fig. 4. Alternatively or additionally, block 505 may include obtaining DOA data corresponding to electromagnetic waves that are transmitted and received by each of the plurality of devices in the environment.

In this example, the localization algorithm receives as input the DOA data obtained in block 505 from each intelligent device to each other intelligent device in the audio environment, along with any configuration parameters 510 specified for the audio environment. In some examples, optional constraints 525 may be applied to the DOA data. The configuration parameters 510, the minimization weights 515, the optional constraints 525 and the initial configuration 530, for example, may be obtained from a memory device by a control system that executes software for implementing the cost function 520 and the nonlinear search algorithm 535. The configuration parameters 510, for example, may include data corresponding to maximum room dimensions, speaker layout constraints, external input for determining a global translation in space (for example, 2 parameters), a global rotation (1 parameter) and a global scale (1 parameter), etc.

According to this example, the configuration parameters 510 are input into the cost function 520 and into the nonlinear search algorithm 535. In some examples, the configuration parameters 510 are subject to optional constraints 525. In this example, the cost function 520 takes into account the differences between the measured DOA and the DOA estimated by the optimizer's localization solution.

In some embodiments, optional constraints 525 impose constraints on the possible location and/or orientation of the audio device, such as imposing a condition that the audio devices are at a minimum distance from each other. Alternatively or in addition, optional constraints 525 may impose constraints on dummy minimization variables introduced for convenience, such as described below.

In this example, the minimization weights 515 are also fed into the nonlinear search algorithm 535. Some examples are described below.

According to some implementations, the nonlinear search algorithm 535 is an algorithm that can find local solutions to the continuous shape optimization problem:

min C(x)

,

so that: g _L ≤g(x)≤g _U ,

and x _L ≤x≤x _U

In the above expressions, represent the cost function 520, and represent the constraint functions corresponding to the optional constraints 525. In these examples, the vectors g _L and g _U represent the lower and upper bounds for the constraints, and the vectors x _L and x _U represent the bounds for the variables x.

The nonlinear search algorithm 535 may vary according to the specific implementation. Examples of the nonlinear search algorithm 535 include gradient descent methods, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method, the interior point optimization method (IPOPT), etc. Although some nonlinear search algorithms require only the values of the cost and constraint functions, some other methods may also require the first derivatives (gradients, Jacobian) of the cost and constraint functions, and some other methods may also require the second derivatives (Hessian) of the identical functions. If derivatives are required, they may be provided explicitly, or they may be automatically calculated using automatic or numerical differentiation techniques.

Some nonlinear search algorithms require starting point information to begin minimization, as suggested by the starting configuration 530 that is input to the nonlinear search algorithm 535 in Fig. 5. In some examples, the starting point information may be provided in the form of a configuration consisting of an equal number of intelligent audio devices (in other words, a number equal to the actual number of intelligent audio devices for which DOA data is obtained) with corresponding locations and orientations. The locations and orientations may be arbitrary and do not necessarily represent the actual or approximated locations and orientations of the intelligent audio devices. In some examples, the starting point information may indicate intelligent audio device locations that extend along an axis or other arbitrary line of the audio environment, intelligent audio device locations that extend along a circle, rectangle or other geometric shape in the audio environment, etc. In some examples, the starting point information may indicate arbitrary orientations of the intelligent audio device, which may be specified orientations of the intelligent audio device or random orientations of the intelligent audio device.

In some embodiments, the cost function 520 may be formulated in terms of complex plane variables as follows:

,

where the star indicates the complex conjugation, the bar indicates the absolute value, and:

z _nm = exp(iDOA _nm ) represents the complex plane value that specifies the direction of arrival of signals from smart device m measured from device n, where i represents the imaginary unit;

x _n =x _nx +ix _ny represents the complex plane value encoding the x and y positions of smart device n;

z _n = exp(iα _n ) represents a complex value encoding the orientation angle α _n of the smart device n;

represents the weighting factor specified for the DOA _nm measurement;

N represents the number of intelligent audio devices for which DOA data is obtained; and

x=(x ₁ , …, x _N ), and z=(z ₁ , ..., z _N ) represent the complex position and complex orientation vectors, respectively, of all N smart audio devices.

According to this example, the results of the minimization are device location data 540 indicating the two-dimensional position of the smart devices, x _k (representing 2 real unknowns per device), and device orientation data 545 indicating the orientation vector z _k of the smart devices (representing 2 additional real variables per device). Of the orientation vector, only the orientation angle α _k of the smart device is relevant to the problem (1 real unknown per device). Therefore, in this example, there are 3 relevant unknowns per smart device.

In some examples, the result evaluation unit 550 includes calculating the residual of the cost function at the resulting positions and orientations. A relatively lower residual indicates relatively more accurate device localization values. According to some implementations, the result evaluation unit 550 may include a feedback process. For example, some such examples may implement a feedback process that includes comparing the residual of a given combination of possible DOA options with another combination of possible DOA options, such as described in the explanation of DOA reliability metrics below.

As noted above, in some implementations, block 505 may include obtaining acoustic DOA data as described above with reference to blocks 405-420 of Fig. 4, which include determining possible DOA variants and selecting possible DOA variants. Accordingly, Fig. 5 includes a dashed line from the result evaluation block 550 to block 505 to represent one procedure of the optional feedback process. Furthermore, Fig. 4 includes a dashed line from block 430 (which may include evaluating results in some examples) to the DOA candidate selection block 420 to represent a procedure of another optional feedback process.

In some embodiments, the nonlinear search algorithm 535 may not resolve complex-valued variables. In such cases, each complex-valued variable may be replaced by a pair of real variables.

In some implementations, additional prior information regarding the availability or reliability of each DOA measurement may be provided. In some such examples, loudspeakers may be localized using only a subset of all possible DOA elements. Missing DOA elements, for example, may be masked with a corresponding zero weight in the cost function. In some such examples, the weights w _nm may be equal to zero or one, for example, zero for those measurements that are either missed or considered insufficiently reliable, and one for reliable measurements. In some other embodiments, the weights w _nm may have a continuous value from zero to one, as a function of the reliability of the DOA measurement. In those embodiments in which prior information is not available, the weights w _nm may simply be set equal to one.

In some implementations the conditions (one condition for each smart audio device) may be added as constraints to ensure that the vector indicating the orientation of the smart audio device is normalized. In other examples, these additional constraints may not be required, and the vector indicating the orientation of the smart audio device may be left unnormalized. Other implementations may add conditions regarding the proximity of smart audio devices as constraints, such as specifying that , where D is the minimum distance between smart audio devices.

The above cost function minimization does not completely determine the absolute position and orientation of the smart audio devices. According to this example, the cost function remains invariant under global rotation (1 independent parameter), global translation in space (2 independent parameters) and global rescaling (1 independent parameter), affecting all locations and orientations of the smart devices simultaneously. This global rotation, translation in space and rescaling cannot be determined from the cost function minimization. Different layout schemes related by symmetry transformations are completely indistinguishable in this framework and are considered to belong to an identical equivalence class. Therefore, the configuration parameters must provide criteria for ensuring the possibility of uniquely defining a layout scheme of the smart audio devices that represents the entire equivalence class. In some embodiments, it may be advantageous to select the criteria such that this layout scheme of the smart audio devices defines a reference frame that is close to the reference frame of the listener near the reference listening position. Examples of such criteria are given below. In some other examples, the criteria may be purely mathematical and separate from the realistic reference frame.

The symmetry ambiguity resolution criteria may include a reference position that fixes a global translation symmetry in space (e.g., intelligent audio device 1 should be located at the origin of the coordinate system); a reference orientation that fixes a two-dimensional rotational symmetry (e.g., intelligent device 1 should be oriented toward a region of the audio environment designated as the front side, for example, in which the television receiver 101 is located in Fig. 1-3); and a reference distance that fixes a global scaling symmetry (e.g., intelligent device 2 should be at a unit distance from intelligent device 1). In total, there are 4 parameters that cannot be determined from the minimization problem in this example and that must be provided as external input. Therefore, in this example, there are 3N-4 unknowns that can be determined from the minimization problem.

As described above, in some examples, in addition to the set of intelligent audio devices, one or more passive audio receivers may be provided, equipped with an array of microphones and/or one or more audio signal emitters. In such cases, the localization process may use technology for determining the location and orientation of the intelligent audio device, the location of the emitter, and the location and orientation of the passive receiver, from audio data emitted by each intelligent audio device and each emitter and captured by each other intelligent audio device and each passive receiver, based on the DOA estimate.

In some such examples, the localization process may proceed similarly to the one described above. In some cases, the localization process may be based on an identical cost function as described above, which is shown below for the convenience of the reader:

However, if the localization process includes passive audio receivers and/or audio emitters that are not audio receivers, the variables in the above equation must be interpreted in a slightly different way. Now N represents the total number of devices, which includes N _smart smart audio devices, N _rec passive audio receivers, and N _emit emitters, so that N=N _smart +N _rec +N _emit . In some examples, the weighting factors may have a sparse structure to mask missing data due to passive or emitter-only receivers (or other audio sources without receivers, such as humans), so that for all m if device n is an audio signal emitter without a receiver, and for all n if device m is an audio receiver. For both smart audio devices and passive receivers, position and angle can be determined, while for audio emitters, only position can be obtained. The total number of unknowns is 3N _smart +3N _rec +2N _emit -4.

Combined localization based on signal arrival time and signal arrival direction

In the explanation below, the differences between the above-described DOA-based localization processes and the combined DOA and TOA localization of this section are emphasized. These details, which are not explicitly given, are presumably identical to the details in the above-described DOA-based localization processes.

Fig. 6 is a block diagram that provides one example of a method for automatically estimating locations and orientations of a device based on DOA data and TOA data. The method 600, for example, can be implemented by implementing a localization algorithm through a control system of a device, such as the device shown in Fig. 10. The blocks of the method 600, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods can include more or fewer blocks than shown and/or described.

According to this example, the DOA data is obtained in blocks 605-620. According to some implementations, blocks 605-620 may include obtaining acoustic DOA data from a plurality of intelligent audio devices, for example, as described above with reference to blocks 405-420 of Fig. 4. In some alternative implementations, blocks 605-620 may include obtaining DOA data corresponding to electromagnetic waves that are transmitted and received by each of the plurality of devices in the environment.

However, in this example, block 605 also includes obtaining TOA data. According to this example, the TOA data includes a measured TOA of audio data emitted and received by each intelligent audio device in the audio environment (for example, each pair of intelligent audio devices in the audio environment). In some embodiments that include emitting structured source signals, the audio data used to extract the TOA data may be the same as the audio data used to extract the DOA data. In other embodiments, the audio data used to extract the TOA data may be different from the audio data used to extract the DOA data.

According to this example, block 616 includes detecting possible TOA variants in the audio data, and block 618 includes selecting one TOA for each pair of intelligent audio devices from among the possible TOA variants. Some examples are described below.

Various techniques can be used to obtain TOA data. One method is to use a room calibration audio sequence, such as a sweep (e.g., a logarithmic sine tone) or a maximum length sequence (MLS). If desired, any of the above sequences can be used with the bandwidth limited to a near ultrasonic audio frequency range (e.g., 18 kHz to 24 kHz). In this audio frequency range, most standard audio devices are able to emit and record sound, but such a signal is not audible to humans because it is beyond the normal capabilities of human hearing. Some alternative implementations may involve reconstructing TOA elements from a hidden signal in the primary audio signal, such as a direct sequence spread spectrum signal.

Given a set of DOA data from each intelligent audio device to each other intelligent audio device and a set of TOA data from each pair of intelligent audio devices, the localization method 625 of Fig. 6 may be based on minimizing a certain cost function, possibly subject to certain constraints. In this example, the localization method 625 of Fig. 6 takes as input the above-described DOA and TOA values and outputs estimated location and orientation data 630 corresponding to the intelligent audio devices. In some examples, the localization method 625 may also output playback and recording delay times of the intelligent audio devices, for example, up to certain global symmetries that cannot be determined from the minimization problem. Some examples are described below.

Fig. 7 is a block diagram that provides another example of a method for automatically estimating locations and orientations of a device based on DOA data and TOA data. The method 700, for example, can be implemented by implementing a localization algorithm through a control system of a device, such as the device shown in Fig. 10. The blocks of the method 700, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods can include more or fewer blocks than shown and/or described.

Except as described below, in some examples, blocks 705, 710, 715, 720, 725, 730, 735, 740, 745, and 750 may be as described above with reference to blocks 505, 510, 515, 520, 525, 530, 535, 540, 545, and 550 of Fig. 5. However, in this example, cost function 720 and nonlinear optimization method 735 are modified relative to cost function 520 and nonlinear optimization method 535 of Fig. 5 such that they operate on both DOA data and TOA data. The TOA data of block 708, in some examples, may be obtained in the manner described above with reference to Fig. 6. Another difference, compared to the process of Fig. 5, is that in this example, the nonlinear optimization method 735 also outputs recording and playback delay time data 747 corresponding to intelligent audio devices, for example, as described below. Accordingly, in some implementations, the results evaluation block 750 may include evaluating both the DOA data and the TOA data. In some such examples, the operations of block 750 may include a feedback process that includes the DOA data and/or the TOA data. For example, some such examples may implement a feedback process that includes comparing the remainder of a given combination of possible TOA/DOA options with another combination of possible TOA/DOA options, for example, as described in the explanation of TOA/DOA reliability indicators below.

In some examples, the results evaluation unit 750 includes calculating the residual of the cost function at the final positions and orientations. A relatively lower residual normally indicates relatively more accurate device localization values. According to some implementations, the results evaluation unit 750 may include a feedback process. For example, some such examples may implement a feedback process that includes comparing the residual of a given combination of possible TOA/DOA options with another combination of possible TOA/DOA options, such as described in the explanation of TOA and DOA reliability metrics below.

Accordingly, Fig. 6 includes dashed lines from block 630 (which may include evaluating results in some examples) to block 620 for selecting possible DOA options and to block 618 for selecting possible TOA options to represent an optional feedback process procedure. In some implementations, block 705 may include obtaining DOA acoustic data as described above with reference to blocks 605-620 of Fig. 6, which include determining DOA options and selecting DOA options. In some examples, block 708 may include obtaining TOA acoustic data as described above with reference to blocks 605-618 of Fig. 6, which include determining TOA options and selecting TOA options. Although not shown in Fig. 7, some optional feedback processes may include returning from block 750 for evaluating results to block 705 and/or block 708.

According to this example, the localization algorithm proceeds by minimizing a cost function, possibly subject to some constraints, and can be described as follows. In this example, the localization algorithm receives as input DOA data 705 and TOA data 708, along with configuration parameters 710 specified for the listening environment, and possibly some optional constraints 725. In this example, the cost function takes into account differences between the measured DOA and the estimated DOA and differences between the measured TOA and the estimated TOA. In some embodiments, the constraints 725 impose constraints on the possible location, orientation and/or delay times of the device, for example, imposing a condition that the audio devices are at a minimum distance from each other, and/or imposing a condition that some delay times of the device must be zero.

In some implementations, the cost function may be formulated as follows:

In the above equation, represent the playback and recording vectors for each device, respectively, and where W _DOA and W _TOA represent the global weighting factors (also known as pre-weighting factors) of the DOA and TOA minimization parts, respectively, reflecting the relative importance of each of the two terms. In some such examples, the TOA cost function may be formulated as follows:

,

Where:

TOA _nm represents the measured arrival time of a signal traveling from smart device m to smart device n;

represents the weighting factor specified for the TOA _nm measurement; and

c represents the speed of sound.

There are up to 5 real unknowns per smart audio device: the device positions x _n (2 real unknowns per device), the device orientations α _n (1 real unknown per device), and the recording and playback delay times l _n and k _n (2 additional unknowns per device). Of these, only the device positions and delay times are relevant to the TOA part of the cost function. The number of effective unknowns may be reduced in some implementations if there are a priori known constraints or relationships between delay times.

In some examples, additional prior information may be provided, such as regarding the availability or reliability of each TOA measurement. In some of these examples, the weighting factors may be equal to zero or one, for example, zero for those measurements that are not available (or are not reliable enough), and one for reliable measurements. Thus, the localization of devices can be estimated only with a subset of all possible DOA and/or TOA elements. In some other implementations, the weighting factors may have a continuous value from zero to one, for example, as a function of the reliability of the TOA measurement. In some examples, in which prior reliability information is not available, the weighting factors may simply be set equal to one.

According to some implementations, one or more additional constraints may be imposed on the possible values of delay times and/or on the relationship of different delay times to each other.

In some examples, the position of the audio devices may be measured in standard units of length, such as meters, and the delay times and arrival times may be specified in standard units of time, such as seconds. However, it is often the case that nonlinear optimization methods work better when the scale of variation of the various variables used in the minimization process is of the same order of magnitude. Therefore, some implementations may include rescaling the position measurements such that the range of variation of the positions of the smart device is between -1 and 1, and rescaling the delay times and arrival times such that these values also range between -1 and 1.

The above cost function minimization does not completely determine the absolute position and orientation of the smart audio devices or the delay times. The TOA information provides an absolute distance scale, which means that the cost function is no longer invariant under a scale transformation, but still remains invariant under a global rotation and global translation in space. In addition, the delay times obey an additional global symmetry: the cost function remains invariant if the same global quantity is added simultaneously to all playback and recording delay times. These global transformations cannot be determined from the cost function minimization. Similarly, the configuration parameters must provide a criterion to ensure that a device layout representing the entire equivalence class can be uniquely determined.

In some examples, the symmetry ambiguity resolution criteria may include the following: a reference position that fixes the global translation symmetry in space (e.g., smart device 1 must be at the origin of the coordinate system); a reference orientation that fixes the two-dimensional rotation symmetry (e.g., smart device 1 must be oriented toward the front); and a reference delay time (e.g., the write delay time for device 1 must be zero). In total, in this example, there are 4 parameters that cannot be determined from the minimization problem and that must be provided as external input. Therefore, there are 5N - 4 unknowns that can be determined from the minimization problem.

In some implementations, in addition to the set of intelligent audio devices, one or more passive audio receivers may be provided, which may not be equipped with a functioning microphone array and/or one or more audio emitters. Including delay times as minimization variables allows some disclosed methods to localize receivers and emitters for which emission and reception times are not precisely known. In some such implementations, the TOA cost function described above may be implemented. For the convenience of the reader, this cost function is shown again below:

As described above with the treatment of the DOA cost function, the cost function variables must be interpreted in a slightly different way if the cost function is used for localization estimates involving passive receivers and/or emitters. Now N represents the total number of devices, including N _smart smart audio devices, N _rec passive audio receivers, and N _emit emitters, so that N=N _smart +N _rec +N _emit . The weighting factors may have a sparse structure to mask missing data due to passive receivers or only emitters, for example, so that for all m if device n is an audio emitter, and for all n, if device m is an audio receiver. According to some implementations, for smart audio devices, the positions, orientations, and recording and playback delays must be determined; for passive receivers, the positions, orientations, and recording delays must be determined; and for audio emitters, the positions and playback delays must be determined. According to some such examples, the total number of unknowns is therefore 5N _smart +4N _rec +3N _emit -4.

Resolution of ambiguity of global spatial translation and rotation

Solutions to DOA-only and combined TOA and DOA problems are subject to global spatial translation and rotation ambiguities. In some examples, the spatial translation ambiguity can be resolved by treating the source with only the emitter as the listener and spatially translating all devices such that the listener is at the origin.

Rotational ambiguities may be resolved by imposing additional constraints on the solution. For example, some environments with multiple speakers may include speakers of a television receiver (TV) and a sofa positioned for viewing the TV. Once the location of the speakers in the environment is determined, some methods may include finding a vector connecting the listener to the viewing direction of the TV. Some such methods may then include instructing the TV to emit sound from its speakers and/or instructing the user to approach the TV and determining the location of the user's speech. Some implementations may include rendering an audio object that is panned around the environment. The user may provide user input (e.g., the phrase "Stop") indicating when the audio object is at one or more specified locations in the environment, such as at the front of the environment, at the location of the TV in the environment, etc. Some implementations include a cell phone application equipped with an inertial measurement unit that prompts a user to point the cell phone in two specified directions: first in the direction of a specific device, such as a device with illuminated LEDs, then in the user's desired viewing direction, such as the front of the environment, the location of a television in the environment, etc. Some detailed examples of ambiguity resolution are described below with reference to Figs. 8A-8D.

Fig. 8A shows an example of an audio environment. According to some examples, the audio device location data output by one of the disclosed localization methods may include an estimate of the audio device location for each of the audio devices 1-5, with reference to the audio device coordinate system 807. In this implementation, the audio device coordinate system 807 is a Cartesian coordinate system having the location of the microphone of the audio device 2 as its origin. Here, the X-axis of the audio device coordinate system 807 corresponds to the line 803 between the location of the microphone of the audio device 2 and the location of the microphone of the audio device 1.

In this example, the location of the listener is determined by indicating to the listener 805, who is shown sitting on the sofa 103 (e.g., via an audio indication from one or more speakers in the environment 800a), to determine one or more speech fragments 827 and estimating the location of the listener according to time of arrival (TOA) data. The TOA data corresponds to microphone data obtained by a plurality of microphones in the environment. In this example, the microphone data corresponds to detections of one or more speech fragments 827 by microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1-5.

Alternatively or additionally, the listener's location may be estimated according to DOA data provided by microphones of at least some (e.g., 2, 3, 4, or all 5) of the audio devices 1-5. According to some such examples, the listener's location may be determined according to the intersection of lines 809a, 809b, etc., corresponding to the DOA data.

According to this example, the location of the listener corresponds to the origin of the listener coordinate system 820. In this example, the angular orientation data of the listener is indicated by the y' axis of the listener coordinate system 820, which corresponds to the line 813a between the head of the listener 810 (and/or the nose 825 of the listener) and the sound panel 830 of the television receiver 101. In the example shown in Fig. 8A, the line 813a is parallel to the y' axis. Therefore, the angle Θ represents the angle between the y-axis and the y' axis. In this example, the block 1225 of Fig. 12 may include a rotation by the angle Θ of the coordinates of the audio device about the origin of the listener coordinate system 820. Accordingly, although the origin of the coordinate system 807 of the audio device is shown as corresponding to the audio device 2 in Fig. 8A, some implementations include co-locating the origin of the audio device coordinate system 807 with the origin of the listener coordinate system 820 before rotating the audio device coordinates by an angle Θ about the origin of the listener coordinate system 820. This co-location may be accomplished by transforming coordinates from the audio device coordinate system 807 to the listener coordinate system 820.

The location of the sound bar 830 and/or the television receiver 101, in some examples, may be determined by directing the sound bar to emit a sound and estimating the location of the sound bar according to DOA and/or TOA data, which may correspond to sound detections by microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1-5. Alternatively or additionally, the location of the sound bar 830 and/or the television receiver 101 may be determined by directing the user to approach the television and estimating the location of the user's speech using DOA and/or TOA data, which may correspond to sound detections by microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1-5. Some such methods may include applying a cost function, such as described above. Some such methods may include triangulation. Such examples may be advantageous in situations where the sound bar 830 and/or the television receiver 101 does not have an associated microphone.

In some other examples, in which the sound bar 830 and/or the television receiver 101 have an associated microphone, the location of the sound bar 830 and/or the television receiver 101 may be determined according to TOA and/or DOA methods, such as the methods disclosed in this document. According to some such methods, the microphone may be co-located with the sound bar 830.

According to some implementations, the sound bar 830 and/or the television receiver 101 may have an associated camera 811. The control system may be configured to capture an image of the head of the listener 810 (and/or the nose 825 of the listener). In some such examples, the control system may be configured to determine a line 813a between the head of the listener 810 (and/or the nose 825 of the listener) and the camera 811. The angular orientation data of the listener may correspond to the line 813a. Alternatively or additionally, the control system may be configured to determine an angle Θ between the line 813a and the y-axis of the coordinate system of the audio device.

Fig. 8B shows a further example of determining the angular orientation data of a listener. According to this example, the location of the listener has already been determined in block 1215 of Fig. 12. Here, the control system controls the speakers in the environment 800b to render the audio object 835 to a plurality of locations in the environment 800b. In some such examples, the control system may cause the speakers to render the audio object 835 in such a way that the audio object 835 appears to rotate around the listener 805, for example, by rendering the audio object 835 in such a way that the audio object 835 appears to rotate around the origin of the listener's coordinate system 820. In this example, the curved arrow 840 shows a portion of the trajectory of the audio object 835 as it rotates around the listener 805.

According to some such examples, the listener 805 may provide user input (e.g., the phrase "Stop") indicating when the audio object 835 is in the direction in which the listener 805 is facing. In some such examples, the control system may be configured to determine a line 813b between the location of the listener and the location of the audio object 835. In this example, the line 813b corresponds to the y' axis of the listener's coordinate system, which indicates the direction in which the listener 805 is facing. In alternative implementations, the listener 805 may provide user input indicating when the audio object 835 is in the front of the environment, at the location of a television in the environment, at the location of an audio device, etc.

Fig. 8C shows a further example of determining the angular orientation data of the listener. According to this example, the location of the listener has already been determined in block 1215 of Fig. 12. Here, the listener 805 uses the hand-held device 845 to provide input regarding the viewing direction of the listener 805 by pointing the hand-held device 845 toward the television receiver 101 or the sound bar 830. The dotted outline of the hand-held device 845 and the listener's hand indicates that at the time before the time at which the listener 805 points the hand-held device 845 toward the television receiver 101 or the sound bar 830, the listener 805 points the hand-held device 845 toward the audio device 2 in this example. In other examples, the listener 805 may point the hand-held device 845 toward another audio device, such as the audio device 1. According to this example, the hand-held device 845 is configured to determine an angle α between the audio device 2 and the television receiver 101 or the sound bar 830, which approximates the angle between the audio device 2 and the viewing direction of the listener 805.

The hand-held device 845, in some examples, may be a cellular telephone that includes an inertial sensor system and a wireless interface configured to communicate with a control system that controls audio devices in the environment 800c. In some examples, the hand-held device 845 may execute an application or "application" that is configured to control the hand-held device 845 in such a way as to perform the necessary functionality, such as by providing user instructions (for example, via a graphical user interface), by receiving an input indicating that the hand-held device 845 points in a desired direction, by storing the corresponding inertial sensor data and/or transmitting the corresponding inertial sensor data to the control system that controls the audio devices in the environment 800c, etc.

According to this example, the control system (which may be the control system of the hand-held device 845, the control system of the intelligent audio device in the environment 800c, or the control system that controls the audio devices in the environment 800c) is configured to determine the orientation of the lines 813c and 850 according to the data of the inertial sensors, for example, according to the data of the gyroscope. In this example, the line 813c is parallel to the y' axis and can be used to determine the angular orientation of the listener. According to some examples, the control system can determine the appropriate rotation for the coordinates of the audio device around the origin of the coordinate system 820 of the listener according to the angle α between the audio device 2 and the viewing direction of the listener 805.

Fig. 8D shows one example of determining a corresponding rotation for the coordinates of the audio device in accordance with the method described with reference to Fig. 8C. In this example, the origin of the audio device coordinate system 807 is co-located with the origin of the listener coordinate system 820. The co-location of the origins of the audio device coordinate system 807 and the listener coordinate system 820 becomes possible after the location of the listener is determined. The co-location of the origins of the audio device coordinate system 807 and the listener coordinate system 820 may include transforming the locations of the audio device from the audio device coordinate system 807 to the listener coordinate system 820. The angle α is determined as described above with reference to Fig. 8C. Accordingly, the angle α corresponds to the desired orientation of the audio device 2 in the listener coordinate system 820. In this example, the angle β corresponds to the orientation of the audio device 2 in the audio device coordinate system 807. The angle Θ, which is β-α in this example, specifies the rotation required to align the y-axis of the audio device coordinate system 807 with the y'-axis of the listener coordinate system 820.

DOA Reliability Indicators

As noted above with reference to Fig. 4, in some examples using "blind" methods that are applied to arbitrary signals, including directional response power, beamforming, or other similar methods, reliability metrics may be added to improve accuracy and stability. Some implementations include integrating the directional response of the beamforming module over time to filter out transients and detect only constant peaks, and averaging random errors and fluctuations in these constant DOAs. Other examples may use only limited frequency bands as input, which may be adjusted to room or signal types for better performance.

For examples using "supervised" methods, which include the use of structured source signals and deconvolution methods to produce impulse responses, preprocessing of the metric may be implemented to improve the accuracy and prominence of DOA peaks. In some examples, such preprocessing may include truncation with an amplitude encoding function with a weighting of some time width starting at the beginning of the impulse response in each microphone channel. Such examples may include an impulse response onset detector such that each channel onset can be detected independently.

In some examples based on “blind” or “supervised” methods as described above, additional processing may be added to improve the DOA accuracy. It is important to note that DOA selection based on peak detection (e.g., during SRP or impulse response analysis) is sensitive to environmental acoustics, which may cause non-primary path signals to be captured due to device reflections and occlusions that attenuate the receive and transmit energy. These cases may degrade the DOA accuracy of device pairs and introduce errors into the optimizer’s localization solution. Therefore, it is reasonable to consider all peaks within given thresholds as DOA candidates based on experimentally obtained validation data. One example of a given threshold is the requirement that a peak must be greater than the average SRP. For all detected peaks, saliency thresholding and removal of candidates below the average signal level have proven to be simple yet effective initial filtering techniques. As used in this document, "salience" is a measure of how large a local peak is relative to its adjacent local minima, which differs from thresholding based on power alone. One example of a salience threshold is to require that the difference in power between a peak and its adjacent local minima be equal to or greater than a threshold. Keeping the acceptable candidates increases the chance that a device pair will contain a usable DOA in its set (within the allowed error tolerance of the experimentally obtained test data), although there is a chance that it will not contain a usable DOA in cases where the signal is corrupted by strong reflections/occlusions. In some examples, a selection algorithm may be implemented to do one of the following: 1) select the best usable DOA candidate per device pair; 2) determine that none of the candidates is usable and thus set the optimization contribution of that pair to the cost function weight matrix to zero; or 3) choosing the best logically inferred option, but applying non-binary weighting to the DOA fraction in cases where it is difficult to resolve ambiguity in the amount of error that the best option tolerates.

After an initial optimization with the best inferred candidate options, in some examples the localization decision may be used to calculate the residual cost contribution of each DOA. An analysis of residual cost outliers may provide evidence of the DOA pairs that have the greatest impact on the localization decision, with extreme outliers marking those DOAs as potentially incorrect or suboptimal. A recursive series of optimizations for the outlier DOA pairs based on the residual cost contributions with the remaining candidate options and with a weighting applied to the contribution of that pair of devices may then be used to process the candidate options according to one of the above three options. This is one example of a feedback process, such as described above with reference to FIGS. 4-7. According to some implementations, repeated optimizations and processing decisions may be performed until all discovered candidate options have been evaluated and the residual cost contributions of the selected DOAs are balanced.

A disadvantage of selecting candidate variants based on optimizer scores is that it is computationally intensive and sensitive to the order in which the candidate variants are traversed. An alternative technique with a lower computational weight involves determining all permutations of candidate variants in a set and performing a triangular alignment method to localize devices for these candidate variants. Relevant triangular alignment methods are disclosed in U.S. Provisional Patent Application Ser. No. 62/992,068, filed March 19, 2020, entitled "Audio Device Auto-Location", which is hereby incorporated by reference herein for all purposes. The localization results can then be evaluated by computing cumulative and residual costs that provide results relative to the DOA candidate variants used in the triangulation. Decision logic for parsing these metrics can be used to determine the best candidate variants and weight them accordingly, which should be fed into the nonlinear optimization problem. In cases where the list of possible options is large, resulting in high numbers of permutations, filtering and intelligent traversal of the permutation list can be used.

TOA Reliability Indicators

As described above with reference to Fig. 6, the use of solutions based on multiple possible TOA options adds robustness over systems that use single or minimum TOA values and ensures that errors have a minimal impact on finding the optimal speaker placement scheme. Once the system impulse response is obtained, in some examples, each of the TOA elements can be reconstructed by searching for a peak corresponding to the direct sound. Under ideal conditions (e.g., no noise, no obstructions in the direct path between the source and receiver, and speakers pointing directly at the microphones), this peak can be easily identified as the largest peak in the impulse response. However, in the presence of noise, obstructions, or improper alignment of the speakers and microphones, the peak corresponding to the direct sound does not necessarily correspond to the largest value. Furthermore, under such conditions, the peak corresponding to the direct sound may be difficult to isolate from other reflections and/or noise. Identifying the direct sound, in some cases, may be a difficult process. Incorrect identification of the direct sound should degrade (and in some cases may completely ruin) the automatic localization process. Thus, in cases where there is potential for error in the direct sound identification process, it may be effective to consider multiple candidate candidates for the direct sound. In some such cases, the peak selection process may involve two parts: (1) a direct sound search algorithm that searches for suitable candidate peak candidates, and (2) a process for evaluating candidate peak candidates to increase the probability of selecting the correct TOA matrix entries.

In some implementations, the process of searching for direct sound peak candidates may include a method for identifying relevant candidates for the direct sound. Some such methods may be based on the following steps: (1) identifying one first reference peak (e.g., the maximum of the absolute value of the impulse response (IR)), the "first peak"; (2) estimating the noise level around (before and after) this first peak; (3) searching for alternative peaks before (and in some cases after) the first peak that are above the noise level; (4) ranking the found peaks according to their probability of matching the correct TOA; and, if necessary, (5) grouping close peaks (to reduce the number of possible candidates).

After the direct sound peak candidates have been identified, some implementations may include a peak set evaluation step. As a result of the direct sound peak candidate search, in some examples there are one or more candidate values for each TOA matrix element, ranked according to their estimated probability. The set of TOA matrices may be formed by choosing between different candidate values. A minimization process (e.g., the minimization process described above) may be implemented to estimate the likelihood of a particular TOA matrix. This process may produce minimization residuals that are good estimates of the internal coherence of the TOA and DOA matrices. An ideal noise-free TOA matrix should result in zero residuals, while a TOA matrix with incorrect matrix elements should result in large residuals. In some implementations, the method should search for a set of candidate TOA matrix elements that produces a TOA matrix with the smallest residuals. This is one example of the evaluation process described above with reference to FIG. 6 and 7, which may include a results evaluation unit 750. In one example, the evaluation process may include performing the following steps: (1) selecting an initial TOA matrix; (2) evaluating the initial matrix with the residuals of the minimization process; (3) changing one matrix element of the TOA matrix from a list of possible TOA options; (4) re-evaluating the matrix with the residuals of the minimization process; (5) if the residuals are smaller, allowing the change, otherwise not allowing it; and (6) iteratively performing steps 3-5. In some examples, the evaluation process may terminate when all possible TOA options have been evaluated, or when a predetermined maximum number of iterations has been reached.

Example of localization method

Fig. 9A is a block diagram that shows one example of a localization method. The blocks of the method 900, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods may include more or fewer blocks than shown and/or described. In this implementation, the method 900 includes estimating locations and orientations of audio devices in an environment. The blocks of the method 900 may be performed by one or more devices, which may be (or may include) the device 1000 shown in Fig. 10.

In this example, in block 905, obtaining, by means of a control system, direction of arrival (DOA) data corresponding to a sound emitted by at least a first intelligent audio device in an audio environment. The control system, for example, may be a control system 1010, which is described below with reference to Fig. 10. According to this example, the first intelligent audio device includes a first audio transmitter and a first audio receiver, and the DOA data corresponds to a sound received by at least a second intelligent audio device in the audio environment. Here, the second intelligent audio device includes a second audio transmitter and a second audio receiver. In this example, the DOA data also corresponds to a sound emitted by at least the second intelligent audio device and received by at least the first intelligent audio device. In some examples, the first and second intelligent audio devices may be two of the audio devices 105a-105d shown in Fig. 1.

The DOA data may be obtained in various ways, depending on the particular implementation. In some cases, determining the DOA data may include one or more DOA-related methods that are described above with reference to Fig. 4 and/or in the DOA Reliability Metrics section. Some implementations may include obtaining, by the control system, one or more DOA data elements using a beamforming-based method, a directional power response-based method, a signal time difference-based method, and/or a structured signal-based method.

According to this example, block 910 includes receiving, by means of the control system, configuration parameters. In this implementation, the configuration parameters correspond to the audio environment itself, one or more audio devices in the audio environment, or both the audio environment and one or more audio devices in the audio environment. According to some examples, the configuration parameters may indicate a number of audio devices in the audio environment, one or more dimensions of the audio environment, one or more restrictions on the location or orientation of the audio device, and/or ambiguity resolution data for at least one of rotation, spatial translation, or scaling. In some examples, the configuration parameters may include playback delay time data, recording delay time data, and/or data for resolving delay time symmetry ambiguity.

In this example, block 915 includes minimizing, by a control system, a cost function based at least in part on the DOA data and configuration parameters for estimating the position and orientation of at least the first intelligent audio device and the second intelligent audio device.

According to some examples, the DOA data may also correspond to the sound emitted by the third-Nth intelligent audio devices in the audio environment, where N corresponds to the total number of intelligent audio devices in the audio environment. In such examples, the DOA data may also correspond to the sound received by each of the first-Nth intelligent audio devices from all other intelligent audio devices in the audio environment. In such cases, minimizing the cost function may include estimating the position and orientation of the third-Nth intelligent audio devices.

In some examples, the DOA data may also correspond to sound received by one or more passive audio receivers in an audio environment. Each of the one or more passive audio receivers may include a microphone array, but may lack an audio signal emitter. Minimizing the cost function may also provide an estimated location and orientation of each of the one or more passive audio receivers. According to some examples, the DOA data may also correspond to sound emitted by one or more audio signal emitters in an audio environment. Each of the one or more audio signal emitters may include at least one sound emitting transducer, but may lack an array of microphones. Minimizing the cost function may also provide an estimated location of each of the one or more audio signal emitters.

In some examples, the method 900 may include receiving by the control system an initial configuration for the cost function. The initial configuration, for example, may indicate the correct number of transmitting and receiving audio devices in the audio environment and an arbitrary location and orientation for each of the transmitting and receiving audio devices in the audio environment.

According to some examples, the method 900 may include receiving, by the control system, a weighting factor associated with one or more DOA data elements. The weighting factor, for example, may indicate the availability and/or reliability of one or more DOA data elements.

In some examples, the method 900 may include receiving, by the control system, time of arrival (TOA) data corresponding to sound emitted by at least one audio device in the audio environment and received by at least one other audio device in the audio environment. In some such examples, the cost function may be based at least in part on the TOA data. Some such implementations may include estimating at least one playback delay time and/or at least one recording delay time. According to some such examples, the cost function may operate with a rescaled position, a rescaled delay time, and/or a rescaled signal arrival time.

In some examples, the cost function may include a first term depending only on the DOA data and a second term depending only on the TOA data. In some such examples, the first term may include a first weighting factor, and the second term may include a second weighting factor. According to some such examples, one or more TOA elements of the second term may have a TOA element weighting factor indicating the availability or reliability of each of the one or more TOA elements.

Fig. 9B is a flow chart that provides another example of a localization method. The blocks of the method 950, like other methods described herein, are not necessarily performed in the order shown. In addition, such methods may include more or fewer blocks than shown and/or described. In this implementation, the method 950 includes estimating locations and orientations of devices in an environment. The blocks of the method 950 may be performed by one or more devices, which may be (or may include) the device 1000 shown in Fig. 10.

In this example, at block 955, obtaining, by means of the control system, direction of arrival (DOA) data corresponding to transmissions of at least a first transceiver of the first device in the environment. The control system, for example, can be the control system 1010, which is described below with reference to Fig. 10. According to this example, the first transceiver includes a first transmitter and a first receiver, and the DOA data correspond to transmissions received by means of at least a second transceiver of the second device in the environment, wherein the second transceiver also includes a second transmitter and a second receiver. In this example, the DOA data also correspond to transmissions from at least the second transceiver received by means of at least the first transceiver. According to some examples, the first transceiver and the second transceiver can be configured to transmit and receive electromagnetic waves. In some examples, the first and second intelligent audio devices may be two of the audio devices 105a-105d shown in Fig. 1.

The DOA data may be obtained in various ways, depending on the particular implementation. In some cases, determining the DOA data may include one or more DOA-related methods that are described above with reference to Fig. 4 and/or in the DOA Reliability Metrics section. Some implementations may include obtaining, by the control system, one or more DOA data elements using a beamforming-based method, a directional power response-based method, a signal time difference-based method, and/or a structured signal-based method.

According to this example, block 960 includes receiving, by means of the control system, configuration parameters. In this implementation, the configuration parameters correspond to the environment itself, to one or more devices in the audio environment, or to both the environment and to one or more devices in the audio environment. According to some examples, the configuration parameters may indicate a number of audio devices in the environment, one or more dimensions of the environment, one or more restrictions on the location or orientation of the device, and/or ambiguity resolution data for at least one of rotation, translation in space, or scaling. In some examples, the configuration parameters may include playback delay time data, recording delay time data, and/or data for resolving delay time symmetry ambiguity.

In this example, block 965 includes minimizing a cost function by the control system based at least in part on the DOA data and configuration parameters for estimating the position and orientation of at least the first device and the second device.

According to some implementations, the DOA data may also correspond to transmissions emitted by a third-Nth transceiver of a third-Nth device in the environment, where N corresponds to the total number of transceivers in the environment, and wherein the DOA data also corresponds to transmissions received by each of the first-Nth transceivers of all other transceivers in the environment. In some such implementations, minimizing the cost function may also include estimating the position and orientation of the third-Nth transceiver.

In some examples, the first device and the second device may be intelligent audio devices, and the environment may be an audio environment. In some such examples, the first transmitter and the second transmitter may be audio transmitters. In some such examples, the first receiver and the second receiver may be audio receivers. According to some such examples, the DOA data may also correspond to sound emitted by a third to Nth intelligent audio devices in the audio environment, where N corresponds to the total number of intelligent audio devices in the audio environment. In such examples, the DOA data may also correspond to sound received by each of the first to Nth intelligent audio devices from all other intelligent audio devices in the audio environment. In such cases, minimizing the cost function may include estimating the position and orientation of the third to Nth intelligent audio devices. Alternatively or additionally, in some examples, the DOA data may correspond to electromagnetic waves emitted and received by devices in the environment.

In some examples, the DOA data may also correspond to sound received by one or more passive receivers in the environment. Each of the one or more passive receivers may include an array of receivers, but may lack a transmitter. Minimizing the cost function may also provide an estimated location and orientation of each of the one or more passive receivers. According to some examples, the DOA data may also correspond to transmissions from one or more transmitters in the environment. In some such examples, each of the one or more transmitters may lack an array of receivers. Minimizing the cost function may also provide an estimated location of each of the one or more transmitters.

In some examples, the method 950 may include receiving, by means of the control system, an initial configuration for the cost function. The initial configuration, for example, may indicate the correct number of transmitters and receivers in the audio environment and an arbitrary location and orientation for each of the transmitters and receivers in the audio environment.

According to some examples, the method 950 may include receiving, by the control system, a weighting factor associated with one or more DOA data elements. The weighting factor, for example, may indicate the availability and/or reliability of one or more DOA data elements.

In some examples, the method 950 may include receiving, by the control system, time of arrival (TOA) data corresponding to sound emitted by at least one audio device in the audio environment and received by at least one other audio device in the audio environment. In some such examples, the cost function may be based at least in part on the TOA data. Some such implementations may include estimating at least one playback delay time and/or at least one recording delay time. According to some such examples, the cost function may operate with a rescaled position, a rescaled delay time, and/or a rescaled signal arrival time.

In some examples, the cost function may include a first term depending only on the DOA data and a second term depending only on the TOA data. In some such examples, the first term may include a first weighting factor, and the second term may include a second weighting factor. According to some such examples, one or more TOA elements of the second term may have a TOA element weighting factor indicating the availability or reliability of each of the one or more TOA elements.

Fig. 10 is a block diagram that shows examples of components of a device capable of implementing various aspects of the present invention. Device 1000, for example, may be configured to implement the methods described above with reference to Fig. 9A and/or 9B. According to some examples, device 1000 may be or may include a smart audio device (e.g., a smart speaker) that is configured to implement at least some of the methods disclosed herein. In other implementations, device 1000 may be or may include another device that is configured to implement at least some of the methods disclosed herein. In some such implementations, device 1000 may be or may include a smart home hub or a server.

In this example, device 1000 includes an interface system 1005 and a control system 1010. Interface system 1005, in some implementations, may be configured to receive input from each of a plurality of microphones in the environment. Interface system 1005 may include one or more network interfaces and/or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some implementations, interface system 1005 may include one or more wireless interfaces. Interface system 1005 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 1005 may include one or more interfaces between control system 1010 and a memory system, such as optional memory system 1015 shown in FIG. 10. However, the control system 1010 may include a memory system.

The control system 1010, for example, may include a single- or multi-chip general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete logic element or transistor logic and/or discrete hardware components. In some implementations, the control system 1010 may reside in more than one device. For example, a portion of the control system 1010 may reside in a device in the audio environment 100 illustrated in Fig. 1 (e.g., in one of the audio devices 105a-105d or in a smart home hub), and another portion of the control system 1010 may reside in a device that is outside the audio environment 100, for example, on a server, in a mobile device (e.g., a smartphone or a tablet computer), etc. The interface system 1005, in some such examples, may also reside in more than one device.

In some implementations, the control system 1010 may be configured to implement at least some of the methods disclosed herein. According to some examples, the control system 1010 may be configured to implement the methods described above, for example, with reference to Fig. 4-9B .

In some examples, device 1000 may include an optional microphone system 1020, which is illustrated in Fig. 10. Microphone system 1020 may include one or more microphones. In some examples, microphone system 1020 may include an array of microphones. In some examples, device 1000 may include an optional speaker system 1025, which is illustrated in Fig. 10. Speaker system 1025 may include one or more speakers. In some examples, microphone system 1020 may include an array of speakers. In some such examples, device 1000 may be or may include an audio device. For example, device 1000 may be or may include one of audio devices 105a-105d shown in Fig. 1.

In some examples, the device 1000 may include an optional antenna system 1030, which is shown in Fig. 10. According to some examples, the antenna system 1030 may include an array of antennas. In some examples, the antenna system 1030 may be configured to transmit and/or receive electromagnetic waves. According to some implementations, the control system 1010 may be configured to estimate the distance between two audio devices in the environment based on antenna data from the antenna system 1030. For example, the control system 1010 may be configured to estimate the distance between two audio devices in the environment according to the direction of arrival of signals from these antennas and/or the strength of received signals from these antennas.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more persistent media. For example, some or all of the methods described herein may be performed by a control system 1010 according to instructions stored on one or more persistent media. Such persistent media may include memory devices, such as the memory devices described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. The one or more persistent media, for example, may reside in the optional memory system 1015 shown in Fig. 10 and/or in the control system 1010. Accordingly, various innovative aspects of the subject matter described herein may be implemented in one or more persistent media having stored software. The software, for example, may include instructions for controlling at least one device to process audio data. The software may, for example, be executed by one or more components of a control system, such as control system 1010 of Fig. 10.

Fig. 11 shows an example of a room plan for an audio environment, which in this example is a living space. Like other drawings provided herein, the types and numbers of elements shown in Fig. 11 are provided as an example only. Other implementations may include more, fewer, and/or different types and numbers of elements.

According to this example, environment 1100 includes a living room 1110 in the upper left corner, a kitchen 1115 in the lower center and a bedroom 1122 in the lower right corner. Rectangles and circles distributed throughout the living space represent a set of speakers 1105a-1105h, at least some of which may be intelligent speakers in some implementations, placed in locations convenient for the space, but without adhering to a prescribed placement scheme of the standard (arbitrarily placed). In some examples, television receiver 1130 may be configured to at least partially implement one or more disclosed embodiments. In this example, environment 1100 includes cameras 1111a-1111e, which are distributed throughout the environment. In some implementations, one or more intelligent audio devices in environment 1100 may also include one or more cameras. One or more intelligent audio devices may be single-task audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may be permanently located in/on the television receiver 1130, in the mobile phone or in the intelligent speaker, for example, in one or more speakers 1105b, 1105d, 1105e or 1105h. Although the cameras 1111a-1111e are not shown in each illustration of the environment 1100 presented in this description, nevertheless, in some implementations, each of the environments 1100 may include one or more cameras.

Some aspects of the present invention include a system or device configured (e.g., programmable) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) that stores code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmable with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including an embodiment of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory device, and a processing subsystem that is programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data input thereto.

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform the desired processing on the audio signal(s), which includes performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or a microprocessor that may include an input device and a memory device) that is programmed with software or firmware and/or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of a system according to the invention are implemented as a general-purpose processor or DSP configured (e.g., programmable) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more speakers and/or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and/or keyboard), a storage device, and a display device.

Another aspect of the present invention is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code for performing (e.g., code executable with the ability to execute) one or more examples of the disclosed methods or steps thereof.

While specific embodiments and applications of the invention have been described herein, it will be apparent to those skilled in the art that many other embodiments and applications described herein are possible without departing from the scope of the invention.

Claims (35)

1. A method for localizing audio devices in an audio environment, wherein the method comprises the steps of: - receiving, by means of a control system, data of the direction of arrival (DOA) corresponding to a sound emitted by at least a first intelligent audio device in an audio environment, wherein the first intelligent audio device includes a first transmitting audio device and a first receiving audio device, wherein the DOA data corresponds to a sound received by at least a second intelligent audio device in the audio environment, wherein the second intelligent audio device includes a second transmitting audio device and a second receiving audio device, wherein the DOA data also corresponds to a sound emitted by at least the second intelligent audio device and received by at least the first intelligent audio device; - receiving, via the control system, configuration parameters, wherein the configuration parameters correspond to an audio environment, correspond to one or more audio devices in the audio environment, or correspond to both an audio environment and one or more audio devices in the audio environment; and - minimizing by means of a control system a cost function at least partially based on DOA data and configuration parameters for estimating the position and orientation of at least the first intelligent audio device and the second intelligent audio device, wherein the DOA data also corresponds to sound received by one or more passive audio receivers in an audio environment, wherein each of the one or more passive audio receivers includes a microphone array but lacks an audio signal emitter, and wherein the minimization of the cost function also provides an estimated location and orientation of each of the one or more passive audio receivers, wherein the DOA data also corresponds to sound emitted by one or more audio signal emitters in an audio environment, wherein each of the one or more audio signal emitters includes at least one sound-emitting transducer, but lacks a microphone array, and wherein the minimization of the cost function also provides an estimated location of each of the one or more audio signal emitters, wherein the DOA data also corresponds to a sound emitted by a third-Nth intelligent audio device in the audio environment, wherein N corresponds to the total number of intelligent audio devices in the audio environment, wherein the DOA data also corresponds to a sound received by each of the first-Nth intelligent audio devices from all other intelligent audio devices in the audio environment, and wherein minimizing the cost function includes a step in which the position and orientation of the third-Nth intelligent audio devices are estimated, wherein the configuration parameters include at least one of a number of audio devices in the audio environment, one or more dimensions of the audio environment, one or more constraints on the location or orientation of the audio device, or ambiguity resolution data for at least one of rotation, translation in space, or scaling. 2. The method of claim 1, further comprising the step of receiving, by means of the control system, an initial configuration for the cost function, wherein the initial configuration specifies the correct number of transmitting and receiving audio devices in the audio environment and an arbitrary location and orientation for each of the transmitting and receiving audio devices in the audio environment. 3. The method according to any one of claims 1, 2, further comprising the step of receiving by the control system a weighting factor associated with one or more DOA data elements, wherein the weighting factor indicates at least one of availability or reliability of the one or more elements. 4. The method according to any one of claims 1-3, further comprising the step of obtaining by the control system one or more DOA data elements using at least one of a beamforming-based method, a power response-based method, a time-of-arrival-difference method, or a structured signal-based method. 5. The method according to any one of claims 1 to 4, further comprising the step of receiving, by means of the control system, time of arrival (TOA) data corresponding to sound emitted by means of at least one audio device in the audio environment and received by means of at least one more audio device in the audio environment, and wherein the cost function is based at least in part on the TOA data. 6. The method according to claim 5, further comprising the step of estimating at least one playback delay time, estimating at least one recording delay time, or estimating at least one playback delay time and at least one recording delay time. 7. The method of claim 6, wherein the cost function operates on at least one of a rescaled position, a rescaled delay time, or a rescaled signal arrival time. 8. The method according to any one of paragraphs. 5-7, in which the cost function includes a first term depending only on the DOA data and a second term depending only on the TOA data. 9. The method of claim 8, wherein the first term includes a first weighting factor, and wherein the second term includes a second weighting factor. 10. The method of claim 8, wherein one or more TOA elements of the second member have a TOA element weighting factor indicating the availability or reliability of each of the one or more TOA elements. 11. The method according to any one of claims 1 to 10, wherein the configuration parameters include at least one of: playback delay time data; recording delay time data; delay time symmetry ambiguity resolution data; rotation ambiguity resolution data; spatial translation ambiguity resolution data; or scaling ambiguity resolution data. 12. A device for localizing audio devices in an audio environment, wherein the device includes an interface system and a control system, wherein the control system is designed with the ability to: - obtaining direction of arrival (DOA) data corresponding to sound emitted by at least a first intelligent audio device in an audio environment, wherein the first intelligent audio device includes a first audio transmitter and a first audio receiver, wherein the DOA data corresponds to sound received by at least a second intelligent audio device in the audio environment, wherein the second intelligent audio device includes a second audio transmitter and a second audio receiver, wherein the DOA data also corresponds to sound emitted by at least the second intelligent audio device and received by at least the first intelligent audio device; - receiving configuration parameters, wherein the configuration parameters correspond to an audio environment, correspond to one or more audio devices in the audio environment, or correspond to both an audio environment and one or more audio devices in the audio environment; and - minimizing a cost function at least in part based on DOA data and configuration parameters for estimating the position and orientation of at least the first intelligent audio device and the second intelligent audio device, wherein the DOA data also corresponds to sound received by one or more passive audio receivers in an audio environment, wherein each of the one or more passive audio receivers includes a microphone array but lacks an audio signal emitter, and wherein the minimization of the cost function also provides an estimated location and orientation of each of the one or more passive audio receivers, wherein the DOA data also corresponds to sound emitted by one or more audio signal emitters in an audio environment, wherein each of the one or more audio signal emitters includes at least one sound-emitting transducer, but lacks a microphone array, and wherein the minimization of the cost function also provides an estimated location of each of the one or more audio signal emitters, wherein the DOA data also corresponds to a sound emitted by a third-Nth intelligent audio device in the audio environment, wherein N corresponds to the total number of intelligent audio devices in the audio environment, wherein the DOA data also corresponds to a sound received by each of the first-Nth intelligent audio devices from all other intelligent audio devices in the audio environment, and wherein minimizing the cost function includes a step in which the position and orientation of the third-Nth intelligent audio devices are estimated, wherein the configuration parameters include at least one of a number of audio devices in the audio environment, one or more dimensions of the audio environment, one or more constraints on the location or orientation of the audio device, or ambiguity resolution data for at least one of rotation, translation in space, or scaling. 13. A system for localizing audio devices in an audio environment, wherein the system includes an interface system and a control system, wherein the control system is designed with the ability to: - obtaining direction of arrival (DOA) data corresponding to sound emitted by at least a first intelligent audio device in an audio environment, wherein the first intelligent audio device includes a first audio transmitter and a first audio receiver, wherein the DOA data corresponds to sound received by at least a second intelligent audio device in the audio environment, wherein the second intelligent audio device includes a second audio transmitter and a second audio receiver, wherein the DOA data also corresponds to sound emitted by at least the second intelligent audio device and received by at least the first intelligent audio device; - receiving configuration parameters, wherein the configuration parameters correspond to an audio environment, correspond to one or more audio devices in the audio environment, or correspond to both an audio environment and one or more audio devices in the audio environment; and - minimizing a cost function at least in part based on DOA data and configuration parameters for estimating the position and orientation of at least the first intelligent audio device and the second intelligent audio device, wherein the DOA data also corresponds to sound received by one or more passive audio receivers in an audio environment, wherein each of the one or more passive audio receivers includes a microphone array but lacks an audio signal emitter, and wherein the minimization of the cost function also provides an estimated location and orientation of each of the one or more passive audio receivers, wherein the DOA data also corresponds to sound emitted by one or more audio signal emitters in an audio environment, wherein each of the one or more audio signal emitters includes at least one sound-emitting transducer, but lacks a microphone array, and wherein the minimization of the cost function also provides an estimated location of each of the one or more audio signal emitters, wherein the DOA data also corresponds to a sound emitted by a third-Nth intelligent audio device in the audio environment, wherein N corresponds to the total number of intelligent audio devices in the audio environment, wherein the DOA data also corresponds to a sound received by each of the first-Nth intelligent audio devices from all other intelligent audio devices in the audio environment, and wherein minimizing the cost function includes a step in which the position and orientation of the third-Nth intelligent audio devices are estimated, wherein the configuration parameters include at least one of a number of audio devices in the audio environment, one or more dimensions of the audio environment, one or more constraints on the location or orientation of the audio device, or ambiguity resolution data for at least one of rotation, translation in space, or scaling. 14. A non-transitory medium storing software, wherein the software includes instructions for controlling one or more devices to implement the method according to any one of paragraphs 1-11.