TW202314684A - Processing of audio signals from multiple microphones - Google Patents

Abstract

A first device includes a memory configured to store instructions and one or more processors configured to receive audio signals from multiple microphones. The one or more processors are configured to process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals. The one or more processors are also configured to and send, to a second device, data based on the direction-of-arrival information and a class or embedding associated with the direction-of-arrival information.

Description

Processing of audio signals from multiple microphones

Cross References to Related Applications

This patent application claims priority to Provisional Patent Application No. 63/203,562, filed July 27, 2021, entitled "DIRECTIONAL AUDIO SIGNAL PROCESSING," the contents of which are hereby incorporated by reference in their entirety.

The large system of this case is about audio signal processing.

Advances in technology have led to smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless telephones (such as cell phones and smart phones), tablet computers, and laptop computers, which are small, lightweight, and easy for users to carry. These devices can communicate voice and data packets over a wireless network. In addition, many of these devices incorporate additional functions such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications (such as web browser applications) that can be used to access the Internet. Accordingly, these devices can include significant computing capabilities.

Devices such as mobile phones and smartphones can be paired with headsets to allow users to listen to audio without holding the mobile phone to their ear. One disadvantage of the user wearing headphones is that the user may not be aware of the surrounding environment. As a non-limiting example, if a user walks across an intersection, the user may not be able to hear approaching vehicles. In scenarios where the user's focus is elsewhere (for example, on the user's mobile phone or looking away from an approaching vehicle), the user may not be able to determine whether the vehicle is approaching or which direction the vehicle is coming from near.

According to one embodiment of the present invention, the first device includes a memory configured to store instructions and one or more processors. The one or more processors are configured to receive audio signals from a plurality of microphones. The one or more processors are also configured to process the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The one or more processors are further configured to send the direction-of-arrival information and the category or embedded data associated with the direction-of-arrival information to the second device.

According to another embodiment of the present invention, a method of processing audio includes receiving, at one or more processors of a first device, audio signals from a plurality of microphones. The method also includes processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The method also includes sending data based on the direction-of-arrival information and the category or embedding associated with the direction-of-arrival information to the second device.

According to another embodiment of the present invention, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause the one or more processors to read from a plurality of microphones Receive audio signal. When executed by one or more processors, the instructions further cause the one or more processors to process the audio signal to generate arrivals corresponding to one or more sources of sound represented by one or more of the audio signals. Direction information. When executed by the one or more processors, the instructions further cause the one or more processors to send to the second device data based on the direction-of-arrival information and the class or embedding associated with the direction-of-arrival information.

According to another embodiment of the invention, the first device comprises means for receiving audio signals from a plurality of microphones. The first apparatus also includes means for processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The first device also includes means for sending data based on the direction-of-arrival information and the category or embedding associated with the direction-of-arrival information to the second device.

Other aspects, advantages and features of this case will become apparent after reading the entire application including the following sections: Description of Drawings, Detailed Description and Claims.

Systems and methods for performing directional audio signal processing are disclosed. A first device, such as a headset, may include a plurality of microphones configured to pick up sounds in the surrounding environment. Each microphone may have a different orientation and position on the first device, such as to pick up sounds from different directions. In response to capturing sound, each microphone can generate a corresponding audio signal, which is provided to the directional audio signal processing unit. The directional audio signal processing unit may process audio signals from the microphones to identify different audio events associated with the sound and location of each audio event. In some embodiments, an audio signal associated with an audio event is processed via one or more classifiers at the first device to identify an audio class of the audio event. In a non-limiting example, if at least one of the plurality of microphones picks up car sounds, the directional audio signal processing unit may identify based on characteristics (e.g., pitch, frequency, etc.) associated with the corresponding audio signal car sounds, and the relative direction of the car sounds can be identified based on the corresponding microphones that pick up the sounds. In response to recognizing the car sound and the corresponding relative direction, the first device may generate data representing the sound and direction, and may provide this data to a second device (such as a mobile phone). In some examples, data representing sound may include audio class or embedding and direction-of-arrival information associated with the sound source. The second device can use this data (eg, direction information) to perform additional operations. As a non-limiting example, the second device may decide whether to generate a visual alert or a physical alert to warn the user of the headset of nearby vehicles.

According to some aspects, distributed audio processing is performed using a first device, such as a headset device, to capture sound using a plurality of microphones, and to perform preliminary processing of the audio corresponding to the captured sound. For example, as illustrative and non-limiting examples, the first device may perform direction-of-arrival processing for locating one or more sound sources, acoustic environment processing for detecting the environment or changes in the environment of the first device based on ambient sounds. , an audio event handler for identifying a sound corresponding to the audio event, or a combination thereof.

Since the first device may be relatively limited in terms of processing resources, memory capacity, battery life, etc., the first device may send information about the audio to a second device (such as a mobile phone) that has greater computing, memory, and power resources. processed information. For example, in some embodiments, the first device sends a representation of the audio data and classifications of audio events detected in the audio data to the second device, and the second device performs additional processing to verify the classification of the audio events. According to some aspects, the second device uses information provided by the first device, such as directional information and classifications associated with sound events, as additional input to a classifier that processes the audio data. Performing classification of audio data in conjunction with directional information, classification from the first device, or both may improve the accuracy, speed, or one or more other aspects of the classifier at the second device.

This distributed audio processing enables the user of the first device to benefit from the Enhanced processing capabilities of the second device. For example, the first device may automatically transition from playback mode (eg, playing music or other audio to the user) to a transparency mode in which sounds corresponding to detected audio events are played to the user. Other benefits and examples of applications in which the disclosed technology may be used are described in more detail below with reference to the accompanying figures.

Specific aspects of the present case are described below with reference to the drawings. In the description, common features are denoted by common reference numerals. As used herein, various terms are used for the purpose of describing particular embodiments only and are not intended to be limiting of the embodiments. For example, the singular forms "a", "an" and "the" are intended to include the plural forms as well unless the context clearly dictates otherwise. Furthermore, some features described herein are in the singular in some embodiments and in the plural in other embodiments. To illustrate, FIG. 1 illustrates device 110 including one or more processors ("processor(s)" 116 in FIG. 1), which indicates that, in some implementations, device 110 includes a single processor 116 , while in other implementations, the device 110 includes a plurality of processors 116 . For ease of reference herein, these features are generally introduced as "one or more" features, and are subsequently referred to in the singular unless an aspect is described in relation to a plurality of such features.

It will also be understood that the terms "comprise/comprises/comprising" may be used interchangeably with "include/includes/including". Furthermore, it should be understood that the term "wherein" may be used interchangeably with "where". As used herein, "exemplary" may indicate an example, an implementation, and/or an aspect, and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element such as structure, element, operation, etc., do not by themselves indicate that the element is relative to another element. any precedence or order, but merely to distinguish that element from another element of the same name (but using ordinal terminology). As used herein, the term "collection" refers to one or more of the specified elements, and the term "plurality" refers to a plurality (eg, two or more) of the specified elements.

As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (for example, communicatively coupled) via one or more other devices, components, wires, buses, networks (for example, wired networks, wireless networks, or combinations thereof), , electrical coupling or physical coupling). As illustrative and non-limiting examples, two devices (or elements) that are electrically coupled may be included in the same device or different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some embodiments, two devices (or components) that are communicatively coupled (such as in electrical communication) can send and receive signals (e.g., digital signals) directly or indirectly via one or more wires, buses, networks, etc. or analog signals). As used herein, "directly coupled" may include two devices that are coupled (eg, communicatively, electrically, or physically) without intervening elements.

In this case, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that these terms should not be construed as limiting and that other techniques may be utilized to perform similar operations. Furthermore, as referred to herein, "generate", "calculate", "estimate", "use", "select", "access" and "determine" may be used interchangeably. For example, "produce", "calculate", "estimate" or "determine" a parameter (or signal) may refer to actively producing, estimating, calculating or determining a parameter (or signal), or may refer to using, selecting or accessing A parameter (or signal) that has been produced (such as by another element or device).

Referring to FIG. 1 , a particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 100 . System 100 includes a first microphone 102 and a second microphone 104 each coupled to or integrated in device 110 . System 100 also includes third microphone 106 and fourth microphone 108 coupled to or integrated in device 120 . Although two microphones 102, 104 are shown coupled to or integrated in device 110, and two microphones 106, 108 are shown coupled to or integrated in device 120, in other implementations, the device 110, device 120, or both may be coupled to any number of additional microphones. As a non-limiting example, four (4) microphones may be coupled to device 110 and another four (4) microphones may be coupled to device 120 . In some implementations, microphones 102, 104, 106, and 108 are implemented as directional microphones. In other implementations, one or more (or all) of microphones 102, 104, 106, and 108 are implemented as omnidirectional microphones.

According to one embodiment, the device 110 corresponds to a headset and the device 120 corresponds to a mobile phone. In some scenarios, device 110 may be paired with device 120 using a wireless connection (eg, a Bluetooth® (registered trademark of Bluetooth SIG, Inc., Washington) connection). For example, device 110 may communicate with device 120 using a low energy protocol, such as the Bluetooth® Low Energy (BLE) protocol. In other examples, the wireless connection corresponds to signal transmission and reception according to an IEEE 802.11 type (eg, WiFi) wireless area network or one or more other wireless radio frequency (RF) communication protocols.

The first microphone 102 is configured to capture sound 182 from one or more sources 180 . In the illustrative example of FIG. 1 , source 180 corresponds to a vehicle, such as an automobile. Thus, if the device 110 corresponds to a headset, the microphones 102, 104 can be used to pick up the sound 182 of nearby cars. It should be understood, however, that a vehicle is only a non-limiting example of a sound source, and that the techniques described herein may be implemented with other sound sources. Upon capturing a sound 182 from a source 180 , the first microphone 102 is configured to generate an audio signal 170 representative of the captured sound 182 . In a similar manner, the second microphone 104 is configured to capture sound 182 from one or more sources 180 . Upon capturing sound 182 from source 180 , second microphone 104 is configured to generate an audio signal 172 representative of the captured sound 182 .

The first microphone 102 and the second microphone 104 may have different positions, different orientations, or both. As a result, the microphones 102, 104 may pick up the sound 182 at different times, with different phases, or both. To illustrate, if first microphone 102 is closer to source 180 than second microphone 104 , first microphone 102 may capture sound 182 before second microphone 104 captures sound 182 . As described below, audio signals 170, 172 produced by microphones 102, 104, respectively, may be used to perform orientation processing at device 110, device 120, or both, if the location and orientation of microphones 102, 104 are known. That is, device 110 may use audio signals 170, 172 to determine the location of source 180, determine the direction of arrival of sound 182, spatially filter audio corresponding to sound 182, and the like. As described further below, device 110 may provide results of targeted processing (eg, material associated with targeted processing) to device 120 for high-complexity processing, and vice versa.

The device 110 includes a first input interface 111 , a second input interface 112 , a memory 114 , one or more processors 116 and a modem 118 . The first input interface 111 is coupled to one or more processors 116 and is configured to be coupled to the first microphone 102 . The first input interface 111 is configured to receive an audio signal 170 (eg, the first microphone output) from the first microphone 102 and provide the audio signal 170 to the processor 116 as an audio frame 174 . The second input interface 112 is coupled to one or more processors 116 and is configured to be coupled to the second microphone 104 . The second input interface 112 is configured to receive an audio signal 172 (eg, the second microphone output) from the second microphone 104 and provide the audio signal 172 to the processor 116 as an audio frame 176 . The audio frames 174, 176 may also be referred to as audio data 178 herein.

The one or more processors 116 optionally include a direction of arrival processing unit 132, an audio event processing unit 134, an acoustic environment processing unit 136, a beamforming unit 138, or a combination thereof. According to one embodiment, one or more elements of one or more processors 116 may be implemented using dedicated circuitry. As non-limiting examples, one or more elements of one or more processors 116 may be implemented using a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. According to another embodiment, one or more elements of one or more processors 116 may be implemented by executing instructions 115 stored in memory 114 . For example, memory 114 may be a non-transitory computer-readable medium storing instructions 115 executable by one or more processors 116 to perform the operations described herein.

The direction of arrival processing unit 132 may be configured to process the plurality of audio signals 170 , 172 to generate direction of arrival information 142 corresponding to a source 180 of a sound 182 represented by the audio signals 170 , 172 . To illustrate, direction of arrival processing unit 132 may select audio frames 174 , 176 generated from audio signals 170 , 172 from each microphone 102 , 104 representing similar sounds, such as sound 182 from source 180 . For example, direction of arrival processing unit 132 may process audio frames 174 , 176 to compare sound characteristics and ensure that audio frames 174 , 176 represent the same instance of sound 182 . In an illustrative, non-limiting example of direction-of-arrival processing, in response to determining that audio frames 174, 176 represent the same instance of sound 182, direction-of-arrival processing unit 132 may compare the timestamps of each audio frame 174, 176 to determine Which microphone 102 , 104 captures the corresponding instance of sound 182 first. If the audio frame 174 has an earlier time stamp than the audio frame 176 , the direction of arrival processing unit 132 may generate the direction of arrival information 142 indicating that the source 180 is closer to the first microphone 102 . If the audio frame 176 has an earlier time stamp than the audio frame 174 , the direction of arrival processing unit 132 may generate the direction of arrival information 142 indicating that the source 180 is closer to the second microphone 104 . Thus, based on the timestamps of similar audio frames 174 , 176 , the direction of arrival processing unit 132 can locate the sound 182 and the corresponding source 180 . Timestamps of audio frames from additional microphones can be used to improve localization in a similar manner to that described above.

In some implementations, one or more other techniques may be used to determine direction of arrival information 142 instead of or in addition to the time difference as described above, such as measuring each microphone in the microphone array of device 110 (e.g., microphone 102 and 104) the phase difference of the received sound 182. In some implementations, microphones 102, 104, 106, and 108 may operate in conjunction with device 120 as a distributed microphone array, and direction-of-arrival information 142 is based on the characteristics of the sound from each of microphones 102, 104, 106, and 108 (such as time of arrival or phase) and is generated based on the relative positions and orientations of the microphones 102 , 104 , 106 and 108 . In such embodiments, information about sound characteristics (e.g., phase information, time information, or both), captured audio data (e.g., at least a portion of audio signals 170, 172), or a combination thereof may be recorded at device 110 To and from device 120 for direction-of-arrival detection using a distributed microphone array.

Arrival direction information 142 may be sent to device 120 . For example, modem 118 may send data based on direction of arrival information 142 to device 120 . In some examples, generating direction-of-arrival information 142 at device 110 corresponds to performing low-complexity processing operations. The device 120 may use the direction of arrival information 142 to perform high-complexity processing operations. For example, in some implementations, device 110 may be a resource-constrained device, such as a device with limited battery life, limited memory capacity, or limited processing capability relative to device 120 . Performing high-complexity processing operations at device 120 may offload resource-intensive operations from device 110 .

To illustrate, device 120 may optionally include one or more sensors 129 . As non-limiting examples, sensors 129 may include non-audio sensors, such as 360-degree cameras, lidar sensors, and the like. Based on the direction of arrival information 142, the device 120 may command a 360-degree camera to focus on the source 180, command a lidar sensor to measure the distance between the user of the device 110, 120 and the source 180, and so on.

The audio event processing unit 134 may be configured to process a plurality of audio signals 170, 172 to perform audio event detection. To illustrate, the audio event processing unit 134 may process the sound characteristics of the audio frames 174, 176 and compare the sound characteristics to a plurality of audio event models to determine whether an audio event has occurred. For example, the audio event processing unit 134 may access a database (not shown) that includes models for different audio events, such as car horns, train horns, pedestrian talking, and the like. In response to the sound characteristic matching (or substantially matching) the particular model, audio event processing unit 134 may generate audio event information 144 indicating that sound 182 represents an audio event associated with the particular model. As used herein, the sound characteristics of an audio frame may "match" a particular sound model if the pitch and frequency components of the audio frame are within thresholds of the pitch and frequency components of the particular sound model.

In some embodiments, audio event processing unit 134 includes one or more classifiers configured to process audio signal data (such as audio signals 170, 172, audio characteristics of audio frames 174, 176) , based on the beamforming data of the audio signals 170, 172 or a combination thereof) to determine an associated class from among a plurality of classes supported by one or more classifiers. In one example, one or more classifiers operate in conjunction with the plurality of audio event models described above to determine the class (e.g., classification, such as "dog barking", "glass breaking", "baby crying", etc.). For example, one or more classifiers may include a neural network that has been trained using labeled sound data to distinguish sounds corresponding to various categories, and configured to process audio signal data to determine A specific category of sound represented by the signal data (or for each category, the probability that the sound belongs to that category is determined). The categories may correspond to or be included in the audio event information 144 . An example of a device 110 including one or more classifiers is described in more detail with reference to FIG. 6 .

In some implementations, audio event processing unit 134 includes one or more encoders configured to process audio signal data (such as audio signals 170, 172, audio characteristics of audio frames 174, 176) , based on the beamforming data of the audio signals 170, 172 or a combination thereof) to generate a signature of the sound represented by the audio signal data. For example, an encoder may include one or more neural networks configured to process audio signal data to generate embeddings corresponding to particular sounds in the audio signal data and associated with audio events. An "embedding" may specify a relatively low-dimensional space represented by a vector (e.g., an ordered sequence of values or a collection of indexed values) into which a higher-dimensional vector can be transformed and that preserves semantic relationships. To illustrate, an audio signal may be represented using a sequence of relatively large vectors (eg, representing spectral data and other audio features), which may be processed to produce embeddings represented by smaller vectors. The embedding may include enough information to be able to detect specific sounds in the audio signal. A signature (eg, embedded) may correspond to or be included in the audio event information 144 . An example of a device 110 including one or more encoders is described in more detail with reference to FIG. 7 .

In a non-limiting example, an audio event may correspond to the sound of an approaching vehicle (eg, source 180 ). Based on the audio event, the audio event processing unit 134 can generate audio event information 144 , and the audio event information 144 can be sent to the device 120 . For example, modem 118 may send data corresponding to the detected event to device 120 . In some examples, generating audio event information 144 at device 110 corresponds to performing a low-complexity processing operation. Device 120 may use audio event information 144 to perform high-complexity processing operations. To illustrate, based on the audio event information 144, the device 120 may perform one or more operations, such as processing the audio data at a larger, more accurate classifier to validate the audio event, editing the audio scene based on the sound signature (e.g., to remove sound corresponding to an embedding included in the audio event information 144 or remove a sound that does not correspond to an embedding), command the 360-degree camera to focus on the source 180, command the user of the lidar sensor measurement device 110, 120 Distance from source 180, etc.

The acoustic environment processing unit 136 may be configured to process the plurality of audio signals 170 , 172 to perform acoustic environment detection. To illustrate, the acoustic environment processing unit 136 may process the sound characteristics of the audio frames 174, 176 to determine the acoustic characteristics of the surrounding environment. As a non-limiting example, the acoustic properties may include a Direct-to-Reverberant Ratio (DRR) estimate of the surrounding environment. The acoustic environment processing unit 136 can generate the environment information 146 based on the acoustic characteristics of the surrounding environment. For example, environmental information 146 may indicate that device 110 is in an indoor environment if the DRR estimate is relatively high. However, if the DRR estimate is relatively low, environmental information 146 may indicate that device 110 is in an outdoor environment. In some implementations, the acoustic environment processing unit 136 may include or be implemented as one or more classifiers configured to generate an indication that may correspond to the environment information 146 or may be included in the environment information 146. 146 in the output of the audio environment category.

Environmental information 146 may be sent to device 120 . For example, modem 118 may transmit data corresponding to (eg, identifying) the detected environment to device 120 . In some examples, generating contextual information 146 at device 110 corresponds to performing low-complexity processing operations. Device 120 may use context information 146 to perform high-complexity processing operations. To illustrate, as an illustrative and non-limiting example, based on environmental information 146, device 120 may perform one or more operations, such as removing environmental or background noise from one or more audio signals, editing audio based on environmental information 146 scene, or change the 360 camera settings to capture outdoor images instead of indoor images.

The beamforming unit 138 may be configured to process the plurality of audio signals 170, 172 to perform beamforming. In some examples, the beamforming unit 138 performs beamforming based on the direction of arrival information 142 . Alternatively or additionally, in some examples, the beamforming unit 138 performs adaptive beamforming that utilizes a multi-channel signal processing algorithm to spatially filter the audio signals 170 , 172 and determine the location of the source 180 . The beamforming unit 138 may direct the increased sensitivity beam at the location of the source 180 and suppress audio signals from other locations. In some examples, beamforming unit 138 is configured to (eg, by introducing time or phase delays, adjusting signal amplitudes, or both based on the different sound propagation paths from source 180 to each of the different microphones 102, 104) The processing of audio signal 170 is adjusted relative to audio signal 172 to emphasize (eg, via constructive interference) sounds arriving from the direction of source 180 and attenuate sounds arriving from one or more other directions. In some examples, if beamforming unit 138 determines that source 180 is located close to first microphone 102, beamforming unit 138 may send a signal to change the orientation or direction of first microphone 102 to pick up sound 182 and make sound 182 from other directions ( For example, the direction associated with the second microphone 104) sounds a null (null) command.

The resulting one or more beamformed audio signals 148 (eg, representations of audio signals 170 , 172 ) may be sent to device 120 . For example, modem 118 may send one or more beamformed audio signals 148 to device 120 . In particular embodiments, a single beamformed audio signal 148 is provided to device 120 for each audio source of interest. In some examples, generating beamformed audio signal 148 at device 110 corresponds to performing a low-complexity processing operation. Device 120 may use beamformed audio signals 148 to perform high-complexity processing operations. In an illustrative example, based on beamforming audio signal 148 , device 120 may command a 360-degree camera to focus on source 180 , command a radar sensor to measure the distance between a user of device 110 , 120 and source 180 , and so forth.

Optionally, device 110 may send at least a portion of the audio data (e.g., audio signals 170, 172) captured by microphones 102, 104 to device 120 for distributed audio processing, or use a larger Processing, memory, and power resources perform additional processing, and in this distributed audio processing, a portion of the processing described as being performed by device 110 is offloaded to device 120 . As an example, in some implementations, device 110 may send at least a portion of audio signals 170, 172 (e.g., audio data 178) to device 120 for higher accuracy direction-of-arrival processing, higher accuracy audio event detection detection, higher-accuracy environmental detection, or a combination thereof. In some implementations, the device 110 may transmit at least a portion of the audio signal 170 , 172 (eg, audio material 178 ) to the device 120 instead of, or in addition to, the beamformed audio signal 148 .

Optionally, device 110 may include or be coupled to user peripherals, such as visual user peripherals (eg, a display such as that shown in FIG. 25 or a holographic projection unit such as that shown in FIG. 26 as non-limiting examples), Audio user peripherals (e.g., such as a speaker with reference to FIG. 3 or a voice user interface such as with reference to FIG. 5, as non-limiting examples), or tactile user peripherals (such as described with reference to FIG. sex example). The one or more processors 116 may be configured to provide a user interface output indicative of at least one of an environmental event or an acoustic event to a user peripheral. To illustrate, the UI output may cause the UI device to provide detections (such as based on audio event information 144, audio event information 145 received from device 120, ambient information 146, ambient information 147 received from device 120, or a combination thereof). Notification of incoming audio events or environmental conditions.

The various techniques described above illustrate that a device 110 (eg, a low-power device) performs directional context-aware processing. That is, the device 110 processes the audio signals 170, 172 from the plurality of microphones 102, 104 to determine the direction from which the sound 182 originates. In a particular embodiment, device 110 corresponds to a headset and device 120 corresponds to a mobile phone. In this embodiment, the headset performs directional context-aware processing, and the resulting data can be sent to the mobile phone to perform additional high-complexity processing. In other embodiments, device 110 corresponds to one or more other devices having less computing power than device 120 (eg, mobile phone, tablet device, personal computer, server, vehicle, etc.), such as a headset headsets (for example, virtual reality headsets, mixed reality headsets, or augmented reality headsets), glasses (for example, augmented reality glasses or mixed reality glasses), "smart watch" devices, virtual assistant devices, or Internet of Things devices.

As described below, device 120 (eg, a mobile phone) may also perform directional context-aware processing based on audio signals 170 , 172 received from device 110 , based on audio signals 190 , 192 from microphones 106 , 108 , or a combination thereof. Device 120 may provide the results of the directional context-aware processing to device 110 (eg, a headset) so that device 110 may perform additional operations, such as the audio zoom operation described in more detail with reference to FIG. 3 .

Device 120 includes memory 124 , one or more processors 126 and a data engine 128 . Optionally, the device 120 further includes one or more of the first input interface 121 , the second input interface 122 and one or more sensors 129 .

In some implementations, both the first input interface 121 and the second input interface 122 are coupled to one or more processors 126 and are configured to be coupled to the third microphone 106 and the fourth microphone 108, respectively. The first input interface 121 is configured to receive an audio signal 190 from the third microphone 106 and provide the audio signal 190 , such as an audio frame 194 , to the one or more processors 126 . The second input interface 122 is configured to receive an audio signal 192 from the fourth microphone 108 and provide the audio signal 192 , such as an audio frame 196 , to the one or more processors 126 . Audio signals 190 , 192 (eg, audio frames 194 , 196 ) may be referred to as audio data 198 provided to one or more processors 126 .

The one or more processors 126 optionally include a direction of arrival processing unit 152, an audio event processing unit 154, an acoustic environment processing unit 156, a beamforming unit 158, or a combination thereof. According to some implementations, one or more elements of the one or more processors 126 may be implemented using dedicated circuitry. As non-limiting examples, one or more elements of one or more processors 126 may be implemented using FPGAs, ASICs, or the like. According to another embodiment, one or more elements of one or more processors 126 may be implemented by executing instructions 125 stored in memory 124 . For example, memory 124 may be a non-transitory computer-readable medium storing instructions 124 executable by one or more processors 126 to perform the operations described herein.

The direction of arrival processing unit 152 may be configured to process a plurality of audio signals (eg, two or more of the audio signals 170, 172, 190, or 192) to generate a source 180 of a sound 182 represented by the plurality of audio signals. Corresponding arrival direction information 143 . To illustrate, direction of arrival processing unit 152 may be configured to process multiple audio signals using one or more techniques described with reference to direction of arrival processing unit 132 (eg, time of arrival, phase difference, etc.). Compared with the direction of arrival processing unit 132, the direction of arrival processing unit 152 may have more powerful processing capabilities and thus may generate more accurate results.

In some embodiments, the audio signals 170, 172 are received from the device 110, and the direction-of-arrival processing unit 152 may process the audio signals 170, 172 to determine the direction-of-arrival information 143 without processing the audio signal 190 at the direction-of-arrival processing unit 152 , 192. For example, one or more of microphones 106, 108 may be blocked or otherwise unable to produce a useful representation of sound 182, such as when device 120 is a mobile device being carried in a user's pocket or bag.

In other embodiments, the audio signals 190, 192 are received from the microphones 106, 108 and processed at the direction of arrival processing unit 152 to determine the direction of arrival information 143 without processing the audio signals 170, 192 at the direction of arrival processing unit 152. 172. For example, audio signals 170 , 172 may not be sent by device 110 or may not be received by device 120 . In another example, the audio signals 170, 172 may be of low quality (such as due to a large amount of noise (e.g., wind noise) at the microphones 102, 104), and the device 120 may choose to use the audio signals 190, 192 and ignore Audio signals 170,172.

In some embodiments, the audio signals 170 , 172 are received from the device 110 and used in combination with the audio signals 190 , 192 at the direction-of-arrival processing unit 152 to generate the direction-of-arrival information 143 . To illustrate, device 110 may correspond to a sensor having one or more sensors, such as a position or positioning sensor (eg, a Global Positioning System (GPS) receiver), that tracks one or more of the orientation, movement, or acceleration of device 110 . Multiple (e.g. head tracker profiles) inertial measurement units (IMUs) or combinations thereof) headsets. Device 120 may also include one or more position or positioning sensors (e.g., a GPS receiver) and an IMU to enable device 120 to decide to operate as a distributed microphone array in conjunction with head tracker data received from device 110 The absolute or relative positions and orientations of the microphones 102, 104, 106 and 108. Direction of arrival information 142, direction of arrival information 143, or both may be relative to the frame of reference of device 110, relative to the frame of reference of device 120, relative to an absolute frame of reference, or a combination thereof, and may be generated by device 110, device 120, or both in Translate appropriately between various reference frames.

Arrival direction information 143 may be sent to device 110 . For example, modem 128 may send data to device 110 based on direction of arrival information 143 . Device 110 may use direction of arrival information 143 to perform audio operations, such as audio zoom operations. For example, one or more processors 116 may send commands to retrieve (or focus) audio from source 180 and the direction of sound 182 .

The audio event processing unit 154 may be configured to process a plurality of audio signals to perform audio event detection, and generate audio event information 145 corresponding to one or more detected audio events. To illustrate, in embodiments where audio signals 170, 172 are received at device 120, audio event processing unit 154 may process the sound characteristics of audio signals 170, 172 (e.g., audio frames 174, 176) and Compare with a plurality of audio event models to determine whether an audio event has occurred. In some implementations where audio signals 190, 192 are received at device 120, audio event processing unit 154 may process the sound characteristics of audio signals 190, 192 (e.g., audio frames 194, 196) and compare the sound characteristics with complex Audio event models are compared to detect audio events. In some implementations where the beamformed audio signal 148 is received, the audio event processing unit 154 may process the acoustic characteristics of the beamformed audio signal 148 to detect the audio event. In some implementations where the beamforming unit 158 generates the beamformed audio signal 149, the audio event processing unit 154 may process the acoustic characteristics of the beamformed audio signal 149 to detect audio events.

The audio event processing unit 154 may access a database (not shown) that includes models for different audio events, such as car horns, train horns, pedestrian talking, and the like. In response to the sound characteristic matching (or substantially matching) the particular model, audio event processing unit 154 may generate audio event information 145 indicating that sound 182 represents an audio event associated with the particular model. In some implementations, audio event processing unit 154 includes one or more classifiers configured to determine the class of an audio event in a manner similar to that described for audio event processing unit 134 . However, compared to audio event processing unit 134, audio event processing unit 154 can perform more complex operations, can support a much larger set of models or audio classes than audio event processing unit 134, and can generate 134 More accurate audio event determination (or classification).

In some examples, the audio event processing unit 134 is a relatively low power detector configured with a relatively high sensitivity, which reduces the probability of an audio event not being detected and may also result in an increased number of false alarms ( For example, to detect an audio event when no audio event actually occurs). The audio event processing unit 154 can use the information received from the device 110 to provide higher audio event detection accuracy, and can process the corresponding audio signal (for example, one of the audio signals 170, 172, 190, 192 or Multiple, beamformed audio signals 148 , 149 one or more or a combination thereof) to verify (eg, classify) audio events received from the audio event processing unit 134 .

Audio event information 145 may be sent to device 110 . For example, modem 128 may send data corresponding to the detected event to device 110 . Device 110 may use audio event information 145 to perform audio operations, such as audio zoom operations. For example, one or more processors 116 may send a command to capture (or focus) sound from an audio event. In another example, audio event information 145 may cause one or more processors 116 to ignore (eg, not focus on) or attenuate or remove sounds from the audio event. For example, audio event processing unit 154 may determine that the audio event corresponds to the buzzing of a fly near device 110, and audio event information 145 may instruct device 110 to ignore the buzzing sound or to direct a null beam in the direction of the source of the buzzing sound . In embodiments where device 110 chooses whether to replay ambient sounds to the user of device 110, such as when device 110 is a headset configured to enter "transparency" mode to enable the user to hear external sounds in certain circumstances, the audio event Information 145 may indicate to device 110 whether sound 182 should trigger device 110 to transition to transparency mode.

The acoustic environment processing unit 156 may be configured to process the plurality of audio signals 170 , 172 , the plurality of audio signals 190 , 192 or a combination thereof to perform acoustic environment detection. To illustrate, the acoustic environment processing unit 156 may process the sound characteristics of the audio frames 174, 176, the audio frames 194, 196, or both to determine the acoustic characteristics of the surrounding environment. In some embodiments, the acoustic environment processing unit 156 functions in a similar manner as the acoustic environment processing unit 136 . However, the AEP unit 156 can perform more complex operations than the AEP unit 136, can support a much larger set of models or audio environment classes than the AEP unit 136, and can generate Unit 136 for more accurate acoustic environment determination (or classification).

In some examples, acoustic environment processing unit 136 is configured to have a relatively high sensitivity to environmental changes compared to acoustic environment processing unit 156 (e.g., as a non-limiting example, when device 110 moves from an indoor environment to an outdoor environment Or relatively low power detectors that detect changes in the characteristics of background sound when moving from an outdoor environment to a vehicle) but may have relatively low accuracy in determining the environment itself. The acoustic environment processing unit 156 can use the information received from the device 110 to provide higher acoustic environment detection accuracy, and can process the corresponding audio signal (for example, one of the audio signals 170, 172, 190, 192 or Multiple, beamformed audio signals 148 , 149 one or more or a combination thereof) to verify the environment information 146 received from the acoustic environment processing unit 136 (eg, classification).

The acoustic environment processing unit 156 can generate the environment information 147 based on the acoustic characteristics of the surrounding environment. Environmental information 147 may be sent to device 110 . For example, modem 128 may send data corresponding to the detected environment to device 110 . Device 110 may use environmental information 147 to perform additional audio operations.

The beamforming unit 158 may be configured to process the plurality of audio signals 170, 172 to perform adaptive beamforming. To illustrate, in some examples, the beamforming unit 158 spatially filters the audio signals 170, 172 using a multi-channel signal processing algorithm to direct a beam of increased sensitivity to the location of the source 180 in the same manner as for the beamforming unit 138. Audio signals from other locations are suppressed in a similar manner to that described. In another example, the beamforming unit 158 spatially filters the audio signals 190 , 192 using a multi-channel signal processing algorithm to direct a beam of increased sensitivity to the location of the source 180 . In another example where device 120 receives audio signals 170 , 172 from device 110 and also receives audio signals 190 , 192 , beamforming unit 158 may perform spatial filtering based on all audio signals 170 , 172 , 190 , and 192 . In some embodiments, the beamforming unit 158 generates a single beamformed audio signal for each sound source detected in the audio signal. For example, if a single sound source is detected, a single beamformed audio signal 149 directed towards that sound source is generated. In another example, if multiple sound sources are detected, multiple beamformed audio signals 149 may be generated, wherein each of the multiple beamformed audio signals 149 is directed to a corresponding one of the sound sources.

The resulting beamformed audio signal 149 may be sent to device 110 . For example, modem 128 may transmit one or more beamformed audio signals 149 to device 110 . Device 110 may use beamformed audio signal 149 to replay the improved audio.

While various elements of device 110 and device 120 have been shown and described above, it should be understood that in other implementations, one or more elements may be omitted or bypassed. It should also be appreciated that various combinations of elements of device 110, device 120, or both enable interoperability that enhances the performance of device 110, device 120, or both, such as described in the non-limiting examples listed below.

In a particular embodiment, device 110 includes audio event processing unit 134 and omits (or disables or bypasses its operation) direction of arrival processing unit 132 , acoustic environment processing unit 136 , and beamforming unit 138 . In this embodiment, the audio event information 144 may be provided to the device 120 as previously described and communicated with the device 120 using the audio signals 170 , 172 , using the audio signals 190 , 192 , or using the audio signals 170 , 172 , 190 , 192 Combination of processing combined use.

In another particular embodiment, the device 110 includes the audio event processing unit 134 and the direction of arrival processing unit 132, and the acoustic environment processing unit 136 and the beamforming unit 138 are omitted (or disabled or bypassed for their operation). In this embodiment, direction of arrival information 142 and audio event information 144 are generated at device 110 and may be provided to device 120 for use as previously described. Direction of arrival information 142 may be used to enhance (eg, via increased accuracy, reduced latency, or both) audio event detection that may be performed at audio event processing unit 134 , audio event processing unit 154 , or both. For example, direction of arrival information 142 may be provided as input to audio event processing unit 134, and audio event processing unit 134 may associate direction of arrival information 142 with a direction associated with one or more previously detected audio events or sound sources Compare. In another example, the audio event processing unit 134 may use the direction of arrival information 142 to increase or decrease the probability of detecting a particular audio event. To illustrate, since sounds originating above the user are more likely to be from birds or airplanes than cars, as an illustrative and non-limiting example, a weighting factor may be applied to reduce overhead sounds determined to match car-based audio events The probability. Additionally or alternatively, the direction of arrival information 142 may be used to enhance the performance of the audio event processing unit 154 in a manner similar to that described for the audio event processing unit 134 .

As further explained with reference to FIG. 9 , the performance of the audio event processing unit 154 may be enhanced by providing audio event information 144 (eg, the audio class detected by the audio event processing unit 134 ) as input to the audio event processing unit 154 . For example, audio event information 144 may be used as a starting point for an event model database search, or as an input that may affect a classification operation performed by a neural network based audio event classifier. Thus, by using the direction of arrival information 142 at the audio event processing unit 134 to improve the accuracy of the audio event information 144 , the improved accuracy of the audio event information 144 may also improve the performance of the audio event processing unit 154 .

In some implementations where the device 110 also includes an acoustic environment processing unit 136, the environment information 146 may be used to improve the performance of the audio event processing unit 134, the audio event processing unit 154, or both. For example, because some audio events (for example, a car horn) are more likely to occur in some environments (for example, on a busy street or in traffic) than others (for example, in an office), audio event processing Unit 134 may adjust operation based on circumstances. For example, audio event processing unit 134 may prioritize searching for acoustic event models that are more likely to occur in a particular environment, which may result in increased accuracy, decreased latency, or both. As another example, audio event processing unit 134 may adjust weighting factors for one or more sound event models based on circumstances to increase or decrease the likelihood that sound 182 is determined to match those sound event models. In some implementations, environmental information 146 may be sent to device 120 and used to improve the performance of audio event processing unit 154 in a similar manner.

In some implementations where device 110 includes beamforming unit 138, beamforming audio signal 148 may be used to improve the operation of audio event processing unit 134, audio event processing unit 154, or both. For example, beamforming audio signal 148 may be directed at source 180 of sound 182, thereby enhancing sound 182, attenuating or removing sound from other sources or ambient noise, or a combination thereof. As a result, in embodiments where audio event processing unit 134 operates on beamformed audio signal 148, beamformed audio signal 148 may provide an improved representation of sound 182 compared to audio signals 170, 172, which allows audio event processing unit 134 Audio event information 144 can be determined more accurately (eg, by reducing the likelihood of misclassification of sound 182 ). Similarly, in embodiments where beamformed audio signals 148 are sent to device 120 and audio event processing unit 154 operates on beamformed audio signals 148 , beamforming audio signals 148 enables improved performance of audio event processing unit 154 .

In a particular embodiment, device 120 includes audio event processing unit 154 and omits (or disables or bypasses its operation) direction of arrival processing unit 152 , acoustic environment processing unit 156 , and beamforming unit 158 . In this embodiment, the audio event processing unit 154 may operate using the audio signals 170, 172, using the beamforming audio signals 148, using the audio signals 190, 192, or a combination thereof, as previously described.

In another particular embodiment, the device 120 includes the audio event processing unit 154 and the direction of arrival processing unit 152, and the acoustic environment processing unit 156 and the beamforming unit 158 are omitted (or disabled or bypassed for their operation). In this embodiment, direction of arrival information 143 and audio event information 145 are generated at device 120 and may be provided to device 110 for use as previously described. Direction of arrival information 143 may be used to enhance (e.g., via increased accuracy, reduced latency, or both) audio events that may be performed at audio event processing unit 154 in a manner similar to that described for direction of arrival information 142 detection.

In some embodiments where device 120 also includes acoustic environment processing unit 156, environment information 147 may be used to improve audio event processing unit 134, audio event processing unit 154, or both in a manner similar to that described for environment information 146. the efficacy of the In some embodiments where device 120 includes beamforming unit 158, the beamforming audio signals generated by beamforming unit 158 may be used to improve audio event processing unit 154 in a manner similar to that described for beamforming audio signal 148. operation.

The techniques described with reference to FIG. 1 enable each device 110, 120 to perform directional context-aware processing based on audio signals 170, 172 produced by microphones 102, 104, audio signals 190, 192 produced by microphones 106, 108, or a combination thereof. As a result, each device 110, 120 is able to detect context for different use cases and to determine characteristics associated with the surrounding environment. As non-limiting examples, these techniques enable each device 110, 120 to distinguish between one or more moving sound sources (e.g., sirens, birds, etc.), one or more stationary sound sources (e.g., televisions, speakers, etc.), or its combination.

It should be appreciated that the technique described with respect to FIG. 1 can implement multi-channel or single-channel audio context detection to distinguish different sounds based on direction of arrival. According to one embodiment, the microphones 102, 104, 106, and 108 may be included in a microphone array having microphones located at different locations in a building, such as a house. In the scenario of someone falling on the floor, if the microphones of the microphone array are connected to a mobile device such as device 120, using the techniques described herein, the mobile device can use the direction of arrival information to determine where the sound is coming from, to determine the context of the sound , and perform the appropriate action (eg, notify the nursing staff).

Referring to FIG. 2 , another particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 200 . System 200 includes one or more processors 202 . One or more processors 202 may be integrated into device 110 or device 120 . For example, one or more processors 202 may correspond to one or more processors 116 or one or more processors 126 .

The one or more processors 202 optionally include an audio input 204 configured to receive audio data 278 (such as audio data 178 of FIG. 1 ) and output audio frames 274 , 276 . The one or more processors 202 include a first processing domain 210 and a second processing domain 220 . The first processing domain 210 may correspond to a low power domain operating in a low power state, such as an "always on" power domain. The first processing domain 210 can remain active to process the audio frame 274 and the audio frame 276 . In some implementations, audio frames 274 and 276 correspond to audio frames 174 and 176, respectively. In another embodiment, audio frames 274 and 276 correspond to audio frames 194 and 196, respectively. The second processing domain 220 may correspond to a high power domain that transitions between an idle state and a high power state.

The first processing domain 210 includes an audio preprocessing unit 230 . Compared with one or more components in the second processing domain 220, the audio pre-processing unit 230 may consume relatively low power. The audio pre-processing unit 230 can process the audio frames 274, 276 to determine whether there is any audio activity. According to some embodiments, the audio pre-processing unit 230 may receive and process audio frames from a single microphone to save additional power. For example, in some implementations, the audio frame 276 may not be provided to the first processing domain 210 , and the audio pre-processing unit 230 may determine whether there is audio activity in the audio frame 274 .

If the audio pre-processing unit 230 determines that there is audio activity in the audio frame 274 or both audio frames 274, 276, the audio pre-processing unit 230 may generate an enable signal 252 to switch the second processing domain 220 from the idle state to high power state. According to some embodiments, the audio pre-processing unit 230 may determine preliminary direction information 250 about the audio activity, and provide the preliminary direction information 250 to the second processing domain 220 . For example, if there is audio activity in audio frame 274, and there is less or no audio activity in audio frame 276, preliminary direction information 250 may indicate that sound 182 originates from the captured audio frame 276. 274 corresponding to the audio signal near the microphone.

The second processing domain 220 includes a direction of arrival processing unit 232 , an audio event processing unit 234 , an acoustic environment processing unit 236 , a beamforming unit 238 or a combination thereof. The direction of arrival processing unit 232 may correspond to the direction of arrival processing unit 132 of FIG. 1 or the direction of arrival processing unit 152 of FIG. 1 and may operate in a substantially similar manner. The audio event processing unit 234 may correspond to the audio event processing unit 134 of FIG. 1 or the audio event processing unit 154 of FIG. 1 and may operate in a substantially similar manner. The acoustic environment processing unit 236 may correspond to the acoustic environment processing unit 136 of FIG. 1 or the acoustic environment processing unit 156 of FIG. 1 and may operate in a substantially similar manner. Beamforming unit 238 may correspond to beamforming unit 138 of FIG. 1 or beamforming unit 158 of FIG. 1 and may operate in a substantially similar manner.

Accordingly, the second processing domain 220 may operate in different modes. For example, the second processing domain 220 may be used to enable different sensors, such as the sensor 129 of FIG. 1 . In addition, the second processing domain 220 can be used to perform direction of arrival processing and calculations, beamforming, DRR operations, indoor/outdoor detection, source distance determination, and the like.

The system 200 enables the first processing domain 210 to selectively activate the second processing domain 220 in response to detecting the presence of audio activity. As a result, battery can be saved at devices such as headsets or mobile phones by switching the second processing domain 220 (e.g., the high power processing domain) to an idle state when no audio activity is detected by using low power processing electricity.

Referring to FIG. 3 , another particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 300 . System 300 includes headset 310 and mobile phone 320 . Headset 310 may correspond to device 110 , and mobile phone 320 may correspond to device 120 .

The headset 310 includes an audio processing unit 330, an audio scaling unit 332, an optional user prompt generating unit 334, or a combination thereof. The audio processing unit 330 includes a direction of arrival processing unit 132 and an audio event processing unit 134 . As described with reference to FIG. 1 , the direction-of-arrival processing unit 132 may generate direction-of-arrival information 142 indicative of a location (eg, a heading) of a source 180 of the sound 182 . The direction of arrival information 142 is provided to the audio scaling unit 332 and the user prompt generating unit 334 . As described with reference to FIG. 1 , the audio event processing unit 134 may generate the audio event information 144 indicating that the sound 182 is related to a vehicle sound. The audio event information 144 is provided to the user prompt generation unit 334 .

The audio scaling unit 332 can also receive the direction of arrival information 143 from the mobile phone 320 . The audio scaling unit 332 may be configured to adjust the beamforming algorithm of the beamforming unit 138 based on the direction of arrival information 142 or the direction of arrival information 143 . As a result, audio scaling unit 332 may adjust the focus of microphones 102, 104 to the sound of interest (eg, sound 182) and attenuate sounds from other directions. Headphones 310 may thus generate beamformed audio signals 148 focused on sound 182 from source 180 and provide beamformed audio signals 148 to speakers 336 for playback. In some implementations, beamforming is performed at multiple speakers 336 (e.g., a left speaker for the user's left ear and a right speaker for the user's right ear) in a manner that preserves the directionality of the source 180 of the sound 182 The playback of the audio signal 148 allows the user to perceive the direction in which the focused sound 182 originates from the source 180 (or, where distance information is determined, originates from the location of the source 180).

User alert generation unit 334 may generate user alert 350 that is provided to speaker 336 for rebroadcast. For example, user alert 350 may be an audio signal indicating that a vehicle (eg, source 180 ) is approaching. The user alert generation unit 334 may also generate one or more user alerts 352 that are provided to the mobile phone 320 . User alert 350 may include text indicating an approaching vehicle, a vibration programmed to indicate an approaching vehicle, or the like.

Thus, the system 300 of FIG. 3 enables the headset 310 to focus (eg, audio zoom) on the sound of interest 182 and can generate user alerts 350 , 352 . To illustrate, in a scenario where the user is wearing the headset 310, the system 300 may alert the user to surrounding events, such as an approaching vehicle, that the user may not otherwise be aware of.

Referring to FIG. 4 , another particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 400 . System 400 includes headset 410 and mobile phone 420 . Headset 410 may correspond to device 110 , and mobile phone 420 may correspond to device 120 .

The headset 410 includes an audio processing unit 430, and optionally includes an audio scaling unit 432, a noise canceling unit 434, one or more speakers 436, or a combination thereof. The audio processing unit 430 includes a direction of arrival processing unit 132 and an audio event processing unit 134 . As described with reference to FIG. 1 , direction of arrival processing unit 132 may generate direction of arrival information indicative of an approximate location of source 180 of sound 182 . The direction of arrival processing unit 132 may also generate direction of arrival information indicating the approximate location of the source 184 of the sound 186 . As described with reference to FIG. 1 , the audio event processing unit 134 may generate audio event information indicating that the sound 182 is related to a vehicle sound. The audio event processing unit 134 can also generate audio event information indicating that the sound 186 is related to human speech.

The audio processing unit 430 may be configured to generate first sound information 440 indicating the direction of arrival information associated with the sound 182 (eg, the first output of the direction of arrival processing unit 132 ), and indicating that the sound 182 is related to traffic Tool-dependent (eg, the first output of the audio event processing unit 134). The audio processing unit 430 may also be configured to generate second sound information 442 indicating the direction of arrival information associated with the sound 186 (eg, the second output of the direction of arrival processing unit 132 ), and indicating that the sound 186 is related to Human speech related (eg, the second output of the audio event processing unit 134). Optionally, headset 410 may transmit audio signal data, such as one or more portions of audio signal 170 , 172 corresponding to sound 182 , 186 , to mobile phone 420 . The audio signal data may be included in the audio information 440 , 442 or may be separate from the audio information 440 , 442 .

The mobile phone 420 includes a single microphone audio context detection unit 450 , an audio adjustment unit 452 and a mode controller 454 . The first sound information 440 and the second sound information 442 are provided to the audio adjustment unit 452 . According to some embodiments, the single-microphone audio context detection unit 450 may provide additional context information 496 to the audio adjustment unit 452, such as the direction of arrival information 143 generated by the direction of arrival processing unit 152 of FIG. The audio event information 145, the environment information 147 generated by the acoustic environment processing unit 156 or a combination thereof. For example, the single-microphone audio context detection unit 450 can process audio signal data received from the headset 410 (eg, one or more portions of the audio signal 170 , 172 ), audio signals received from one or more microphones of the mobile phone 420 data (eg, audio signals 190, 192) or a combination thereof.

The audio adjustment unit 452 may be configured to generate an audio scaling angle 460 and a noise reduction parameter 462 based on the audio information 440 , 442 from the audio processing unit 430 . That is, based on the context information 496 from the single-microphone audio context detection unit 450, the audio adjustment unit 452 can determine the audio zoom angle 460 to focus for beamforming purposes, and can determine the noise reduction parameters 462 to reduce noise from other direction noise. Thus, based on contextual information 496 , if audio adjustment unit 452 decides to prioritize focusing on sound 182 , audio zoom angle 460 may indicate an angle associated with source 180 , and noise reduction parameter 462 may include parameters for reducing noise from source 184 . parameters of the message. The audio scaling angle 460 is provided to the audio scaling unit 432 and the denoising parameters 462 are provided to the denoising unit 434 .

The audio adjustment unit 452 may also be configured to generate a mode signal 464 provided to the mode controller 454 . The mode signal 464 may indicate whether a vibration alert should be generated for the user of the mobile phone 420 , whether a text alert should be generated for the user of the mobile phone 420 , whether a voice alert should be generated for the user of the mobile phone 420 , and so on.

The audio scaling unit 432 may be configured to adjust a beamforming algorithm of a beamforming unit, such as the beamforming unit 138 of FIG. 1 , based on the audio scaling angle 460 . As a result, audio zoom unit 432 may adjust the focus of microphones 102, 104 to the sound of interest (eg, sound 182). Based on the noise reduction parameters 462 , the noise reduction unit 434 may be configured to generate a noise reduction signal 490 to attenuate sound 186 from other directions. The beamformed audio signal 148 and the noise-reduced signal 490 may be provided to one or more speakers 436 for playback.

The system 400 of FIG. 4 is capable of analyzing detected sound events and corresponding arrival directions to improve auditory experience. Based on the contextual information 496, the system 400 can determine which sounds are of particular interest to the user. For example, if the user is crossing the street, the system 400 may decide that the sound 182 of vehicles is more important than the sound 186 of people speaking. As a result, system 400 can focus on important sounds 182 and suppress other sounds.

Although headphone 410 is described as providing the focusing of sound 182 and the suppression of other sounds, it should be noted that the focus of sound 182 provided by audio scaling unit 432 and the suppression of other sounds provided by noise canceling unit 434 Each provides an enhanced perception of sound 182 to the user of earphone 410 . For example, in an embodiment in which the headset 410 includes the audio scaling unit 432 but omits the noise canceling unit 434 (or bypasses the operation of the noise canceling unit 434 ), even in the absence of the noise canceling signal 490 , the sound 182 is routed through the audio Zoom operations are enhanced. As another example, in an implementation in which the headset 410 includes the noise cancellation unit 434 but omits the audio scaling unit 432 (or bypasses the operation of the audio scaling unit 432), the sound 182 is relatively Other sounds are enhanced.

Referring to FIG. 5 , another particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 500 . The system 500 includes a spatial filter processing unit 502 , an audio event processing unit 504 , an API 506 and a voice user interface 508 . According to one embodiment, the system 500 may be integrated into the device 110 or the device 120 .

The spatial filtering processing unit 502 may be configured to perform one or more spatial filtering operations on audio frames (shown as audio frames 574 and 576 ) associated with the received audio signal. In some implementations, audio frames 574 and 576 correspond to audio frames 174 and 176, respectively. In another embodiment, audio frames 574 and 576 correspond to audio frames 194 and 196, respectively. In a non-limiting example, the spatial filtering processing unit 502 may perform adaptive beamforming on the audio frames 574, 576, perform audio scaling operations on the audio frames 574, 576, perform beamforming operations on the audio frames 574, 576, A zero beamforming operation or a combination thereof is performed on the audio frames 574, 576.

Based on the spatial filtering operation, the spatial filtering processing unit 502 may generate a plurality of outputs 510 , 512 , 514 and corresponding direction of arrival information 542 for each output 510 , 512 , 514 . In the illustrative example of FIG. 5 , spatial filter processing unit 502 may generate speech content output based on audio frames 574, 576 and two other outputs 512, 514 (e.g., audio from two other detected audio sources) 510. The outputs 510 , 512 , 514 are provided to the audio event processing unit 504 and the direction of arrival information 542 for each output 510 , 512 , 514 is provided to the API 506 .

The audio event processing unit 504 is configured to process each output 510 , 512 , 514 to determine audio event information 544 associated with the output 510 , 512 , 514 . For example, audio event processing unit 504 may indicate that output 510 is associated with speech content, output 512 is associated with non-speech content, and output 514 is associated with non-speech content. The audio event processing unit 504 provides audio content output 510 to the audio user interface 508 for user playback, and provides audio event information 544 to the API 506 .

API 506 may be configured to provide direction of arrival information 542 and audio event information 544 to other applications or devices for further specialized processing, as described with reference to FIGS. 1-4 .

FIG. 6 illustrates an embodiment 600 of the device 110 . The one or more processors 116 are configured to receive audio signals (shown as audio signals 170 , 172 ) from a plurality of microphones. The one or more processors 116 are also configured to send to the second device data based on a category 612 of sounds represented by one or more of the audio signals 170, 172 and associated with the audio event. For example, one or more processors 116 sends an indication 616 of category 612 to a second device (eg, device 120 ). In an illustrative example, one or more processors 116 are integrated into a headset device, and the second device corresponds to a mobile phone. In another illustrative example, one or more processors 116 are integrated in a vehicle.

One or more processors 116 are configured to process signal data 602 at one or more classifiers 610 to determine a class 612 from among a plurality of supported classes 614 supported by the one or more classifiers 610 . Signal data 602 corresponds to audio signals 170 , 172 . For example, in some embodiments, one or more processors are configured (e.g., at beamforming unit 138) to perform beamforming operations on audio signals 170, 172 to generate signal data 602, which may correspond to The audio signal 148 is beamformed. Alternatively or additionally, the one or more processors 116 are configured to determine one or more characteristics of the audio signal 170 , 172 to include in the signal data 602 . Alternatively or additionally, signal data 602 includes audio signals 170 , 172 .

According to some aspects, the one or more classifiers 610 include one or more neural networks configured to process the signal data 602 and generate an indication that the class 612 is more accurate than one of the plurality of supported classes 614 The remaining categories are outputs that are more closely associated with audio events (for example, one-shot outputs). Category 612 is sent to the second device via indication 616 . In some examples, indication 616 includes a bit configuration, quantity, or other indicator of category 612 . In other examples, indication 616 includes a textual name, label, or other descriptor that enables category 612 to be recognized by the second device. In some implementations, one or more classifiers 610 correspond to (or are included in) audio event processing unit 134 of FIG. Audio Event Information 144).

Optionally, one or more processors 116 are further configured to process the image data at one or more classifiers 610 to determine a class 612 . For example, device 110 may optionally include one or more cameras configured to generate image material, or may receive image material (eg, via a modem) from another device. Classes 612 may correspond to objects (eg, sound sources) represented by graphical material and associated with audio events. For example, in some implementations, one or more processors 116 may generate direction-of-arrival information 142 (or receive direction-of-arrival information 143 from a second device) based on audio signals 170, 172, and use direction-of-arrival information 142 or 143 to Objects corresponding to sound sources are located in the image data. In embodiments where the one or more classifiers 610 process image data in addition to audio data, the image data may be included in the signal data 602 or provided as a separate input to the one or more classifiers 610.

In some implementations, the plurality of supported categories 614 includes an "unknown" category, which indicates that the audio event fails to correspond to any other supported category 614 within a confidence threshold. In one example, one or more classifiers 610 calculate, for each of a plurality of supported categories 614, a probability that an audio event corresponds to that particular category. If none of the calculated probabilities exceed a threshold amount, one or more classifiers 610 designate class 612 as an "unknown" class.

In some implementations, the one or more processors 116 are configured to process the audio signals 170, 172 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals , and category 612 is associated with direction-of-arrival information. For example, the direction of arrival information and category 612 correspond to the same sound in the audio signals 170 , 172 . To illustrate, the one or more processors 116 may optionally include the direction of arrival processing unit 132 of FIG. 1 . The one or more processors 116 may be configured to send data based on the direction of arrival information to the second device. In one example, the data based on the direction of arrival information includes a report indicating at least one detected event and a direction of the detected event.

According to various embodiments, device 110 may optionally include one or more additional elements or aspects previously described with reference to FIG. 1 . For example, the one or more processors may be configured to perform spatial processing on the audio signal based on the direction of arrival information to generate one or more beamformed audio signals, and may transmit the one or more beamformed audio signals to the second device. To illustrate, one or more processors 116 may optionally include beamforming unit 138 of FIG. 1 . In another example, the one or more processors 116 may be configured to generate environment data corresponding to the detected environment based on the acoustic environment detection operation. To illustrate, the one or more processors 116 may optionally include the acoustic environment processing unit 136 of FIG. 1 .

In another example, the one or more processors 116 may be configured to send the representation of the audio signal 170, 172 to the second device. In some implementations, the representations of audio signals 170 , 172 correspond to one or more beamformed audio signals, such as beamformed audio signal 148 . In another example, the one or more processors 116 may be configured to receive direction information associated with the audio signal from the second device, and perform an audio zoom operation based on the direction information, as described with reference to FIGS. 3 and 4 .

By sending an indication 616 of a category 612 corresponding to a sound represented by an audio signal 170, 172, the device 110 provides information that can be used by the second device to improve the accuracy of audio event processing at the second device, as described in reference Figure 9 is further described.

FIG. 7 illustrates an embodiment 700 of the device 110 . Compared with the embodiment 600, the embodiment 700 includes one or more encoders 710 and omits one or more classifiers 610 . Signal data 602 is processed by one or more encoders 710 to generate embeddings 712 corresponding to sounds represented by one or more of audio signals 170, 172 and associated with audio events. The one or more processors 116 are also configured to send the embedding 712 based profile to the second device. In one example, the one or more processors 116 send the indication 716 of the embedding 712 to the second device.

According to some aspects, the one or more encoders 710 include one or more neural networks configured to process the signal data 602 to generate an embedding 712 of sound. Embedding 712 represents a "signature" of the sound, which includes enough information about various characteristics of the sound to enable the sound to be detected in other audio signals, but may not include enough information to be able to reproduce the sound from embedding 712 alone. According to some aspects, embedding 712 may correspond to the user's speech, specific sounds from the environment (such as barking, etc.), and embedding 712 may be used to detect and amplify or extract other sounds that may occur in other audio data. Example, as further described with reference to FIG. 11 . In some implementations, one or more encoders 710 correspond to (or are included in) audio event processing unit 134 of FIG. Audio Event Information 144).

In some embodiments, the one or more processors 116 are configured to process the audio signals 170, 172 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals, And embedding 712 is associated with direction-of-arrival information. In one example, the direction of arrival information and embedding 712 correspond to the same sound in the audio signal 170 , 172 . To illustrate, the one or more processors 116 may optionally include the direction of arrival processing unit 132 of FIG. 1 . The one or more processors 116 may be configured to send data based on the direction of arrival information to the second device.

Optionally, one or more processors 116 are also configured to process image data at one or more encoders 710 to generate embedding 712 . For example, device 110 may optionally include one or more cameras configured to generate image material, or may receive image material (eg, via a modem) from another device. Embedding 712 may correspond to an object (eg, a sound source) represented in graphical material and associated with an audio event. For example, in some implementations, one or more processors 116 may generate direction-of-arrival information 142 (or receive direction-of-arrival information 143 from a second device) based on audio signals 170, 172, and use direction-of-arrival information 142 or 143 to Objects corresponding to sound sources are located in the image data. In embodiments where one or more encoders 710 process image data in addition to audio data, the image data may be included in signal material 602 or provided as a separate input to one or more encoders 710.

FIG. 8 illustrates an embodiment 800 of an apparatus 110 that includes one or more classifiers 610 of FIG. 6 and further includes one or more encoders 710 of FIG. 7 . Signal data 602 (or one or more portions of signal data 602) is processed by one or more classifiers 610 to determine category 612, and signal data 602 (or one or more portions of signal data 602) is processed by one or more Encoder 710 processes to produce embedding 712 . The one or more processors 116 are also configured to send the profile based on category 612, embedding 712, or both to the second device. For example, indication 616 of category 612 , indication 716 of embedded 712 , or both may correspond to being sent to or included in audio event handling unit 134 of device 120 of FIG. 1 .

FIG. 9 illustrates an embodiment 900 of a device 120 (eg, a second device) including one or more processors 126 . The one or more processors 126 include an audio event processing unit 154 and are configured to receive an indication 902 of an audio category corresponding to an audio event from a first device (eg, device 110 ). In some examples, indication 902 corresponds to indication 616 of FIG. 6 or FIG. 8 , which indicates classes 612 detected at one or more classifiers 610 of device 110 . In some implementations, the one or more processors 126 are coupled to memory (eg, memory 124 ) and integrated into a mobile phone, and the first device corresponds to a headset device. In another embodiment, the memory and one or more processors 126 are integrated into the vehicle.

Optionally, one or more processors 126 include one or more classifiers 920 , which may correspond to or be included in the audio event processing unit 154 . According to one aspect, the one or more classifiers 920 are more powerful and more accurate than the classifier(s) in the first device that generated the indication 902, such as described with reference to the audio event processing unit 154 of FIG. 1 . One or more processors 126 may be configured to also receive audio data 904 representing sounds associated with the audio event. In some implementations, audio material 904 may correspond to audio signal 170, 172 from first device, beamformed audio signal 148 from first device, audio signal 190, 192, or combination. The one or more processors 126 may be configured to process the audio data 904 at the one or more classifiers 920 to verify the indication (such as by comparing the indication 902 with the classification 922 determined by the one or more classifiers 920). 902 is correct. The category 922 can be selected from a plurality of supported categories 924 as the audio category that best corresponds to the audio event detected in the audio data 904 .

In some implementations, validating the indication 902 or validating the class indicated by the indication 902 includes determining whether the class indicated by the indication 902 matches a class determined by one or more classifiers 920 (eg, classification 922 ). Alternatively or additionally, validating the indication 902 or the class indicated by the indication 902 includes deciding that the class determined by the one or more classifiers 920 is a particular instance or subclass of the class indicated by the indication 902 . For example, an indication 902 corresponding to the category "vehicle event" may be determined by one or more classifiers 920 that the categories 922 correspond to "car engine", "motorcycle engine", "brake sound", "car horn", "Motorcycle Horn", "Train Horn", "Vehicle Collision", etc. (which can be classified as different types of vehicle incidents) to verify.

According to some aspects, the accuracy of the one or more classifiers 920 is improved by providing the one or more classifiers 920 with other information related to the audio event in addition to the audio data 904 . For example, the one or more processors 126 may optionally be configured to provide the audio data 904 and the indication 902 of the audio class as input to the one or more classifiers 920 to determine the class 922 associated with the audio data 904 . In embodiment 900 , audio data 904 includes one or more beamformed signals 910 (eg, beamformed audio signals 148 ) that are input to one or more classifiers 920 . In another example, one or more processors 126 may optionally be configured to receive direction data 912 (eg, direction-of-arrival information 142 ) corresponding to a sound source from a first device, and combine audio data 904 , direction The data 912 and the indication 902 of the audio class are provided as input to one or more classifiers 920 to determine a class 922 associated with the audio data 904 .

Optionally, one or more processors 126 are configured to generate one or more outputs in addition to audio event information 145 instead of audio event information 145 or to generate one or more outputs for inclusion in audio event information 145 , such as a notification 930, a control signal 932, a classifier output 934, or a combination thereof. For example, in an embodiment where the audio category (eg, category 922 ) corresponds to a vehicle event (eg, collision), the one or more processors 126 may base the location of the first device (eg, device 110 ) on one or more The location of the third device sends a notification 930 of the vehicle event to one or more third devices, such as further described with reference to FIGS. 14 and 15 . In another example, a user of device 120 may be participating in an outdoor event, such as hiking along a trail, and the audio category (eg, classification 922 ) corresponds to a safety-related event, such as an animal growling. In this example, one or more processors 126 may send a notification 930 of a security-related event to one or more third devices determined to be nearby based on location profile associated with the one or more third devices (such as other hikers' phones or headsets).

In another example, the control signal 932 is sent to the first device based on the classifier output 934 . To illustrate, classifier output 934 may include a bit pattern, a numeric indicator, or a text label or description indicating the classification 922 determined by one or more classifiers 920 . In an illustrative example, control signal 932 instructs the first device to perform an audio zoom operation. In another example, the control signal 932 instructs the first device to perform spatial processing based on the direction of the sound source. In another example, the control signal 932 instructs the first device to change the mode of operation, such as transitioning from a media playback mode (e.g., playing streaming audio to the user of the first device) to a transparent mode (e.g., making the user of the first device Able to hear ambient sounds).

Optionally, the one or more processors 126 are configured to perform one or more operations associated with tracking the source of directional audio sounds in the audio scene, as further explained with reference to FIG. 16 . In one example, the one or more processors 126 may receive direction data 912 corresponding to sound sources detected by the first device. Based on the audio events, the one or more processors 126 may update the map of directional sound sources in the audio scene to generate an updated map. The one or more processors 126 may send data corresponding to the updated map to one or more third devices geographically remote from the first device. As an illustrative and non-limiting example, the updated map may be used by one or more third devices to notify users of the one or more third devices of sound sources detected near the first device, or to participate in shared Users of a virtual environment (eg, in a virtual conference room) provide a shared audio experience.

FIG. 10 illustrates another embodiment 1000 of the device 120 . Compared to the embodiment 900 of FIG. 9 , the audio event processing unit 154 (eg, one or more classifiers 920 ) receives as input the multi-channel audio signal 1002 instead of the beamformed signal 910 . For example, multi-channel audio signal 1002 may include audio signals 170, 172 received in audio data 904, audio signals 190, 192 received from microphones 106, 108, or a combination thereof. The multi-channel audio signal 1002 may be provided as input to one or more classifiers 920 in combination with the indication 902, the direction data 912, or both.

For example, beamforming data is not available in some situations, such as when audio events are detected but the directionality of the audio events cannot be determined with sufficient accuracy (e.g., the sound is mostly diffuse or non-directional, or masked by other sounds interfering with beamforming). An example of processing based on whether an audio signal or a beamformed signal is transmitted between devices is described with reference to FIGS. 12 and 13 .

FIG. 11 illustrates an embodiment 1100 of device 120 and a diagram 1150 representing audio processing that may be performed at device 120 . The one or more processors 126 include a content separator 1120 configured to separate foreground signals from background signals in the audio content based on embeddings corresponding to the audio signals.

Content separator 1120 may include an audio production network 1122 configured to receive one or more embeddings 1104 corresponding to one or more signatures of a particular sound. For example, one or more embeddings 1104 may correspond to or include embeddings 712 of FIG. 7 . In some examples, one or more embeddings 1104 may include a signature of one or more audio events, a signature of a particular person's voice, or the like. Audio generation network 1122 is also configured to receive audio data (shown as input mixed waveform 1102 ) which may include background and foreground sounds from various sound sources. The audio generation network 1122 is configured to determine whether the input mixed waveform 1102 includes any sounds corresponding to the one or more embeddings 1104, and to extract, isolate, or remove those particular sounds.

Target output 1106 is produced by content separator 1120 . Target output 1106 may include an audio signal corresponding to a particular sound. For example, specific sounds corresponding to one or more embeddings 1104 may be isolated from the remaining sounds in the input mixed waveform 1102 to produce the target output 1106 . In one example, a particular sound may correspond to a foreground sound in the input mixed waveform 1102, and the target output 1106 may include the foreground sound with the background removed or attenuated.

In another example, the target output 1106 corresponds to a modified version of the input mixing waveform 1102 and may include the sound represented by the input mixing waveform 1102 and remaining after a particular sound is removed (or attenuated). For example, a particular sound may correspond to a foreground sound in the input mixing waveform 1102, and the target output 1106 may include the background sound remaining in the input mixing waveform 1102 after the foreground sound is removed (or attenuated).

In another example, the target output 1106 may include an audio signal containing a particular sound as a foreground sound that has been removed from the background sound of the input mixing waveform 1102 and added to a different set of background sounds.

In illustration 1150 , a first foreground sound ( FG1 ) 1154 , a second foreground sound ( FG2 ) 1156 , and a third foreground sound ( FG3 ) are illustrated in an audio scene 1151 that includes a first ambiance 1152 (eg, background). 1158. A first embedding of the one or more embeddings 1104 for the first foreground sound 1154, a second embedding of the one or more embeddings 1104 for the second foreground sound 1156, and one or more of the embeddings 1104 are used by the content splitter 1120 A foreground extraction operation 1160 is performed for the third embedding of the third foreground sound 1158 to isolate the foreground sounds 1154 , 1156 , 1158 from the first ambience 1152 (shown as isolated foreground sound 1162 ). A scene generation operation 1164 adds foreground sounds 1154 , 1156 , 1158 to an audio scene 1171 with a second ambiance 1172 (eg, an updated audio scene). Scene generation operation 1164 may be performed by audio generation network 1122, content separator 1120, one or more processors 1126, or a combination thereof.

In one example, input mixed waveform 1102 represents audio data corresponding to audio scene 1151, which is processed by one or more processors 1126 to produce adjusted audio data (e.g., a target output including isolated foreground sound 1162 1106 ), and the adjusted data is again adjusted by one or more processors 1126 (eg, scene generation operation 1164 ) to generate an updated audio scene (eg, audio scene 1171 ). The audio scene 1171 may include directional information associated with various objects and audio events (eg, audio and events associated with other participants in the shared audio scene), as further described with reference to FIGS. 16-18 .

The content separator 1120, including the audio generation network 1122, can isolate any target sound from the background and is not limited to separating speech from noise. In some embodiments, single-microphone object separation of specific audio events, speech, etc. is achieved using the content separator 1120 of the audio production network 1122 and can overcome limitations associated with conventional techniques that cannot distinguish audio sources.

FIG. 12 illustrates a flowchart corresponding to a method 1200 that may be performed by a first device, such as device 110 (eg, one or more processors 116 ), for sending information to a second device, such as device 120 .

Method 1200 includes processing one or more frames of an audio signal at block 1202 . For example, audio data 178 (e.g., frames of audio signals 170, 172) may be processed at direction of arrival processing unit 132, audio event processing unit 134, acoustic environment processing unit 136, unit 138, or a combination thereof, as in FIG. mentioned.

Method 1200 includes determining, at block 1204, whether processing of one or more frames of an audio signal results in ambient detection. In some examples, environmental detection may include determining that an environmental change has been detected. In response to determining that environmental detection has occurred, method 1200 includes, at block 1206, sending environmental information to the second device. For example, device 110 sends environmental information 146 to device 120 .

In response to determining at block 1204 that no ambient detection occurred, or after sending the ambient information at block 1206, method 1200 includes determining at 1208 whether processing of one or more frames of the audio signal resulted in detection of an audio event. In response to determining that an audio event is detected, the method 1200 includes, at block 1210, sending audio event information to the second device. For example, device 110 sends audio event information 144 to device 120 .

Additionally, in response to determining that an audio event is detected, the method 1200 includes determining at block 1212 whether valid direction of arrival information is available. For example, valid direction-of-arrival information may correspond to detection of a sound source with a direction-of-arrival determined with a confidence level higher than a confidence threshold to distinguish discrete sound sources from diffuse sound with no distinguishable source. In particular embodiments, available direction-of-arrival information available for a sound represented by one or more audio signals indicates that the sound is coming from an identifiable direction (e.g., from a discrete sound source), and available information that is not available for the sound The direction of arrival information indicates that the sound is not coming from a recognizable direction. In response to determining at 1212 that valid direction of arrival information is available, the method 1200 includes, at block 1214 , sending the direction of arrival information to the second device. For example, device 110 sends direction of arrival information 142 to device 120 .

In response to determining at block 1208 that no audio event has been detected, at block 1212 determining that no valid direction of arrival information is available, or at block 1214 after sending direction of arrival information to the second device, method 1200 proceeds to block 1222 A decision is made whether to send one or more audio signals (eg, audio signals 170, 172), one or more beamforming signals (eg, beamforming audio signal 148), or no audio signals to the second device.

12 illustrates several optional decision operations that may be used to decide at block 1220 whether to send one or more audio signals, one or more beamformed signals, or no audio signals to the second device in some embodiments.

At block 1230, it is determined whether at least one environmental detection or audio event detection has occurred. In response to determining that ambient detection did not occur and no audio events were detected, method 1200 determines at block 1240 that no audio will be sent to the second device. Therefore, in this example, when there is no environment detection and no audio event, the first device (eg, device 110 ) does not transmit audio information to the second device (eg, device 120 ) for additional processing.

Otherwise, in response to determining that at least one of environmental detection or audio event detection has occurred, method 1200 includes determining at block 1232 whether an amount of power or bandwidth available for transmission to the second device is limited. For example, if the first device has available battery power below the power threshold, or if the amount of available transmission bandwidth for sending audio data to the second device is below the transmission threshold, the first device may decide to conserve power with the second device. The resources associated with the device transmitting audio data. Otherwise, the first device may operate in a preset (eg, non-saving) mode.

In response to determining at block 1232 that neither power nor transmission bandwidth is limited, method 1200 includes, at block 1248 , sending an audio signal to the second device. For example, device 110 may send audio signals 170 , 172 to device 120 .

Otherwise, in response to determining at block 1232 that at least one of power or transmission bandwidth is limited, method 1200 includes determining at block 1234 whether a microphone at the second device is available to capture audio data. For example, where the microphones (e.g., microphones 106, 108) at the second device are obscured or obstructed (such as in a user's pocket or bag), or located too far away to retrieve the same In the case of substantially the same audio information for the microphones, the microphone at the second device may be considered unavailable.

In response to determining at block 1234 that a microphone at the second device is available, method 1200 includes determining at block 1236 whether a beamformed audio signal is available. For example, when ambient detection occurs based on diffuse ambient sound rather than sound from a particular source whose direction can be localized, no beamforming operation is performed at the first device. As another example, when an audio event is detected, but a direction of a sound source corresponding to the audio event cannot be determined with a confidence greater than a threshold confidence, no valid beamforming signal is generated at the first device.

In response to determining at block 1236 that no beamformed audio signals are available, method 1200 determines at block 1240 that no audio data is to be sent to the second device. Otherwise, when it is determined at block 1236 that a beamformed audio signal is available, the method 1200 proceeds to block 1242 where the beamformed signal or no signal is sent to the second device. For example, because power resources or transmission resources are limited, but a microphone is available at the second device for audio capture and analysis, the first device may decide that no audio is to be sent to the second device, and the second device may capture the audio for analysis at the second device. Otherwise, the first device may decide to send a beamformed audio signal to the second device although power resources or transmission resources are limited and a microphone is available for audio retrieval at the second device. In particular embodiments, the decision at block 1242 whether to transmit a beamformed signal or not to transmit a signal may be based at least in part on the amount of power or bandwidth available for transmission of the beamformed signal (e.g., may be performed in conjunction with one or more frequency bands). comparison of wide or power thresholds to decide whether to transmit one or more beamforming audio signals).

Returning to block 1234, in response to determining that the microphone of the second device is not available, method 1200 determines at block 1238 whether one or more beamformed audio signals are available. In response to the one or more beamforming audio signals being available, the method 1200 includes, at block 1244 , sending the one or more beamforming audio signals. Otherwise, in response to determining at block 1238 that one or more beamformed audio signals are not available, method 1200 includes, at block 1246 , sending the reduced signal to the second device. For example, sending a reduced signal may include sending audio corresponding to a reduced number of microphone channels (e.g., sending a single of audio signals 170 or 172), sending a reduced-resolution version of one or more microphone channels (e.g., audio signal 170 lower resolution versions of one or more of 172, 172), transmitting extracted audio characteristic data (e.g., characteristic data extracted from one or both of audio signals 170, 172, such as spectral information), and transmitting completely This can provide useful information to the second device with reduced power and bandwidth usage compared to the audio signals 170, 172 of the audio system.

FIG. 13 illustrates a flowchart corresponding to a method 1300 that may be performed by a second device, such as device 120 (eg, one or more processors 126 ), for receiving information from a first device, such as device 110 .

Method 1300 includes, at block 1302, receiving a data transmission from a first device. Method 1300 includes determining at block 1304 whether the transmission includes audio signal data. As an example, the second device may parse the received data to determine whether one or more audio signals (eg, audio signals 170, 172, one or more beamforming signals 148, or a combination thereof) were received.

If the transmission does not include audio signal data, method 1300 optionally includes determining at block 1304 whether one or more microphones of the second device are available for audio retrieval. For example, where the microphones (e.g., microphones 106, 108) of the second device are obscured or blocked (such as in a user's pocket or bag), or are located too far away to retrieve the microphone at the first device With substantially the same audio information, the microphone at the second device may be considered unavailable.

In response to determining that one or more microphones are unavailable at block 1304 , method 1300 optionally includes, at 1306 , signaling the first device that the microphones are unavailable, and the method ends at 1308 . Otherwise, when one or more microphones are available, the method 1300 optionally includes performing a data retrieval operation at the second device at block 1310 to retrieve the audio signal.

Method 1300 optionally includes determining at block 1312 whether the transmission includes environmental data. As an example, device 120 may parse the received data to determine whether environmental information 146 is received. In response to the transmission including context data, method 1300 optionally includes performing context processing at 1314 . For example, device 120 may process audio signals 170 , 172 , 190 , 192 or a combination thereof at acoustic environment processing unit 156 to generate environment information 147 .

Method 1300 includes determining at block 1320 whether the transmission includes audio event data. As an example, device 120 may parse the received data to determine whether audio event information 144 is received. If the transmission does not include audio event data, then at 1322, processing of data received in the transmission ends. In response to the transmission including audio event data, method 1300 optionally includes determining at block 1330 whether the transmission includes direction of arrival data. As an example, device 120 may parse the received data to determine whether direction-of-arrival information 142 is received. In response to the transmission not including direction-of-arrival data, method 1300 optionally includes performing direction-of-arrival processing at 1332 to generate direction-of-arrival data. For example, device 120 may process audio signals 170 , 172 , 190 , 192 or a combination thereof at direction of arrival processing unit 152 to generate direction of arrival information 143 . However, if the transmission includes direction-of-arrival data, then the direction-of-arrival processing of block 1332 is bypassed. Accordingly, the second device may selectively bypass direction-of-arrival processing of received audio data corresponding to audio events based on whether direction-of-arrival information is received from the first device.

When the transmission includes direction of arrival information at block 1330 , or after generating the direction of arrival information at block 1332 , method 1300 optionally includes determining, at block 1340 , whether the transmission includes beamforming data. As an example, device 120 may parse the received data to determine whether beamformed audio signal 148 is received. In response to the transmission not including beamforming data, method 1300 optionally includes performing, at 1342, a beamforming operation to generate beamforming data. For example, device 120 may process audio signals 170 , 172 , 190 , 192 or a combination thereof at beamforming unit 158 to generate beamformed audio signal 149 . However, if the transmission includes beamforming data, then performance of the beamforming operation of block 1342 is bypassed. Thus, the second device may selectively bypass the beamforming operation based on whether the received audio material corresponds to a multi-channel microphone signal from the first device or to a beamformed signal from the first device.

When the transmission includes beamforming data at block 1340 , or after generating the beamforming data at block 1342 , method 1300 includes performing audio event processing at block 1350 . For example, device 120 may process audio signals 170 , 172 , 190 , 192 or a combination thereof at audio event processing unit 154 to generate audio event information 145 .

By selectively bypassing one or more operations, such as direction of arrival processing or beamforming operations, method 1300 achieves reduced power consumption, reduced latency associated with processing audio event data received from the first device or both.

Referring to FIG. 14 , a particular illustrative aspect of a system configured to perform directional processing on a plurality of audio signals received from a plurality of microphones is disclosed and generally designated 1400 . System 1400 includes vehicle 1410 coupled to first microphone 1402 and second microphone 1404 . Although two microphones 1402 , 1404 are shown, in other implementations additional microphones may be coupled to the vehicle 1410 . As a non-limiting example, eight (8) microphones may be coupled to vehicle 1410 . In some implementations, the microphones 1402, 1404 are directional microphones. In other implementations, one or both of the microphones 1402, 1404 are omnidirectional microphones.

According to some implementations, vehicle 1410 may be an autonomous vehicle. That is, vehicle 1410 can navigate without user interaction. According to other implementations, vehicle 1410 may include one or more user-assisted modes (e.g., obstacle detection, obstacle avoidance, lane maintenance, speed control, etc.) switch between. System 1400 also includes device 1420 . According to one embodiment, the device 1420 includes a second vehicle. According to another embodiment, the device 1420 comprises a server. As described below, vehicle 1410 may communicate wirelessly with device 1420 to perform one or more operations (such as autonomous navigation) based on sounds detected at vehicle 1410 . In a particular embodiment, vehicle 1410 corresponds to device 110 and device 1420 corresponds to device 120 .

The first microphone 1402 is configured to capture sound 1482 from one or more sources 1480 . In the illustrative example of FIG. 14 , source 1480 corresponds to another vehicle, such as an automobile. It should be understood, however, that a vehicle is only a non-limiting example of a sound source, and that the techniques described herein may be implemented with other sound sources. Upon capturing sound 1482 from source 1480 , first microphone 1402 is configured to generate an audio signal 1470 representative of the captured sound 1482 . In a similar manner, second microphone 1404 is configured to capture sound 1482 from one or more sources 1480 . Upon capturing sound 1482 from source 1480 , second microphone 1404 is configured to generate an audio signal 1472 representative of the captured sound 1482 .

The first microphone 1402 and the second microphone 1404 may have different locations on the vehicle 1410, different orientations, or both. As a result, the microphones 1402, 1404 may pick up the sound 1482 at different times, with different receive phases, or both. To illustrate, if first microphone 1402 is closer to source 1480 than second microphone 1404 , first microphone 1402 may capture sound 1482 before second microphone 1404 captures sound 1482 . As described below, if the location and orientation of the microphones 1402, 1404 are known, the audio signals 1470, 1472 produced by the microphones 1402, 1404, respectively, may be used to perform orientation processing. That is, the vehicle 1410 may use the audio signals 1470, 1472 to determine the relative location of the source 1480, determine the direction of arrival of the sound 1482, and the like.

The vehicle 1410 includes a first input interface 1411 , a second input interface 1412 , a memory 1414 and one or more processors 1416 . The first input interface 1411 is coupled to one or more processors 1416 and is configured to be coupled to the first microphone 1402 . The first input interface 1411 is configured to receive an audio signal 1470 from the first microphone 1402 (eg, the first microphone output), and may provide the audio signal 1470 to the processor 1416 as an audio frame 1474 . The second input interface 1412 is coupled to one or more processors 1416 and is configured to be coupled to the second microphone 1404 . The second input interface 1412 is configured to receive an audio signal 1472 from the second microphone 1404 (eg, the second microphone output), and may provide the audio signal 1472 to the processor 1416 as an audio frame 1476 . Audio signals 1470, 1472, audio frames 1474, 1476, or both may also be referred to herein as audio data 1478.

The one or more processors 1416 include an arrival direction processing unit 1432, and optionally include an audio event processing unit 1434, a report generator 1436, a navigation instruction generator 1438, or a combination thereof. According to one embodiment, one or more elements of the one or more processors 1416 may be implemented using dedicated circuitry. As non-limiting examples, one or more elements of one or more processors 1416 may be implemented using an FPGA, ASIC, or the like. According to another embodiment, one or more elements of one or more processors 1416 may be implemented by executing instructions 1415 stored in memory 1414 . For example, memory 1414 may be a non-transitory computer-readable medium storing instructions 1415 executable by one or more processors 1416 to perform operations described herein.

The direction of arrival processing unit 1432 may be configured to process the plurality of audio signals 1470 , 1472 to generate direction of arrival information 1442 corresponding to a source 1480 of a sound 1482 represented by the audio signals 1470 , 1472 . In some embodiments, the direction of arrival processing unit 1432 is configured to operate in a similar manner as the direction of arrival processing unit 132 of FIG. 1 . In an illustrative, non-limiting example, direction of arrival processing unit 1432 may select audio frames 1474 , 1476 produced from each microphone 1402 , 1404 that represent similar sounds, such as sound 1482 from source 1480 . For example, direction of arrival processing unit 1432 may process audio frames 1474 , 1476 to compare sound characteristics and ensure that audio frames 1474 , 1476 represent the same instance of sound 1482 . In response to determining that audio frames 1474, 1476 represent the same instance of sound 1482, direction-of-arrival processing unit 1432 may compare the timestamps of each audio frame 1474, 1476 to determine which microphone 1402, 1404 first picked up the corresponding instance of sound 1482. corresponding instance. If the audio frame 1474 has an earlier timestamp than the audio frame 1476 , the direction-of-arrival processing unit 1432 may generate direction-of-arrival information 1442 indicating that the source 1480 is closer to the first microphone 1402 . If the audio frame 1476 has an earlier timestamp than the audio frame 1474 , the direction-of-arrival processing unit 1432 may generate direction-of-arrival information 1442 indicating that the source 1480 is closer to the second microphone 1404 . Thus, based on the timestamps of similar audio frames 1474 , 1476 , the direction of arrival processing unit 1432 can locate the sound 1482 and the corresponding source 1480 . Timestamps of audio frames from additional microphones can be used to improve localization in a similar manner to that described above.

In some embodiments, one or more other techniques may be used to determine the direction of arrival information 1442 instead of or in addition to the time difference as described above, such as measuring each microphone in the microphone array of the vehicle 1410 (e.g., The phase difference of sound 1482 received at microphones 1402 and 1404). In some implementations, the microphones 1402, 1404 may operate as, or be included in, a microphone array, and the direction of arrival information 1442 is based on characteristics of the sound from each microphone of the microphone array, such as time of arrival or phase ) and are generated based on the relative positions and orientations of the microphones in the microphone array. In such embodiments, information about sound characteristics or captured audio data may be transmitted between the vehicle 1410 and the device for direction of arrival detection 1420 .

The audio event processing unit 1434 may be configured to process a plurality of audio signals 1470 , 1472 to perform audio event detection in a similar manner to the audio event processing unit 134 . To illustrate, the audio event processing unit 1434 may process the sound characteristics of the audio frames 1474, 1476 and compare the sound characteristics to a plurality of audio event models to determine whether an audio event has occurred. For example, the audio event processing unit 1434 may access a database (not shown) that includes models of different audio events (such as car horns, train horns, pedestrian talking, etc.). In response to the sound characteristic matching (or substantially matching) the particular model, audio event processing unit 1434 may generate audio event information 1444 indicating that sound 1482 represents an audio event associated with the particular model. As a non-limiting example, an audio event may correspond to the sound of an approaching vehicle (eg, source 1480).

Report generator 1436 may be configured to generate report 1446 based on direction of arrival information 1442 and audio event information 1444 . Accordingly, report 1446 may indicate at least one detected event and a direction of the detected event. In scenarios where the microphones 1402, 1404 pick up multiple sounds from various directions, the report 1446 may indicate a list of detected events and direction information for the detected events over a period of time.

Processor 1416 may be configured to send report 1446 to device 1420 . Based on report 1446, device 1420 may send navigation instructions 1458 to vehicle 1410, according to one embodiment. Upon receiving navigation instructions 1458 from device 1420 , processor 1416 may navigate (eg, autonomously navigate) vehicle 1410 based on navigation instructions 1458 . Alternatively or additionally, navigational instructions 1458 may be provided to the operator of vehicle 1410 , such as visual or audible alerts or instructions to adjust the operation of vehicle 1410 . In some examples, navigation instructions 1458 instruct vehicle 1410 on a path to take (eg, pull over to one side when it is possible to safely allow emergency vehicles to pass). In some examples, navigation instructions 1458 inform vehicle 1410 of the path of one or more other vehicles (eg, a vehicle ahead has detected an accident and is about to slow down). The processor 1416 may autonomously navigate the vehicle 1410 to change the route (eg, change the route or change the speed) to take into account the paths of one or more other vehicles.

According to another embodiment, the device 1420 may send the second report 1456 to the vehicle 1410 based on the report 1446 or independently of the report 1446 . In response to receiving second report 1456 , processor 1416 may navigate (eg, autonomously navigate) vehicle 1410 based on report 1446 and second report 1456 , according to one embodiment. According to another embodiment, in response to receiving second report 1456 , navigation instruction generator 1438 may be configured to generate navigation instructions 1448 to be used by processor 1416 to navigate vehicle 1410 . In some examples, the second report 1456 indicates an event detected by another vehicle (eg, a vehicle ahead detects a sound indicating an accident). Navigation instruction generator 1438 may generate navigation instructions 1448 to autonomously navigate vehicle 1410 to change the path of travel to avoid the location of an event or change speed (eg, slow down). Processor 1416 may also send navigation instructions 1448 to device 1420 to inform device 1420 of the route of vehicle 1410 . In some examples, navigation instructions 1448 indicate a suggested route (eg, route or speed) for one or more other vehicles to take. For example, navigation instructions 1448 indicate that vehicle 1410 is slowing down and advise any vehicles within 20 feet of vehicle 1410 to slow down or change course.

Optionally, device 1420 may send notification 1492 of an audio event (eg, a vehicle collision) to one or more other devices 1490 based on the location of vehicle 1410 and the location of one or more other devices 1490 . In one example, notification 1492 corresponds to notification 930 of FIG. 9 . As an illustrative, non-limiting example, one or more devices 1490 may include, or be incorporated into, one or more other vehicles that may be determined to be near or approaching the location of vehicle 1410. other vehicles to notify the vehicle of one or more detected audio events in the vicinity of the vehicle 1410 (eg, siren, collision, etc.).

The system 1400 of FIG. 14 enables a vehicle 1410 to detect external sounds, such as sirens, and navigate accordingly. It should be appreciated that the use of multiple microphones enables the location and relative distance of a siren (eg, source 1480 ) to be determined and displayed when a detected siren is approaching or moving away.

FIG. 15 illustrates a certain illustrative aspect of a system 1500 including a vehicle 1510 (eg, a first device) in communication with a device 1520 (eg, a second device). Vehicle 1510 includes input interfaces 1412 , 1411 , memory 1414 and one or more processors 1416 of FIG. 14 . In particular embodiments, vehicle 1510 corresponds to device 110 and device 1520 corresponds to device 120 .

The one or more processors 1416 include an implementation of the audio event processing unit 1434 in which the generated audio event information 1444 indicates that the detected audio event corresponds to the vehicle event 1502 and the audio category 1504 associated with the vehicle event 1502 . For example, audio event processing unit 1434 may include one or more classifiers (such as one or more classifiers 610 of FIG. The audio category 1504 that corresponds to the sound 1482 that is represented and associated with the vehicle event 1502 .

One or more processors 1416 are configured to send audio material 1550 representing sounds associated with vehicle event 1502 to device 1520 . For example, audio data 1550 may include audio data 1478, audio signals 1470, 1472, one or more beamformed audio signals directed at source 1480 of sound 1482, or a combination thereof. The one or more processors 1416 are also configured to send an indication 1552 to the device 1520 that the audio data 1550 corresponds to the audio category 1504 associated with the vehicle event 1502 . For example, indication 1552 may correspond to indication 616 of FIG. 6 or FIG. 8 .

Device 1520 includes memory 1514 configured to store instructions 1515 and also includes one or more processors 1516 coupled to memory 1514 . One or more processors 1516 are configured to receive from vehicle 1510 (eg, first device) audio data 1550 representing sound 1482 and an indication 1552 that audio data 1554 corresponds to audio category 1504 associated with vehicle event 1502 . In particular embodiments, device 1520 corresponds to another vehicle, server, or distributed computing (eg, cloud-based) system, as non-limiting examples.

One or more processors 1516 are also configured to process audio data 1550 at one or more classifiers 1530 to verify that sound 1482 represented by audio data 1550 corresponds to vehicle event 1502 . For example, in particular embodiments, one or more classifiers 1530 correspond to one or more classifiers 920 of FIG. 9 . The one or more processors 1516 are configured to send information to the one or more devices 1490 based on the location of the vehicle 1510 (eg, the first device) and the location of the one or more devices 1490 (eg, the one or more third devices) A notification 1492 of the vehicle event 1502 is sent.

FIG. 16 illustrates a particular embodiment of a device 120 (eg, a second device) in which one or more processors 126 are configured to update orientation based on audio events detected by a first device (eg, device 110 ). A map of sound sources 1614.

The one or more processors 126 include an audio event processing unit 154 , a map updater 1612 and an audio scene renderer 1618 . The one or more processors 126 are configured to perform one or more operations associated with tracking directional audio sources in an audio scene. In one example, one or more processors 126 may receive from the first device an indication 1602 of an audio category corresponding to the audio event, such as indication 616 of FIG. 6 , and a direction corresponding to a sound source associated with the audio event. Data 1604 (such as arrival direction information 142).

The one or more processors 126 may update the map 1614 of directional sound sources in the audio scene based on the audio events to generate an updated map 1616 . For example, when the audio event corresponds to a newly detected audio event, the map updater 1612 is configured to insert information corresponding to the audio event into the map 1614 to generate an updated map 1616 . The inserted information may include, for example, the location of the sound source associated with the audio event, an indication of the type of the audio event (e.g., an audio class corresponding to the audio event), and the audio associated with the audio event (e.g., to represent a sound Links to audio signal data for ) information.

Optionally, one or more processors 126 may send data 1660 corresponding to updated map 1616 to one or more third devices (shown as devices 1670 , 1672 , and 1674 ) that are geographically remote from the first device. . Profile 1660 enables each of devices 1670, 1672, and 1674 to update the device's local copy of map 1614 to enable a user of device 1670, 1672, or 1674 to be notified, access, or experience sounds associated with audio events.

In some implementations, map 1614 (and updated map 1616 ) corresponds to a database (such as a "crowdsourced" database) of audio events and locations distributed over a geographic area to notify when a collision is detected nearby. The vehicle or update vehicle navigation instructions to avoid specific audio events, such as described in FIGS. 14 and 15 . In other implementations, map 1614 (and updated map 1616 ) may be used in other applications, such as providing a map of sound events detected in a neighborhood, town, city, or the like. For example, maps of audio events (eg, gunshots, yells, sirens, glass breaking, etc.) associated with crimes can be used by law enforcement to plan resource allocation or to detect incidents worthy of investigation. As another example, a map of audio events may be associated with nature. For example, a bird lover may use a map of various types of birds that have been located based on the detection and classification of their particular song.

In some implementations, the audio scene renderer 1618 is configured to generate audio data corresponding to the three-dimensional audio scene based on the updated map 1616 for playback to the user of the first device. For example, the first device may correspond to an audio headset worn by a user (such as described with reference to FIG. 21 ), or a virtual reality, augmented reality, or mixed reality headset (such as described with reference to FIG. 25 ).

FIG. 17 illustrates a graphical example of a 3D audio map 1700 of the audio scene around a user 1702 wearing headphones. 3D audio map 1700 may correspond to map 1614 (or updated map 1616 ) of FIG. 16 . The 3D audio map 1700 includes a first vehicle 1710 moving in a direction generally towards the user 1702 and a second vehicle 1712 also moving in a direction generally towards the user. (The direction of movement of a moving audio source is indicated by an arrow.) Other sound sources include barking dogs 1714, people talking 1716, crosswalk timers counting down the time remaining to cross the street 1718, and man-made audio maps that have been compiled into the 3D audio map 1700. Sound 1720. For example, sound sources 1710-1718 may be real-world sound sources detected via a microphone of a headset worn by user 1702, and artificial sound 1720 may be added by an augmented reality engine (or game engine) to a specific location in the sound scene , such as a sound effect (eg, a commercial bell) associated with a store or restaurant at the location.

FIG. 18 illustrates an example of a directional audio scene 1802 captured with sound event and ambient class detection (such as based on map 1614 (or updated map 1616 ) of FIG. 16 ). The user 1804 is in the center of the directional audio scene 1802 and a plurality of sets of virtual (or real) speakers associated with the sound field of the directional audio scene 1802 are illustrated, including a first set of speakers located substantially above and below the user 1804 A first representative speaker 1810 of the directional audio scene 1802, a second representative speaker 1812 of a second set of speakers placed along the upper and lower perimeters of the directional audio scene 1802, and a third set of speakers located around the user 1804 at approximately head height The third representative speaker 1814 in .

In particular embodiments, operation 1820 (e.g., an update of map 1614 to add or remove sound events based on type, direction, etc.) Directional audio scene 1830. For example, virtual participants 1832, 1834 may correspond to remote users sharing information about their respective local sound fields, which information may be combined with directional audio scene 1802 to create Generate immersive shared virtual experiences. This shared virtual experience can be used in applications such as live travel channel guides or live meeting, party or event immersion for people who are unable to attend in person due to social, health or other constraints.

FIG. 19 illustrates an implementation 1900 of at least one of the devices 110, 120 as an integrated circuit 1902 including directional audio signal processing circuitry. For example, integrated circuit 1902 includes one or more processors 1916 . One or more processors 1916 may correspond to one or more processors 116, one or more processors 126, one or more processors 202 of FIG. Multiple processors 1416, one or more processors 1516, or a combination thereof. The one or more processors 1916 include a directional audio signal processing unit 1990 . The directional audio signal processing unit 1990 may include at least one element of the processor 116, at least one element of the processor 126, at least one element of the processor 202, at least one element of the earphone 310, at least one element of the earphone 410, and at least one element of the mobile phone 420. at least one element, at least one element of system 500, at least one element of processor 1416, at least one element of processor 1516, or a combination thereof.

Integrated circuit 1902 also includes audio input 1904 (such as one or more bus interfaces) to enable audio data 178 to be received for processing. The integrated circuit 1902 also includes a signal output 1906 (such as a bus interface) to enable sending directional audio signal data 1992 . The directional audio signal data 1992 may correspond to at least one of: direction of arrival information 142, 143, audio event information 144, 145, environmental information 146, 147, beamforming audio signals 148, 149, direction information 250, first sound information 440, second sound information 442, context information 496, audio zoom angle 460, noise reduction parameters 462, direction of arrival information 542, audio event information 544, indication 616, indication 716, notification 930, control signal 932, classifier output 934, Target output 1106, reports 1446, 1456, navigation instructions 1448, 1458, notifications 1492, instructions 1552, audio data 1550, data 1660, or combinations thereof.

The integrated circuit 1902 implements directional audio signal processing as a component in a system including a microphone, such as a mobile phone or tablet computer as shown in FIG. 20 , a headset as shown in FIG. 21 , a wearable electronic device as shown in FIG. 22 , the sound control speaker system shown in Figure 23, the camera shown in Figure 24, the virtual reality headset, mixed reality headset or augmented reality headset shown in Figure 25, the augmented reality glasses or mixed reality glasses shown in Figure 26, the A set of in-ear devices as shown in 27, or a vehicle as shown in Figure 28 or Figure 29.

As an illustrative, non-limiting example, FIG. 20 illustrates an implementation 2000 in which device 120 is a mobile device 2002 such as a phone or tablet. The mobile device 2002 includes a third microphone 106 positioned to primarily capture the user's voice, one or more fourth microphones 108 positioned to primarily capture ambient sounds, and a display screen 2004 . The directional audio signal processing unit 1990 is integrated in the mobile device 2002 and is shown using dashed lines to indicate internal elements that are not generally visible to a user of the mobile device 2002 . In a particular example, directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992 that is subsequently processed to perform one or more operations at mobile device 2002, such as activating a graphical user interface Or otherwise display other information associated with the detected audio event at the display screen 2004 (eg, via an integrated "smart assistant" application).

FIG. 21 illustrates an embodiment 2100 in which the device 110 is a headset device 2102 . The earphone device 2102 includes a first microphone 102 positioned to primarily capture a user's voice and one or more second microphones 104 positioned to primarily capture ambient sounds. The directional audio signal processing unit 1990 is integrated in the earphone device 2102 . In a particular example, directional audio signal processing unit 1990 may be configured to generate directional audio signal data 1992 that may cause headphone device 2102 to perform one or more operations at headphone device 2102, transmit directional audio signal data 1992 to the second device ( not shown) for further processing or a combination thereof. Headphone device 2102 may be configured to provide an audible representation of a detected audio event or environment to a wearer of headphone device 2102 (such as based on audio event information 144, audio event information 145, environment information 146, environment information 147, or a combination thereof). notify.

FIG. 22 illustrates an embodiment 2200 in which at least one of the devices 110, 120 is a wearable electronic device 2202 (shown as a "smart watch"). The directional audio signal processing unit 1990 , the first microphone 102 and one or more second microphones 104 are integrated into the wearable electronic device 2202 . In a particular example, directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992, which is then processed to perform one or more operations at wearable electronic device 2202, such as activating a graphical user Interface or otherwise display other information associated with the detected audio event at the display screen 2204 of the wearable electronic device 2202. To illustrate, display screen 2204 of wearable electronic device 2202 may be configured to display notifications based on speech detected by wearable electronic device 2202 . In a particular example, wearable electronic device 2202 includes a haptic device that provides a haptic notification (eg, vibration) in response to detecting an audio event. For example, a haptic notification may cause a user to look at wearable electronic device 2202 to view a detected audio event displayed (such as based on audio event information 144, audio event information 145, environmental information 146, environmental information 147, or a combination thereof) or environmental notifications. The wearable electronic device 2202 may thus alert a hearing-impaired user or a user wearing headphones to the detection of certain audio activity.

FIG. 23 is an embodiment 2300 in which at least one of the devices 110 , 120 is a wireless speaker and voice-activated device 2302 . Wireless speaker and voice-activated device 2302 may have a wireless network connection and be configured to perform secondary operations. Directional audio signal processing unit 1990 , first microphone 102 , one or more second microphones 104 , third microphone 106 , fourth microphone 108 , or a combination thereof are included in wireless speaker and voice activated device 2302 . The wireless speaker and voice activated device 2302 also includes a speaker 2304 . In certain aspects, speaker 2304 corresponds to speaker 336 of FIG. 3 , speaker 436 of FIG. 4 , or both. During operation, the directional audio signal processing unit 1990 may be used to generate the directional audio signal data 1992 and determine whether a keyword was spoken. In response to deciding to speak the keyword, the wireless speaker and voice-activated device 2302 may perform an auxiliary operation (such as via execution of an integrated auxiliary application). Secondary actions can include adjusting the temperature, playing music, turning on lights, and more. For example, secondary actions are performed in response to receiving a command following a keyword or key phrase (eg, "Hello Assistant").

FIG. 24 illustrates an embodiment 2400 in which at least one of the devices 110 , 120 is a portable electronic device corresponding to a camera device 2402 . The directional audio signal processing unit 1990 , the first microphone 102 , the one or more second microphones 104 , or a combination thereof are included in the camera device 2402 . During operation, the directional audio signal processing unit 1990 may be used to generate the directional audio signal data 1992 and determine whether a keyword was spoken. As an illustrative example, in response to determining that a keyword was spoken, camera device 2402 may perform an action in response to a spoken user command, such as adjusting image or video capture settings, image or video replay settings, or image or video playback settings. Fetch command.

25 illustrates where device 110 includes a device corresponding to an extended reality ("XR") headset 2502, such as a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") headset device. Embodiment 2500 of the portable electronic device. The directional audio signal processing unit 1990 , the first microphone 102 , one or more second microphones 104 or a combination thereof are integrated into the earphone 2502 . In a particular aspect, the headset 2502 includes a first microphone 102 positioned to primarily capture the user's voice and a second microphone 104 positioned to primarily capture ambient sounds. The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on audio signals received from the first microphone 102 and the second microphone 104 of the headset 2502 . Visual peripherals are placed in front of the user's eyes to be able to display augmented reality or virtual reality images or scenes to the user while wearing the headset 2502 . In a particular example, the visual peripheral device is configured to display a notification indicating user voice detected in the audio signal. In certain examples, the visual peripheral device is configured to display pointing detections superimposed on displayed content (e.g., in a virtual reality application) or superimposed on the user's field of view (e.g., in an augmented reality application). notification of incoming audio events to visually indicate to the user the location of the sound source associated with the audio event. To illustrate, visual peripheral devices may be configured to display notifications of detected audio events or circumstances (such as based on audio event information 144 , audio event information 145 , environment information 146 , environment information 147 , or combinations thereof).

FIG. 26 illustrates an embodiment 2600 in which device 110 includes a portable electronic device corresponding to augmented reality or mixed reality glasses 2602 . Glasses 2602 include holographic projection unit 2604 configured to project visual material onto the surface of lens 2606 or to reflect visual material from the surface of lens 2606 onto the wearer's retina. The directional audio signal processing unit 1990 , the first microphone 102 , the one or more second microphones 104 , or a combination thereof are integrated into the glasses 2602 . The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on the audio signals received from the first microphone 102 and the second microphone 104 . In a particular example, holographic projection unit 2604 is configured to display a notification indicating user voice detected in the audio signal. In a particular example, holographic projection unit 2604 is configured to display a notification indicating a detected audio event. For example, a notification may be overlaid at a specific location on the user's field of view that coincides with the location of the sound source associated with the audio event. To illustrate, the sound may be perceived by the user as emanating from the direction of the notification. In an illustrative embodiment, holographic projection unit 2604 is configured to display a notification of a detected audio event or environment (such as based on audio event information 144 , audio event information 145 , environment information 146 , environment information 147 , or a combination thereof).

FIG. 27 illustrates an embodiment 2700 in which device 110 includes a portable electronic device corresponding to a pair of earbuds 2706 , including a first earbud 2702 and a second earbud 2704 . Although earbuds are described, it should be understood that the present technology may be applied to other in-ear or on-ear playback devices.

The first earbud 2702 includes a first microphone 2720, such as a high signal-to-noise ratio microphone positioned to pick up the voice of the wearer of the first earbud 2702, a microphone configured to detect ambient sound and spatially distributed to support beamforming. or an array of other microphones (shown as microphones 2722A, 2722B, and 2722C), an "internal" microphone 2724 close to the wearer's ear canal (e.g., to aid in active noise cancellation), and a self-speech microphone 2726 (such as configured as A bone conduction microphone that converts the sound vibrations of the wearer's ear bones or skull into audio signals).

In a particular embodiment, first microphone 2720 corresponds to microphone 102, and microphones 2722A, 2722B, and 2722C correspond to multiple instances of microphone 104, and audio signals generated by microphones 2720 and 2722A, 2722B, and 2722C are provided to directional audio Signal processing unit 1990. The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on the audio signal. In some embodiments, the directional audio signal processing unit 1990 may also be configured to process audio signals from one or more other microphones of the first earbud 2702, such as the internal microphone 2724, the self-voice microphone 2726, or both.

Second earbud 2704 may be configured in a substantially similar manner as first earbud 2702 . In some embodiments, the directional audio signal processing unit 1990 of the first earbud 2702 is also configured to (such as via wireless transmission between the earbuds 2702, 2704, or via wired transmission in embodiments where the earbuds 2702, 2704 are coupled via a transmission line) ) to receive one or more audio signals generated by one or more microphones of the second earbud 2704 . In other implementations, the second earbud 2704 also includes a directional audio signal processing unit 1990 to enable a user wearing a single one of the earbuds 2702, 2704 to perform the techniques described herein.

In some implementations, the earbuds 2702, 2704 are configured to automatically switch between various modes of operation, such as a pass-through mode in which ambient sounds are played through the speakers 2730, non-ambient sounds (e.g., talking to a phone, media, etc.) Replay mode, corresponding to streaming audio for video games, etc.), and audio zoom mode or beamforming mode in which one or more ambient sounds are emphasized and/or other ambient sounds are suppressed for playback at speaker 2730. In other implementations, the earbuds 2702, 2704 may support fewer modes, or the earbuds 2702, 2704 may support one or more other modes instead of or in addition to the modes described.

In an illustrative example, earbuds 2702, 2704 may automatically transition from playback mode to pass-through mode in response to detecting the wearer's voice, and may automatically transition back to playback mode after the wearer stops speaking. In some examples, the earbuds 2702, 2704 may (such as by performing audio scaling on a particular ambient sound (e.g., a dog barking) and playing the audio-scaled sound superimposed on the sound being played while the wearer is listening to music ( It is possible to lower the volume of the audio scaled sound while playing it)) to operate in two or more modes at the same time. In this example, the wearer may be alerted to ambient sounds associated with the audio event without stopping playback of the music.

FIG. 28 illustrates an embodiment 2800 in which the disclosed technology is implemented in a vehicle 2802 , shown as a manned or unmanned aerial device (eg, a package delivery drone). The directional audio signal processing unit 2850 is integrated into the vehicle 2802 . The directional audio signal processing unit 2850 includes or corresponds to the directional audio signal processing unit 1990 and may be further configured for navigating the vehicle 2802 autonomously. Directional audio signal processing unit 2850 may include, for example, one or more processors 1416 of FIG. 14 , and vehicle 2802 may correspond to vehicle 1410 . Directional audio signal processing unit 2850 may generate and execute navigation instructions based on audio signals received from first microphone 102 and second microphone 104 of vehicle 2802 , such as for delivering instructions from authorized users of vehicle 2802 .

FIG. 29 illustrates another embodiment 2900 in which vehicle 1410 or vehicle 1510 corresponds to vehicle 2902 (shown as an automobile). The vehicle 2902 includes a directional audio signal processing unit 2950 . The directional audio signal processing unit 2950 includes or corresponds to the directional audio signal processing unit 1990 and may be further configured for navigating the vehicle 2902 autonomously. Vehicle 2902 also includes first microphone 102 and second microphone 104 . In some examples, one or more of the first microphone 102 and the second microphone 104 are located outside the vehicle 2902 to pick up ambient sounds, such as sirens and other vehicle sounds. In some implementations, tasks can be performed based on audio signals received from external microphones (e.g., first microphone 102 and second microphone 104), such as detection of environmental information and audio sound events, autonomous control of vehicle 2902 navigation etc.

In some examples, one or more of the first microphone 102 and the second microphone 104 are located inside the vehicle 2902 to pick up sounds within the vehicle, such as voice commands or sounds indicating a medical emergency. In some implementations, tasks such as autonomous navigation of the vehicle 2902 may be performed based on audio signals received from internal microphones (eg, first microphone 102 and second microphone 104 ). One or more operations of vehicle 2902 may be based on one or more detected keywords (e.g., "unlock ”, “Start Engine”, “Play Music”, “Show Weather Forecast” or another voice command).

Referring to FIG. 30 , a particular embodiment of a method 3000 of processing audio is illustrated. In certain aspects, one or more operations of method 3000 are performed by device 110 , system 200 , headset 310 , headset 410 , system 500 , vehicle 1410 , vehicle 1510 , or combinations thereof.

Method 3000 includes receiving, at block 3002, audio signals from a plurality of microphones at one or more processors of a first device. For example, referring to FIG. 1, processor 130 may receive audio frames 174, 176 of audio signals 170, 172 from microphones 102, 104, respectively.

Method 3000 also includes processing the audio signal at block 3004 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, referring to FIG. 1 , direction of arrival processing unit 132 may process audio frames 174 , 176 to generate direction of arrival information 142 corresponding to source 180 of sound 182 represented by audio signals 170 , 172 .

Method 3000 also includes, at block 3006, sending to the second device data based on the direction of arrival information and the category or embedding associated with the direction of arrival information. For example, modem 118 may send direction of arrival information 142 and one or both of indication 616 or indication 716 to device 120 . The category may correspond to the category of a particular sound represented by the audio signal and associated with a particular audio event, and the embedding may include a signature or information corresponding to the particular sound or the particular audio event, and may be configured to be able to communicate via other audio Signal processing to detect specific sounds or specific audio events in other audio signals. In some implementations, the method 3000 also includes sending the representation of the audio signal to the second device. For example, a representation of an audio signal may include one or more portions of the audio signals 170, 172, one or more portions of the beamformed audio signal 148, or a combination thereof. According to one embodiment of method 3000 , sending data to device 120 may trigger activation of one or more sensors 129 .

In some embodiments, method 3000 includes processing signal data corresponding to an audio signal to determine a class or embedding. In one example, method 3000 includes performing a beamforming operation (eg, at beamforming unit 138 ) on the audio signal to generate signal data. In one example, signal data is processed at one or more classifiers, such as one or more classifiers 610, to generate one or more audio signals from among a plurality of classes supported by the one or more classifiers. A number of sounds represented and associated with an audio event determine the class. The category is sent to the second device (eg, device 120 ), such as via indication 616 .

In some implementations, signal data is processed at one or more encoders, such as one or more encoders 710, to produce embeddings. Embeddings correspond to sounds represented by one or more of the audio signals and associated with the audio events. The embedding is sent (such as via indication 716 ) to the second device (eg, device 120 ).

In some implementations, method 3000 includes receiving, at one or more processors of a second device, data based on direction-of-arrival information and categories. For example, modem 128 of device 120 may receive data and provide direction of arrival information 142 and indication 616 to one or more processors 126 . Method 3000 may include obtaining, at one or more processors of a second device, audio data representing sounds associated with direction of arrival information and categories. For example, the one or more processors 126 obtain one or more of the audio signals 170, 172 from the first device, obtain one or more of the audio signals 190, 192 from a local microphone (eg, microphone 106, 108), A beamformed audio signal 148 or a combination thereof is obtained from the first device. Method 3000 may also include validating the class at one or more processors of the second device based on at least the audio data and direction-of-arrival information, such as at audio event processing unit 154 or as described with reference to one or more classifiers 610 .

In some implementations, method 3000 includes receiving, at one or more processors of a second device, data based on direction of arrival information and embedding. For example, modem 128 of device 120 may receive data and provide direction of arrival information 142 and indication 716 to one or more processors 126 . Method 3000 may also include processing, at the one or more processors of the second device, the audio data representing the sound scene based on the direction of arrival information and the embedding to generate modified audio data corresponding to the updated sound scene. For example, one or more processors 126 may process input mixed waveform 1102 representing audio scene 1151 in combination with one or more embeddings 1104 and direction information 912 to generate updated audio scene 1171 .

Method 3000 can perform directional context-aware processing based on audio signals produced by multiple microphones. As a result, context detection and decisions on properties associated with the surrounding environment are enabled for various use cases.

Referring to FIG. 31 , a particular embodiment of a method 3100 of processing audio is illustrated. In a particular aspect, one or more operations of method 3100 are performed by vehicle 1410 of FIG. 14 .

Method 3100 includes receiving a plurality of audio signals from a plurality of microphones at one or more processors of a vehicle at block 3102 . For example, referring to FIG. 14, processor 1416 may receive audio frames 1474, 1476 of audio signals 1470, 1472 from microphones 1402, 1404, respectively.

Method 3100 also includes processing the plurality of audio signals at block 3104 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, referring to FIG. 14 , direction of arrival processing unit 1432 may process audio frames 1474 , 1476 to generate direction of arrival information 1442 corresponding to source 1480 of sound 1482 represented by audio signals 1470 , 1472 .

The method 3100 also includes generating a report indicating at least one detected event and a direction of the detected event based on the direction of arrival information at block 3106 . For example, referring to FIG. 14 , report generator 1436 may generate report 1446 indicating at least one detected event (from audio event information 1444 ) and the direction of the detected event (from direction-of-arrival information 1442 ).

According to one embodiment, method 3100 may include sending a report to a second device (eg, a second vehicle or server), and receiving navigation instructions or a second report from the second device. Based on the second report, the processor may generate navigation instructions to autonomously navigate the vehicle. If the second device sends navigation instructions, the processor may autonomously navigate the vehicle using the sent navigation instructions.

Method 3100 enables vehicle 1410 to detect external sounds, such as sirens, and navigate accordingly. It should be appreciated that the use of multiple microphones enables the location and relative distance of a siren (eg, source 1480 ) to be determined and displayed when a detected siren is approaching or moving away.

Referring to FIG. 32 , a particular embodiment of a method 3200 of processing audio is illustrated. In a particular aspect, one or more operations of method 3200 are performed by device 120 , such as at one or more processors 126 .

Method 3200 includes receiving, at block 3202, an indication of an audio class at one or more processors of a second device, the indication being received from the first device and corresponding to an audio event. For example, one or more processors 126 of device 120 of FIG. 9 receive indication 902 (eg, indication 616 ) from device 110 of FIG. 6 .

Method 3200 includes processing the audio data at one or more processors of the second device at block 3204 to verify that the sound represented by the audio data corresponds to the audio event. For example, one or more processors 126 of device 120 of FIG. 2 process audio data 904 to generate classification 922 to verify that the sound represented by audio data 904 corresponds to an audio event. In one example, one or more processors 126 compare classification 922 to the audio category indicated by indication 902 .

Optionally, method 3200 includes receiving audio data from a first device (e.g., device 110), and processing the audio data optionally includes providing the audio data as input to one or more classifiers to determine Associated categories. For example, in some implementations, the audio data 904 includes one or more portions of the audio signals 170, 172, one or more portions of the beamformed audio signal 148, or a combination thereof, and the audio data 904 is input to one or more classifier 920 . In some embodiments, the processing of the audio data further includes providing an indication of the audio category (eg, indication 902 ) as a second input to one or more classifiers to determine a classification associated with the audio data.

Optionally, method 3200 includes sending a control signal (such as control signal 932 ) to the first device (eg, device 110 ) based on the output of the one or more classifiers. In some implementations, the control signal includes an audio zoom command. In some embodiments, the control signal includes instructions for performing spatial processing based on the direction of the sound source.

In some implementations, the audio category corresponds to a vehicle event, and the method 3200 optionally includes sending information about the vehicle event to one or more third devices based on the location of the first device and the location of the one or more third devices. notify. For example, a notification 1492 is sent to one or more devices 1490 as described with reference to FIGS. 14 and 15 .

Optionally, method 3200 includes receiving, from a first device (eg, device 110 ), directional data (such as directional data 912 ) corresponding to a sound source associated with the audio event. Method 3200 may include updating a map of directional sound sources in an audio scene based on the audio event to generate an updated map, such as described with reference to map updater 1612, and sending the map to one or more third devices geographically remote from the first device. Materials corresponding to the updated map. For example, device 120 sends profile 1660 to one or more of devices 1670 , 1672 , and 1674 .

Optionally, method 3200 includes selectively bypassing direction-of-arrival processing of received audio data corresponding to an audio event based on whether direction-of-arrival information is received from a first device (eg, device 110 ). For example, one or more processors 126 may selectively bypass performing the direction-of-arrival information shown at block 1332 of FIG. 13 based on a determination at block 1330 of FIG. deal with.

Optionally, method 3200 includes selectively bypassing beamforming based on whether the received audio material corresponds to a multi-channel microphone signal from a first device (e.g., device 110) or to a beamformed signal from the first device operate. For example, one or more processors 126 may selectively bypass performing beamforming shown at block 1342 of FIG. 13 based on a determination at block 1340 of FIG. operate.

By receiving an indication of an audio class corresponding to an audio event and processing the audio data to verify that the sound represented by the audio data corresponds to the audio event, the method 3200 can perform decentralized audio event detection such that a first level (such as at Headphones) can recognize audio events with relatively high sensitivity and relatively low accuracy (eg, due to power, memory or computational constraints) compared to a second level (such as at a mobile phone). The second stage can use higher power, more accurate audio event detection to verify audio events, and can communicate detection results, control signals, etc. based on detected audio events. As a result, accurate audio event detection can be provided to users of wearable electronic devices, such as headphones, without requiring the wearable electronic device to support the computational load, memory footprint, and performance associated with full-power audio event detection. consumption.

Referring to FIG. 33 , a particular embodiment of a method 3300 of processing audio is illustrated. In certain aspects, one or more operations of method 3300 are performed by device 120 , such as at one or more processors 126 . In another particular aspect, one or more operations of method 3300 are performed by device 1520 , such as at one or more processors 1526 .

Method 3300 includes receiving, at one or more processors of a second device, audio data from a first device and an indication from the first device that the audio data corresponds to an audio category associated with a vehicle event at block 3302 . For example, device 1520 receives audio material 1550 and instructions 1552 from vehicle 1510 .

Method 3300 includes, at block 3304 , processing the audio data at one or more classifiers of a second device (eg, device 1520 ) to verify that the sound represented by the audio data corresponds to a vehicle event. For example, audio data 1550 is processed at one or more classifiers 1530 to determine classification 1522 .

Method 3300 includes, at block 3306, sending a notification of the vehicle event to one or more third devices based on the location of the first device (eg, vehicle 1510) and the location of the one or more third devices. For example, device 1520 sends notification 1592 to one or more devices 1490 based on the location of vehicle 1510 and the location of one or more devices 1490 .

Referring to FIG. 34 , a particular embodiment of a method 3400 of processing audio is illustrated. In a particular aspect, one or more operations of method 3400 are performed by device 110 , such as at one or more processors 116 .

Method 3400 includes receiving, at block 3402, one or more audio signals from one or more microphones at one or more processors of a first device. For example, device 110 receives audio signals 170, 172 from microphones 102, 104, respectively.

Method 3400 includes processing one or more audio signals at one or more processors at block 3404 to determine whether a sound represented by one or more of the audio signals is from a recognizable direction. For example, at block 1212 of FIG. 12, device 110 determines whether the processing of the audio signal at block 1202 of FIG. 12 yielded valid direction-of-arrival information about the source of the audio event.

Method 3400 includes, at block 3406, selectively transmitting direction of arrival information of the sound source to the second device based on the determination. For example, device 110 may select whether to send direction-of-arrival information to the second device based on determining whether valid direction-of-arrival information is available, such as described in connection with blocks 1212 and 1214 of FIG. 12 .

By selectively sending direction-of-arrival information based on whether the sound represented by one or more of the audio signals is from a recognizable direction, method 3400 can save money that would otherwise be generated by sending invalid or unreliable direction-of-arrival to the second device. Power consumption and transmission resources consumed by information.

Referring to FIG. 35 , a particular embodiment of a method 3500 of processing audio is illustrated. In certain aspects, one or more operations of method 3500 are performed by device 110 , such as at one or more processors 116 .

Method 3500 includes receiving, at block 3502, one or more audio signals from one or more microphones at one or more processors of a first device. For example, device 110 receives audio signals 170, 172 from microphones 102, 104, respectively.

Method 3500 includes determining at one or more processors at block 3504 whether to send one or more audio signals to the second device based on one or more criteria or to send an audio signal generated based on the one or more audio signals to the second device. beamformed audio signal. For example, if beamforming audio signals are available at device 110, device 110 may decide whether to transmit one or more audio signals or whether to transmit beamforming audio signals based on criteria such as available power and amount of bandwidth resources, as shown in FIG. 12 Block 1220 of . In an illustrative, non-limiting example where no microphone is available at the second device, if the available power or bandwidth for transmission to the second device exceeds a threshold, then a decision is made to send the audio as described in connection with block 1232 of FIG. signal (e.g., via the "No" path from block 1232); "path).

Method 3500 includes sending, at block 3506, audio data corresponding to the one or more audio signals or corresponding to the beamformed audio signal to the second device based on the determination. Continuing with the example above, device 110 may send an audio signal to device 120 at block 1248 of FIG. 12 , or send a beamforming signal to device 120 at block 1244 of FIG. 12 .

By selecting whether to transmit an audio signal or a beamformed signal based on one or more criteria, such as power availability or transmission resources, the method 3400 enables the transmitting device to make appropriate decisions on whether to provide full audio resolution to the receiving device (e.g., by By sending data corresponding to the full set of microphone channels including the sound of interest) or whether to provide finer-grained targeted audio (eg, by sending data corresponding to a single beamforming channel aimed at the sound source of interest).

Referring to FIG. 36 , a particular embodiment of a method 3600 of processing audio is illustrated. In a particular aspect, one or more operations of method 3600 are performed by device 120 , such as at one or more processors 126 .

Method 3600 includes receiving at one or more processors of a second device at block 3602 audio data representing a sound, direction data corresponding to a sound source, and a classification of the sound corresponding to an audio event, wherein the audio data, Direction data and categories are received from the first device. For example, one or more processors 126 of device 120 may receive audio data 904 of FIG. 9 or FIG. 10 , indication 1602 and direction data 1604 of FIG. 16 from device 110 .

Method 3600 includes, at block 3604, processing the audio data at one or more processors to verify that the sound corresponds to an audio event. For example, the audio event processing unit 154 processes the audio data to verify the audio type indicated by the indication 1602 .

Method 3600 includes updating, at one or more processors, a map of directional sound sources in an audio scene based on the audio events at block 3606 to generate an updated map. For example, map updater 1612 updates map 1614 to generate updated map 1616 .

Method 3600 includes, at block 3608, sending material corresponding to the updated map to one or more third devices that are geographically remote from the first device. For example, updated map material 1660 is sent to devices 1670 , 1672 , and 1674 that are geographically remote from device 110 .

Method 3600 enables applications such as virtual environments in which multiple participants are immersed in a shared sound scene by updating a map of directional sound sources in an audio scene and sending the updated map data to a geographically remote device, Such as described with reference to FIG. 18 .

The methods of Fig. 12, Fig. 13 and Fig. 30-Fig. 36 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processing unit (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, the methods of FIGS. 12 , 13 and 30-36 may be performed by a processor executing instructions, such as described with reference to FIG. 37 .

Referring to FIG. 37 , a block diagram of a particular illustrative embodiment of an apparatus is shown and generally designated 3700 . In various implementations, device 3700 may have more or fewer components than shown in FIG. 37 . In an illustrative embodiment, device 3700 may correspond to device 110 , device 120 , vehicle 1410 , device 1420 , vehicle 1510 , or device 1520 . In an illustrative embodiment, device 3700 may perform one or more operations described with reference to FIGS. 1-36 .

In a particular embodiment, device 3700 includes a processor 3706 (eg, a CPU). Device 3700 may include one or more additional processors 3710 (eg, one or more DSPs). In particular aspects, processor(s) 116, 126 of FIG. 1 or processor(s) 1416 of FIG. 14 correspond to processor 3706, processor 3710, or a combination thereof. Processor 3710 may include speech and music transcoder-decoder (CODEC) 3708, which includes speech transcoder ("vocoder") encoder 3736, vocoder decoding device 3738, directional audio signal processing unit 1990 or a combination thereof.

Device 3700 may include memory 3786 and CODEC 3734 . Memory 3786 may include instructions 3756 executable by one or more additional processors 3710 (or processor 3706 ) to implement the functions described with reference to directional audio signal processing unit 1990 . In certain aspects, memory 3786 corresponds to memory 114 , memory 124 of FIG. 1 , memory 1414 of FIG. 14 , or combinations thereof. In certain aspects, instructions 3756 include instruction 115, instruction 125 of FIG. 1, instruction 1415 of FIG. 14, or a combination thereof. Device 3700 may include modem 3770 coupled to antenna 3752 via transceiver 3750 . Modem 3770 may be configured to send signals to a second device (not shown). According to particular implementations, modem 3770 may correspond to modem 128 of FIG. 1 .

Device 3700 can include a display 3728 coupled to a display controller 3726 . Speaker 3792 , first microphone 102 and second microphone 104 may be coupled to CODEC 3734 . The CODEC 3734 may include a digital-to-analog converter (DAC) 3702, an analog-to-digital converter (ADC) 3704, or both. In particular embodiments, CODEC 3734 may receive analog signals from first microphone 102 and second microphone 104, convert the analog signals to digital signals using analog-to-digital converter 3704, and provide the digital signals to voice and music transcoder 3708 . The speech and music transcoder 3708 can process the digital signal, and the digital signal can be further processed by the directional audio signal processing unit 1990 . In particular embodiments, speech and music transcoder 3708 may provide digital signals to CODEC 3734 . The CODEC 3734 may convert the digital signal into an analog signal using the digital-to-analog converter 3702 and may provide the analog signal to a speaker 3792 .

In particular embodiments, device 3700 may be included in a system-in-package or system-on-chip device 3722 . In a particular embodiment, memory 3786 , processor 3706 , processor 3710 , display controller 3726 , CODEC 3734 , and modem 3770 are included in a system-in-package or system-on-chip device 3722 . In a particular embodiment, an input device 3730 and a power supply 3744 are coupled to the system-on-chip device 3722 . Additionally, in certain embodiments, as shown in FIG. 37 , display 3728 , input device 3730 , speaker 3792 , first microphone 102 , second microphone 104 , antenna 3752 , and power supply 3744 are external to system-on-chip device 3722 . In particular embodiments, each of display 3728, input device 3730, speaker 3792, first microphone 102, second microphone 104, antenna 3752, and power supply 3744 may be coupled to an element of system-on-chip device 3722, such as an interface (e.g., input interface 121 or input interface 122) or a controller.

Device 3700 may include a smart speaker, speaker bar, mobile communication device, smartphone, cellular phone, laptop, computer, tablet, personal digital assistant, display device, television, game console, music player, radio , Digital Video Players, Digital Video Disc (DVD) Players, Tuners, Cameras, Navigation Devices, Vehicles, Headphones, Augmented Reality Headsets, Mixed Reality Headsets, Virtual Reality Headsets, Aircraft, Home Automation Systems, Voice Activated Devices , wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof .

In connection with the described embodiments, an apparatus includes means for receiving audio signals from a plurality of microphones. For example, means for receiving an audio signal may correspond to the input interface 112, the input interface 111, the processor 116 or elements thereof, the input interface 121, the input interface 122, the processor 126 or elements thereof, the first processing domain 210 or elements thereof , second processing domain 220 or components thereof, earphone 310 or components thereof, earphone 410 or components thereof, spatial filter processing unit 502, audio input 1904, one or more processors 1916, directional audio signal processing unit 1990, one or more A processor 3710, one or more other circuits or elements configured to receive audio signals from a plurality of microphones, or any combination thereof.

The device also includes means for processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, means for processing may correspond to processor(s) 116 or components thereof, processor(s) 126 or elements thereof, first processing domain 210 or elements thereof, second processing domain 220 or elements thereof, The earphone 310 or its components, the earphone 410 or its components, the spatial filter processing unit 502, the audio event processing unit 504, the directional audio signal processing unit 1990, the one or more processors 1916, the one or more processors 3710, are configured to One or more other circuits or components, or any combination thereof, that process audio signals.

The apparatus also includes means for sending data based on the direction-of-arrival information and the category or embedding associated with the direction-of-arrival information to the second device. For example, means for transmitting may correspond to modem 118, modem 128, signal output 1906, directional audio signal processing unit 1990, one or more processors 1916, modem 3770, transceiver 3750, antenna 3752, configured One or more other circuits or components, or any combination thereof, for transmitting data and type or embedding.

In connection with the described embodiments, an apparatus includes means for receiving a plurality of audio signals from a plurality of microphones. For example, means for receiving a plurality of audio signals may correspond to input interface 1412, input interface 1411, one or more processors 1416 or components thereof, directional audio signal processing unit 2850, directional audio signal processing unit 2950, one or more A processor 3710, one or more other circuits or elements configured to receive a plurality of audio signals from a plurality of microphones, or any combination thereof.

The device also includes means for processing the plurality of audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, means for processing includes one or more processors 1416 or elements thereof, a directional audio signal processing unit 2850, a directional audio signal processing unit 2950, one or more processors 3710, a device configured to process a plurality of audio signals one or more other circuits or components, or any combination thereof.

The apparatus also includes means for generating a report indicating at least one detected event and a direction of the detected event based on the direction of arrival information. For example, means for generating includes one or more processors 1416 or elements thereof, directional audio signal processing unit 2850, directional audio signal processing unit 2950, one or more processors 3710, one or more processors configured to generate reports other circuits or components, or any combination thereof.

In connection with the described embodiments, an apparatus includes means for receiving an indication of an audio category, the indication being received from a remote device and corresponding to an audio event. For example, means for receiving an indication may correspond to modem 128, one or more processors 126, one or more processors 1516, audio input 1904, one or more processors 1916, antenna 3752, transceiver 3750, A modem 3770, a processor 3706, one or more processors 3710, one or more other circuits or elements configured to receive an indication, or any combination thereof.

The device also includes means for processing the audio data to verify that the sound represented by the audio data corresponds to the audio event. For example, means for processing audio data may correspond to one or more processors 126, one or more processors 1516, one or more processors 1916, one or more processors 3706, one or more processors 3710, configured One or more other circuits or components, or any combination thereof, for processing audio data to verify that the sound represented by the audio data corresponds to the audio event.

In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 114 or memory 3786) includes instructions (e.g., instructions 115 or instructions 3756), which when executed by one or more When executed by a processor (e.g., one or more processors 116, one or more processors 3710, or processor 3706), these instructions cause the one or more processors to receive data from multiple microphones (e.g., microphones 102, 104 ) to receive audio signals (eg, audio signals 170, 172). When executed by the one or more processors, the instructions also cause the one or more processors to process the audio signal to produce one or more sounds (e.g., sound 182) that are represented by one or more of the audio signals. The direction-of-arrival information (eg, direction-of-arrival information 142 ) corresponding to each source (eg, one or more sources 180 ). When executed by the one or more processors, the instructions also cause the one or more processors to send to a second device (e.g., device 120) data based on the direction-of-arrival information and classes or embedded data associated with the direction-of-arrival information. .

In some implementations, a non-transitory computer-readable medium (eg, computer-readable storage device, such as memory 3786 ) includes instructions (eg, instructions 3756 ) that when executed by a vehicle (eg, vehicle 1410 ) When executed by one or more processors (e.g., one or more processors 3710 or processor 3706), the instructions cause the one or more processors to receive multiple audio signals from multiple microphones (e.g., microphones 1402, 1404) signal (eg, audio signal 1470, 1472). When executed by one or more processors, the instructions also cause the one or more processors to process a plurality of audio signals to generate a sound (e.g., sound 1482 ) corresponding to one or more of the audio signals. or multiple sources (eg, one or more sources 1480 ) corresponding to the direction of arrival information (eg, direction of arrival information 1442 ). When executed by the one or more processors, the instructions also cause the one or more processors to generate a report (e.g., report 1446) indicating at least one detected event and the direction of the detected event based on the direction of arrival information ).

In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 124, memory 1514, or memory 3786) includes instructions (e.g., instructions 125, instructions 1515, or instructions 3756 ), which, when executed by one or more processors (eg, one or more processors 126, one or more processors 1516, one or more processors 3710, or processor 3706), cause one or more The plurality of processors receives an indication (eg, indication 902 , indication 1552 , or indication 1602 ) of an audio category corresponding to an audio event from a first device.

This case includes the first set of examples below.

Example 1 includes a first device comprising: a memory configured to store instructions; and one or more processors configured to: receive a plurality of audio signals from a plurality of microphones; process the plurality of audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and sending data based on the direction of arrival information to a second device.

Example 2 includes the first device according to example 1, wherein the memory and the one or more processors are integrated into the headset device, and wherein the second device corresponds to a mobile phone.

Example 3 includes the first device according to example 1, wherein the memory and the one or more processors are integrated into the mobile phone, and wherein the second device corresponds to a headset device.

Example 4 includes the first device of any one of Examples 1-3, wherein the data sent to the second device triggers activation of one or more sensors at the second device.

Example 5 includes the first device of any of Examples 1-4, wherein at least one of the one or more sensors comprises a non-audio sensor.

Example 6 includes the first device of any one of Examples 1-5, wherein the non-audio sensor includes a 360-degree camera.

Example 7 includes the first device of any one of Examples 1-6, wherein the non-audio sensor comprises a lidar sensor.

Example 8 includes the first device of any one of Examples 1-7, wherein the one or more processors include a first processing domain operating in a low power state.

Example 9 includes the first device of any one of Examples 1 to 8, wherein the one or more processors further include a second processing domain operating in a high power state, the second power domain being configured to process a plurality of audio signals to generate direction-of-arrival information.

Example 10 includes the first device of any one of Examples 1 to 9, wherein the one or more processors are further configured to: process the plurality of audio signals to perform audio event detection; Data corresponding to detected audio events.

Example 11 includes the first device of any one of Examples 1 to 9, wherein the one or more processors are further configured to: generate event data corresponding to the detected audio event based on the audio event detection operation ; and sending the event data to the second device.

Example 12 includes the first device of any one of Examples 1-11, wherein the one or more processors are further configured to: process the plurality of audio signals to perform acoustic environment detection; The data corresponding to the detected environment.

Example 13 includes the first device of any one of Examples 1-11, wherein the one or more processors are further configured to generate environment profile corresponding to the detected environment based on the acoustic environment detection operation.

Example 14 includes the first device of any one of Examples 1 to 13, wherein the one or more processors are further configured to: perform spatial processing on the plurality of audio signals based on the direction of arrival information to generate beamformed audio signals ; and sending the beamformed audio signal to the second device.

Example 15 includes the first device of any one of Examples 1-14, wherein the one or more processors are further configured to adjust the focus of at least one of the plurality of microphones based on the direction of arrival information.

Example 16 includes the first device of any one of Examples 1 to 15, further comprising a modem, wherein the material is sent to the second device via the modem.

Example 17 includes the first device of any one of Examples 1-16, wherein the one or more processors are further configured to send the plurality of representations of the audio signal to the second device.

Example 18 includes the first apparatus according to example 17, wherein the representations of the plurality of audio signals correspond to one or more beamformed audio signals.

Example 19 includes the first device of any one of Examples 1-18, wherein the one or more processors are further configured to generate a user interface output indicative of at least one of an environmental event or an acoustic event.

Example 20 includes the first device of any one of Examples 1-19, wherein the one or more processors are further configured to receive material indicative of the acoustic event from the second device.

Example 21 includes the first device of any one of Examples 1-20, wherein the one or more processors are further configured to receive material indicative of an environmental event from the second device.

Example 22 includes the first device of any one of Examples 1-21, wherein the one or more processors are further configured to receive data indicative of beamformed audio signals from the second device.

Example 23 includes the first device of any one of Examples 1-22, wherein the one or more processors are further configured to: receive direction information associated with the plurality of audio signals from the second device; and Info Performs an audio zoom operation.

Example 24 includes the first device of any one of Examples 1-23, wherein the one or more processors are further configured to: receive direction information associated with the plurality of audio signals from the second device; and information to perform noise removal operations.

Example 25 includes the first device of any one of Examples 1-24, further comprising a plurality of microphones.

Example 26 includes the first device of any one of Examples 1-25, further comprising at least one speaker configured to output a sound associated with at least one audio signal of the plurality of audio signals.

Example 27 includes the first device of any one of Examples 1-26, wherein the one or more processors are integrated in a vehicle.

Example 28 includes the first device of any one of Examples 1 to 27, wherein the direction-of-arrival information-based profile includes a report indicating at least one detected event and a direction of the detected event.

Example 29 includes a method of processing audio, the method comprising: receiving, at one or more processors of a first device, a plurality of audio signals from a plurality of microphones; processing the plurality of audio signals to generate and use one of the audio signals or direction-of-arrival information corresponding to one or more sources of sound represented; and sending data based on the direction-of-arrival information to a second device.

Example 30 includes the method of example 29, further comprising: processing the plurality of audio signals to perform audio event detection; and sending data corresponding to the detected audio events to the second device.

Example 31 includes the method according to example 30, wherein the audio event detection includes processing one or more of the plurality of audio signals at the one or more classifiers to select from among the plurality of classes supported by the one or more classifiers A category is determined for the sound represented by one or more of the audio signals, wherein data corresponding to the detected audio event includes an indication of the category.

Example 32 includes the method of any one of Examples 29-31, further comprising: processing the plurality of audio signals to perform acoustic environment detection; and sending data corresponding to the detected environment to the second device.

Example 33 includes the method of any one of Examples 29 to 32, wherein the profile is sent to the second device via a modem.

Example 34 includes the method of any one of Examples 29-33, further comprising sending a plurality of representations of the audio signal to the second device.

Example 35 includes the method of any one of Examples 29 to 34, wherein the data sent to the second device based on the direction of arrival information triggers activation of one or more sensors at the second device.

Example 36 includes the method of any of Examples 29-35, wherein at least one of the one or more sensors includes a non-audio sensor.

Example 37 includes an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method of any one of claims 29-36.

Example 38 includes an apparatus comprising a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of the first apparatus, cause the one or more processors to perform the requested The method of any one of 29 to 36.

Example 39 includes an apparatus comprising means for performing the method of any one of claims 29 to 36.

Example 40 includes a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of a first device, cause the one or more processors to: receive a plurality of audio from a plurality of microphones signals; process a plurality of audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and send data based on the direction of arrival information to a second device.

Example 41 includes the non-transitory computer-readable medium of example 40, wherein the data sent to the second device triggers activation of one or more sensors at the second device.

Example 42 includes the non-transitory computer-readable medium of Examples 41 or 42, wherein at least one of the one or more sensors includes a non-audio sensor.

Example 43 includes the non-transitory computer-readable medium of any of Examples 40-42, wherein the instructions are executable to further cause the one or more processors to send representations of the plurality of audio signals to the second device .

Example 44 includes the non-transitory computer-readable medium of example 43, wherein the representations of the plurality of audio signals correspond to one or more beamformed audio signals.

Example 45 includes a first device comprising: means for receiving a plurality of audio signals from a plurality of microphones; means for processing the plurality of audio signals to produce sound associated with one or more of the audio signals means for direction of arrival information corresponding to one or more sources; and means for sending data based on the direction of arrival information to a second device.

Example 46 includes a vehicle comprising: a memory configured to store instructions; and one or more processors configured to execute instructions to: receive a plurality of audio signals from a plurality of microphones; process a plurality of audio signals signal to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and generating an indication of at least one detected event and the detected event based on the direction of arrival information direction report.

Example 47 includes the vehicle of example 46, wherein the one or more processors are further configured to send the report to the second device.

Example 48 includes the vehicle of any one of Examples 46-47, wherein the second device comprises a second vehicle.

Example 49 includes the vehicle of any one of Examples 46-48, wherein the second device comprises a server.

Example 50 includes the vehicle of any one of Examples 46-49, wherein the one or more processors are further configured to: receive navigation instructions from the second device; and navigate based on the navigation instructions.

Example 51 includes the vehicle of any of tool examples 46-50, wherein the one or more processors are further configured to: receive a second report from the second device; and navigate based on the report and the second report.

Example 52 includes the vehicle of any one of Examples 46-51, wherein the one or more processors are further configured to: receive a second report from the second device; generate navigation instructions based on the second report; The command is sent to the second device.

Example 53 includes the vehicle of any one of examples 46 to 52, wherein the report indicates a list of detected events and directional information for the detected events over a period of time.

Example 54 includes a method of processing audio, the method comprising: receiving a plurality of audio signals from a plurality of microphones at one or more processors of a vehicle; processing the plurality of audio signals to generate one or more of the audio signals direction of arrival information corresponding to one or more sources of a plurality of represented sounds; and generating a report indicating at least one detected event and a direction of the detected event based on the direction of arrival information.

Example 55 includes the method of example 54, further comprising sending the report to the second device.

Example 56 includes the method of any of Examples 54-55, wherein the second device comprises a second vehicle.

Example 57 includes the method of any one of examples 54-56, wherein the second device comprises a server.

Example 58 includes the method of any one of Examples 54-57, further comprising: receiving navigation instructions from the second device; and navigating based on the navigation instructions.

Example 59 includes the method of any one of Examples 54 to 58, further comprising: receiving a second report from the second device; and navigating based on the report and the second report.

Example 60 includes the method of any one of Examples 54 to 59, further comprising: receiving a second report from the second device; generating navigation instructions based on the second report; and sending the navigation instructions to the second device.

Example 61 includes the method of any one of examples 54 to 60, wherein the report is a list indicating detected events and directional information for the detected events over a period of time.

Example 62 includes a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of a vehicle, cause the one or more processors to: receive a plurality of audio signals from a plurality of microphones; audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and generating an indication of at least one detected event and a detected event based on the direction of arrival information Reporting of the direction of the event.

Example 63 includes the non-transitory computer-readable medium of example 62, wherein when executed by the one or more processors, the instructions further cause the one or more processors to send a report to the second device.

Example 64 includes the non-transitory computer readable medium of any one of Examples 62-63, wherein the second device includes a second vehicle.

Example 65 includes the non-transitory computer readable medium of any one of Examples 62-64, wherein the second device comprises a server.

Example 66 includes the non-transitory computer-readable medium of any one of Examples 62-65, wherein when executed by the one or more processors, the instructions further cause the one or more processors to: The second device receives the navigation instruction; and performs navigation based on the navigation instruction.

Example 67 includes the non-transitory computer-readable medium of any of Examples 62-66, wherein when executed by the one or more processors, the instructions further cause the one or more processors to: The second device receives the second report; and navigates based on the report and the second report.

Example 68 includes the non-transitory computer-readable medium of any of Examples 62-67, wherein when executed by the one or more processors, the instructions further cause the one or more processors to: The second device receives the second report; generates navigation instructions based on the second report; and sends the navigation instructions to the second device.

Example 69 includes the non-transitory computer-readable medium of any one of Examples 62-68, wherein the report indicates a list of detected events and directional information for the detected events over a period of time.

Example 70 includes a vehicle comprising: means for receiving a plurality of audio signals from a plurality of microphones; a means for processing the plurality of audio signals to produce a sound associated with one or more of the audio signals or means for direction of arrival information corresponding to a plurality of sources; and means for generating a report indicating at least one detected event and a direction of the detected event based on the direction of arrival information.

Example 71 includes the vehicle of example 70, further comprising means for sending the report to the second device.

Example 72 includes the vehicle of any one of examples 70-71, wherein the second device comprises a second vehicle.

Example 73 includes the vehicle of any one of Examples 70-72, wherein the second device includes a server.

Example 74 includes the vehicle of any one of examples 70 to 73, wherein the report indicates a list of detected events and directional information for the detected events over a period of time.

Example 75 includes the vehicle of any one of Examples 70-74, further comprising a device that performs autonomous navigation based on the report.

This case includes the following second set of examples.

According to example 1, a first device includes: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; direction-of-arrival information corresponding to one or more sources of the one or more represented sounds; and sending to the second device data based on the direction-of-arrival information and the category or embedding associated with the direction-of-arrival information.

Example 2 includes the first apparatus according to example 1, wherein the one or more processors are further configured to process signal data corresponding to the audio signal to determine a class or embedding.

Example 3 includes the first apparatus according to example 2, wherein the one or more processors are further configured to perform a beamforming operation on the audio signal to generate signal data.

Example 4 includes the first device according to example 2 or example 3, wherein the one or more processors are further configured to process the signal data at the one or more classifiers to obtain information from multiple classes supported by the one or more classifiers Among the categories are sounds represented by one or more of the audio signals and associated with the audio event determining a category, and wherein the category is sent to the second device.

Example 5 includes the first device according to any one of Examples 2 to 4, wherein the one or more processors are further configured to process the signal material at the one or more encoders to produce an embedding corresponding to One or more sounds represented in the audio signal and associated with the audio event and embedded therein are sent to the second device.

Example 6 includes the first device of any one of Examples 1 to 5, wherein the one or more processors are further configured to process the image data at the one or more encoders to produce an embedding corresponding to An object represented by image data and associated with an audio event, and wherein the embedding is sent to the second device.

Example 7 includes the first device according to example 6, further including one or more cameras configured to generate image material.

Example 8 includes the first device of any one of Examples 1-7, wherein the one or more processors are further configured to generate environment profile corresponding to the detected environment based on the acoustic environment detection operation.

Example 9 includes the first device of any one of Examples 1 to 8, wherein the one or more processors are further configured to: perform spatial processing on the audio signal based on the direction of arrival information to generate one or more beamforming an audio signal; and sending one or more beamformed audio signals to a second device.

Example 10 includes the first device of any one of Examples 1 to 9, wherein the memory and the one or more processors are integrated into the headset device, and wherein the second device corresponds to a mobile phone.

Example 11 includes the first device of any one of Examples 1-9, wherein the one or more processors are integrated in a vehicle.

Example 12 includes the first device of any one of Examples 1 to 11, further comprising a modem, wherein the material is sent to the second device via the modem.

Example 13 includes the first device of any one of Examples 1-12, wherein the one or more processors are further configured to send the representation of the audio signal to the second device.

Example 14 includes the first apparatus according to example 13, wherein the representation of the audio signal corresponds to one or more beamformed audio signals.

Example 15 includes the first device of any one of Examples 1-14, wherein the one or more processors are further configured to generate a user interface output indicative of at least one of an environmental event or an acoustic event.

Example 16 includes the first device of any one of Examples 1-15, wherein the one or more processors are further configured to receive material indicative of the acoustic event from the second device.

Example 17 includes the first device of any one of Examples 1-16, wherein the one or more processors are further configured to: receive direction information associated with the audio signal from the second device; and perform Audio zoom operation.

Example 18 includes the first device of any one of Examples 1 to 17, wherein the direction-of-arrival information-based profile includes a report indicating at least one detected event and a direction of the detected event.

Example 19 includes the first device of any one of Examples 1-18, further comprising a plurality of microphones.

Example 20 includes the first device of any one of Examples 1-19, further comprising at least one speaker configured to output a sound associated with the at least one audio signal.

Example 21 includes the first device according to any one of Examples 1 to 20, wherein: the category corresponds to a classification of a particular sound represented in the audio signal and associated with a particular audio event; The signature or information corresponding to the audio event is configured to be able to detect specific sounds or specific audio events in other audio signals through the processing of other audio signals.

According to example 22, a system comprising: the first device according to any one of examples 1-21; and a second device comprising: one or more processors configured to: receive data; and The data is processed to verify the class, modify the audio data representing the sound scene based on the direction of arrival information and embedding to generate modified audio data corresponding to the updated sound scene, or both.

According to example 23, a system includes: a first device including: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; direction of arrival information corresponding to one or more sources of sound represented by the one or more representations in the audio signal; and transmitting data based on the direction of arrival information and a category associated with the direction of arrival information; and a second device comprising one or a plurality of processors, one or more processors configured to: receive data based on the direction of arrival information and category; obtain audio data and category representing sounds associated with the direction of arrival information; and based at least on the audio data and the direction of arrival information to verify the category.

According to example 24, a system includes: a first device including: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; direction of arrival information corresponding to one or more sources of sound represented by the one or more representations in the audio signal; and transmitting embedded data based on and associated with the direction of arrival information; and a second device comprising one or A plurality of processors, one or more processors configured to: receive data based on direction of arrival information and embedding; and process audio data representing a sound scene based on direction of arrival information and embedding to generate an updated sound scene corresponding to The corresponding modified audio data.

According to example 25, a method of processing audio comprises: receiving, at one or more processors of a first device, audio signals from a plurality of microphones; processing the audio signals to produce a sound corresponding to one or more of the audio signals and sending direction-of-arrival information corresponding to one or more sources of the direction-of-arrival information to the second device based on the direction-of-arrival information and the category or embedded data associated with the direction-of-arrival information.

Example 26 includes the method according to Example 25, further comprising processing signal data corresponding to the audio signal to determine the class or embedding.

Example 27 includes the method of Example 26, further comprising performing a beamforming operation on the audio signal to generate signal data.

Example 28 includes the method according to example 26 or example 27, wherein the signal data is processed at one or more classifiers to generate one or more of the audio signals from a plurality of classes supported by the one or more classifiers The sound represented and associated with the audio event determines the category, and wherein the category is sent to the second device.

Example 29 includes the method according to any one of Examples 26 to 28, wherein the signal data is processed at one or more encoders to produce embeddings corresponding to and associated with one or more of the audio signals The audio event is associated with a sound, and an embed thereof is sent to the second device.

Example 30 includes the method of any one of Examples 25-29, further comprising sending the representation of the audio signal to the second device.

Example 31 includes the method of any one of Examples 25-30, further comprising: receiving, at one or more processors of the second device, data based on direction-of-arrival information and category; obtaining, at a processor, audio data representing a sound associated with the direction of arrival information and a category; and at one or more processors of a second device, verifying the category based on at least the audio data and the direction of arrival information.

Example 32 includes the method of any one of Examples 25-31, further comprising: receiving, at one or more processors of the second device, the data based on the direction of arrival information and the embedding; and at one or more processors of the second device The audio data representing the sound scene is processed at a plurality of processors based on the direction of arrival information and the embedding to generate modified audio data corresponding to the updated sound scene.

According to example 33, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of examples 25-30.

According to Example 34, a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause one or more processors to perform the method according to any one of Examples 25-30 method.

According to example 35, an apparatus comprises means for implementing the method according to any one of examples 25-30.

According to example 36, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause the one or more processors to: receive audio signals from a plurality of microphones; processing the audio signal to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and sending to a second device based on the direction-of-arrival information and a category associated with the direction-of-arrival information or embedded data.

Example 37 includes the non-transitory computer-readable medium of example 36, wherein the instructions are executable to further cause the one or more processors to send the representation of the audio signal to the second device.

Example 38 includes the non-transitory computer-readable medium of example 37, wherein the representation of the audio signal corresponds to one or more beamformed audio signals.

According to example 39, a first apparatus comprises: means for receiving audio signals from a plurality of microphones; and processing the audio signals to produce one or more sources corresponding to sounds represented by one or more of the audio signals means for the direction-of-arrival information; and means for sending to the second device data based on the direction-of-arrival information and the category or embedding associated with the direction-of-arrival information.

This case includes the following third set of examples.

According to example 1, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from a first device an indication of an audio category corresponding to an audio event.

Example 2 includes the second device according to example 1, wherein the one or more processors are further configured to: receive audio data representing sounds associated with the audio event from the first device; and process at the one or more classifiers Audio data to verify that the sound corresponds to an audio event.

Example 3 includes the second apparatus according to example 2, wherein the one or more processors are configured to provide as input the audio data and the indication of the audio category to the one or more classifiers to determine a classification associated with the audio material.

Example 4 includes the second device according to example 2 or example 3, wherein the audio category corresponds to a vehicle event, and wherein the one or more processors are further configured to The location sends a notification of the vehicle event to one or more third devices.

Example 5 includes the second device of any one of Examples 2-4, wherein the one or more processors are further configured to send a control signal to the first device based on the output of the one or more classifiers.

Example 6 includes the second device of example 5, wherein the control signal instructs the first device to perform an audio zoom operation.

Example 7 includes the second device according to example 5 or example 6, wherein the control signal instructs the first device to perform spatial processing based on the direction of the sound source.

Example 8 includes the second device of any one of Examples 2-7, wherein the one or more processors are further configured to: receive direction data corresponding to sound sources from the first device; and convert the audio data, The directional data and an indication of the audio class are provided as input to one or more classifiers to determine a class associated with the audio data.

Example 9 includes the second device of any one of Examples 2-8, wherein the audio data includes one or more beamforming signals.

Example 10 includes the second device of any one of Examples 1-9, wherein the one or more processors are further configured to: receive, from the first device, directional data corresponding to a sound source associated with the audio event; updating a map of directional sound sources in the audio scene based on the audio event to generate an updated map; and sending data corresponding to the updated map to one or more third devices geographically remote from the first device.

Example 11 includes the second device of any one of Examples 1-10, wherein the memory and the one or more processors are integrated into a mobile phone, and wherein the first device corresponds to a headset device.

Example 12 includes the second device of any one of Examples 1-10, wherein the memory and the one or more processors are integrated into a vehicle.

Example 13 includes the second device of any one of Examples 1-12, further comprising a modem, wherein the indication of the audio category is received via the modem.

Example 14 includes the second device of any one of Examples 1-13, wherein the one or more processors are configured to selectively bypass checking of received direction-of-arrival information based on whether direction-of-arrival information is received from the first device. Direction-of-arrival processing of audio data corresponding to an audio event.

Example 15 includes the second device of any one of Examples 1-14, wherein the one or more processors are configured to, based on whether the received audio material corresponds to a multi-channel microphone signal from the first device or to The beamforming signal from the first device selectively bypasses the beamforming operation.

Example 16 includes the second device of any one of Examples 1-15, wherein the audio category corresponds to a classification of a particular sound represented by the audio signal and associated with the audio event.

According to example 17, a system comprising: the second device according to any one of examples 1 to 16; and a first device comprising: one or more processors configured to: A microphone receives an audio signal; processes the audio signal to determine an audio type; and sends an indication of the audio type to a second device.

According to example 18, a system includes: a first device comprising: one or more processors configured to: receive audio signals from one or more microphones; process the audio signals to determine an audio category corresponding to an audio event; and sending an indication of an audio class; and a second device comprising one or more processors configured to: receive an indication of an audio class corresponding to an audio event.

According to example 19, a method comprises: receiving at one or more processors of a second device an indication of an audio class received from a first device and corresponding to an audio event; and processing at one or more processors of the second device The audio data is processed by the device to verify that the sound represented by the audio data corresponds to the audio event.

Example 20 includes the method of example 19, further comprising receiving audio data from the first device, and wherein processing the audio data includes providing the audio data as input to one or more classifiers to determine a classification associated with the audio data.

Example 21 includes the method of example 20, wherein the processing of the audio data further includes providing an indication of the audio category as a second input to one or more classifiers to determine a classification associated with the audio material.

Example 22 includes the method of example 20 or example 21, further comprising sending a control signal to the first device based on the output of the one or more classifiers.

Example 23 includes the method of example 22, wherein the control signal includes an audio zoom instruction.

Example 24 includes the method according to example 22 or example 23, wherein the control signal includes instructions for performing spatial processing based on the direction of the sound source.

Example 25 includes the method of any one of Examples 19 to 24, wherein the audio category corresponds to a vehicle event, and the method further comprises reporting to one or more third devices based on the location of the first device and the location of the one or more third devices. A plurality of third devices send notifications of vehicle events.

Example 26 includes the method of any one of Examples 19 to 25, further comprising: receiving from the first device directional data corresponding to sound sources associated with the audio event; updating the directional sound source in the audio scene based on the audio event to generate an updated map; and sending data corresponding to the updated map to one or more third devices geographically distant from the first device.

Example 27 includes the method of any one of Examples 19 to 26, further comprising selectively bypassing the arrival of the received audio data corresponding to the audio event based on whether direction of arrival information is received from the first device. Orientation processing.

Example 28 includes the method of any one of Examples 19 to 27, further comprising selecting based on whether the received audio material corresponds to a multi-channel microphone signal from the first device or to a beamformed signal from the first device to bypass the beamforming operation.

Example 29 includes the method of any one of Examples 19-28, further comprising: receiving, at one or more processors of the first device, audio signals from one or more microphones; Process the audio signal at multiple processors to determine the audio type; and send an indication of the audio type from the first device to the second device.

According to example 30, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of examples 16-28.

According to Example 31, a non-transitory computer-readable medium comprising instructions which, when executed by one or more processors, cause one or more processors to perform the method according to any one of Examples 16-29 method.

According to example 32, an apparatus comprises means for performing the method according to any one of examples 16-28.

According to example 33, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause the one or more processors to receive from a first device An indication of the corresponding audio class.

Example 34 includes the non-transitory computer-readable medium of Example 33, wherein the instructions are executable to further cause the one or more processors to: receive audio data representing sounds associated with the audio event from the first device; and The audio data is processed at one or more classifiers to verify that sounds correspond to audio events.

Example 35 includes the non-transitory computer-readable medium of Example 34, wherein the instructions are executable to further cause the one or more processors to provide as input the audio data and the indication of the audio category to the one or more classifiers, to determine the category associated with the audio data.

Example 36 includes the non-transitory computer-readable medium of Example 34 or Example 35, wherein the instructions are executable to further cause the one or more processors to: receive direction data corresponding to a sound source from the first device; and The audio data, direction data, and indication of audio category are provided as input to one or more classifiers to determine a category associated with the audio data.

According to example 37, an apparatus comprising: means for receiving an indication of an audio class, the indication being received from a remote device and corresponding to an audio event; and for processing audio data to verify that a sound represented by the audio data corresponds to the audio The component of the event.

According to example 38, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from the first device: audio data representing sounds; an indication of an audio category associated with the vehicle event; processing the audio data at one or more classifiers to verify that the sound represented by the audio data corresponds to the vehicle event; and based on the location of the first device and one or more third The location of the device sends a notification of the vehicle event to one or more third devices.

According to example 39, a method comprises: receiving, at one or more processors of a second device, audio data from a first device and an indication that the audio data from the first device corresponds to an audio category associated with a vehicle event; Processing the audio data at one or more classifiers of the second device to verify that the sound represented by the audio data corresponds to a vehicle event; A plurality of third devices send notifications of vehicle events.

According to example 40, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to example 39.

According to Example 41, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause the one or more processors to perform the method according to Example 39.

According to example 42, an apparatus comprises means for performing the method according to example 39.

According to example 43, a first apparatus comprises: a memory configured to store instructions; and one or more processors configured to: receive one or more audio signals from one or more microphones; process one or more an audio signal to determine whether a sound represented by one or more of the audio signals is from an identifiable direction; and selectively send direction-of-arrival information of the sound source to the second device based on the determination.

According to example 44, a method comprises: receiving, at one or more processors of a first device, one or more audio signals from one or more microphones; processing, at the one or more processors, the one or more audio signals , to determine whether the sound represented by one or more of the audio signals is from an identifiable direction; and selectively send arrival direction information of the sound source to the second device based on the determination.

According to example 45, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to example 44.

According to Example 46, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause the one or more processors to perform the method according to Example 44.

According to example 47, an apparatus comprises means for performing the method according to example 44.

According to example 48, a first apparatus comprises: a memory configured to store instructions; and one or more processors configured to: receive one or more audio signals from one or more microphones; determining whether to send one or more audio signals to the second device or sending a beamformed audio signal generated based on the one or more audio signals to the second device; and sending the audio signal to the second device based on the decision. The audio data corresponding to the signal or corresponding to the beamformed audio signal.

According to example 49, a method comprises: receiving, at one or more processors of a first device, one or more audio signals from one or more microphones; at the one or more processors, based on one or more criteria, deciding whether to send one or more audio signals to a second device or to send a beamformed audio signal generated based on the one or more audio signals to the second device; and sending the one or more audio signals to the second device based on the decision Audio data corresponding to or corresponding to the beamformed audio signal.

According to example 50, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to example 49.

According to example 51 , a non-transitory computer-readable medium comprising instructions which, when executed by one or more processors of a first device, cause the one or more processors to perform the method according to example 49.

According to example 52, an apparatus comprises means for performing the method according to example 49.

According to example 53, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from a first device: audio data representing a sound; a direction corresponding to a sound source data; and classification of sounds corresponding to the audio event; processing the audio data to verify that the sound corresponds to the audio event; updating a map of directional sound sources in the audio scene based on the audio event to generate an updated map; and moving geographically away The one or more third devices of the first device transmit data corresponding to the updated map.

According to example 54, a method comprises receiving, at one or more processors of a second device, audio data representative of sounds, direction data corresponding to sound sources, and classifications of sounds corresponding to audio events, the audio data, The orientation data and classification are received from the first device; the audio data is processed at the one or more processors to verify that the sound corresponds to the audio event; the orientation in the audio scene is updated at the one or more processors based on the audio event a map of sound sources to generate an updated map; and sending data corresponding to the updated map to one or more third devices geographically remote from the first device.

According to example 55, an apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to example 54.

According to example 56, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause the one or more processors to perform the method according to example 54.

According to example 57, an apparatus comprises means for performing the method according to example 54.

Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or A combination of both. Various illustrative elements, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor-executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The steps of the method or algorithm described in connection with the embodiments disclosed herein may be directly embodied in hardware, a software module executed by a processor, or a combination of both. Software modules can be resident in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory ( EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), scratchpad, hard disk, removable disk, compact disk read-only memory (CD-ROM), or any other form known in the art in non-transitory storage media. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an application specific integrated circuit (ASIC). The ASIC can be resident in a computing device or a user terminal. In the alternative, the processor and storage medium may be resident as separate components in the computing device or user terminal.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present application is not intended to be limited to the aspects shown herein, but is to be accorded the widest possible scope consistent with the principles and novel features defined by the appended claims.

100: system 102: Microphone 104: Microphone 106: Microphone 108: Microphone 110: Equipment 111: the first input interface 112: Second input interface 114: memory 115: instruction 116: Processor 118: modem 120: Equipment 121: the first input interface 122: Second input interface 124: memory 125: instruction 126: Processor 128: modem 129: sensor 132: Processing unit 134: Audio event processing unit 136: Acoustic environment processing unit 138: Beamforming unit 142: Arrival direction information 143: Arrival direction information 144:Audio event information 145:Audio event information 146:Environmental information 147: Environmental information 148: Beamforming audio signal 149: Beamforming audio signal 152: Arrival direction processing unit 154: Audio event processing unit 156: Acoustic environment processing unit 158: Beamforming unit 170: audio signal 172:Audio signal 174: Audio frame 176: Audio frame 178: Audio data 180: source 182: sound 190: audio signal 192: audio signal 194: Audio frame 196: Audio frame 198: Audio data 200: system 202: Processor 210: First processing domain 220: second processing domain 230: audio preprocessing unit 232: Arrival direction processing unit 234: Audio event processing unit 236: Acoustic environment processing unit 238: Beamforming unit 250: Direction information 252: start signal 274: Audio frame 276: Audio frame 278: Audio data 300: system 310: Headphones 320: mobile phone 330:Audio processing unit 332: audio scaling unit 334: user prompt generating unit 336:Speaker 350: User Alert 352: User Alert 400: system 410: Headphones 420: mobile phone 430:Audio processing unit 432: Audio scaling unit 434: Noise removal unit 436:Speaker 440: First Voice Information 442:Second Voice Information 450:Single microphone audio context detection unit 452:Audio adjustment unit 454: Mode Controller 460: audio zoom angle 462: Noise reduction parameters 464: mode signal 490: Noise reduction signal 496:Context information 500: system 502: Spatial filtering processing unit 504: Audio event processing unit 506: Application Programming Interface 508:Voice user interface 510: output 512: output 514: output 542: Arrival direction information 544:Audio event information 574:Audio frame 576:Audio frame 600: Implementation 602: signal data 610: Classifier 612: Category 614: Category 616: instruction 700: Implementation 710: Encoder 712:Embed 716: instruction 900: Implementation 902: instruction 904: audio data 910: Beamforming signal 912: Direction data 920: Classifier 922: classification 924: category 930: Notification 932: control signal 934: Classifier output 1000: another implementation 1002: Multi-channel audio signal 1100: Implementation 1102: input mixed waveform 1104: Embed 1106: target output 1120: content separator 1122:Audio generation network 1150: icon 1151: Audio scene 1152: The first atmosphere 1154:Foreground sound 1156:Foreground sound 1158:Foreground sound 1160: Foreground extraction operation 1162: Foreground sound 1164: Scene generation operation 1171: Audio scene 1172:Second Atmosphere 1200: method 1202: block 1204: block 1206: block 1208: block 1210: block 1212: block 1214: block 1220: block 1230: block 1232: square 1234: block 1236: block 1238: square 1240: block 1242: block 1244: block 1246: block 1248: block 1300: method 1302: block 1304: block 1306: cube 1308: cube 1310: block 1312: block 1314: block 1320: block 1322: block 1330: block 1332: block 1340: block 1342: block 1350: block 1400: system 1402: Microphone 1404:Microphone 1410: Transportation 1411: the first input interface 1412: Second input interface 1414: memory 1415: instruction 1416: Processor 1420: Equipment 1432: Arrival direction processing unit 1434: Audio event processing unit 1436: Report Generator 1438: Navigation instruction generator 1442: Arrival direction information 1444:Audio event information 1446: report 1448:Navigation command 1456: Second report 1458:Navigation command 1470: audio signal 1472:Audio signal 1474:Audio frame 1476: Audio frame 1478: Audio data 1480: source 1482: sound 1490:Other equipment 1492: Notice 1500: system 1502: Vehicle event 1504: Audio category 1510: Transportation 1514: memory 1515: instruction 1516: Processor 1520: Equipment 1522: classification 1530: classifier 1550: audio data 1552: instruction 1602: instruction 1604: Direction information 1612: Map Updater 1614: map 1616: map 1618: Audio scene renderer 1660: data 1670: Equipment 1672: Equipment 1674: equipment 1700: 3D audio map 1702: user 1710: The first means of transportation 1712: Second means of transportation 1714: dog bark 1716: Talker 1718: Crosswalk timer 1720: Artificial Sound 1802: Directed audio scene 1804: user 1810: First representative speaker 1812: Second representative speaker 1814: Third representative speaker 1820: Operation 1830: Directed Audio Scenarios 1832: Virtual Participants 1834: Virtual Participants 1900: Implementation 1902: Integrated circuits 1904: Audio input 1906: Signal output 1916: Processor 1990: Directional audio signal processing unit 1992: Directional audio signal data 2000: Implementation 2002: Mobile devices 2004: display screen 2100: Implementation 2102: Headphone equipment 2200: Implementation 2202: Wearable Electronic Devices 2204: display screen 2300: Implementation 2302:Voice activated device 2304:Speaker 2400: Implementation method 2402: Camera equipment 2500: Implementation 2502: Headphones 2600: Implementation method 2602: Augmented or mixed reality glasses 2604: Holographic projection unit 2606: Lens 2700: Implementation 2702: The first earplug 2704:Second earplug 2706: A pair of earplugs 2720: the first microphone 2722A: Microphone 2722B: Microphone 2722C: Microphone 2724: "Internal" microphone 2726: voice microphone 2730:Speaker 2800: Implementation 2802: Transportation 2850: Directional Audio Signal Processing Unit 2900: another implementation 2910: Loudspeaker 2920:Display 2950: Directional Audio Signal Processing Unit 3000: method 3002: block 3004: block 3006: block 3100: method 3102: block 3104: block 3106: block 3200: method 3202: block 3204: block 3300: method 3302: block 3304: block 3306: block 3400: method 3402: block 3404: block 3406: block 3500: method 3502: block 3504: block 3506: block 3600: method 3602: block 3604: block 3606: block 3608: block 3700: Equipment 3702: Digital to Analog Converter (DAC) 3704: Analog to Digital Converter (ADC) 3706: Processor 3708: Speech and music transcoder-decoder (transcoder, CODEC) 3710: Processor 3722: System-in-Package or System-on-Chip Devices 3726: display controller 3728:Display 3730: input device 3734:CODEC 3736: Speech Transcoder ("Vocoder") Encoder 3738: Vocoder decoder 3744:Power 3750: Transceiver 3752: Antenna 3756: command 3770: modem 3786:Memory 3792:Speaker FG1: first foreground sound FG2: second foreground voice FG3: third foreground voice

1 is a block diagram of certain illustrative aspects of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

2 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

3 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

4 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

5 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

6 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

7 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

8 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

9 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

10 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

FIG. 11 is a block diagram of another specific illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, and includes an aspect of audio content separation, according to some examples of the present disclosure. Graphic illustration.

12 is a diagram of a particular implementation of operations that may be performed in an audio processing device, according to some examples of the present disclosure.

13 is a diagram of another particular implementation of operations that may be performed in an audio processing device, according to some examples of the present disclosure.

14 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

15 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

16 is a block diagram of another particular illustrative aspect of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

17 illustrates an example of an audio scene including multiple directional sound sources that may be determined through directional processing of one or more audio signals received from one or more microphones, according to some examples of the present disclosure.

18 illustrates an example of a shared audio scene including multiple directional sound sources, according to some examples of the present disclosure.

19 illustrates an example of an integrated circuit including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

20 is a diagram of a mobile device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

21 is a diagram of a headset including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

22 is a diagram of a wearable electronic device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

23 is a diagram of a voice-activated speaker system including a directional audio signal processing unit for generating directional audio signal material, according to some examples of the present disclosure.

24 is a diagram of a camera including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

25 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, including a directional audio signal processing unit for generating directional audio signal material, according to some examples of the present disclosure.

26 is a diagram of a mixed reality or augmented reality glasses device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

27 is a diagram of an earbud including a directional audio signal processing unit for generating directional audio signal data, according to some examples of the present disclosure.

28 is a diagram of a first example of a vehicle including a directional audio signal processing unit for navigating the vehicle, according to some examples of the present disclosure.

29 is a diagram of a second example of a vehicle including a directional audio signal processing unit for navigating the vehicle, according to some examples of the present disclosure.

30 is a diagram of a particular implementation of a method of processing audio according to some examples of the present disclosure.

FIG. 31 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

32 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

FIG. 33 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

FIG. 34 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

FIG. 35 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

FIG. 36 is a diagram of another particular embodiment of a method of processing audio according to some examples of the present disclosure.

37 is a block diagram of a particular illustrative example of an apparatus operable to perform directional processing on one or more audio signals received from one or more microphones, in accordance with some examples of the present disclosure.

Domestic deposit information (please note in order of depositor, date, and number) none Overseas storage information (please note in order of storage country, institution, date, and number) none

100: system

102: Microphone

104: Microphone

106: Microphone

108: Microphone

110: Equipment

111: the first input interface

112: Second input interface

114: memory

115: instruction

116: Processor

118: modem

120: Equipment

121: the first input interface

122: Second input interface

124: memory

125: instruction

126: Processor

128: modem

129: sensor

132: Processing unit

134: Audio event processing unit

136: Acoustic environment processing unit

138: Beamforming unit

142: Arrival direction information

143: Arrival direction information

144:Audio event information

145:Audio event information

146:Environmental information

147: Environmental information

148: Beamforming audio signal

149: Beamforming audio signal

152: Arrival direction processing unit

154: Audio event processing unit

156: Acoustic environment processing unit

158: Beamforming unit

170: audio signal

172:Audio signal

174: Audio frame

176: Audio frame

178: Audio data

180: source

182: sound

190: audio signal

192: audio signal

194: Audio frame

196: Audio frame

198: Audio data

Claims (30)

A first device comprising: a memory configured to store instructions; and One or more processors configured to: Receive audio signals from multiple microphones; processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and Sending to a second device based on the direction of arrival information and a class or embedded data associated with the direction of arrival information. The first apparatus according to claim 1, wherein the one or more processors are further configured to process signal data corresponding to the audio signals to determine the class or embedding. The first apparatus according to claim 2, wherein the one or more processors are further configured to perform a beamforming operation on the audio signals to generate the signal data. The first apparatus according to claim 2, wherein the one or more processors are further configured to process the signal data at one or more classifiers to select from a plurality of classes supported by the one or more classifiers The category is determined for a sound represented by one or more of the audio signals and associated with an audio event, and wherein the category is sent to the second device. The first apparatus according to claim 2, wherein the one or more processors are further configured to process the signal data at one or more encoders to generate the embedding corresponding to the One or more representations of a sound are associated with an audio event, and wherein the embedding is sent to the second device. The first apparatus according to claim 1, wherein the one or more processors are further configured to process image data at one or more encoders to generate the embedding corresponding to a representation of the image data The object is also associated with an audio event, and wherein the embedding is sent to the second device. The first device according to claim 6, further comprising one or more cameras configured to generate the image data. The first device according to claim 1, wherein: the category corresponds to a classification of a particular sound represented by the audio signals and associated with a particular audio event; and The embedding includes a signature or information corresponding to the specific sound or specific audio event and is configured to enable detection of the specific sound or the specific audio event in other audio signals through processing of other audio signals. The first device according to claim 1, wherein the one or more processors are further configured to: performing spatial processing on the audio signals based on the direction of arrival information to generate one or more beamformed audio signals; and The one or more beamformed audio signals are sent to the second device. The first device according to claim 1, wherein the memory and the one or more processors are integrated into an earphone device, and wherein the second device corresponds to a mobile phone. According to claim 1, the first device further comprises a modem, wherein the data is sent to the second device via the modem. The first device according to claim 1, wherein the one or more processors are further configured to send a representation of the audio signals to the second device. The first apparatus according to claim 12, wherein the representations of the audio signals correspond to one or more beamformed audio signals. The first apparatus according to claim 1, wherein the one or more processors are further configured to generate a user interface output indicative of at least one of an environmental event or an acoustic event. The first device according to claim 1, wherein the one or more processors are further configured to receive data indicative of an acoustic event from the second device. The first device according to claim 1, wherein the one or more processors are further configured to: receive direction information associated with the audio signal from the second device; and Performs an audio zoom operation based on the orientation information. The first device according to claim 1, wherein the one or more processors are integrated in a vehicle. The first apparatus according to claim 1, wherein the data based on the direction of arrival information includes a report indicating at least one detected event and the direction of the detected event. According to the first device of claim 1, further comprising a plurality of microphones. The first device according to claim 1, further comprising at least one speaker configured to output a sound associated with at least one of the audio signals. A method of processing audio, the method comprising: receiving, at one or more processors of a first device, audio signals from a plurality of microphones; processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and Sending to a second device based on the direction of arrival information and a class or embedded data associated with the direction of arrival information. The method according to claim 21, further comprising processing signal data corresponding to the audio signals to determine the class or embedding. The method according to claim 22, further comprising performing a beamforming operation on the audio signals to generate the signal data. The method according to claim 22, wherein the signal data is processed at one or more classifiers to represent one or more of the audio signals from a plurality of classes supported by the one or more classifiers The sound associated with the audio event determines the category, and wherein the category is sent to the second device. The method according to claim 22, wherein the signal data is processed at one or more encoders to generate the embedding corresponding to sounds represented by one or more of the audio signals and associated with audio events , and wherein the embedding is sent to the second device. The method according to claim 21, further comprising sending a representation of the audio signal to the second device. According to the method of claim 21, further comprising: at one or more processors of the second device, receiving data based on the direction of arrival information and the category; obtaining, at one or more processors of the second device, audio data representing a sound associated with the direction-of-arrival information and the category; and At one or more processors of the second device, the class is verified based on at least the audio data and the direction-of-arrival information. According to the method of claim 21, further comprising: at one or more processors of the second device, receiving data based on the direction of arrival information and the embedded data; and At one or more processors of the second device, audio data representing a sound scene is processed based on the direction of arrival information and the embedding to generate modified audio data corresponding to an updated sound scene. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a first device, cause the one or more processors to: Receive audio signals from multiple microphones; processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and Sending to a second device based on the direction of arrival information and a class or embedded data associated with the direction of arrival information. A first device comprising: means for receiving audio signals from multiple microphones; means for processing the audio signals to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and Means for sending to a second device based on the direction of arrival information and a class or embedded data associated with the direction of arrival information.

Images