KR20230113853A - Psychoacoustic reinforcement based on audio source directivity - Google Patents

Abstract

The device includes memory configured to store directional data of one or more audio sources corresponding to one or more input audio signals. The device also includes one or more processors configured to determine one or more equalizer settings based at least in part on the directivity data. The one or more processors are also configured to generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings.

Description

Psychoacoustic reinforcement based on audio source directivity

I. related CROSS REFERENCES TO APPLICATIONS

This application claims the benefit of priority from commonly owned US Provisional Patent Application Serial No. 17/162,241, filed on January 29, 2021, the contents of which are expressly incorporated herein by reference in their entirety.

II. Field

This disclosure relates generally to psychoacoustic enhancement based on audio source directivity.

III. Description of related technology

Advances in technology have resulted in smaller and more powerful computing devices. For example, a variety of portable personal computing devices currently exist, including wireless telephones such as mobile and smart phones, tablets, and laptop computers that are small, lightweight, and easily carried by users. These devices can communicate voice and data packets over wireless networks. Additionally, many such devices incorporate additional functionality such as digital still cameras, digital video cameras, digital recorders, and audio file players. Also, such devices may process executable instructions including software applications, such as a web browser application that may be used to access the Internet. As such, these devices may include significant computing capabilities.

Such computing devices often incorporate functionality for receiving an audio signal from one or more microphones. For example, an audio signal may represent user speech captured by microphones, ambient sounds captured by microphones, or a combination thereof. User speech may be difficult to hear in the audio signal due to the distance of the microphones from the user at which the user speech is captured by the microphones. For example, microphones further away from the user may capture more ambient sounds, such as traffic, speech of other users, and the like. As another example, user speech sounds softer when captured by more distant microphones. The ability to focus on specific sounds in an audio signal is useful for a variety of applications, such as communicating user speech more clearly in a communication application or voice controlled assistant system.

IV. summary

According to one implementation of the present disclosure, a device includes a memory configured to store directivity data of one or more audio sources corresponding to one or more input audio signals. The device also includes one or more processors configured to determine one or more equalizer settings based at least in part on the directivity data. The one or more processors are also configured to generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings.

According to another implementation of the present disclosure, a method includes obtaining, at a device, directional data of one or more audio sources corresponding to one or more input audio signals. The method also includes determining, at the device, one or more equalizer settings based at least in part on the directional data. The method further includes generating one or more output audio signals corresponding to the psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings.

According to another implementation of the present disclosure, a non-transitory computer readable medium, when executed by one or more processors, causes the one or more processors to perform one or more audio signals corresponding to one or more input audio signals. Stores instructions that allow obtaining directivity data from sources. The instructions, when executed by one or more processors, also cause the one or more processors to determine one or more equalizer settings based at least in part on the directivity data. The instructions, when executed by one or more processors, also cause the one or more processors to generate, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals. let it

According to another implementation of the present disclosure, an apparatus includes means for obtaining directional data of one or more audio sources corresponding to one or more input audio signals. The apparatus also includes means for determining one or more equalizer settings based at least in part on the directivity data. The apparatus further comprises means for generating, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals.

Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims.

V. BRIEF DESCRIPTION OF THE DRAWINGS
1 is a block diagram of a particular illustrative aspect of a system operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
2A is a diagram of an illustrative example of a graphical user interface (GUI) generated by the system of FIG. 1, in accordance with some examples of the present disclosure.
2B is a diagram of another illustrative example of a GUI generated by the system of FIG. 1, in accordance with some examples of the present disclosure.
3 is a diagram of an exemplary aspect of components of the system of FIG. 1, in accordance with some examples of the present disclosure.
4 is a diagram of an exemplary aspect of components of the system of FIG. 1, in accordance with some examples of the present disclosure.
5 is a diagram of an exemplary aspect of components of the system of FIG. 1, in accordance with some examples of the present disclosure.
6 is a diagram of an exemplary aspect of components of the system of FIG. 1, in accordance with some examples of the present disclosure.
7 illustrates an example of an integrated circuit operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
8 is a diagram of a mobile device operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
9 is a diagram of a headset operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
10 is a diagram of a wearable electronic device operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
11 is a diagram of a voice controlled speaker system operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
12 is a diagram of a camera operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
13 is a diagram of a headset, such as a virtual reality or augmented reality headset, operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
14 is a diagram of a first example of a vehicle operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
15 is a diagram of a second example of a vehicle operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.
16 is a diagram of a particular implementation of a method of psychoacoustic enhancement based on audio source directivity that may be performed by the device of FIG. 1, in accordance with some examples of this disclosure.
17 is a block diagram of a particular illustrative example of a device operable to perform psychoacoustic enhancement based on audio source directivity, in accordance with some examples of the present disclosure.

VI. details

Microphones produce audio signals representative of captured sound, such as user speech, ambient sounds, or a combination thereof. Due to the distance of the microphones from the audio source, various sounds may be difficult to hear in the audio signal. The ability to focus on specific sounds in an audio signal is useful for a variety of applications, such as user speech in a communication application or bird sounds in a bird tracking application.

Systems and methods of psychoacoustic enhancement based on audio source directivity are disclosed. Different types of audio sources may have different sound directivity characteristics. For example, human speech is directed more in front of the human head than behind, and may exhibit a frequency response that varies based on distance and angular offset from the direction the human talker is facing, whereas a dodecahedral sound source is omnidirectional. Close up.

An audio enhancer performs psychoacoustic enhancement based on the directivity of the audio source to approximate the sound that would be captured by moving the microphone relative to the audio source (eg, closer to or away from the audio source). do For example, audio enhancers include directivity analyzers and equalizers. The directivity analyzer creates equalizer settings based on directivity data of an audio source and zoom target. For example, a directivity analyzer creates equalizer settings to adjust the loudness of specific audio frequencies to emulate that applying the equalizer settings moves a microphone to a zoom target. An equalizer applies equalizer settings to the input audio signals to produce output audio signals corresponding to psychoacoustic enhanced versions of the input audio signals. For example, the input audio signals are based on the microphone outputs of the microphones, and the output audio signals approximate the frequency response of the audio source at the zoom target. Thus, the output audio signals approximate the sounds to be captured by the microphones at the zoom target.

Certain aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated with common reference numbers. As used herein, various terms are used merely for the purpose of describing particular implementations and are not intended to limit the implementations. For example, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly dictates otherwise. Additionally, some features described herein are singular in some implementations and plural in others. To illustrate, FIG. 1 shows a device 102 that includes one or more processors (“processor(s)” 190 in FIG. and indicates that device 102 includes multiple processors 190 in other implementations.

As used herein, the terms "comprise", "comprises", and "comprising" refer to "include", "includes" , or "including" may be used interchangeably. Additionally, the term “wherein” may be used interchangeably with “where”. As used herein, "exemplary" indicates examples, implementations, and/or aspects, and is not to be construed as limiting or indicating a preference or preferred implementation. As used herein, structure An ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a component, operation, etc., alone indicates any priority or order of elements relative to other elements. does not indicate, but rather merely distinguishes an element from other elements having the same name (unless the ordinal term is used) As used herein, the term "set" refers to one or more of a particular element; The term "plurality" refers to multiples (eg, two or more) of a particular element.

As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively ) may include any combination of these. Two devices (or components) may be coupled (e.g., directly or indirectly) through one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), or the like. eg, communicatively coupled, electrically coupled, or physically coupled). Two electrically coupled devices (or components) may be included in the same device or different devices and may be connected via electronics, one or more connectors or inductive coupling as illustrative, non-limiting examples. . In some implementations, as in telecommunication, two devices (or components) that are communicatively coupled transmit signals (e.g., a digital signal) directly or indirectly, via one or more wires, buses, networks, or the like. or analog signals) may be transmitted and received. As used herein, “directly coupled” may include two devices that are coupled (eg, communicatively coupled, electrically coupled, or physically coupled) without intervening components.

In this disclosure, terms such as “determining”, “calculating”, “estimating”, “shifting”, “adjusting”, etc. are used to describe how one or more operations are performed. may also be used for It should be noted that such terms should not be construed as limiting and that other techniques may be utilized to perform similar operations. Additionally, as referred to herein, "generate", "calculate", "estimate", "use", "select", "access" and "determine" "To do" may be used interchangeably. For example, “generating”, “calculating”, “estimating”, or “determining” a parameter (or signal) means actively generating, estimating, or estimating a parameter (or signal). , calculating, or determining, or may refer to using, selecting, or accessing a parameter (or signal) already generated, eg, by another component or device.

Referring to FIG. 1 , a particular exemplary aspect of a system configured to perform psychoacoustic enhancement based on audio source directivity is disclosed and generally designated 100 . System 100 includes a device 102 coupled to one or more microphones 120, a camera 140, one or more speakers 160, a display device 162, an input device 130, or a combination thereof. include In some implementations, display device 162 includes input device 130 (eg, a touchscreen).

Device 102 includes one or more processors 190 coupled to memory 132 . Memory 132 is configured to store equalizer (Eq) setting data 149, directivity data 141, other data used or generated by audio enhancer 192, or a combination thereof. In a particular aspect, one or more processors 190 are coupled to one or more microphones 120 via one or more input interfaces 124 . For example, one or more input interfaces 124 may receive one or more microphone outputs 122 from one or more microphones 120 and convert one or more microphone outputs 122 to one or more input audio signals (SIGs) 126 as audio. Enhancer 192 is configured to provide.

In a particular aspect, one or more processors 190 are coupled to camera 140 via input interface 144 . For example, input interface 144 is configured to receive camera output 142 from camera 140 and provide camera output 142 as image data 145 to audio enhancer 192 . In a particular aspect, input interface 144 is configured to provide camera output 142 to audio enhancer 192 at the same time as one or more input interfaces 124 provide image data 145 to audio enhancer 192. do.

Device 102 is configured to perform psychoacoustic enhancement based on audio source directivity using audio enhancer 192 included in one or more processors 190 . Audio enhancer 192 includes directivity analyzer 152 coupled to equalizer 148 and pre-equalization signal processor 146 coupled to equalizer 148 . According to some implementations, directional data updater 150 is included in audio enhancer 192 .

Input device 130 is configured to provide user input 131 indicating zoom target 133 to device 102 . Directivity analyzer 152 is configured to generate equalizer settings 153 based on directivity data 141 , zoom target 133 , equalizer settings data 149 , or a combination thereof. For example, directivity analyzer 152 applies equalizer settings 153 to adjust the loudness of certain audio frequencies to emulate moving one or more microphones 120 closer to zoom target 133. configured to generate equalizer settings (153).

Equalizer 148 is configured to apply equalizer settings 153 to one or more equalizer input audio signals 147 to generate one or more output audio signals 138 . In a particular aspect, one or more equalizer input audio signals 147 includes one or more input audio signals 126 . In an alternative implementation, pre-equalization signal processor 146 processes one or more input audio signals 126 to obtain one or more equalizer input audio signals 147, as further described with reference to FIG. configured to create

In some implementations, device 102 corresponds to or is included in one or several types of devices. In the illustrative example, processor 190 is integrated into a headset device that includes one or more speakers 160, as further described with reference to FIG. 9 . In other examples, processor 190 may be a mobile phone or tablet computer device as described with reference to FIG. 8 , a wearable electronic device as described with reference to FIG. 10 , a voice controlled speaker as described with reference to FIG. 11 . system, a camera device as described with reference to FIG. 12 , or a virtual reality headset or augmented reality headset as described with reference to FIG. 13 . In another illustrative example, processor 190 is integrated into a vehicle that also includes one or more speakers 160, as described further with reference to FIGS. 14 and 15 .

During operation, one or more microphones 120 capture sound 186 from one or more audio sources including audio source 184 (eg, a person) and output one or more microphone outputs representing sound 186. (122). In certain aspects, the one or more audio sources include people, animals, birds, vehicles, musical instruments, other types of audio sources, or combinations thereof. One or more input interfaces 124 provide one or more microphone outputs 122 to audio enhancer 192 as one or more input audio signals 126 .

In a particular implementation, camera 140 captures images (e.g., video, still images, or both) of one or more audio sources, such as audio source 184, and camera output 142 representing the images. generate In this implementation, input interface 144 provides camera output 142 as image data 145 to audio enhancer 192 . In a particular aspect, camera 140 provides camera output 142 to device 102 at the same time as one or more microphones 120 provide one or more microphone outputs 122 to device 102 .

In a particular implementation, image data 145 , one or more input audio signals 126 , or a combination thereof may be added to data captured via external sensors (e.g., microphone 120 and camera 140). Instead of corresponding, it corresponds to stored data, such as video game data or previously recorded data. For example, audio enhancer 192 retrieves image data 145, one or more input audio signals 126, or a combination thereof from memory 132.

One or more processors 190 generate one or more output audio signals 138 based on one or more input audio signals 126 and send one or more output audio signals 138 via one or more speakers 160. outputs In certain implementations, one or more processors 190 generate graphical user interface 161 based on image data 145 and simultaneously output one or more output audio signals 138 through one or more speakers 160. A graphical user interface 161 is provided on the display device 162 to display the image captured by the camera 140 to the user 101 .

Device 102 responds to user 101 to initiate an audio zoom operation. For example, user 101 uses input device 130 to provide user input 131 representing zoom target 133 to audio enhancer 192 . In a particular implementation, user 101 moves a zoom selector displayed on graphical user interface 161 to select zoom target 133 by using an input device (as further described with reference to FIGS. 2A and 2B ). 130) (e.g., mouse, keyboard, button, slider input, or combinations thereof). In another implementation, user 101 initiates an audio zoom operation independently of graphical user interface 161 . For example, one or more processors 190 provide one or more output audio signals 138 to one or more speakers 160 independent of providing any GUI to display device 162 . User 101 provides user input 131 indicating zoom target 133 to audio enhancer 192 using input device 130 (eg, arrow keys on a keyboard, buttons on a headset, etc.) do. To illustrate, user 101 uses input device 130 to select different regions of the sound field corresponding to the audio output of one or more speakers 160, as further described with reference to FIG. zoom into fields

The zoom target 133 includes information indicating how audio zoom is to be performed. In various implementations, zoom target 133 is a user's selection of at least one audio source (eg, audio source 184 ), microphone 120 , as further described with reference to FIGS. 4-6 . ), the user's choice to adjust the audio in a way that simulates moving the audio, or a combination thereof. For example, zoom target 133 is a user's selection of audio source 184 and how closer one or more microphones 120 are positioned to audio source 184 (eg, 2 feet closer). may include a zoom distance 135 indicating what should be perceived. In another example, zoom target 133 includes zoom orientation 137 indicating how much and in which direction one or more microphones 120 should be perceived to have moved from location 134 (eg, physical location) and may include the user's selection of the zoom distance 135. In certain illustrative examples, a first value (eg, 0 degrees), a second value (eg, 90 degrees), a third value (eg, 180 degrees), or a second value of zoom orientation 137 . The four values (eg, 270 degrees) correspond to forward movement, right movement, backward movement, or left movement of one or more microphones 120 relative to position 134 , respectively. In certain instances, such as when user 101 selects zoom distance 135 and zoom orientation 137, audio enhancer 192 adjusts zoom orientation 137 and zoom distance 135 (one or more microphones 120). ) to determine the zoom position 136 by applying to the position 134 of In another example, when zoom target 133 includes the user's selection of zoom position 136 , audio enhancer 192 determines zoom orientation 137 based on a comparison of position 134 and zoom position 136 . and zoom distance 135. In a particular example, when zoom target 133 includes the user's selection of audio source 184, audio enhancer 192 estimates the location of audio source 184, and the estimated location of audio source 184. Determines zoom distance 135, zoom position 136, zoom orientation 137, or a combination thereof based on . In a particular aspect, audio enhancer 192 estimates a location of audio source 184 using image analysis techniques, audio analysis techniques, position information of audio source 184, or a combination thereof. In a particular aspect, position 134 corresponds to a representative position (e.g., average position) of positions of plurality of microphones 120, and zoom position 136 indicates that plurality of microphones 120 have moved. Corresponds to a representative location (eg, average location) of the emulated locations.

Directivity analyzer 152 obtains directivity data 141 of one or more audio sources (eg, audio source 184 ) corresponding to one or more input audio signals 126 . For example, directivity analyzer 152 may (e.g., based on the type of source, such as from analyzing input audio signal 126, analyzing image data 145, or a combination thereof) The audio source 184 is identified and the directivity data 141 that most closely corresponds to the audio source 184 is retrieved from the memory 132 . In another example, the directivity analyzer 152 requests (eg, downloads) the directivity data 141 from another device or network.

The specific audio source's directivity data 141 represents the orientation and distance frequency response characteristics of the specific audio source. In a particular aspect, directional data 141 is associated with a generic audio source. For example, directivity data 141 represents orientation and frequency response characteristics of a typical audio source. To illustrate, the directivity data 141 may provide a frequency response corresponding to mid-frequencies in response to a change from a first distance to a generic audio source to a second distance to a generic audio source at a first frequency response. indicates a change (eg, decrease or increase) by an amount. In an alternative aspect, directional data 141 represents directional data associated with particular types of audio sources. For example, directivity data 141 may include orientation, distance from a particular audio source type (eg, human talker, bird, musical instrument, etc.) of audio source 184, as further described with reference to FIG. 4 . , or changes in the frequency response of various frequencies in response to changes in both.

Directivity analyzer 152 determines equalizer settings (based at least in part on directivity data 141 , zoom target 133 , and equalizer setting data 149 , as further described with reference to FIG. 4 ). 153). For example, directivity analyzer 152 applies equalizer settings 153 to a specific audio signal to emulate moving one or more microphones 120 to (or closer to) zoom position 136 . Create equalizer settings 153 to adjust the loudness of the frequencies. In a particular implementation, directivity analyzer 152, in response to determining that directivity data for an audio source type of audio source 184 is not available, selects equalizer settings 153 based on default directivity data. do. To illustrate, directivity analyzer 152 equalizes (e.g., independently of the audio source type of audio source 184) to adjust (e.g., increase or decrease) the frequency response corresponding to the intermediate frequencies. Presets 153 are selected. For example, directivity analyzer 152 responds to intermediate frequencies in response to determining that the distance between zoom position 136 and audio source 184 is less than the distance between position 134 and audio source 184. select equalizer settings 153 to increase the loudness of the As another example, directivity analyzer 152 responds to intermediate frequencies in response to determining that the distance between zoom position 136 and audio source 184 is greater than the distance between position 134 and audio source 184. Equalizer settings 153 to reduce the loudness of the noise. In an alternative implementation, directivity analyzer 152 determines the audio source type of audio source 184 indicated by directivity data 141 (eg, a human talker or a bird, as further described with reference to FIG. 4 ). ) selects equalizer settings 153 based on the directionality (eg, frequency response) of . Directivity analyzer 152 provides equalizer settings 153 to equalizer 148 .

Equalizer 148 generates one or more output audio signals 138 by applying equalizer settings 153 to one or more equalizer input audio signals 147 . In a particular implementation, the one or more equalizer input audio signals 147 include one or more input audio signals 126 . In another implementation, pre-equalization signal processor 146 applies pre-equalization processing to one or more input audio signals 126, as further described with reference to FIG. 147). Equalizer 148 provides one or more output audio signals 138 to one or more speakers 160 .

One or more output audio signals 138 correspond to a psychoacoustic enhanced version of one or more input audio signals 126 . The psychoacoustic enhancement version (eg, one or more output audio signals 138) is audio at a zoom position 136 (eg, zoom orientation 137 and zoom distance 135) associated with an audio zoom operation. Approximate the frequency response of source 184. Thus, the sound (corresponding to the one or more output audio signals 138) produced by the one or more speakers 160 emulates moving the one or more microphones 120 to (or closer to) the zoom position 136. .

In a particular implementation, directional data updater 150 creates or updates directional data 141 . Directivity updater 150 is configured to sample and analyze audio captured at various distances and orientations from an audio source and generate or update directivity data associated with the audio source. In a particular example, the directional data updater 150 generates, at a first time, a first sound spectrum of an input audio signal of the one or more input audio signals 126 corresponding to the audio source 184 . A first sound spectrum represents sound captured by one or more microphones 120 at a first distance from audio source 184 when audio source 184 has a first orientation relative to one or more microphones 120. Directional data updater 150 generates, at a second time, a second sound spectrum of an input audio signal of one or more input audio signals 126 corresponding to audio source 184 . The second sound spectrum represents sound captured by one or more microphones 120 at a second distance from audio source 184 when audio source 184 has a second orientation relative to one or more microphones 120. indicate Directivity data updater 150 updates directivity data 141 to indicate that a difference between a first distance and a first direction and a second distance and a second direction corresponds to a difference between a first sound spectrum and a second sound spectrum. do.

System 100 enables an audio zoom operation relative to zoom target 133 that approximates moving one or more microphones 120 to zoom position 136 . Generating one or more output audio signals 138 by adjusting loudness with respect to frequencies based on the directivity of audio source 184 is more natural than adjusting only the gains of one or more input audio signals 126. This results in a sounding audio zoom.

Although one or more microphones 120, camera 140, one or more speakers 160, display device 162, and input device 130 are illustrated as being coupled to device 102, in other implementations, One or more microphones 120 , camera 140 , one or more speakers 160 , display device 162 , input device 130 , or a combination thereof may be incorporated into device 102 . Various implementations of system 100 may include fewer, additional, or different components. For example, in some implementations, directional data updater 150, camera 140, or both may be omitted.

Referring to FIG. 2A , an example of GUI 161 is shown. In a particular aspect, graphical user interface 161 is generated by audio enhancer 192 of FIG. 1 , one or more processors 190 , device 102 , system 100 , or a combination thereof.

Graphical user interface 161 includes video display 204 configured to display images corresponding to image data 145 of FIG. 1 . For example, video display 204 displays images of audio source 184. The graphical user interface 161 includes a zoom selector 202 that can be used to initiate audio zoom operations. For example, user 101 of FIG. 1 can move zoom selector 202 up to zoom in to audio source 184 or down to zoom out of audio source 184 . In certain aspects, moving zoom selector 202 up corresponds to selecting a first value for zoom orientation 137 (e.g., 0 degrees, forward, or zoom in), and zoom selector 202 Moving down corresponds to selecting a second value for zoom orientation 137 (eg, 180 degrees, back, or zoom out). The amount of movement of the zoom selector 202 represents the zoom distance 135 . A zoom target 133 includes a zoom distance 135 , a zoom orientation 137 , or both.

Referring to FIG. 2B , an example of a graphical user interface 161 is shown. In a particular aspect, graphical user interface 161 is generated by audio enhancer 192 of FIG. 1 , one or more processors 190 , device 102 , system 100 , or a combination thereof.

Graphical user interface 161 indicates that user 101 has moved zoom selector 202 to initiate an audio zoom operation. For example, user 101 responds to selection of zoom orientation 137 (e.g. forward, 0 degree, or zoom in) and zoom distance 135 (e.g. 2 feet) with a zoom selector ( 202), and input device 130 provides user input 131 representing zoom target 133 to audio enhancer 192. The zoom target 133 represents a zoom orientation 137 (eg, 0 degree, forward, or zoomed in) and a zoom distance 135 (eg, based on the amount of movement of the zoom selector 202 ). Directivity analyzer 152 generates equalizer settings 153 based at least in part on zoom target 133 , as further described with reference to FIG. 4 . Equalizer 148 generates one or more output audio signals 138 by applying equalizer settings 153 to one or more equalizer input audio signals 147, as described with reference to FIG. (e.g. update). Equalizer 148 provides one or more output audio signals 138 to one or more speakers 160 .

In a particular aspect, one or more processors 190, in response to user input 131, perform an image zoom operation on image data 145, and equalizer 148 outputs one or more output audio signals 138 ) to one or more speakers 160 while updating video display 204 to display a zoomed version of image data 145 . As illustrated, the audio source 184 is enlarged in the video display 204 of FIG. 2B compared to FIG. 2A to indicate that the audio zoom operation has zoomed into the audio source 184 .

Zoom selector 202 is provided as an illustrative example of selecting a zoom target 133 . In other implementations, user 101 may use other ways of specifying zoom target 133 . In a particular example, graphical user interface 161 is displayed on a touchscreen (eg, input device 130 ), and user 101 interacts with (eg, input device 130 ) the touchscreen to specify zoom target 133 . , using a tap or pinch-to-zoom gesture). For example, user 101 may tap the touch screen to select a location on video display 204 corresponding to selection of zoom position 136, audio source 184, or both as zoom target 133. can As another example, user 101 makes a first pinch-zoom (eg, widen) gesture to indicate a first value of zoom orientation 137 (eg, forward, 0 degrees, or zoomed in). or use a second pinch-zoom (eg, narrow) gesture to indicate a second value of the zoom orientation 137 (eg, back, 180 degrees, or zoom out). The distance of the pinch-zoom gesture represents the zoom distance 135 . A zoom target 133 includes a zoom distance 135 , a zoom orientation 137 , or both.

In a particular example, user 101 receives user input (eg, a voice command) indicating an identifier (eg, name) of zoom position 136 , audio source 184 , or both as zoom target 133 . , option selection, or both). Audio enhancer 192 performs image recognition on image data 145, audio analysis of input audio signals 126, or both to identify audio source 184, zoom position 136, or both. do. For example, user 101 receives user input (eg, "zoom to Sarah Lee") with an identifier (eg, contact name) of audio source 184 (eg, "Sarah Lee"). provides Audio enhancer 192 performs image recognition (e.g., on image data 145) to identify portions of image data 145 (e.g., “Sarah Lee”) that correspond to audio source 184. to the input audio signals 126 to perform human recognition or object recognition), or to identify portions of the input audio signals 126 that correspond to the audio source 184 (eg, “Sarah Lee”). speech recognition, or both. Zoom target 133 includes audio source 184 .

Referring to FIG. 3 , a diagram 300 of the components of system 100 of FIG. 1 is shown according to a particular implementation. The pre-equalization signal processor 146 includes a spatial analyzer 340, an activity detector 342, a gain adjuster 344, a noise suppressor 346, a context detector 350, or a combination thereof. Context detector 350 includes source detector 362, source position detector 364, or both. One or more of the components shown in dotted lines in FIG. 3 may be omitted in some implementations.

Spatial analyzer 340 is configured to apply beamforming to one or more input audio signals 126 to generate one or more beamformed audio signals 341 . In a particular aspect, spatial analyzer 340 applies beamforming based on zoom target 133 . For example, spatial analyzer 340 applies beamforming based on zoom orientation 137 of FIG. 1 such that one or more beamformed audio signals 341 represent sound captured around zoom orientation 137 . do. Spatial analyzer 340 provides one or more beamformed audio signals 341 to one or more components of pre-equalization signal processor 146 or to equalizer 148 . For example, spatial analyzer 340 may send one or more beamformed audio signals 341 as one or more activity input audio signals 361 to activity detector 342, one or more gain adjuster input audio signals 363 to gain adjuster 344 as one or more context detector input audio signals 369 to context detector 350 as one or more noise suppression input audio signals 365 to noise suppressor 346, to one or more Equalizer input audio signals 147 to equalizer 148, or a combination thereof.

Activity detector 342 is configured to detect activity in one or more activity input audio signals 361 . In a particular implementation, one or more activity input audio signals 361 include one or more input audio signals 126 . In an alternative implementation, the one or more activity input audio signals 361 include one or more beamformed audio signals 341 .

Activity detector 342 is configured to generate one or more activity audio signals 343 based on activity detected in one or more activity input audio signals 361 . In a particular example, activity detector 342 (e.g., a speech activity detector) detects speech in a first activity input audio signal of one or more activity input audio signals 361, and detects speech in one or more activity audio signals containing speech. 343 is configured to generate a second activity audio signal comprising the first activity audio signal and remaining sounds of the first activity input audio signal. For illustrative purposes, the first activity audio signal is reduced or contains no remaining sounds, and the second activity audio signal is reduced or contains no speech.

In a particular implementation, activity detector 342 is configured to detect sounds corresponding to different types of audio sources, different audio sources of the same type, or both. In the illustrative example, activity detector 342 may detect in a first activity input audio signal of one or more activity input audio signals 361 a first speech associated with a first speaker, a second speech associated with a second speaker, an instrument and and detect associated music sounds, bird sounds associated with a bird, or a combination thereof. Activity detector 342 receives a first activity audio signal comprising a first speech (e.g., the remaining sounds are absent or reduced), a second activity audio signal comprising a second speech (e.g., remaining sounds are absent or reduced), and absent or reduced), a third activity audio signal comprising music sound (eg no remaining sounds present or reduced), a fourth activity audio signal comprising new sounds (eg no remaining sounds present or reduced) ), a fifth activity audio signal comprising the remaining sounds of the first activity input audio signal, or a combination thereof. The one or more activity audio signals 343 include a first activity audio signal, a second activity audio signal, a third activity audio signal, a fourth activity audio signal, a fifth activity audio signal, or a combination thereof.

Activity detector 342 provides one or more activity audio signals 343 to one or more components of pre-equalization signal processor 146, equalizer 148, or a combination thereof. For example, activity detector 342 may send one or more activity audio signals 343 as one or more gain adjuster input audio signals 363 to gain adjuster 344 and one or more context detector input audio signals 369 to context detector 350 as one or more noise suppression input audio signals 365 to noise suppressor 346, one or more equalizer input audio signals 147 to equalizer 148, or provided as a combination.

A gain adjuster 344 applies one or more gains to one or more gain adjuster input audio signals 363 . The one or more gain adjuster input audio signals 363 include one or more input audio signals 126 , one or more beamformed audio signals 341 , or one or more activity audio signals 343 . Gain adjuster 344 applies one or more gains based on zoom target 133 . For example, when an audio zoom operation corresponds to zooming in on zoom target 133 , gain adjuster 344 outputs one or more of the gain adjuster input audio signals 363 corresponding to sounds from zoom orientation 137 . Increases the gains of the first input audio signals, decreases the gains of the second input audio signals of the one or more gain adjuster input audio signals 363 corresponding to sounds from the other directions, or both . In another example, when the audio zoom operation corresponds to a zoom away from zoom target 133 , gain adjuster 344 outputs one or more of the gain adjuster input audio signals 363 corresponding to sounds from zoom orientation 137 . reduce the gains of the first input audio signals, increase the gains of the second input audio signals of the one or more gain adjuster input audio signals 363 corresponding to sounds from the other directions, or both . In certain aspects, the amount of gain adjustment is based on zoom distance 135 .

Gain adjuster 344 provides one or more gain-adjusted audio signals 345 to one or more components of pre-equalization signal processor 146, equalizer 148, or a combination thereof. For example, gain adjuster 344 may send one or more gain-adjusted audio signals 345 as one or more context detector input audio signals 369 to context detector 350, as well as one or more noise suppression input audio signals ( 365 to noise suppressor 346, one or more equalizer input audio signals 147 to equalizer 148, or a combination thereof.

Context detector 350 processes one or more context detector input audio signals 369 , image data 145 , or a combination thereof to generate context data 351 . In a particular aspect, one or more context detector input audio signals 369 may be one or more input audio signals 126, one or more beamformed audio signals 341, one or more activity audio signals 343, or one Gain-adjusted audio signals 345 above.

Source detector 362 performs audio source recognition on one or more context detector input audio signals 369, image data 145, or combinations thereof to determine the audio source type of one or more audio sources, such as audio source 184. identify For example, source detector 362 may perform image analysis (eg, object recognition and distance analysis) on image data 145 so that image data 145 is at a first location relative to camera 140 . Determines that it represents an audio source type (eg, human talker). In a particular aspect, source detector 362 performs sound analysis (e.g., audio source recognition and distance analysis) on one or more context detector input audio signals 369 to determine the one or more context detector input audio signals 369. ) contains sounds that match the audio source type from the second location for the one or more microphones 120 . In a particular aspect, source detector 362 determines that the first location for camera 140 corresponds to the same physical location as the second location for one or more microphones 120 . Source detector 362 transmits source detection data indicative of an audio source type, a first location for camera 140, a second location for one or more microphones 120, a physical location, or a combination thereof to a source position detector ( 364) is provided.

Source position detector 364 performs image analysis to detect the orientation of audio source 184 in image data 145 relative to camera 140 . For example, if audio source 184 corresponds to a human speaker, source position detector 364 performs image recognition on image data 145 to determine the orientation of the human speaker's head (e.g., camera 140). looking toward or away from the camera 140).

In the illustrative example, source position detector 364 detects one or more microphones 120 based on the orientation of audio source 184 relative to camera 140 and the difference between the positions of camera 140 and one or more microphones 120. Determines the orientation of the audio source 184 relative to Context detector 350 determines that the second position for one or more microphones 120 represents the distance of audio source 184 from one or more microphones 120 . Context detector 350 includes the distance of audio source 184 from one or more microphones 120, the orientation of audio source 184 relative to one or more microphones 120, the audio source type of audio source 184, Alternatively, context data 351 indicating a combination thereof is generated. Context detector 350 provides context data 351 to orientation analyzer 152 .

A noise suppressor 346 performs noise suppression on one or more noise suppression input audio signals 365 to produce one or more noise suppressed audio signals 347 . In a particular aspect, one or more noise suppression input audio signals 365 may be one or more input audio signals 126, one or more beamformed audio signals 341, one or more activity audio signals 343, or one Gain-adjusted audio signals 345 above. Noise suppressor 346 provides one or more noise suppressed audio signals 347 to equalizer 148 as one or more equalizer input audio signals 147 .

The specific order of operations of the components of signal processor 146 before equalization is provided as an illustrative example. In other examples, the order of operations of the components of signal processor 146 before equalization may be different. In the particular example, zoom target 133 represents a selection of audio source 184 . Context detector 350 provides the audio source type (eg, human talker or bird) of audio source 184 to activity detector 342 , based on zoom target 133 . Activity detector 342 includes first activity signals of one or more activity audio signals 343 corresponding to sounds of an audio source type (e.g., remaining sounds reduced or absent), remaining sounds corresponding to (e.g., remaining sounds) generating second activity signals, eg, with no sounds or reduced sounds of the audio source type), or a combination. Activity detector 342 provides one or more activity audio signals 343 to gain adjuster 344 . The gain adjuster 344, in response to determining that the audio zoom operation includes zooming toward the zoom target 133, increases the gain of the first activity signals or increases the gain of the second activity signals. decrease, or both. Alternatively, gain adjuster 344, in response to determining that the audio zoom operation includes zooming away from zoom target 133, reduces the gain of the first activity signals and reduces the gain of the second activity signals. increase, or both.

In a particular aspect, directivity analyzer 152 obtains directivity data 141 based on the audio source type of audio source 184, as further described with reference to FIG. Directivity analyzer 152 generates equalizer settings 153 based on directivity data 141 , as further described with reference to FIG. 4 . Directivity analyzer 152 provides equalizer settings 153 to equalizer 148 .

Equalizer 148 applies equalizer settings 153 to one or more equalizer input audio signals 147 to produce one or more output audio signals 138 . In a particular aspect, one or more equalizer input audio signals 147 may be one or more input audio signals 126, one or more activity audio signals 343, one or more gain-adjusted audio signals 345, or one or more input audio signals 126. noise suppressed audio signals 347 above.

Accordingly, the pre-equalization signal processor 146 performs pre-equalization signal processing to improve the performance of the audio enhancer 192 by beamforming, gain adjustment, noise reduction, or a combination thereof prior to performing equalization. In a particular aspect, pre-equalization signal processor 146 uses context data 351 to enable directivity analyzer 152 to determine equalizer settings 153 based on directivity of audio source types of one or more audio sources. Decide.

In some implementations, the pre-equalization signal processor 146 may be omitted. As an example, directivity analyzer 152 generates equalizer settings 153 based on default directivity data, and equalizer 148 generates one or more input audio signals 138 to produce one or more output audio signals 138. Apply equalizer settings 153 to signals 126 (eg, adjust their intermediate frequencies).

In some implementations, one or more components of pre-equalization signal processor 146 may be omitted. In one example, spatial analyzer 340 and activity detector 342 are omitted, and one or more input audio signals 126 are provided to gain adjuster 344 as one or more gain adjuster input audio signals 363. In some implementations, spatial analyzer 340 is omitted and one or more input audio signals 126 are provided to activity detector 342 as one or more activity input audio signals 361 . In some implementations, activity detector 342 is omitted, and one or more beamformed audio signals 341 are provided to gain adjuster 344 as one or more gain adjuster input audio signals 363 . In some implementations, gain adjuster 344 is omitted, and one or more activity audio signals 343 are sent to context detector 350 as one or more context detector input audio signals 369 and one or more noise suppression input audio signals. are provided to the noise suppressor 346 as fields 365 . Certain combinations of components are described as illustrative examples. In other implementations, other combinations of components are included in the pre-equalization signal processor 146.

Referring to FIG. 4 , a diagram 400 of a particular example of components of system 100 of FIG. 1 is shown. Orientation analyzer 152 is illustrated as obtaining orientation data 141 , context data 351 , and user input 131 . Context data 351 includes source position data 420 of audio source 184 . For example, the source position data 420 may be the source orientation of the audio source 184 relative to (e.g., at or toward 0 degrees) one or more microphones 120, as described with reference to FIG. 422), the source distance 424 of the audio source 184 from the one or more microphones 120 (eg, 6 feet), or both.

Source position detector 364 of FIG. 3 determines the audio source relative to zoom position 136 based on source orientation 422 (eg, 0 degrees) and zoom orientation 137 (eg, 0 degrees). Determine the source orientation 432 of (184) (eg, 0 degrees). Source position detector 364 determines zoom distance 135 (eg, 2 feet), zoom orientation 137 (eg, 0 degrees), and source distance 424 (eg, 6 feet). determine the source distance 434 (eg, 4 feet) of the audio source 184 from the zoom position 136 based on the zoom position 136 .

In a particular aspect, directivity analyzer 152 obtains directivity data 141 based on the audio source type of audio source 184 . As an example, the graphical depiction of sound directivity patterns 402 represents the frequency-dependent directivity of an audio source type (eg, human talker) in a horizontal plane and a vertical plane. In a particular aspect, directivity data 141 includes a plurality of sets of directivity data associated with various orientations of an audio source type. The directivity analyzer 152 determines the specific orientation of the type of audio source for which the directivity data set 404 matches the source orientation 422 and the source orientation 432 (e.g., on axis, 0 degrees along the horizontal axis and vertical axis). Selects a directivity data set 404 from the directivity data 141 in response to determining that it is associated with.

A graphical depiction of the directivity data set 404 is an audio source type (e.g., of a microphone) corresponding to changes in distance (e.g., of a microphone) from a specific distance (e.g., 1 meter) along a specific orientation to various distances. For example, it represents the frequency response characteristics of audio source 184. For example, the directional data set 404 is a source distance 434 (eg, 1 centimeter) from a source distance 424 (eg, 1 meter) along a particular orientation (eg, on an axis). frequency response characteristics 449 of an audio source type (e.g., audio source 184) for a change to . In a particular aspect, the frequency response characteristics 449 indicate changes in loudness (eg, in decibels (dB)) for various sound frequencies. For example, frequency response characteristics 449 can be measured along a particular orientation (eg, on axis) from source distance 424 (eg, 1 meter) to source distance 434 (eg, 1 centimeter). ) is a drop in loudness (e.g., -0.2 dB) for a particular frequency (e.g., 500 hertz (Hz)), other frequency ranges (e.g., 800 Hz to 1 kilohertz (kHz)). )) corresponds to a rise in loudness (eg, + 4 dB), or both. In a particular example, the frequency response characteristics 449 are measured along a particular orientation (eg, on axis) from a source distance 424 (eg, 1 meter) to a source distance 434 (eg, 1 centimeter). ) corresponds to negligible (eg, below threshold) changes in loudness for another specific frequency range (eg, 200 Hz to 400 Hz). To illustrate, changes in loudness for a particular frequency range (eg, 200 Hz to 400 Hz) may not be perceptible to the human auditory system.

In a particular example, source orientation 422 matches source orientation 432, and directivity analyzer 152 determines at a source distance along a particular orientation (eg, source orientation 422 and source orientation 432). A directivity data set 404 corresponding to the change is selected. In this example, the directional data set 404 is a change in source distance along a particular orientation (e.g., source orientation 422 and source orientation 432) (e.g., source distance from source distance 424). (up to 434). In some other examples, source orientation 422 is different than source orientation 432, as further described with reference to FIG. 6, and directivity analyzer 152 determines (e.g., source distance from source distance 424 Directivity data 141 indicating frequency response characteristics 449 corresponding to a change in source distance (to 434) and a change in source orientation (eg, from source orientation 422 to source orientation 432). ) to select a directional data set from

In a particular aspect, directivity analyzer 152 obtains equalizer setting data 149 from memory 132, another device, a network, or a combination thereof. In certain implementations, equalizer setting data 149 includes context data 351 (e.g., audio source type of audio source 184), directivity data 141 (e.g., directivity data set 404) , zoom distance 135, source distance 424, source distance 434, zoom orientation 137, source orientation 422, source orientation 432, frequency response characteristics (e.g., frequency response characteristics (449)), or a combination thereof, with equalizer settings (153). Based on the equalizer setting data 149, the directivity analyzer 152 determines the audio source type of the audio source 184, the zoom distance 135, the source distance 424, the source distance 434, and the zoom orientation 137. ), source orientation 422, source orientation 432, frequency response characteristics 449, or a combination thereof.

In a particular aspect, directivity analyzer 152 selects, based on equalizer setting data 149 , equalizer settings 153 that match frequency response characteristics 449 . For example, the equalizer settings 153 may be a drop in loudness (eg, -0.2 dB) for a specific frequency (eg, 500 Hz), a first frequency range (eg, 800 Hz to 1 kilohertz). (kHz)), no changes in loudness for the second frequency range (eg, 200 Hz to 400 Hz), or a combination thereof. Accordingly, directivity analyzer 152 applies equalizer settings 153 to approximate the frequency response characteristics of moving one or more microphones 120 to (or closer to) zoom position 136 . Create equalizer settings (153).

Referring to FIG. 5 , a diagram 500 of a particular example of components of the system 100 of FIG. 1 is shown. A graphical representation of the directivity data set 504 of the directivity data 141 is an audio source type (eg, human talker) of the audio source 184 for a particular orientation (eg, -45 degree horizontal axis and 0 degree vertical axis) ) represents the frequency response characteristics of For example, the directivity data set 504 provides frequency response characteristics corresponding to changes in distance (eg, of a microphone) from a specific distance (eg, 1 meter) to various distances along a specific orientation. indicate

The context detector 350 of FIG. 3 determines the source position data 520 of the audio source 184 as described with reference to FIG. 3 . For example, the source position data 520 may indicate that the audio source 184 has a source orientation 522 relative to the position 134 of the one or more microphones 120 (e.g., -45 degree horizontal axis and 0 degree vertical). axis) at approximately the source distance 424 (eg, 1 meter). The source position data 520 is the source distance 434 (eg, - 45 degree horizontal axis and 0 degree vertical axis) of the audio source 184 to the source orientation 532 (eg - 45 degree horizontal axis and 0 degree vertical axis) relative to the zoom position 136 . e.g., 10 centimeters).

The directivity analyzer 152 determines the source orientation 422 (eg, −45 degrees on the horizontal axis and 0 degrees on the vertical axis) and the source orientation 432 (eg, −45 degrees on the horizontal axis and 0 degrees on the vertical axis). A directional data set 504 is selected from the directional data set 504 in response to a determination that it coincides with a particular orientation associated with the directional data set 504 (eg - 45 degree horizontal axis and 0 degree vertical axis). The directional data set 504 is a source distance 434 (eg, −45 degrees on the horizontal axis and 0 degrees on the vertical axis) from a source distance 424 (eg, 1 meter) along a particular orientation (eg, minus 45 degrees on the horizontal axis and 0 degrees on the vertical axis). , 1 centimeter) of the frequency response characteristics 549. In a particular aspect, the frequency response characteristics 549 have a particular orientation (eg, on a horizontal axis) from a source distance 424 (eg, 1 meter) toward a source distance 434 (eg, 1 centimeter). - 45 degrees and 0 degrees on the vertical axis) is a drop in loudness (eg - 0.2 dB) for a first frequency (eg 500 Hz), a second frequency (eg 800 Hz) ) a first rise in loudness (eg + 2 dB), a second rise in loudness (eg + 4 dB) for a third frequency (eg 1 kHz), a specific frequency range ( eg, between 200 Hz and 315 Hz), or a negligible (eg, below threshold) change in loudness, or a combination thereof.

In a particular aspect, directivity analyzer 152 selects, based on equalizer setting data 149 , equalizer settings 153 that match frequency response characteristics 549 . For example, the equalizer settings 153 may be a drop in loudness (eg, -0.2 dB) for a first frequency (eg, 500 Hz), and a loudness for a second frequency (eg, 800 Hz). A first rise (e.g., +2dB), a second rise (e.g., +4dB) in loudness for a third frequency, no change in loudness for a specific frequency range (e.g., 200Hz to 315Hz), or a combination thereof. Accordingly, the directivity analyzer 152 generates equalizer settings 153 based on the directivity data set 504 so that applying the equalizer settings 153 causes the audio source 184 to transmit one or more microphones ( Frequency response that moves one or more microphones 120 to (or closer to) the zoom position 136 when having a particular orientation (eg, 45 degrees on the horizontal axis and 0 degrees on the vertical axis) relative to 120). to approximate the properties.

Referring to FIG. 6 , the context detector 350 of FIG. 3 determines based on the one or more context detector input audio signals 369 of FIG. 3 multiple audio sources, e.g., audio source 184 and an audio source ( An example diagram 600 of the components of system 100 of FIG. 1 according to an implementation that has detected 684 is shown.

Context detector 350 determines source position data 620 of audio source 684 in a manner similar to that described with reference to FIG. 3 . For example, the source position data 620 is the source orientation 622 of the audio source 684 relative to the position 134 of the one or more microphones 120 (e.g., -30 degrees horizontal axis and 0 degrees vertical). axis) with a source distance 624 (e.g., 2 meters). In certain aspects, the source position data 620 is the source distance 634 at which the audio source 684 has the source orientation 632 relative to the zoom position 136 (e.g., -2 degree horizontal axis and 0 degree vertical axis). (e.g., 2.2 meters).

In a particular implementation, zoom target 133 represents audio source 184 , and directivity analyzer 152 ignores audio source 684 when determining equalizer settings 153 . In a particular aspect, one or more output audio signals 138 include reduced (eg, no) sounds of audio source 684 . As an example, activity detector 342 may detect one or more activity audio signals 343 corresponding to sounds of audio source 184 in which sounds of audio source 684 have been reduced (eg, no sounds). generate As another example, gain adjuster 344 generates one or more gain-adjusted audio signals 345 with reduced (eg, no) sounds of audio source 684. In another example, spatial analyzer 340 applies beamforming to generate one or more beamformed audio signals 341 with reduced (eg, no) sounds of audio source 684 . In this implementation, directivity analyzer 152 generates equalizer settings 153 based on directivity data set 504 and source position data 520, as described with reference to FIG.

In a particular implementation, zoom target 133 represents audio source 184, and audio enhancer 192 adjusts based on directivity of audio source 184 with little or no change to the sounds of audio source 684. produces one or more output audio signals 138 with the sounds of the audio source 184. As an example, activity detector 342 may detect a first of one or more activity audio signals 343 corresponding to sounds of audio source 184 with reduced (eg, no) sounds of audio source 684. A second portion of one or more activity audio signals 343 corresponding to the subset and remaining sounds (including sounds of audio source 684) with reduced (eg, no) sounds of audio source 184. create a subset

Directivity analyzer 152 generates equalizer settings 153 based on directivity data set 504 and source position data 520, as described with reference to FIG. The one or more equalizer input audio signals 147 may include a first subset of one or more activity audio signals 343, a gain-adjusted version of the first subset of one or more activity audio signals 343, one or more a noise-suppressed version of the first subset of activity audio signals 343, or a combination thereof. Equalizer 148 generates a first subset of one or more output audio signals 138 by applying equalizer settings 153 to one or more equalizer input audio signals 147, and user 101 produces a psychoacoustic enhanced version of sounds from audio source 184 that is perceived as being located at zoom position 136.

The second subset of one or more output audio signals 138 is based on the second subset of one or more activity audio signals 343 and includes sounds from an audio source 684 . For example, the second subset of one or more output audio signals 138 is a second subset of one or more activity audio signals 343, the gain of the second subset of one or more activity audio signals 343 -adjusted version, noise-suppressed version of the second subset of one or more activity audio signals 343, or a combination thereof.

Accordingly, one or more output audio signals 138 move one or more microphones 120 from position 134 to zoom position 136 and the audio with no (or few) changes to audio source 684. Approximate the frequency response characteristics of source 184. In this implementation, the audio zoom operation appears to zoom on the audio source 184 with little or no change to the audio source 684. For example, the sound of audio source 184 in one or more output audio signals 138 is believed to be coming from audio source 184 at approximately source distance 434 with source orientation 532 relative to zoom position 136 . see. Sounds of audio source 684 in one or more output audio signals 138 appear to be coming from audio source 684 at approximately source distance 624 in source direction 622 relative to zoom position 136 .

In another particular implementation, zoom target 133 represents audio source 184, and audio enhancer 192 adjusts sounds of audio source 184 based on directivity of audio source 184 and audio source 184. Generates one or more output audio signals 138 in an operation that includes adjusting the sounds of the adjusted audio source 684 based on the directivity of 684 . In certain aspects, audio source 684 has the same audio source type as audio source 184 (eg, human talker). In this aspect, the directivity analyzer 152 determines the change in orientation associated with the audio source 684 (e.g., from source orientation 622 to source orientation 632) and (e.g., source distance 624). Select the directivity data set 604 from the directivity data 141 that matches the change in distance (from the source distance 634).

In an alternative aspect, audio source 684 has a second audio source type (eg, bird) that is different from the first audio source type (eg, human talker) of audio source 184 . In this aspect, directivity analyzer 152 obtains second directivity data associated with a second audio source type, at an orientation (e.g., from source orientation 622 to source orientation 632) and at a source distance A directivity data set 604 representing the frequency response characteristics of the audio source 684 for changes at various distances from 624 is selected from the second directivity data. For illustrative purposes, the directivity data set 604 includes changes in orientation (e.g., from source orientation 622 to source orientation 632) and (e.g., source distance 624 to source distance 634). frequency response characteristics 649 for changes in distance).

Directivity analyzer 152 determines, based on equalizer setting data 149 , equalizer settings 653 that match frequency response characteristics 649 . Directivity analyzer 152 provides equalizer settings 653 corresponding to audio source 684 and equalizer settings 153 corresponding to audio source 184 to equalizer 148 .

In a particular aspect, activity detector 342 determines a first subset of one or more activity audio signals 343, audio A second subset of one or more activity audio signals 343 corresponding to sounds of source 684 and other sounds that are reduced (eg, absent), audio source 184 and audio source 684 Generate a third subset of one or more activity audio signals 343 corresponding to the remaining sounds and reduced (eg, absent) sounds, or a combination thereof. In a particular aspect, the first subset of one or more equalizer input audio signals 147 is based on the first subset of one or more activity audio signals 343 and the one or more equalizer input audio signals 147 The second subset of the one or more activity audio signals 343 is based on the second subset of the one or more activity audio signals 343 and the third subset of the one or more equalizer input audio signals 147 is based on the one or more activity audio signals 343 is based on the third subset of, or a combination thereof. Equalizer 148 applies equalizer settings 153 to a first subset of one or more equalizer input audio signals 147 corresponding to audio source 184 and one or more equalizers corresponding to audio source 684. equalizer settings 653 on the second subset of the input audio signal 147 and no change on the third subset of the equalizer input audio signal 147 corresponding to one or more of the remaining audio; or Applying a combination of these produces one or more output audio signals 138. Accordingly, equalizer settings 153 and equalizer settings 653 are equivalent to one or more output audio signals 138 moving one or more microphones 120 from position 134 to zoom position 136 and vice versa. It allows approximation of the frequency response characteristics of the associated audio source 184 and audio source 684. For example, sounds of audio source 184 in one or more output audio signals 138 are believed to be coming from audio source 184 at approximately source distance 434 with source orientation 532 relative to zoom position 136 . see. Sounds of audio source 684 in one or more output audio signals 138 appear to be coming from audio source 684 at approximately source distance 634 in source direction 632 relative to zoom position 136 .

7 shows an implementation 700 of device 102 as an integrated circuit 702 that includes one or more processors 190 . Integrated circuit 702 also includes audio inputs 704, such as one or more bus interfaces, through which one or more input audio signals 126 can be received for processing. Integrated circuit 702 also includes an audio output 706 such as a bus interface that enables transmission of an output signal, such as one or more output audio signals 138 . The integrated circuit 702 is a mobile phone or tablet as shown in FIG. 8, a headset as shown in FIG. 9, a wearable electronic device as shown in FIG. 10, a voice controlled speaker system as shown in FIG. 11, A psychoacoustic based audio source directivity as a component in a system such as a camera as shown in FIG. 12, a virtual reality headset or augmented reality headset as shown in FIG. 13, or a vehicle as shown in FIG. 14 or 15 Enables implementation of reinforcement.

8 shows an implementation 800 in which device 102 includes a mobile device 802 such as a phone or tablet as illustrative, non-limiting examples. Mobile device 802 includes one or more speakers 160 , one or more microphones 120 , and a display screen 804 . Components of processor 190 , including audio enhancer 192 , are integrated into mobile device 802 and are illustrated using dotted lines to indicate internal components that are not normally visible to a user of mobile device 802 . In a specific example, audio enhancer 192 operates to enhance user voice activity, then initiates a graphical user interface or otherwise (eg, via an integrated “smart assistant” application) on display screen 804 to enable user processed to perform one or more actions on the mobile device 802, such as displaying other information associated with the speech of the user. In a particular example, audio enhancer 192 enhances a speaker's voice activity during an online conference. To illustrate, a user may view a talker on a display screen 804 during an online conference and select the talker as a zoom target. Audio enhancer 192 enhances the speaker's speech in response to selection of the zoom target. In another example, a user of a bird tracking application on mobile device 802 selects a tree as a zoom target. Audio enhancer 192 enhances bird calls from birds on trees in response to selection of a zoom target.

9 shows an implementation 900 in which device 102 includes a headset device 902 . Headset device 902 includes one or more microphones 120, one or more speakers 160, or a combination thereof. Components of processor 190 , including audio enhancer 192 , are integrated into headset device 902 . In a particular example, audio enhancer 192 is operative to enhance user voice activity, which causes headset device 902 to perform one or more actions on headset device 902, or to respond to user voice activity for further processing. audio data to be transmitted to the second device (not shown), or a combination thereof may be performed. In a particular aspect, headset device 902 has inputs (eg, buttons or arrows) that can be used to zoom to different parts of the sound field corresponding to audio output of headset device 902 . For example, headset device 902 outputs orchestral music and a user wearing headset device 902 uses inputs of headset device 902 to select a particular section or instrument of the orchestra as a zoom target. Audio enhancer 192 generates one or more output audio signals 138 corresponding to an audio zoom operation relative to a zoom target (eg, a particular section or instrument).

10 shows an implementation 1000 in which device 102 includes a wearable electronic device 1002 illustrated as a “smart watch”. An audio enhancer 192, one or more microphones 120, one or more speakers s 160, or a combination thereof, is incorporated into wearable electronic device 1002. In a particular example, audio enhancer 192 is operative to enhance user voice activity, which then initiates a graphical user interface or otherwise displays the wearable electronic device 1002. processed to perform one or more actions on wearable electronic device 1002, such as displaying other information associated with the user's speech on display screen 1004 of device 1002. To illustrate, wearable electronic device 1002 may: It may also include a display screen configured to display a notification based on user speech enhanced by wearable electronic device 1002. In certain examples, wearable electronic device 1002 provides a haptic notification in response to user voice activity. A haptic device (e.g., vibrating) A haptic notification may cause a user to look at the wearable electronic device 1002 to see a displayed notification indicating detection of a keyword spoken by the user. Accordingly, the wearable electronic device 1002 can notify a hearing-impaired user or a user wearing a headset that the user's voice activity has been detected In certain aspects, the wearable electronic device 1002 can ) inputs (eg buttons or arrows) that can be used to zoom to different parts of the sound field corresponding to the audio output of .

11 is an implementation 1100 in which a device 102 includes a wireless speaker and a voice activated device 1102 . The wireless speaker and voice activated device 1102 can have wireless network connectivity and is configured to perform assistant operations. Included in the wireless speaker and voice activation device 1102 are one or more processors 190 including an audio enhancer 192, one or more microphones 120, one or more speakers 160, or a combination thereof. During operation, in response to receiving verbal commands identified as user speech through audio enhancement operations of audio enhancer 192, wireless speaker and voice activation device 1102, for example, a voice activation system (e.g., an integrated assistant) Application) may execute assistant operations. Assistant actions may include adjusting the temperature, playing music, turning on lights, and the like. For example, assistant actions are performed in response to receiving a command followed by a keyword or key phrase (eg, “hello assistant”).

12 shows an implementation 1200 in which device 102 includes a portable electronic device corresponding to camera device 1202 . Audio enhancer 192 , one or more microphones 120 , one or more speakers 160 , or a combination thereof are included in camera device 1202 . During operation, in response to receiving verbal commands as user speech enhanced through operation of audio enhancer 192, camera device 1202 may, as illustrative examples, perform zoom operations, set image or video capture, It may execute actions in response to issued user commands, such as adjusting image or video playback settings, or image or video capture instructions. In a particular example, when camera device 1202 zooms in on audio source 184 visible in the viewfinder, audio enhancer 192 performs an audio zoom operation to enhance audio captured from audio source 184. Including a video camera that allows

FIG. 13 shows an implementation 1300 in which device 102 includes a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality headset 1302 . An audio enhancer 192 , one or more microphones 120 , one or more speakers 160 , or a combination thereof are integrated into the headset 1302 . Audio enhancement may be performed based on audio signals received from one or more microphones 120 of headset 1302 . In certain examples, audio enhancement may be performed on audio signals corresponding to virtual, augmented, or mixed reality received from memory, a network, another device, or a combination thereof. The visual interface device is positioned in front of the user's eyes to enable display of augmented reality or virtual reality images or scenes to the user while the headset 1302 is being worn. In a particular example, the visual interface device is configured to display a notification indicating enhanced speech of an audio signal. In certain implementations, when a user uses headset 1302 to zoom to a virtual or real object being viewed on the visual interface device, audio enhancer 192 provides audio corresponding to that object (eg, as part of gameplay). Performs an audio zoom of In some examples, audio enhancer 192 performs audio zoom in conjunction with visual zoom displayed by the visual interface device.

14 shows an implementation 1400 in which a device 102 corresponds to or is incorporated within a vehicle 1402 exemplified as a manned or unmanned aerial device (eg, a baggage delivery drone). An audio enhancer 192 , one or more microphones 120 , one or more speakers 160 , or a combination thereof are integrated into the vehicle 1402 . Audio (eg, user voice activity) enhancement is based on audio signals received from one or more microphones 120 of vehicle 1402, such as for conveying commands from an authorized user of vehicle 1402. can be performed by

15 shows another implementation 1500 in which the device 102 corresponds to or is incorporated within a vehicle 1502 exemplified as an automobile. Vehicle 1502 includes a processor 190 that includes an audio enhancer 192 . Vehicle 1502 also includes one or more microphones 120 . Audio (eg, user voice activity) enhancement may be performed based on audio signals received from one or more microphones 120 of vehicle 1502 . In some implementations, audio (eg, voice activity) enhancement is an audio signal received from internal microphones (eg, one or more microphones 120), such as for a voice command from an authorized passenger. can be performed based on For example, user voice activity enhancement may be used to enhance a voice command from a driver or passenger of vehicle 1502 . In some implementations, audio enhancement is a sound from an audio source 184 (eg, birds, waves on the beach, outdoor music, an authorized user of vehicle 1502, a drive-thru retail employee, or a curbside pickup person). It may be performed based on an audio signal received from external microphones (eg, one or more microphones 120), such as . In certain implementations, in response to receiving a spoken command as user speech enhanced through operation of audio enhancer 192, the voice activation system may, for example, display 1520 or one or more speakers (e.g., a speaker ( 1510), so that one or more keywords detected in one or more output audio signals 138 (e.g., "unlock", "start engine", "play music", "weather") Initiate one or more operations of vehicle 1502 based on "show forecast", or other voice command. In a particular implementation, enhanced external sounds (eg, outdoor music, birdsong, etc.) are played inside the vehicle 1502 through one or more speakers 160 .

Referring to FIG. 16 , a specific implementation of a method 1600 of psychoacoustic enhancement based on audio source directivity is shown. In a particular aspect, one or more operations of method 1600 may include directivity analyzer 152 of FIG. 1 , equalizer 148 , audio enhancer 192 , one or more processors 190 , device 102 , system 100 ), or a combination thereof.

The method 1600 includes obtaining directional data of one or more audio sources corresponding to one or more input audio signals, at 1602 . For example, the directivity analyzer 152 of FIG. 1 uses the directivity data of the audio source 184 corresponding to one or more input audio signals 126 ( 141) is obtained.

The method 1600 also includes determining one or more equalizer settings based at least in part on the directivity data, at 1604 . For example, directivity analyzer 152 of FIG. 1 determines equalizer settings 153 based at least in part on directivity data 141, as described with reference to FIGS. 1 and 4-6. do.

The method 1600 further includes generating one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings, at 1606 . For example, equalizer 148 of FIG. 1 outputs one or more output audio signals 138 corresponding to a psychoacoustic enhanced version of one or more input audio signals 126 based on equalizer settings 153. generate

Method 1600 enables generating one or more output audio signals 138 by adjusting loudness for frequencies based on directivity of audio source 184 . The one or more output audio signals 138 corresponds to a more natural sounding audio zoom, eg, compared to adjusting only the gains of the one or more input audio signals 126 .

The method 1600 of FIG. 16 may include a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a DSP, a controller, other hardware device, a firmware device, or It may be implemented by any combination of As an example, the method 1600 of FIG. 16 may be performed by a processor executing instructions as described with reference to FIG. 17 .

Referring to FIG. 17 , a block diagram of a particular example implementation of a device is shown and generally designated 1700 . In various implementations, device 1700 may have more or fewer components than illustrated in FIG. 17 . In an example implementation, device 1700 may correspond to device 102 . In an example implementation, device 1700 may perform one or more operations described with reference to FIGS. 1-16 .

In a particular implementation, the device 1700 includes a processor 1706 (eg, a central processing unit (CPU)). Device 1700 may include one or more additional processors 1710 (eg, one or more DSPs). In a particular aspect, one or more processors 190 of FIG. 1 correspond to processor 1706, processors 1710, or a combination thereof. Processors 1710 may include a speech and music coder-decoder (CODEC) 1708 including a voice coder (“vocoder”) encoder 1736, a vocoder decoder 1738, an audio enhancer 192, or a combination thereof. may also include

Device 1700 may include memory 132 and codec 1734 . Memory 132 may include instructions 1756 executable by one or more additional processors 1710 (or processor 1706) to implement the functionality described with reference to audio enhancer 192. Device 1700 may include a modem 1746 coupled to an antenna 1752 via a transceiver 1750 .

Device 1700 may include a display device 162 coupled to a display controller 1726 . One or more speakers 160 , one or more microphones 120 , or a combination thereof may be coupled to the codec 1734 . For example, one or more microphones 120 may be coupled to the codec 1734 via one or more input interfaces 124 of FIG. 1 . One or more speakers 160 may be coupled to CODEC 1734 via one or more output interfaces. The codec 1734 may include a digital-to-analog converter (DAC) 1702 , an analog-to-digital converter (ADC) 1704 , or both. In a particular implementation, codec 1734 receives analog signals from one or more microphones 120, converts the analog signals to digital signals using analog-to-digital converter 1704, and converts the digital signals into speech and It can also be provided to the music codec 1708. Speech and music codec 1708 may process digital signals, which may be further processed by audio enhancer 192 . In a particular implementation, speech and music codec 1708 may provide digital signals to codec 1734. The codec 1734 may convert the digital signals to analog signals using the digital-to-analog converter 1702 and may provide the analog signals to one or more speakers 160 .

In a particular implementation, device 1700 may be included in a system-in-package or system-on-chip device 1722 . In a particular implementation, memory 132, processor 1706, processors 1710, display controller 1726, codec 1734, and modem 1746 are system-in-package or system-on-chip devices ( 1722) included. In a particular implementation, input device 130 , camera 140 , and power supply 1744 are coupled to system-on-chip device 1722 . Moreover, in a particular implementation, as illustrated in FIG. 17 , a display device 162 , an input device 130 , a camera 140 , one or more speakers 160 , one or more microphones 120 , an antenna 1752 ), and the power supply 1744 is external to the system-on-chip device 1722. In a particular implementation, each of display device 162, input device 130, camera 140, one or more speakers 160, one or more microphones 120, antenna 1752, and power supply 1744 may be coupled to a component of the system-on-chip device 1722, such as an interface (eg, one or more input interfaces 124, input interface 144, one or more additional interfaces, or a combination thereof) or a controller. there is.

Device 1700 may be a virtual assistant, consumer appliance, smart device, Internet of Things (IoT) device, communication device, headset, vehicle, computer, display device, television, game console, music player, radio, video player, entertainment unit, personal use Media players, digital video players, cameras, navigation devices, smart speakers, speaker bars, mobile communication devices, smartphones, cellular phones, laptop computers, tablets, personal digital assistants, digital video disc (DVD) players, tuners, augmented reality headsets , virtual reality headsets, aviation vehicles, home automation systems, voice activated devices, wireless speakers and voice activated devices, portable electronic devices, automobiles, computing devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof. may include.

In conjunction with the described implementations, an apparatus includes means for obtaining directional data of one or more audio sources corresponding to one or more input audio signals. For example, means for obtaining may include directivity analyzer 152 of FIG. 1 , audio enhancer 192 , one or more processors 190 , device 102 , system 100 , processor 1706 , processors ( 1710), modem 1746, transceiver 1750, antenna 1752, one or more other circuits or components configured to obtain directional data of one or more audio sources, or any combination thereof.

The apparatus also includes means for determining one or more equalizer settings based at least in part on the directivity data. For example, the means for determining may include directivity analyzer 152 of FIG. 1 , audio enhancer 192 , one or more processors 190 , device 102 , system 100 , processor 1706 , processors ( 1710), one or more other circuits or components configured to determine one or more equalizer settings based at least in part on the directivity data, or any combination thereof.

The apparatus further comprises means for generating, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals. For example, means for generating may include directivity analyzer 152 of FIG. 1 , audio enhancer 192 , one or more processors 190 , device 102 , system 100 , processor 1706 , processors ( 1710), which may correspond to one or more other circuits or components configured to generate, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of one or more input audio signals, or any combination thereof. can

In some implementations, a non-transitory computer-readable medium (eg, a computer-readable storage device such as memory 132) may be one or more processors (eg, one or more processors 1710 or processor 1706) ), causes the one or more processors to generate one or more audio sources (e.g., audio source) corresponding to one or more input audio signals (e.g., one or more input audio signals 126). 184, instructions that cause obtaining directivity data (eg, directivity data 141) of audio source 684, or both (eg, instructions 1756). The instructions, when executed by one or more processors, also cause the one or more processors to set one or more equalizer settings (e.g., equalizer settings 153, equalizer settings) based at least in part on the directivity data. 653, or a combination thereof). The instructions, when executed by the one or more processors, also cause the one or more processors to generate, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals ( For example, to generate one or more output audio signals 138).

Certain aspects of the present disclosure are described below in a first set of interrelated clauses:

According to clause 1, a device: obtains directivity data of one or more audio sources corresponding to one or more input audio signals; determine one or more equalizer settings based at least in part on the directivity data; and one or more processors configured to generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings.

Clause 2 includes the device of clause 1, wherein the psychoacoustic enhancement version approximates the frequency response of one or more audio sources at a zoom orientation and zoom distance associated with an audio zoom operation.

Clause 3 includes the device of clause 1 or clause 2, wherein the one or more processors are configured to: receive user input indicating a zoom target of an audio zoom operation; and further configured to determine one or more equalizer settings based on the zoom target.

Clause 4 includes the device of clause 3, wherein the zoom target comprises a zoom position, a zoom distance, a zoom orientation, a selection of at least one of one or more audio sources, or a combination thereof.

Clause 5 includes the device of any of clauses 1-4, wherein the directivity data of a particular audio source of the one or more audio sources indicates orientation and distance frequency response characteristics of the particular audio source.

Clause 6 includes the device of any of clauses 1-5, wherein the one or more processors are configured to perform beamforming on the one or more input audio signals to generate one or more beamformed audio signals; and further configured to process the equalizer input audio signal based on the one or more beamformed audio signals to generate one or more output audio signals.

Clause 7 includes the device of any of clauses 1-6, wherein the one or more processors are configured to: identify speech in the speech detection input audio signal based on the one or more input audio signals to generate the one or more speech audio signals; ; and further configured to process the equalizer input audio signal based on the one or more speech audio signals to generate one or more output audio signals.

Clause 8 includes the device of any of clauses 1 through 7, wherein the one or more processors include a gain adjuster input audio signal based on the one or more input audio signals to generate a gain adjusted audio signal based on the zoom target. apply one or more gains to the signal; and further configured to process the equalizer input audio signal based on the gain-adjusted audio signal to generate one or more output audio signals.

Clause 9 includes the device of any of clauses 1 through 8, wherein the one or more processors comprise a noise suppressor for the input audio signal based on the one or more input audio signals to generate a noise suppressed audio signal. perform; and further configured to process the equalizer input audio signal based on the noise suppressed audio signal to generate one or more output audio signals.

Clause 10 includes the device of any of clauses 1 to 9, wherein the one or more processors process a context detector input audio signal based on the one or more input audio signals to generate context data of the one or more audio sources; - context data of a specific audio source of the one or more audio sources indicates an orientation of the specific audio source, a distance of the specific audio source, a type of the specific audio source, or a combination thereof; and acquire directivity data of the specific audio source based on the type of the specific audio source.

Clause 11 includes the device of clause 10, wherein the one or more processors are further configured to generate context data based at least in part on image data associated with the one or more input audio signals.

Clause 12 includes the device of clause 11, wherein the one or more processors are further configured to retrieve the image data and the one or more input audio signals from the memory.

Clause 13 includes the device of clause 11 or clause 12, further comprising one or more microphones coupled to the one or more processors and configured to generate one or more input audio signals.

Clause 14 includes the device of any of clauses 11-13, further comprising a camera coupled to the one or more processors and configured to generate image data.

Clause 15 includes the device of any of clauses 11-14, wherein the one or more processors output one or more input audio signals, image data, or both to identify a type of a particular one of the one or more audio sources. and further configured to perform audio source recognition based on.

Clause 16 includes the device of any of clauses 11 to 15, wherein the one or more processors are configured to send one or more input audio signals, image data, or both to determine a particular one of the one or more audio sources. perform audio source recognition based on; and further configured to perform image analysis on the image data to determine an orientation of the particular audio source.

Clause 17 includes the device of any of clauses 11-16, wherein the one or more processors, to determine a particular one of the one or more audio sources, output one or more input audio signals, image data, or both. perform audio source recognition based on; and further configured to perform distance analysis on one or more input audio signals, image data, or both to determine a distance of a particular audio source.

Clause 18 includes the device of any of clauses 10-17, wherein the one or more processors associate one or more equalizer settings with context data, directivity data, zoom orientation, zoom distance, or a combination thereof, equalizer settings data. is further configured to select one or more equalizer settings based on .

Clause 19 includes the device of clause 18, wherein the one or more processors are further configured to obtain equalizer setting data from memory, another device, or both.

Clause 20 includes the device of any of clauses 10 to 19, wherein the one or more processors are further configured to select one or more equalizer settings to reduce a frequency response corresponding to mid-frequencies.

Clause 21 includes the device of any of clauses 1 to 20, wherein the one or more processors generate, at a first time, a first sound spectrum of a first input audio signal corresponding to a particular one of the one or more audio sources. do; at a second time, generating a second sound spectrum of the second input audio signal corresponding to the specific audio source; and directivity to indicate that the difference between the first distance and the first orientation at the first time and the second distance and the second orientation at the second time corresponds to the difference between the first sound spectrum and the second sound spectrum. It is further configured to update data.

Clause 22 includes the device of any of clauses 1-21, wherein the one or more processors are further configured to obtain directional data from memory, another device, or both.

Clause 23 includes the device of any of clauses 1-5, 21, or 22, wherein the one or more processors perform beamforming on the one or more input audio signals to generate the one or more beamformed audio signals. to perform; detect speech in the one or more input audio signals to produce one or more speech audio signals; based on the zoom target, apply one or more gains to one or more beamformed audio signals, one or more speech audio signals, or a combination thereof to generate one or more gain-adjusted audio signals; generate context data of one or more audio sources based at least in part on the one or more gain-adjusted audio signals, wherein the context data of a particular audio source of the one or more audio sources includes an orientation of the particular audio source, a distance of the particular audio source, , indicating the type of specific audio source, or a combination thereof -; obtain directivity data of a specific audio source based on the type of the specific audio source; determine one or more equalizer settings further based on the context data, zoom orientation, and zoom distance; apply noise suppression to the one or more gain-adjusted audio signals to produce one or more noise suppressed audio signals; and generate one or more output audio signals by processing the one or more noise suppressed audio signals based on the one or more equalizer settings.

Certain aspects of the present disclosure are described below in a second set of interrelated clauses:

According to clause 24, a method comprises: obtaining, at a device, directional data of one or more audio sources corresponding to one or more input audio signals; determining, at the device, one or more equalizer settings based at least in part on the directivity data; and based on the equalizer settings, generating one or more output audio signals corresponding to the psychoacoustic enhanced version of the one or more input audio signals.

Clause 25 includes receiving, at the device, user input indicating a zoom target of an audio zoom operation; and determining, at the device, one or more equalizer settings based on a zoom target, wherein the zoom target is a zoom position, a zoom distance, a zoom orientation, a selection of at least one of one or more audio sources, or a combination thereof. includes

Certain aspects of the present disclosure are described below in a third set of interrelated clauses:

According to clause 26, the non-transitory computer-readable medium, when executed by one or more processors, causes the one or more processors to: obtain directional data of one or more audio sources corresponding to one or more input audio signals; determine one or more equalizer settings based at least in part on the directivity data; Stores instructions that cause, based on the equalizer settings, to generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals.

Clause 27 includes the non-transitory computer-readable medium of clause 26, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: generate one or more beamformed audio signals. perform beamforming on one or more input audio signals; and process an equalizer input audio signal based on the one or more beamformed audio signals to generate one or more output audio signals.

Clause 28 includes the non-transitory computer-readable medium of clause 26 or clause 27, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: generate one or more speech audio signals. detect speech based on the one or more input audio signals to identify speech in the input audio signal; Process an equalizer input audio signal based on the one or more speech audio signals to generate one or more output audio signals.

Certain aspects of the present disclosure are described below in a fourth set of interrelated clauses:

According to clause 29, an apparatus comprising: means for obtaining directional data of one or more audio sources corresponding to one or more input audio signals; means for determining one or more equalizer settings based at least in part on the directivity data; and means for generating, based on the equalizer settings, one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals.

Clause 30 includes the apparatus of clause 29, wherein the means for obtaining, the means for determining, and the means for generating are virtual assistants, consumer electronics, smart devices, Internet of Things (IoT) devices, communication devices, headsets, vehicles. , a computer, display device, television, gaming console, music player, radio, video player, entertainment unit, personal media player, digital video player, camera, or navigation device.

Those skilled in the art will also understand that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein are electronic hardware, computer software executed by a processor, or combinations of both. It will be appreciated that it may be implemented as Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or processor executable instructions depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. A software module may include random access memory (RAM), flash memory, read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory ( EEPROM), registers, hard disk, removable disk, compact disk read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). An ASIC may reside on a computing device or user terminal. Alternatively, the processor and storage medium may reside as separate components in a computing device or user terminal.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosed embodiments. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims (30)

As a device,
a memory configured to store directional data of one or more audio sources corresponding to one or more input audio signals; and
determine one or more equalizer settings based at least in part on the directivity data; and
generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings;
A device comprising one or more processors configured. According to claim 1,
wherein the psychoacoustic enhancement version approximates a frequency response of the one or more audio sources at a zoom orientation and zoom distance associated with an audio zoom operation. According to claim 1,
The one or more processors may further:
receive user input indicating a zoom target of an audio zoom operation; and
determine the one or more equalizer settings based on the zoom target;
configured device. According to claim 3,
wherein the zoom target comprises a zoom position, a zoom distance, a zoom orientation, a selection of at least one of the one or more audio sources, or a combination thereof. According to claim 1,
wherein the directional data of a particular audio source of the one or more audio sources indicates orientation and distance frequency response characteristics of the particular audio source. According to claim 1,
The one or more processors may further:
performing beamforming on the one or more input audio signals to generate one or more beamformed audio signals; and
process an equalizer input audio signal based on the one or more beamformed audio signals to generate the one or more output audio signals;
configured device. According to claim 1,
The one or more processors may further:
identify speech in a speech detection input audio signal based on the one or more input audio signals to generate one or more speech audio signals; and
process an equalizer input audio signal based on the one or more speech audio signals to generate the one or more output audio signals;
configured device. According to claim 1,
The one or more processors may further:
based on a zoom target, apply one or more gains to a gain adjuster input audio signal based on the one or more input audio signals to generate a gain-adjusted audio signal; and
process an equalizer input audio signal based on the gain-adjusted audio signal to generate the one or more output audio signals;
configured device. According to claim 1,
The one or more processors may further:
a noise suppressor based on the one or more input audio signals performs noise suppression on an input audio signal to generate a noise suppressed audio signal; and
process an equalizer input audio signal based on the noise suppressed audio signal to generate the one or more output audio signals;
configured device. According to claim 1,
The one or more processors may further:
processing a context detector input audio signal based on the one or more input audio signals to generate context data of the one or more audio sources, wherein the context data of a particular audio source of the one or more audio sources is processing the context detector input audio signal indicative of an orientation of the specific audio source, a distance of the specific audio source, a type of the specific audio source, or a combination thereof; and
Obtain directivity data of the specific audio source based on the type of the specific audio source
configured device. According to claim 10,
wherein the one or more processors are further configured to generate the context data based at least in part on image data associated with the one or more input audio signals. According to claim 11,
wherein the one or more processors are further configured to retrieve the image data and the one or more input audio signals from a memory. According to claim 11,
and one or more microphones coupled to the one or more processors and configured to generate the one or more input audio signals. According to claim 11,
and a camera coupled to the one or more processors and configured to generate the image data. According to claim 11,
The one or more processors are further configured to perform audio source recognition based on the one or more input audio signals, the image data, or both to identify the type of the specific audio source of the one or more audio sources. device. According to claim 11,
The one or more processors may further:
perform audio source recognition based on the one or more input audio signals, the image data, or both, to determine the specific one of the one or more audio sources; and
perform image analysis on the image data to determine an orientation of the particular audio source;
configured device. According to claim 11,
The one or more processors may further:
perform audio source recognition based on the one or more input audio signals, the image data, or both, to determine the specific one of the one or more audio sources; and
perform a distance analysis on the one or more input audio signals, the image data, or both to determine a distance of the particular audio source;
configured device. According to claim 10,
The one or more processors may further configure the one or more equalizer settings based on equalizer setting data associating the one or more equalizer settings with the context data, the directivity data, zoom orientation, zoom distance, or a combination thereof. A device configured to select. According to claim 18,
wherein the one or more processors are further configured to obtain the equalizer setting data from the memory, another device, or both. According to claim 10,
wherein the one or more processors are further configured to select the one or more equalizer settings to reduce a frequency response corresponding to intermediate frequencies. According to claim 1,
The one or more processors may further:
generate, at a first time, a first sound spectrum of a first input audio signal corresponding to a specific one of the one or more audio sources;
at a second time, generate a second sound spectrum of a second input audio signal corresponding to the specific audio source; and
and that the difference between the first distance and the first orientation at the first time and the second distance and the second orientation at the second time corresponds to the difference between the first sound spectrum and the second sound spectrum. to update the directivity data to
configured device. According to claim 1,
wherein the one or more processors are further configured to obtain the directional data from the memory, another device, or both. According to claim 1,
The one or more processors may further:
performing beamforming on the one or more input audio signals to generate one or more beamformed audio signals;
detect speech in the one or more input audio signals to generate one or more speech audio signals;
based on a zoom target, apply one or more gains to the one or more beamformed audio signals, the one or more speech audio signals, or a combination thereof to generate one or more gain-adjusted audio signals;
generating context data of the one or more audio sources based at least in part on the one or more gain-adjusted audio signals, wherein the context data of the specific one of the one or more audio sources is of the specific audio source. generating context data of the one or more audio sources indicating an orientation, a distance of the particular audio source, the type of the particular audio source, or a combination thereof;
obtain the directivity data of the specific audio source based on the type of the specific audio source;
determine the one or more equalizer settings further based on the context data, zoom orientation, and zoom distance;
apply noise suppression to the one or more gain-adjusted audio signals to produce one or more noise suppressed audio signals; and
generate the one or more output audio signals by processing the one or more noise suppressed audio signals based on the one or more equalizer settings;
configured device. As a method,
obtaining, at the device, directional data of one or more audio sources corresponding to one or more input audio signals;
determining, at the device, one or more equalizer settings based at least in part on the directivity data; and
generating one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings. 25. The method of claim 24,
Receiving, in the device, a user input indicating a zoom target of an audio zoom operation; and
further comprising, in the device, determining the one or more equalizer settings based on the zoom target;
wherein the zoom target comprises a zoom position, a zoom distance, a zoom orientation, a selection of at least one of the one or more audio sources, or a combination thereof. A non-transitory computer readable medium storing instructions,
The instructions, when executed by one or more processors, cause the one or more processors to:
obtain directional data of one or more audio sources corresponding to one or more input audio signals;
determine one or more equalizer settings based at least in part on the directivity data; and
generate one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings. 27. The method of claim 26,
The instructions, when executed by the one or more processors, further cause the one or more processors to:
perform beamforming on the one or more input audio signals to generate one or more beamformed audio signals; and
processing an equalizer input audio signal based on the one or more beamformed audio signals to generate the one or more output audio signals. 27. The method of claim 26,
The instructions, when executed by the one or more processors, further cause the one or more processors to:
identify speech in an input audio signal to generate one or more speech audio signals; and
processing an equalizer input audio signal based on the one or more speech audio signals to generate the one or more output audio signals. As a device,
means for obtaining directional data of one or more audio sources corresponding to one or more input audio signals;
means for determining one or more equalizer settings based at least in part on the directivity data; and
means for generating one or more output audio signals corresponding to a psychoacoustic enhanced version of the one or more input audio signals based on the equalizer settings. The method of claim 29,
The means for obtaining, the means for determining, and the means for generating are virtual assistants, household appliances, smart devices, Internet of Things (IoT) devices, communication devices, headsets, vehicles, computers, display devices, televisions, gaming A device integrated into at least one of a console, music player, radio, video player, entertainment unit, personal media player, digital video player, camera, or navigation device.

Images