US20210219053A1 - Multiple-source tracking and voice activity detections for planar microphone arrays - Google Patents
Multiple-source tracking and voice activity detections for planar microphone arrays Download PDFInfo
- Publication number
- US20210219053A1 US20210219053A1 US16/740,297 US202016740297A US2021219053A1 US 20210219053 A1 US20210219053 A1 US 20210219053A1 US 202016740297 A US202016740297 A US 202016740297A US 2021219053 A1 US2021219053 A1 US 2021219053A1
- Authority
- US
- United States
- Prior art keywords
- tdoa
- audio
- microphone
- computed
- covariance matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 56
- 230000000694 effects Effects 0.000 title claims abstract description 16
- 238000003491 array Methods 0.000 title description 5
- 230000005236 sound signal Effects 0.000 claims description 76
- 238000012545 processing Methods 0.000 claims description 50
- 238000000034 method Methods 0.000 claims description 47
- 239000011159 matrix material Substances 0.000 claims description 38
- 238000013507 mapping Methods 0.000 claims description 7
- 230000002708 enhancing effect Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 abstract description 8
- 230000007246 mechanism Effects 0.000 abstract description 4
- 238000004891 communication Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000001755 vocal effect Effects 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 238000005755 formation reaction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012880 independent component analysis Methods 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 206010019133 Hangover Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000005295 random walk Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S3/00—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received
- G01S3/80—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using ultrasonic, sonic or infrasonic waves
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S3/00—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received
- G01S3/80—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using ultrasonic, sonic or infrasonic waves
- G01S3/8006—Multi-channel systems specially adapted for direction-finding, i.e. having a single aerial system capable of giving simultaneous indications of the directions of different signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/78—Detection of presence or absence of voice signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/78—Detection of presence or absence of voice signals
- G10L25/87—Detection of discrete points within a voice signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R29/00—Monitoring arrangements; Testing arrangements
- H04R29/004—Monitoring arrangements; Testing arrangements for microphones
- H04R29/005—Microphone arrays
- H04R29/006—Microphone matching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
- H04S7/30—Control circuits for electronic adaptation of the sound field
- H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S3/00—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received
- G01S3/80—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using ultrasonic, sonic or infrasonic waves
- G01S3/802—Systems for determining direction or deviation from predetermined direction
- G01S3/808—Systems for determining direction or deviation from predetermined direction using transducers spaced apart and measuring phase or time difference between signals therefrom, i.e. path-difference systems
- G01S3/8083—Systems for determining direction or deviation from predetermined direction using transducers spaced apart and measuring phase or time difference between signals therefrom, i.e. path-difference systems determining direction of source
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
- G10L2021/02166—Microphone arrays; Beamforming
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2203/00—Details of circuits for transducers, loudspeakers or microphones covered by H04R3/00 but not provided for in any of its subgroups
- H04R2203/12—Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2430/00—Signal processing covered by H04R, not provided for in its groups
- H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
- H04R2430/21—Direction finding using differential microphone array [DMA]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2430/00—Signal processing covered by H04R, not provided for in its groups
- H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
- H04R2430/23—Direction finding using a sum-delay beam-former
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
- H04S2420/11—Application of ambisonics in stereophonic audio systems
Definitions
- the present disclosure in accordance with one or more embodiments, relates generally to audio signal processing, and more particularly for example, to systems and methods for multiple source tracking and multi-stream voice activity detections for generic planar microphone arrays.
- Smart speakers and other voice-controlled devices and appliances have gained popularity in recent years.
- Smart speakers often include an array of microphones for receiving audio inputs (e.g., verbal commands of a user) from an environment.
- audio inputs e.g., verbal commands of a user
- target audio e.g., the verbal command
- the smart speaker may translate the detected target audio into one or more commands and perform different tasks based on the commands.
- One challenge of these smart speakers is to efficiently and effectively isolate the target audio (e.g., the verbal command) from noise or other active speakers in the operating environment.
- the target audio e.g., the verbal command
- one or more speakers can be active in the presence of one or more noise sources.
- the target speaker When the goal is to enhance a specific speaker, that speaker is called the target speaker while the rest of the speakers can be viewed as interference sources.
- Existing speech enhancement algorithms mostly exploit the spatial information of the sources using multiple input channels (microphones), such as the blind source separation (BSS) methods related to independent component analysis (ICA), and the spatial filtering or beamforming methods.
- BSS blind source separation
- ICA independent component analysis
- VAD voice activity detection
- FIG. 1 illustrates an example operating environment for an audio processing device in accordance with one or more embodiments of the disclosure.
- FIG. 2 is a block diagram of an example audio processing device in accordance with one or more embodiments of the disclosure.
- FIG. 3 is a block diagram of example audio signal processor for multi-track audio enhancement in accordance with one or more embodiments of the disclosure.
- FIG. 4 is a block diagram of an example multi-track activity detection engine for processing multiple audio signals from a generic microphone array according to various embodiments of the disclosure.
- FIG. 5A is a diagram illustrating example geometry of a microphone pair according to one or more embodiments of the disclosure.
- FIG. 5B is a diagram illustrating example meshes of time difference of arrival (TDOA) locus information in a multi-dimensional space corresponding to different microphone array geometries according to one or more embodiments of the disclosure.
- TDOA time difference of arrival
- FIG. 6 is a logic flow diagram of an example method for enhancing multi-source audio signals by multi-source tracking and activity detection according to various embodiments of the disclosure.
- FIG. 7 is a logic flow diagram of an example process for computing TDOA locus information in the multi-dimensional space using microphone pairs according to various embodiments of the disclosure.
- the present disclosure provides improved systems and methods for detecting and processing target audio signal(s) in a multi-stream audio environment.
- VAD Voice activity detection
- VAD Voice activity detection
- M2T Measurement-to-Track
- existing DOA methods usually construct a single steering vector for the entire array of microphones based on a closed form mapping of azimuth and elevation angles, which can be used to exploit the special geometry of linear or circular arrays.
- Such DOA methods cannot be extended to generic or arbitrary geometries of microphone arrays.
- these closed-form-mapping based DOA methods often require an extensive search in a multi-dimensional space.
- existing TDOA-based methods may be used, which may not restrict to a specific array geometry and may construct multiple steering vectors for each microphone pair to form a multidimensional TDOA vector (one dimension for each pair).
- these existing methods bear the risk of introducing TDOA ghosts formed by the cross intersection of the peaks from the spectrum of each TDOA pair. As a result, further post-processing involving the specific array geometry is often needed to remove TDOA ghosts.
- embodiments described herein provide a combined multi-source TDOA tracking and VAD mechanism that is applicable for generic array geometries, e.g., a microphone array that lies on a plane.
- the combined multi-source TDOA tracking and VAD mechanism may reduce the number of calculations that was usually involved in traditional TDOA by performing the TDOA search for each dimension separately.
- a multi-dimensional TDOA method for generic array geometries that lie on a plane is employed, which avoids unwanted ghost TDOAs.
- the Cartesian coordinates of the generically configured microphones are obtained, one of which may be selected as the reference microphone.
- the azimuth and elevation angles of the microphones may be scanned, based on which a planar locus of physically admissible TDOAs can be formed in the multi-dimensional TDOA space of multiple microphone pairs.
- the formed planar locus avoids the formation of ghost TDOAs, thus no further post-processing is needed to remove ghost TDOAs.
- multi-dimensional TDOA methods disclosed herein reduce computational complexity by conducting the search in the pairwise TDOA domain pertaining to each dimension separately, rather than a search in a full multi-dimensional space.
- FIG. 1 illustrates an example operating environment 100 in which an audio processing system may operate according to various embodiments of the disclosure.
- the operating environment 100 includes an audio processing device 105 , a target audio source 110 , and one or more noise sources 135 - 145 .
- the operating environment 100 is illustrated as a room, but it is contemplated that the operating environment may include other areas, such as an inside of a vehicle, an office conference room, rooms of a home, an outdoor stadium or an airport.
- the audio processing device 105 may include two or more audio sensing components (e.g., microphones) 115 a - 115 d and, optionally, one or more audio output components (e.g., speakers) 120 a - 120 b.
- two or more audio sensing components e.g., microphones
- one or more audio output components e.g., speakers
- the audio processing device 105 may be configured to sense sound via the audio sensing components 115 a - 115 d and generate a multi-channel audio input signal, comprising two or more audio input signals.
- the audio processing device 105 may process the audio input signals using audio processing techniques disclosed herein to enhance the audio signal received from the target audio source 110 .
- the processed audio signals may be transmitted to other components within the audio processing device 105 , such as a speech recognition engine or voice command processor, or to an external device.
- the audio processing device 105 may be a standalone device that processes audio signals, or a device that turns the processed audio signals into other signals (e.g., a command, an instruction, etc.) for interacting with or controlling an external device.
- the audio processing device 105 may be a communications device, such as mobile phone or voice-over-IP (VoIP) enabled device, and the processed audio signals may be transmitted over a network to another device for output to a remote user.
- the communications device may also receive processed audio signals from a remote device and output the processed audio signals via the audio output components 120 a - 120 b.
- the target audio source 110 may be any source that produces sound detectable by the audio processing device 105 .
- the target audio to be detected by the system may be defined based on criteria specified by user or system requirements.
- the target audio may be defined as human speech, a sound made by a particular animal or a machine.
- the target audio is defined as human speech, and the target audio source 110 is a person.
- the operating environment 100 may include one or more noise sources 135 - 145 .
- sound that is not target audio may be processed as noise.
- the noise sources 135 - 145 may include a loud speaker 135 playing music, a television 140 playing a television show, movie or sporting event, and background conversations between non-target speakers 145 . It will be appreciated that different noise sources may be present in various operating environments.
- the target audio and noise may reach the audio sensing components 115 a - 115 d of the audio processing device 105 from different directions and at different times.
- the noise sources 135 - 145 may produce noise at different locations within the operating environment 100
- the target audio source (person) 110 may speak while moving between locations within the operating environment 100 .
- the target audio and/or the noise may reflect off fixtures (e.g., walls) within the room 100 . For example, consider the paths that the target audio may traverse from the target audio source 110 to reach each of the audio sensing components 115 a - 115 d .
- the target audio may directly travel from the target audio source 110 to the audio sensing components 115 a - 115 d , respectively. Additionally, the target audio may reflect off the walls 150 a and 150 b , and reach the audio sensing components 115 a - 115 d indirectly from the target audio source 110 , as indicated by arrows 130 a - 130 b .
- the audio processing device 105 may use one or more audio processing techniques to estimate and apply a room impulse response to further enhance the target audio and suppress noise.
- FIG. 2 illustrates an example audio processing device 200 according to various embodiments of the disclosure.
- the audio processing device 200 may be implemented as the audio processing device 105 of FIG. 1 .
- the audio processing device 200 includes an audio sensor array 205 , an audio signal processor 220 and host system components 250 .
- the audio sensor array 205 comprises two or more sensors, each of which may be implemented as a transducer that converts audio inputs in the form of sound waves into an audio signal.
- the audio sensor array 205 comprises a plurality of microphones 205 a - 205 n , each generating an audio input signal which is provided to the audio input circuitry 222 of the audio signal processor 220 .
- the audio sensor array 205 generates a multichannel audio signal, with each channel corresponding to an audio input signal from one of the microphones 205 a - n.
- the audio signal processor 220 includes the audio input circuitry 222 , a digital signal processor 224 and optional audio output circuitry 226 .
- the audio signal processor 220 may be implemented as an integrated circuit comprising analog circuitry, digital circuitry and the digital signal processor 224 , which is operable to execute program instructions stored in firmware.
- the audio input circuitry 222 may include an interface to the audio sensor array 205 , anti-aliasing filters, analog-to-digital converter circuitry, echo cancellation circuitry, and other audio processing circuitry and components.
- the digital signal processor 224 is operable to process a multichannel digital audio signal to generate an enhanced audio signal, which is output to one or more host system components 250 .
- the digital signal processor 224 may be operable to perform echo cancellation, noise cancellation, target signal enhancement, post-filtering, and other audio signal processing functions.
- the optional audio output circuitry 226 processes audio signals received from the digital signal processor 224 for output to at least one speaker, such as speakers 210 a and 210 b .
- the audio output circuitry 226 may include a digital-to-analog converter that converts one or more digital audio signals to analog and one or more amplifiers for driving the speakers 210 a - 210 b.
- the audio processing device 200 may be implemented as any device operable to receive and enhance target audio data, such as, for example, a mobile phone, smart speaker, tablet, laptop computer, desktop computer, voice controlled appliance, or automobile.
- the host system components 250 may comprise various hardware and software components for operating the audio processing device 200 .
- the host system components 250 include a processor 252 , user interface components 254 , a communications interface 256 for communicating with external devices and networks, such as network 280 (e.g., the Internet, the cloud, a local area network, or a cellular network) and mobile device 284 , and a memory 258 .
- the processor 252 and digital signal processor 224 may comprise one or more of a processor, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., field programmable gate array (FPGA)), a digital signal processing (DSP) device, or other logic device that may be configured, by hardwiring, executing software instructions, or a combination of both, to perform various operations discussed herein for embodiments of the disclosure.
- PLD programmable logic device
- FPGA field programmable gate array
- DSP digital signal processing
- the host system components 250 are configured to interface and communicate with the audio signal processor 220 and the other host system components 250 , such as through a bus or other electronic communications interface.
- the audio signal processor 220 and the host system components 250 are shown as incorporating a combination of hardware components, circuitry and software, in some embodiments, at least some or all of the functionalities that the hardware components and circuitries are operable to perform may be implemented as software modules being executed by the processor 252 and/or digital signal processor 224 in response to software instructions and/or configuration data, stored in the memory 258 or firmware of the digital signal processor 224 .
- the memory 258 may be implemented as one or more memory devices operable to store data and information, including audio data and program instructions.
- Memory 258 may comprise one or more various types of memory devices including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, hard disk drive, and/or other types of memory.
- the processor 252 may be operable to execute software instructions stored in the memory 258 .
- a speech recognition engine 260 is operable to process the enhanced audio signal received from the audio signal processor 220 , including identifying and executing voice commands.
- Voice communications components 262 may be operable to facilitate voice communications with one or more external devices such as a mobile device 284 or user device 286 , such as through a voice call over a mobile or cellular telephone network or a VoIP call over an IP network.
- voice communications include transmission of the enhanced audio signal to an external communications device.
- the user interface components 254 may include a display, a touchpad display, a keypad, one or more buttons and/or other input/output components operable to enable a user to directly interact with the audio processing device 200 .
- the communications interface 256 facilitates communication between the audio processing device 200 and external devices.
- the communications interface 256 may enable Wi-Fi (e.g., 802.11) or Bluetooth connections between the audio processing device 200 and one or more local devices, such as mobile device 284 , or a wireless router providing network access to a remote server 282 , such as through the network 280 .
- the communications interface 256 may include other wired and wireless communications components facilitating direct or indirect communications between the audio processing device 200 and one or more other devices.
- FIG. 3 illustrates an example audio signal processor 300 according to various embodiments of the disclosure.
- the audio signal processor 300 is embodied as one or more integrated circuits including analog and digital circuitry and firmware logic implemented by a digital signal processor, such as audio signal processor 220 of FIG. 2 .
- the audio signal processor 300 includes audio input circuitry 315 , a sub-band frequency analyzer 320 , a multi-track VAD engine 325 , an audio enhancement engine 330 , and a synthesizer 335 .
- the audio signal processor 300 receives a multi-channel audio input from a plurality of audio sensors, such as a sensor array 305 comprising at least two audio sensors 305 a - n .
- the audio sensors 305 a - 305 n may include microphones that are integrated with an audio processing device, such as the audio processing device 200 of FIG. 2 , or external components connected thereto.
- the arrangement of the audio sensors 305 a - 305 n may be known or unknown to the audio signal processor 300 according to various embodiments of the disclosure.
- the audio signals may be processed initially by the audio input circuitry 315 , which may include anti-aliasing filters, analog to digital converters, and/or other audio input circuitry.
- the audio input circuitry 315 outputs a digital, multichannel, time-domain audio signal, where M is the number of sensor (e.g., microphone) inputs.
- M is the number of sensor (e.g., microphone) inputs.
- the multichannel audio signal is input to the sub-band frequency analyzer 320 , which partitions the multichannel audio signal into successive frames and decomposes each frame of each channel into a plurality of frequency sub-bands.
- the sub-band frequency analyzer 320 includes a Fourier transform process and outputs a plurality of frequency bins.
- the decomposed audio signals are then provided to the multi-track VAD engine 325 and the audio enhancement engine 330 .
- the multi-track VAD engine 325 is operable to analyze the frames of one or more of the audio tracks and generate VAD outputs indicating whether a target audio activity is present in the current frame.
- target audio may be any audio to be identified by the audio system.
- the multi-track VAD engine 325 may be implemented specifically for detecting voice activities.
- the multi-track VAD engine 325 is operable to receive a frame of audio data and generate a VAD indication output for each audio track regarding the presence or absence of the target audio on the respective audio track corresponding to the frame of audio data.
- Detailed components and operations of the multi-track VAD engine 325 is further illustrated in relation to 400 in FIG. 4 .
- the audio enhancement engine 330 receives the sub-band frames from the sub-band frequency analyzer 320 and the VAD indications from the multi-track VAD engine 325 .
- the audio enhancement engine 330 is configured to process the sub-band frames based on the received multi-track VAD indications to enhance the multi-track audio signals.
- the audio enhancement engine 330 may enhance the portion of the audio signal determined to be from the direction of the target audio sources and suppress the other portions of the audio signal which are determined to be noise.
- the audio enhancement engine 330 may pass the processed audio signal to the synthesizer 335 .
- the synthesizer 335 reconstructs one or more of the multichannel audio signals on a frame-by-frame basis by combining the sub-bands to form an enhanced time-domain audio signal.
- the enhanced audio signal may then be transformed back to the time domain and sent to a system component or external device for further processing.
- FIG. 4 illustrates an example multi-track VAD engine 400 for processing multiple audio signals from a generic microphone array according to various embodiments of the disclosure.
- the multi-track VAD engine 400 may be implemented as a combination of digital circuitry and logic performed by a digital signal processor.
- the multi-track VAD engine 400 may be installed in an audio signal processor such as 300 in FIG. 3 .
- the multi-track VAD engine 400 may provide further structural and functional details to the multi-track VAD engine 325 in FIG. 3 .
- the multi-track VAD engine 400 includes a subband analysis module 405 , a block-based TDOA estimation module 410 , a TDOA locus calculation module 420 and a multi-source tracking and multi-stream VAD estimation module 430 .
- the subband analysis module 405 is configured to obtain and transform the audio signals 402 to a time-frequency domain representation 404 , denoted as X m (l, k) corresponding to the original time-domain audio signal x m (t), where l indicates the subband time index and k indicates the frequency band index.
- the subband analysis module 405 may be similar to the sub-band frequency analyzer 320 in FIG. 3 , which performs Fourier transform to convert input time-domain audio signals to frequency domain representations.
- the subband analysis module 405 may then send the generated time-frequency domain representations 404 to the block-based TDOA estimation module 410 and the multi-source tracking and multi-stream VAD estimation module 430 .
- the TDOA locus calculation module 420 is configured to scan a generic microphone array (e.g., audio sensors 305 a - n that form a generic array geometry). For example, for a given arbitrary microphone array geometry on a plane, a locus of permissible TDOA locations is computed once at system boot-up. This locus of points can avoid the ghost formations.
- a generic microphone array e.g., audio sensors 305 a - n that form a generic array geometry.
- the first microphone may be selected as the reference microphone, which in turn gives M ⁇ 1 microphone pairs all with respect to the first microphone.
- FIG. 5A illustrates an example microphone pair.
- the microphone pair indexed as the i ⁇ 1th pair, includes microphone i 502 and reference microphone 1 501 for an incident ray 505 with azimuth angle ⁇ and elevation angle of zero emitted from a distant source (assuming a far-field model).
- the distance between the microphone pair of 501 and 502 as well as the angle between the two microphones are denoted as d i ⁇ 1 and ⁇ i ⁇ 1 , respectively, which can be computed given the Cartesian coordinates of the i th microphone 502 .
- the TDOA of the (i ⁇ 1)th microphone pair can be computed as
- ⁇ i - 1 ⁇ ( ⁇ , ⁇ ) d i - 1 c ⁇ cos ⁇ ( ⁇ - ⁇ i - 1 ) ⁇ cos ⁇ ⁇ ⁇ ( 1 )
- the TDOA locus calculation module 420 may construct a mesh of permissible TDOAs.
- the resulting TDOA locus ( ⁇ 1 ( ⁇ , ⁇ ), . . . ⁇ p ( ⁇ , ⁇ ), . . . , ⁇ M ⁇ 1 ( ⁇ , ⁇ ) for all scans of ⁇ and ⁇ ) also lies on a plane in the (M ⁇ 1) dimensional space.
- Different layouts of the M microphones may result in different planes in the (M ⁇ 1) dimensional space.
- FIG. 5B illustrates two different example microphone layouts along with their respective TDOA meshes.
- the distance between the first and the third microphone increases to 16 cm as shown at 520 , the resulting mesh of permissible TDOA is shown at 525 .
- the TDOA locus calculation module 420 may then send the (M ⁇ 1)-dimensional TDOAs 403 to the block-based TDOA estimation module 410 .
- the block-based TDOA estimation module 410 receives the time-frequency domain representations 404 of the multi-source audios and the TDOAs 403 , based on which TDOA estimation module 410 extracts the TDOA information of the source microphones (e.g., audio sensors 305 a - n shown in FIG. 3 ) using the data obtained from consecutive frames.
- the source microphones e.g., audio sensors 305 a - n shown in FIG. 3
- STMV steered minimum variance
- the first microphone may be selected as the reference microphone, and accordingly, X 1 (l, k) denotes the time-frequency representation of the audio from the reference microphone.
- the block-based TDOA estimation module 410 then computes the covariance matrix of the pth input signal pairs for every frequency band k:
- the summation in computing R p (k) is implemented over a block of certain number of consecutive frames.
- the index of the block is neglected here for brevity.
- the block-based TDOA estimation module 410 may then construct a steering matrix for each pair and frequency band as follows:
- ⁇ p is the TDOA for the pth pair obtained from the TDOA locus calculation module 420 after different scans of ⁇ and ⁇ (neglected for brevity);
- f k is the frequency at band k;
- diag([a, b]) denotes a 2 ⁇ 2 diagonal matrix with diagonal elements a and b.
- the block-based TDOA estimation module 410 constructs the directional covariance matrix that is coherently aligned across all frequency bands by:
- the computation of the directional covariance matrix C p ( ⁇ p ) is repeated over all microphone pairs p and all scans of azimuth/elevation ( ⁇ , ⁇ ) for ⁇ p .
- the TDOA space of each dimension p corresponding to the pth microphone pair is linearly quantized to q segments.
- the TDOA locus points ( ⁇ 1 , ⁇ p , . . . , ⁇ M ⁇ 1 ) obtained from scanning each azimuth and elevation angles ( ⁇ , ⁇ ) is mapped to the closest quantized points for each dimension.
- mapping of ( ⁇ , ⁇ ) ⁇ (ind 1 ( ⁇ , ⁇ ), . . . , ind M ⁇ 1 ( ⁇ , ⁇ )) is saved in memory, where 1 ⁇ ind p ( ⁇ , ⁇ ) ⁇ q is the quantized TDOA index of dimension p pertaining to scanning angles ⁇ and ⁇ .
- TDOA locus points ( ⁇ 1 , . . . ⁇ p , . . . , ⁇ M ⁇ 1 ) are quantized, not all calculations need to be carried out as some of the TDOA dimensions may be quantized to the same segment among the q quantized segments.
- the block-based TDOA estimation module 410 may then compute the STMV joint pseudo-likelihood for all M ⁇ 1 pairs of microphones as:
- ⁇ *, ⁇ * argmax ⁇ , ⁇ STMV ( ⁇ 1 , . . . ,T M ⁇ 1 ) (7)
- the azimuth and elevation pair ⁇ *, ⁇ * may then be used for multi-source tracking and multi-stream VAD estimation.
- One possible solution may include tracking the angles between the two microphones of each microphone pair directly. However, due to the wrap-around effect of azimuth in 360 degrees, if tracking is done directly on the angles between paired microphones, track loss may occur when a microphone source crosses 0° towards 360° and vice versa. Therefore, in order to avoid such confusion, detection z is computed based on the angles between paired microphones in a circular manner using a polar transformation as follows:
- ⁇ scale >1 is a scaling constant that may expand the measurement space allowing the tracking to be done with parameters pertaining to meaningful concepts like angles.
- the block-based TDOA estimation module 410 then sends the computed detection z to the multi-source tracking and multi-stream VAD estimation module 430 . If an maximum number of tracks exist, the TDOAs obtained from the block-based TDOA estimation module 410 are to be tracked by updating the track obtained from a previous step in a recursive manner. Specifically, if the detection obtained at block (time-step) n ⁇ 1 is denoted by z n ⁇ 1 and up until then there existed t n ⁇ 1 tracks, for a new detection z n 406 emerging at time-step n, the multi-source tracking and multi-stream VAD estimation module 430 processes the new detection z n based on the gate of existing tracks as follows:
- the tracking results 408 incorporating the up-to-date detections 406 at time-step n can be output from the module 430 , denoted by z n Tr (1), . . . z n Tr ( ).
- the module 430 When all the audio tracks have been updated, the module 430 generates multi-stream VADs 412 using the nearest neighbor M2T assignment. Specifically, at time-step n, the M2T assignment may be carried out by assigning a 1 to the track that is closest to detection z n , and 0 to the other tracks. In some implementations, a hangover may be applied for the VADs to have an intermediate value, e.g., ⁇ 1, etc., before they are assigned completely to zero after being 1 in the previous time-step. In this way, the multi-stream VADs 412 , denoted by VAD 1 , . . . , , each representing whether any voice activity detection is found in the respective track, are output from the module 430 for audio enhancement, e.g., to the audio enhancement engine 330 in FIG. 3 .
- the multi-stream VADs 412 denoted by VAD 1 , . . . , each representing whether any voice activity detection is found in the respective track
- FIG. 6 illustrates an example method 600 for enhancing multi-source audio signals by multi-source tracking and VAD according to various embodiments of the disclosure.
- the method 600 may be performed by one or more components in the audio signal processor 300 , and/or one or more components of the multi-track VAD engine 400 .
- Method 600 starts with step 602 , at which the TDOA locus information may be computed based on spatial information of a microphone array.
- the TDOA locus information may be computed once at system boot-up by scanning the microphone array with incident rays of varying azimuth and elevation angles. The computation may be performed with reduced complexity in a multi-dimensional space constructed by pairing the microphones from the microphone array, as further described in relation to FIG. 7 .
- a first microphone may be selected from the microphone array as a reference microphone.
- each remaining microphone from the microphone array may be paired with the reference microphone.
- a TDOA location corresponding to a specific azimuth angle and a specific elevation angle of an incident ray may be computed based on a distance and an angle between two microphones in the respective pair, e.g., according to Eq. (1) described in relation to FIG. 4 .
- An example microphone pair with a specific azimuth angle and a specific elevation angle of an incident ray is also shown in FIG. 5A .
- the method retrieves the next microphone pair at step 710 and repeats at step 706 until TDOA locations for all microphone pairs have been computed.
- the method retrieves the next scan of azimuth and elevation angles at step 714 and repeats at step 706 , until TDOA locations for all scans of azimuth and elevation angles are computed.
- a mesh of TDOA location points can be formed at step 716 .
- Example meshes of TDOA location points corresponding to different geometries of the microphone arrays are shown in FIG. 5B .
- step 604 one or more multi-source audio signals may be received from the microphone array.
- time-domain samples of the multi-source audio signals 402 in FIG. 4 may be received via the audio input circuitry 315 in FIG. 3 .
- the one or more multi-source audio signals may be transformed from time-domain to time-frequency representations.
- the subband analysis module 405 may transform the time-domain signals to time-frequency domain representations, as described in relation to FIG. 4 .
- TDOA detection data may be computed for the one or more multi-source audio signals according to a STMV beamformer based on the computed TDOA locus. For example, for each microphone pair, a covariance matrix may be computed using time-frequency representations of the one or more multi-source audio signals from the respective microphone pair for every frequency band, e.g., according to Eq. (2) described in relation to FIG. 4 . A steering matrix may then be constructed for every frequency band based on TDOA locations for different scans of azimuth and elevation angles corresponding to the respective microphone pair, e.g., according to Eq. (3) described in relation to FIG. 4 .
- a directional covariance matrix may be constructed aligned across all frequency band based on the constructed steering matrix and the computed covariance matrix, e.g., according to Eq. (4) described in relation to FIG. 4 .
- a pseudo-likelihood solution that minimizes a beam power may be determined based on the constructed directional covariance matrix, e.g., according to Eq. (5) described in relation to FIG. 4 .
- a STMV joint pseudo-likelihood may then be computed by taking a product of all determined pseudo-likelihood solutions across all microphone pairs, e.g., according to Eq. (6) described in relation to FIG. 4 .
- a pair of azimuth and elevation angles that maximizes the STMV joint pseudo-likelihood may then be determined, e.g., according to Eq. (7) described in relation to FIG. 4 .
- the determined pair of azimuth and elevation angles may then be transformed to a polar representation that represents the TDOA detection data, e.g., according to Eq. (8) described in relation to FIG. 4 .
- a plurality of audio tracks may be updated and VAD data may be constructed based on the computed TDOA detection data up to a current time-step. For example, the TDOA detection corresponding to a current time-step and a set of existing audio tracks that have been previously established till the current time-step may be identified. Method 600 may then determine whether to incorporate the TDOA detection into one of the existing audio tracks or establishing a new audio track based on a comparison between the TDOA detection and gates of the existing audio tracks, as described in relation to module 430 in FIG. 4 .
- method 600 may assign a first value to the VAD of a respective audio track when the respective audio track is closest to the TDOA detection and assign a second value to VADs of other audio tracks, as described in relation to module 430 in FIG. 4 .
- one or more enhanced multi-source audio signals may be generated using the updated plurality of audio tracks and the constructed VAD data.
- the enhanced multi-source audio signals may then be transmitted to various devices or components.
- the enhanced multi-source audio signals may be packetized and transmitted over a network to another audio output device (e.g., a smart phone, a computer, etc.).
- the enhanced multi-source audio signals may also be transmitted to a voice processing circuitry such as an automated speech recognition component for further processing.
Abstract
Description
- The present disclosure, in accordance with one or more embodiments, relates generally to audio signal processing, and more particularly for example, to systems and methods for multiple source tracking and multi-stream voice activity detections for generic planar microphone arrays.
- Smart speakers and other voice-controlled devices and appliances have gained popularity in recent years. Smart speakers often include an array of microphones for receiving audio inputs (e.g., verbal commands of a user) from an environment. When target audio (e.g., the verbal command) is detected in the audio inputs, the smart speaker may translate the detected target audio into one or more commands and perform different tasks based on the commands.
- One challenge of these smart speakers is to efficiently and effectively isolate the target audio (e.g., the verbal command) from noise or other active speakers in the operating environment. For example, one or more speakers can be active in the presence of one or more noise sources. When the goal is to enhance a specific speaker, that speaker is called the target speaker while the rest of the speakers can be viewed as interference sources. Existing speech enhancement algorithms mostly exploit the spatial information of the sources using multiple input channels (microphones), such as the blind source separation (BSS) methods related to independent component analysis (ICA), and the spatial filtering or beamforming methods.
- The BSS methods, however, were primarily designed for batch processing, which can often be undesirable or even inapplicable in real applications due to the large response delays. Spatial filtering or beamforming methods, on the other hand, often require the supervision under voice activity detection (VAD) as the cost function that is to be minimized, which can be overly dependent on the estimate of a covariance matrix pertaining to the noise/interference-only segments.
- Therefore, there is a need for improved systems and methods for detecting and processing target audio signal(s) in a multi-stream audio environment.
- Aspects of the disclosure and their advantages can be better understood with reference to the following drawings and the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, where showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure.
-
FIG. 1 illustrates an example operating environment for an audio processing device in accordance with one or more embodiments of the disclosure. -
FIG. 2 is a block diagram of an example audio processing device in accordance with one or more embodiments of the disclosure. -
FIG. 3 is a block diagram of example audio signal processor for multi-track audio enhancement in accordance with one or more embodiments of the disclosure. -
FIG. 4 is a block diagram of an example multi-track activity detection engine for processing multiple audio signals from a generic microphone array according to various embodiments of the disclosure. -
FIG. 5A is a diagram illustrating example geometry of a microphone pair according to one or more embodiments of the disclosure. -
FIG. 5B is a diagram illustrating example meshes of time difference of arrival (TDOA) locus information in a multi-dimensional space corresponding to different microphone array geometries according to one or more embodiments of the disclosure. -
FIG. 6 is a logic flow diagram of an example method for enhancing multi-source audio signals by multi-source tracking and activity detection according to various embodiments of the disclosure. -
FIG. 7 is a logic flow diagram of an example process for computing TDOA locus information in the multi-dimensional space using microphone pairs according to various embodiments of the disclosure. - The present disclosure provides improved systems and methods for detecting and processing target audio signal(s) in a multi-stream audio environment.
- Voice activity detection (VAD) may be used to supervise speech enhancement of target audio in a process that utilizes spatial information of the sources derived from multiple input channels. VAD may allow for the spatial statistics of interference/noise sources to be induced during silence periods of the desired speaker so that when the desired speaker becomes active, the effect of noise/interference can subsequently be nulled. For example, the VAD of each source can be inferred to track the spatial information of the sources in the form of Time Difference of Arrival (TDOA) or Direction of Arrival (DOA) by exploiting the history of the detections and constructing the VAD by determining when a detection appears in proximity of an existing track. This process is generally known as Measurement-to-Track (M2T) assignment. In this way, multiple VADs can be inferred for all the sources of interest.
- Specifically, existing DOA methods usually construct a single steering vector for the entire array of microphones based on a closed form mapping of azimuth and elevation angles, which can be used to exploit the special geometry of linear or circular arrays. Such DOA methods cannot be extended to generic or arbitrary geometries of microphone arrays. In addition, these closed-form-mapping based DOA methods often require an extensive search in a multi-dimensional space. For arbitrary geometries, existing TDOA-based methods may be used, which may not restrict to a specific array geometry and may construct multiple steering vectors for each microphone pair to form a multidimensional TDOA vector (one dimension for each pair). However, these existing methods bear the risk of introducing TDOA ghosts formed by the cross intersection of the peaks from the spectrum of each TDOA pair. As a result, further post-processing involving the specific array geometry is often needed to remove TDOA ghosts.
- In view of the need for multi-stream VAD that is not constrained by particular array geometries, embodiments described herein provide a combined multi-source TDOA tracking and VAD mechanism that is applicable for generic array geometries, e.g., a microphone array that lies on a plane. The combined multi-source TDOA tracking and VAD mechanism may reduce the number of calculations that was usually involved in traditional TDOA by performing the TDOA search for each dimension separately.
- In some embodiments, a multi-dimensional TDOA method for generic array geometries that lie on a plane is employed, which avoids unwanted ghost TDOAs. In one embodiment, the Cartesian coordinates of the generically configured microphones are obtained, one of which may be selected as the reference microphone. The azimuth and elevation angles of the microphones may be scanned, based on which a planar locus of physically admissible TDOAs can be formed in the multi-dimensional TDOA space of multiple microphone pairs. In this way, the formed planar locus avoids the formation of ghost TDOAs, thus no further post-processing is needed to remove ghost TDOAs. Also, compared to a full DOA scan method, multi-dimensional TDOA methods disclosed herein reduce computational complexity by conducting the search in the pairwise TDOA domain pertaining to each dimension separately, rather than a search in a full multi-dimensional space.
-
FIG. 1 illustrates anexample operating environment 100 in which an audio processing system may operate according to various embodiments of the disclosure. Theoperating environment 100 includes anaudio processing device 105, atarget audio source 110, and one or more noise sources 135-145. In the example illustrated inFIG. 1 , theoperating environment 100 is illustrated as a room, but it is contemplated that the operating environment may include other areas, such as an inside of a vehicle, an office conference room, rooms of a home, an outdoor stadium or an airport. In accordance with various embodiments of the disclosure, theaudio processing device 105 may include two or more audio sensing components (e.g., microphones) 115 a-115 d and, optionally, one or more audio output components (e.g., speakers) 120 a-120 b. - The
audio processing device 105 may be configured to sense sound via the audio sensing components 115 a-115 d and generate a multi-channel audio input signal, comprising two or more audio input signals. Theaudio processing device 105 may process the audio input signals using audio processing techniques disclosed herein to enhance the audio signal received from thetarget audio source 110. For example, the processed audio signals may be transmitted to other components within theaudio processing device 105, such as a speech recognition engine or voice command processor, or to an external device. Thus, theaudio processing device 105 may be a standalone device that processes audio signals, or a device that turns the processed audio signals into other signals (e.g., a command, an instruction, etc.) for interacting with or controlling an external device. In other embodiments, theaudio processing device 105 may be a communications device, such as mobile phone or voice-over-IP (VoIP) enabled device, and the processed audio signals may be transmitted over a network to another device for output to a remote user. The communications device may also receive processed audio signals from a remote device and output the processed audio signals via the audio output components 120 a-120 b. - The
target audio source 110 may be any source that produces sound detectable by theaudio processing device 105. The target audio to be detected by the system may be defined based on criteria specified by user or system requirements. For example, the target audio may be defined as human speech, a sound made by a particular animal or a machine. In the illustrated example, the target audio is defined as human speech, and thetarget audio source 110 is a person. In addition totarget audio source 110, theoperating environment 100 may include one or more noise sources 135-145. In various embodiments, sound that is not target audio may be processed as noise. In the illustrated example, the noise sources 135-145 may include aloud speaker 135 playing music, atelevision 140 playing a television show, movie or sporting event, and background conversations between non-targetspeakers 145. It will be appreciated that different noise sources may be present in various operating environments. - It is noted that the target audio and noise may reach the audio sensing components 115 a-115 d of the
audio processing device 105 from different directions and at different times. For example, the noise sources 135-145 may produce noise at different locations within the operatingenvironment 100, and the target audio source (person) 110 may speak while moving between locations within the operatingenvironment 100. Furthermore, the target audio and/or the noise may reflect off fixtures (e.g., walls) within theroom 100. For example, consider the paths that the target audio may traverse from thetarget audio source 110 to reach each of the audio sensing components 115 a-115 d. As indicated by arrows 125 a-125 d, the target audio may directly travel from thetarget audio source 110 to the audio sensing components 115 a-115 d, respectively. Additionally, the target audio may reflect off thewalls target audio source 110, as indicated by arrows 130 a-130 b. In various embodiments, theaudio processing device 105 may use one or more audio processing techniques to estimate and apply a room impulse response to further enhance the target audio and suppress noise. -
FIG. 2 illustrates an exampleaudio processing device 200 according to various embodiments of the disclosure. In some embodiments, theaudio processing device 200 may be implemented as theaudio processing device 105 ofFIG. 1 . Theaudio processing device 200 includes anaudio sensor array 205, anaudio signal processor 220 andhost system components 250. - The
audio sensor array 205 comprises two or more sensors, each of which may be implemented as a transducer that converts audio inputs in the form of sound waves into an audio signal. In the illustrated environment, theaudio sensor array 205 comprises a plurality ofmicrophones 205 a-205 n, each generating an audio input signal which is provided to theaudio input circuitry 222 of theaudio signal processor 220. In one embodiment, theaudio sensor array 205 generates a multichannel audio signal, with each channel corresponding to an audio input signal from one of themicrophones 205 a-n. - The
audio signal processor 220 includes theaudio input circuitry 222, adigital signal processor 224 and optionalaudio output circuitry 226. In various embodiments theaudio signal processor 220 may be implemented as an integrated circuit comprising analog circuitry, digital circuitry and thedigital signal processor 224, which is operable to execute program instructions stored in firmware. Theaudio input circuitry 222, for example, may include an interface to theaudio sensor array 205, anti-aliasing filters, analog-to-digital converter circuitry, echo cancellation circuitry, and other audio processing circuitry and components. Thedigital signal processor 224 is operable to process a multichannel digital audio signal to generate an enhanced audio signal, which is output to one or morehost system components 250. In various embodiments, thedigital signal processor 224 may be operable to perform echo cancellation, noise cancellation, target signal enhancement, post-filtering, and other audio signal processing functions. - The optional
audio output circuitry 226 processes audio signals received from thedigital signal processor 224 for output to at least one speaker, such asspeakers audio output circuitry 226 may include a digital-to-analog converter that converts one or more digital audio signals to analog and one or more amplifiers for driving the speakers 210 a-210 b. - The
audio processing device 200 may be implemented as any device operable to receive and enhance target audio data, such as, for example, a mobile phone, smart speaker, tablet, laptop computer, desktop computer, voice controlled appliance, or automobile. Thehost system components 250 may comprise various hardware and software components for operating theaudio processing device 200. In the illustrated embodiment, thehost system components 250 include aprocessor 252, user interface components 254, acommunications interface 256 for communicating with external devices and networks, such as network 280 (e.g., the Internet, the cloud, a local area network, or a cellular network) andmobile device 284, and amemory 258. - The
processor 252 anddigital signal processor 224 may comprise one or more of a processor, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., field programmable gate array (FPGA)), a digital signal processing (DSP) device, or other logic device that may be configured, by hardwiring, executing software instructions, or a combination of both, to perform various operations discussed herein for embodiments of the disclosure. Thehost system components 250 are configured to interface and communicate with theaudio signal processor 220 and the otherhost system components 250, such as through a bus or other electronic communications interface. - It will be appreciated that although the
audio signal processor 220 and thehost system components 250 are shown as incorporating a combination of hardware components, circuitry and software, in some embodiments, at least some or all of the functionalities that the hardware components and circuitries are operable to perform may be implemented as software modules being executed by theprocessor 252 and/ordigital signal processor 224 in response to software instructions and/or configuration data, stored in thememory 258 or firmware of thedigital signal processor 224. - The
memory 258 may be implemented as one or more memory devices operable to store data and information, including audio data and program instructions.Memory 258 may comprise one or more various types of memory devices including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, hard disk drive, and/or other types of memory. - The
processor 252 may be operable to execute software instructions stored in thememory 258. In various embodiments, aspeech recognition engine 260 is operable to process the enhanced audio signal received from theaudio signal processor 220, including identifying and executing voice commands.Voice communications components 262 may be operable to facilitate voice communications with one or more external devices such as amobile device 284 oruser device 286, such as through a voice call over a mobile or cellular telephone network or a VoIP call over an IP network. In various embodiments, voice communications include transmission of the enhanced audio signal to an external communications device. - The user interface components 254 may include a display, a touchpad display, a keypad, one or more buttons and/or other input/output components operable to enable a user to directly interact with the
audio processing device 200. - The
communications interface 256 facilitates communication between theaudio processing device 200 and external devices. For example, thecommunications interface 256 may enable Wi-Fi (e.g., 802.11) or Bluetooth connections between theaudio processing device 200 and one or more local devices, such asmobile device 284, or a wireless router providing network access to aremote server 282, such as through thenetwork 280. In various embodiments, thecommunications interface 256 may include other wired and wireless communications components facilitating direct or indirect communications between theaudio processing device 200 and one or more other devices. -
FIG. 3 illustrates an exampleaudio signal processor 300 according to various embodiments of the disclosure. In some embodiments, theaudio signal processor 300 is embodied as one or more integrated circuits including analog and digital circuitry and firmware logic implemented by a digital signal processor, such asaudio signal processor 220 ofFIG. 2 . As illustrated, theaudio signal processor 300 includesaudio input circuitry 315, asub-band frequency analyzer 320, amulti-track VAD engine 325, anaudio enhancement engine 330, and asynthesizer 335. - The
audio signal processor 300 receives a multi-channel audio input from a plurality of audio sensors, such as asensor array 305 comprising at least twoaudio sensors 305 a-n. Theaudio sensors 305 a-305 n may include microphones that are integrated with an audio processing device, such as theaudio processing device 200 ofFIG. 2 , or external components connected thereto. The arrangement of theaudio sensors 305 a-305 n may be known or unknown to theaudio signal processor 300 according to various embodiments of the disclosure. - The audio signals may be processed initially by the
audio input circuitry 315, which may include anti-aliasing filters, analog to digital converters, and/or other audio input circuitry. In various embodiments, theaudio input circuitry 315 outputs a digital, multichannel, time-domain audio signal, where M is the number of sensor (e.g., microphone) inputs. The multichannel audio signal is input to thesub-band frequency analyzer 320, which partitions the multichannel audio signal into successive frames and decomposes each frame of each channel into a plurality of frequency sub-bands. In various embodiments, thesub-band frequency analyzer 320 includes a Fourier transform process and outputs a plurality of frequency bins. The decomposed audio signals are then provided to themulti-track VAD engine 325 and theaudio enhancement engine 330. - The
multi-track VAD engine 325 is operable to analyze the frames of one or more of the audio tracks and generate VAD outputs indicating whether a target audio activity is present in the current frame. As discussed above, target audio may be any audio to be identified by the audio system. When the target audio is human speech, themulti-track VAD engine 325 may be implemented specifically for detecting voice activities. In various embodiments, themulti-track VAD engine 325 is operable to receive a frame of audio data and generate a VAD indication output for each audio track regarding the presence or absence of the target audio on the respective audio track corresponding to the frame of audio data. Detailed components and operations of themulti-track VAD engine 325 is further illustrated in relation to 400 inFIG. 4 . - The
audio enhancement engine 330 receives the sub-band frames from thesub-band frequency analyzer 320 and the VAD indications from themulti-track VAD engine 325. In accordance with various embodiments of the disclosure, theaudio enhancement engine 330 is configured to process the sub-band frames based on the received multi-track VAD indications to enhance the multi-track audio signals. For example, theaudio enhancement engine 330 may enhance the portion of the audio signal determined to be from the direction of the target audio sources and suppress the other portions of the audio signal which are determined to be noise. - After enhancing the target audio signal, the
audio enhancement engine 330 may pass the processed audio signal to thesynthesizer 335. In various embodiments, thesynthesizer 335 reconstructs one or more of the multichannel audio signals on a frame-by-frame basis by combining the sub-bands to form an enhanced time-domain audio signal. The enhanced audio signal may then be transformed back to the time domain and sent to a system component or external device for further processing. -
FIG. 4 illustrates an examplemulti-track VAD engine 400 for processing multiple audio signals from a generic microphone array according to various embodiments of the disclosure. Themulti-track VAD engine 400 may be implemented as a combination of digital circuitry and logic performed by a digital signal processor. In some embodiments, themulti-track VAD engine 400 may be installed in an audio signal processor such as 300 inFIG. 3 . Themulti-track VAD engine 400 may provide further structural and functional details to themulti-track VAD engine 325 inFIG. 3 . - According to various embodiments of the disclosure, the
multi-track VAD engine 400 includes asubband analysis module 405, a block-basedTDOA estimation module 410, a TDOAlocus calculation module 420 and a multi-source tracking and multi-streamVAD estimation module 430. - The
subband analysis module 405 receives multipleaudio signals 402, denoted by xm(t), m=1, . . . , M, the sampled time domain audio signal recorded at the mth microphone for a total of M microphones (e.g., similar toaudio sensors 305 a-n inFIG. 3 ). The audio signals xm(t), m=1, . . . , M, may be received via theaudio input circuitry 315 inFIG. 3 . - The
subband analysis module 405 is configured to obtain and transform theaudio signals 402 to a time-frequency domain representation 404, denoted as Xm(l, k) corresponding to the original time-domain audio signal xm(t), where l indicates the subband time index and k indicates the frequency band index. For example, thesubband analysis module 405 may be similar to thesub-band frequency analyzer 320 inFIG. 3 , which performs Fourier transform to convert input time-domain audio signals to frequency domain representations. Thesubband analysis module 405 may then send the generated time-frequency domain representations 404 to the block-basedTDOA estimation module 410 and the multi-source tracking and multi-streamVAD estimation module 430. - The TDOA
locus calculation module 420 is configured to scan a generic microphone array (e.g.,audio sensors 305 a-n that form a generic array geometry). For example, for a given arbitrary microphone array geometry on a plane, a locus of permissible TDOA locations is computed once at system boot-up. This locus of points can avoid the ghost formations. - For an array of M microphones, the first microphone may be selected as the reference microphone, which in turn gives M−1 microphone pairs all with respect to the first microphone. For example,
FIG. 5A illustrates an example microphone pair. The microphone pair, indexed as the i−1th pair, includes microphone i 502 andreference microphone 1 501 for anincident ray 505 with azimuth angle θ and elevation angle of zero emitted from a distant source (assuming a far-field model). The distance between the microphone pair of 501 and 502 as well as the angle between the two microphones are denoted as di−1 and ψi−1, respectively, which can be computed given the Cartesian coordinates of the ith microphone 502. For the general case when theincident ray 505 is angled at azimuth θ and elevation ϕ, the TDOA of the (i−1)th microphone pair can be computed as -
- where c is the propagation velocity.
- After scanning different elevation and azimuth angles, the TDOA
locus calculation module 420 may construct a mesh of permissible TDOAs. When all the M microphones lie on a plane, the resulting TDOA locus (τ1(θ, ϕ), . . . τp (θ, ϕ), . . . , τM−1(θ, ϕ) for all scans of θ and ϕ) also lies on a plane in the (M−1) dimensional space. Different layouts of the M microphones may result in different planes in the (M−1) dimensional space. - For example,
FIG. 5B illustrates two different example microphone layouts along with their respective TDOA meshes. A set of M=4 microphones are shown at 510, where the distance between the first and the third microphone is 8 cm and the resulting mesh of permissible TDOA is in the M−1=3-dimensional space as shown at 515. When the distance between the first and the third microphone increases to 16 cm as shown at 520, the resulting mesh of permissible TDOA is shown at 525. - Referring back to
FIG. 4 , the TDOAlocus calculation module 420 may then send the (M−1)-dimensional TDOAs 403 to the block-basedTDOA estimation module 410. The block-basedTDOA estimation module 410 receives the time-frequency domain representations 404 of the multi-source audios and theTDOAs 403, based on whichTDOA estimation module 410 extracts the TDOA information of the source microphones (e.g.,audio sensors 305 a-n shown inFIG. 3 ) using the data obtained from consecutive frames. - In one embodiment, the block-based
TDOA estimation module 410 employs a steered minimum variance (STMV) beamformer to obtain TDOA information from the time-frequency domain representations 404 of the multi-source audios. Specifically, the block-basedTDOA estimation module 410 may select a microphone as a reference microphone, and then designate a total of M−1 microphone pairs by pairing the remaining M−1 microphones with the reference microphone. The microphone pairs are indexed by p=1, . . . , M−1. - For example, the first microphone may be selected as the reference microphone, and accordingly, X1(l, k) denotes the time-frequency representation of the audio from the reference microphone. For the pth pair of microphones, the block-based
TDOA estimation module 410 computes a frequency representation of the pth pair in the matrix form as Xp(k, l)=[X1(l, k) Xp+1(l, k)]T where ( )T represents the transpose. The block-basedTDOA estimation module 410 then computes the covariance matrix of the pth input signal pairs for every frequency band k: -
R p(k)=Σl X p(k,l)X p(k,l)H (2) - where ( )H represents the Hermitian transpose.
- In some implementations, the summation in computing Rp (k) is implemented over a block of certain number of consecutive frames. The index of the block is neglected here for brevity.
- The block-based
TDOA estimation module 410 may then construct a steering matrix for each pair and frequency band as follows: -
T k(τp)=diag(1e −j2πfk τp ]T) (3) - where τp is the TDOA for the pth pair obtained from the TDOA
locus calculation module 420 after different scans of θ and ϕ (neglected for brevity); fk is the frequency at band k; and diag([a, b]) denotes a 2×2 diagonal matrix with diagonal elements a and b. - For each microphone pair p, the block-based
TDOA estimation module 410 constructs the directional covariance matrix that is coherently aligned across all frequency bands by: -
C p(τp)=Σk T k(τp)R p(k)T k(τp)H (4) - The computation of the directional covariance matrix Cp (τp) is repeated over all microphone pairs p and all scans of azimuth/elevation (θ, ϕ) for τp. In order to reduce computations over all scans, the TDOA space of each dimension p corresponding to the pth microphone pair is linearly quantized to q segments. At the beginning of the processing (at system boot-up), the TDOA locus points (τ1, τp, . . . , τM−1) obtained from scanning each azimuth and elevation angles (θ, ϕ) is mapped to the closest quantized points for each dimension. For each azimuth/elevation (θ, ϕ), the mapping of (θ, ϕ)→(ind1(θ, ϕ), . . . , indM−1(θ, ϕ)) is saved in memory, where 1≤indp(θ, ϕ)≤q is the quantized TDOA index of dimension p pertaining to scanning angles θ and ϕ.
- For example, if there exists M=4 microphones and the azimuth and elevation scans are θ=0°:5:355°, Φ=0°:10:80°, respectively. The number of distinct computations of Cp (τp) that need to be performed is length (θ)×length (Φ)×(M−1)=72×9×3=1944. When TDOA locus points (τ1, . . . τp, . . . , τM−1) are quantized, not all calculations need to be carried out as some of the TDOA dimensions may be quantized to the same segment among the q quantized segments. Therefore, if for example q=50, the maximum number of distinct computations needed to compute Cp(τp) is reduced to q×(M−1)×50×3=150. The pseudo-code to carry out the computation of Cp(τp) with TDOA quantization may be demonstrated in
Algorithm 1 as follows: -
Algorithm 1 Calculating Cp(τp) using TDOA quantizationsRavg = zeros(2,2,length(Θ),length(Φ),M − 1) for k > 0 do V = zeros(2,2,q,M − 1) for p = 1 : M − 1 do for i = 1 : length(Θ) do for j = 1 : length(Φ) do if V (:,:,indp(Θ(i),Φ(j)),p) = 0 then Tk = diag([1 e−j2πf k τp (Θ(i),Φ(j))]T)V(:,:,indp(Θ(i),Φ(j)),p) = Tk Rp (k) Tk H end if Ravg(:,:,i,j,p) = Ravg(:,:,i,j,p) + V (:,:,indp(Θ(i),Φ(j)),p) end for end for end for end for - Next, for each pair p, the direction that minimizes beam power subject to a distortionless criteria with its equivalent pseudo-likelihood solution is computed as:
-
- where 1=[1 1]T. The block-based TDOA estimation module 410 may then compute the STMV joint pseudo-likelihood for all M−1 pairs of microphones as:
- The azimuth and elevation that yields the maximum STMV joint pseudo-likelihood of all M−1 pairs is then identified, denoted by
- The azimuth and elevation pair θ*, ϕ* may then be used for multi-source tracking and multi-stream VAD estimation. One possible solution may include tracking the angles between the two microphones of each microphone pair directly. However, due to the wrap-around effect of azimuth in 360 degrees, if tracking is done directly on the angles between paired microphones, track loss may occur when a microphone source crosses 0° towards 360° and vice versa. Therefore, in order to avoid such confusion, detection z is computed based on the angles between paired microphones in a circular manner using a polar transformation as follows:
-
z=α scale[cos θ*cos ϕ*sin θ*cos ϕ*]T (8) - where αscale>1 is a scaling constant that may expand the measurement space allowing the tracking to be done with parameters pertaining to meaningful concepts like angles.
- The block-based
TDOA estimation module 410 then sends the computed detection z to the multi-source tracking and multi-streamVAD estimation module 430. If an maximum number of tracks exist, the TDOAs obtained from the block-basedTDOA estimation module 410 are to be tracked by updating the track obtained from a previous step in a recursive manner. Specifically, if the detection obtained at block (time-step) n−1 is denoted by zn−1 and up until then there existed tn−1 tracks, for anew detection z n 406 emerging at time-step n, the multi-source tracking and multi-streamVAD estimation module 430 processes the new detection zn based on the gate of existing tracks as follows: - If zn falls into the gate of only one of the previous tn−1 tracks, the particular track is updated to incorporate detection zn.
- If zn falls into the overlapping gates of multiple previous tn−1 tracks, then the track that is closest to detection zn is updated to incorporate detection zn.
-
-
- For all the other tracks that were not updated, initiated or replaced (as in previous steps), then these tracks are updated with the same mean value, but the variance of each respective track is increased to account for uncertainty, e.g., based on a random walk model. The power of each respective track is also decayed so that future emerging source has chance to be initiated. In this way, the tracking
results 408 incorporating the up-to-date detections 406 at time-step n can be output from themodule 430, denoted by zn Tr(1), . . . zn Tr(). - When all the audio tracks have been updated, the
module 430 generatesmulti-stream VADs 412 using the nearest neighbor M2T assignment. Specifically, at time-step n, the M2T assignment may be carried out by assigning a 1 to the track that is closest to detection zn, and 0 to the other tracks. In some implementations, a hangover may be applied for the VADs to have an intermediate value, e.g., −1, etc., before they are assigned completely to zero after being 1 in the previous time-step. In this way, themulti-stream VADs 412, denoted by VAD1, . . . , , each representing whether any voice activity detection is found in the respective track, are output from themodule 430 for audio enhancement, e.g., to theaudio enhancement engine 330 inFIG. 3 . -
FIG. 6 illustrates anexample method 600 for enhancing multi-source audio signals by multi-source tracking and VAD according to various embodiments of the disclosure. In some embodiments, themethod 600 may be performed by one or more components in theaudio signal processor 300, and/or one or more components of themulti-track VAD engine 400. -
Method 600 starts with step 602, at which the TDOA locus information may be computed based on spatial information of a microphone array. For example, the TDOA locus information may be computed once at system boot-up by scanning the microphone array with incident rays of varying azimuth and elevation angles. The computation may be performed with reduced complexity in a multi-dimensional space constructed by pairing the microphones from the microphone array, as further described in relation toFIG. 7 . - Referring to
FIG. 7 which provides further detailed steps for step 602, at step 702, a first microphone may be selected from the microphone array as a reference microphone. Atstep 704, each remaining microphone from the microphone array may be paired with the reference microphone. Atstep 706, for each microphone pair, a TDOA location corresponding to a specific azimuth angle and a specific elevation angle of an incident ray may be computed based on a distance and an angle between two microphones in the respective pair, e.g., according to Eq. (1) described in relation toFIG. 4 . An example microphone pair with a specific azimuth angle and a specific elevation angle of an incident ray is also shown inFIG. 5A . - At
step 708, if there are more microphone pairs to be processed, the method retrieves the next microphone pair atstep 710 and repeats atstep 706 until TDOA locations for all microphone pairs have been computed. - At
step 712, if there is more scan of the azimuth and elevation angles, the method retrieves the next scan of azimuth and elevation angles at step 714 and repeats atstep 706, until TDOA locations for all scans of azimuth and elevation angles are computed. - At
step 712, when no more scan of azimuth and elevation angles is to be processed, e.g., the TDOA locations have been computed for all microphone pairs over all scans of azimuth and elevation angles, a mesh of TDOA location points can be formed atstep 716. Example meshes of TDOA location points corresponding to different geometries of the microphone arrays are shown inFIG. 5B . - Referring back to
FIG. 6 , upon computing the TDOA locus information at system boot-up,method 600 moves on to step 604. Atstep 604, one or more multi-source audio signals may be received from the microphone array. For example, time-domain samples of the multi-source audio signals 402 inFIG. 4 may be received via theaudio input circuitry 315 inFIG. 3 . - At
step 606, the one or more multi-source audio signals may be transformed from time-domain to time-frequency representations. For example, thesubband analysis module 405 may transform the time-domain signals to time-frequency domain representations, as described in relation toFIG. 4 . - At step 608, TDOA detection data may be computed for the one or more multi-source audio signals according to a STMV beamformer based on the computed TDOA locus. For example, for each microphone pair, a covariance matrix may be computed using time-frequency representations of the one or more multi-source audio signals from the respective microphone pair for every frequency band, e.g., according to Eq. (2) described in relation to
FIG. 4 . A steering matrix may then be constructed for every frequency band based on TDOA locations for different scans of azimuth and elevation angles corresponding to the respective microphone pair, e.g., according to Eq. (3) described in relation toFIG. 4 . A directional covariance matrix may be constructed aligned across all frequency band based on the constructed steering matrix and the computed covariance matrix, e.g., according to Eq. (4) described in relation toFIG. 4 . A pseudo-likelihood solution that minimizes a beam power may be determined based on the constructed directional covariance matrix, e.g., according to Eq. (5) described in relation toFIG. 4 . A STMV joint pseudo-likelihood may then be computed by taking a product of all determined pseudo-likelihood solutions across all microphone pairs, e.g., according to Eq. (6) described in relation toFIG. 4 . A pair of azimuth and elevation angles that maximizes the STMV joint pseudo-likelihood may then be determined, e.g., according to Eq. (7) described in relation toFIG. 4 . The determined pair of azimuth and elevation angles may then be transformed to a polar representation that represents the TDOA detection data, e.g., according to Eq. (8) described in relation toFIG. 4 . - At
step 610, a plurality of audio tracks may be updated and VAD data may be constructed based on the computed TDOA detection data up to a current time-step. For example, the TDOA detection corresponding to a current time-step and a set of existing audio tracks that have been previously established till the current time-step may be identified.Method 600 may then determine whether to incorporate the TDOA detection into one of the existing audio tracks or establishing a new audio track based on a comparison between the TDOA detection and gates of the existing audio tracks, as described in relation tomodule 430 inFIG. 4 . For another example,method 600 may assign a first value to the VAD of a respective audio track when the respective audio track is closest to the TDOA detection and assign a second value to VADs of other audio tracks, as described in relation tomodule 430 inFIG. 4 . - At step 612, one or more enhanced multi-source audio signals may be generated using the updated plurality of audio tracks and the constructed VAD data. For example, the enhanced multi-source audio signals may then be transmitted to various devices or components. For another example, the enhanced multi-source audio signals may be packetized and transmitted over a network to another audio output device (e.g., a smart phone, a computer, etc.). The enhanced multi-source audio signals may also be transmitted to a voice processing circuitry such as an automated speech recognition component for further processing.
- The foregoing disclosure is not intended to limit the present invention to the precise forms or particular fields of use disclosed. As such, it is contemplated that various alternate embodiments and/or modifications to the present disclosure, whether explicitly described or implied herein, are possible in light of the disclosure. For example, embodiments described herein may be used to provide the position of multiple sound sources in the environment in order to supervise human-machine interaction tasks, e.g., in applications incorporating additional information from other modalities such as video streams, 3D cameras, Lidars, etc. Having thus described embodiments of the present disclosure, persons of ordinary skill in the art will recognize advantages over conventional approaches and that changes may be made in form and detail without departing from the scope of the present disclosure. Thus, the present disclosure is limited only by the claims.
Claims (20)
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/740,297 US11064294B1 (en) | 2020-01-10 | 2020-01-10 | Multiple-source tracking and voice activity detections for planar microphone arrays |
KR1020200166979A KR20210091034A (en) | 2020-01-10 | 2020-12-02 | Multiple-source tracking and voice activity detections for planar microphone arrays |
JP2020212089A JP2021110938A (en) | 2020-01-10 | 2020-12-22 | Multiple sound source tracking and speech section detection for planar microphone array |
CN202110023469.7A CN113113034A (en) | 2020-01-10 | 2021-01-08 | Multi-source tracking and voice activity detection for planar microphone arrays |
US17/349,589 US11937054B2 (en) | 2020-01-10 | 2021-06-16 | Multiple-source tracking and voice activity detections for planar microphone arrays |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/740,297 US11064294B1 (en) | 2020-01-10 | 2020-01-10 | Multiple-source tracking and voice activity detections for planar microphone arrays |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/349,589 Continuation US11937054B2 (en) | 2020-01-10 | 2021-06-16 | Multiple-source tracking and voice activity detections for planar microphone arrays |
Publications (2)
Publication Number | Publication Date |
---|---|
US11064294B1 US11064294B1 (en) | 2021-07-13 |
US20210219053A1 true US20210219053A1 (en) | 2021-07-15 |
Family
ID=76709199
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/740,297 Active US11064294B1 (en) | 2020-01-10 | 2020-01-10 | Multiple-source tracking and voice activity detections for planar microphone arrays |
US17/349,589 Active 2040-05-20 US11937054B2 (en) | 2020-01-10 | 2021-06-16 | Multiple-source tracking and voice activity detections for planar microphone arrays |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/349,589 Active 2040-05-20 US11937054B2 (en) | 2020-01-10 | 2021-06-16 | Multiple-source tracking and voice activity detections for planar microphone arrays |
Country Status (4)
Country | Link |
---|---|
US (2) | US11064294B1 (en) |
JP (1) | JP2021110938A (en) |
KR (1) | KR20210091034A (en) |
CN (1) | CN113113034A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024065256A1 (en) * | 2022-09-28 | 2024-04-04 | Citrix Systems, Inc. | Positional and echo audio enhancement |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7407580B2 (en) | 2018-12-06 | 2024-01-04 | シナプティクス インコーポレイテッド | system and method |
US11197232B2 (en) * | 2019-04-01 | 2021-12-07 | Ofinno, Llc | Location reporting handling |
US11064294B1 (en) | 2020-01-10 | 2021-07-13 | Synaptics Incorporated | Multiple-source tracking and voice activity detections for planar microphone arrays |
CN111586547B (en) * | 2020-04-28 | 2022-05-06 | 北京小米松果电子有限公司 | Detection method and device of audio input module and storage medium |
US11830471B1 (en) * | 2020-08-31 | 2023-11-28 | Amazon Technologies, Inc. | Surface augmented ray-based acoustic modeling |
CN113782047B (en) * | 2021-09-06 | 2024-03-08 | 云知声智能科技股份有限公司 | Voice separation method, device, equipment and storage medium |
US11823707B2 (en) | 2022-01-10 | 2023-11-21 | Synaptics Incorporated | Sensitivity mode for an audio spotting system |
DE202022101069U1 (en) | 2022-02-24 | 2022-03-23 | Pankaj Agarwal | Intelligent sound detection system based on artificial intelligence processing of multiple sounds |
US20220236360A1 (en) * | 2022-04-19 | 2022-07-28 | Intel Corporation | Method and system of detection of acoustic source aiming direction |
CN115113139B (en) * | 2022-05-12 | 2024-02-02 | 苏州清听声学科技有限公司 | Sound source identification method and device based on microphone array and electronic equipment |
CN115295000B (en) * | 2022-10-08 | 2023-01-03 | 深圳通联金融网络科技服务有限公司 | Method, device and equipment for improving speech recognition accuracy under multi-object speaking scene |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160057549A1 (en) * | 2013-04-09 | 2016-02-25 | Sonova Ag | Method and system for providing hearing assistance to a user |
US20190385635A1 (en) * | 2018-06-13 | 2019-12-19 | Ceva D.S.P. Ltd. | System and method for voice activity detection |
Family Cites Families (94)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3484112B2 (en) | 1999-09-27 | 2004-01-06 | 株式会社東芝 | Noise component suppression processing apparatus and noise component suppression processing method |
US6370500B1 (en) | 1999-09-30 | 2002-04-09 | Motorola, Inc. | Method and apparatus for non-speech activity reduction of a low bit rate digital voice message |
GB2379148A (en) | 2001-08-21 | 2003-02-26 | Mitel Knowledge Corp | Voice activity detection |
US7088831B2 (en) | 2001-12-06 | 2006-08-08 | Siemens Corporate Research, Inc. | Real-time audio source separation by delay and attenuation compensation in the time domain |
AUPS270902A0 (en) | 2002-05-31 | 2002-06-20 | Canon Kabushiki Kaisha | Robust detection and classification of objects in audio using limited training data |
US7146315B2 (en) | 2002-08-30 | 2006-12-05 | Siemens Corporate Research, Inc. | Multichannel voice detection in adverse environments |
CN1303582C (en) | 2003-09-09 | 2007-03-07 | 摩托罗拉公司 | Automatic speech sound classifying method |
KR100754385B1 (en) | 2004-09-30 | 2007-08-31 | 삼성전자주식회사 | Apparatus and method for object localization, tracking, and separation using audio and video sensors |
US7464029B2 (en) | 2005-07-22 | 2008-12-09 | Qualcomm Incorporated | Robust separation of speech signals in a noisy environment |
JP2007047427A (en) | 2005-08-10 | 2007-02-22 | Hitachi Ltd | Sound processor |
KR100821177B1 (en) | 2006-09-29 | 2008-04-14 | 한국전자통신연구원 | Statistical model based a priori SAP estimation method |
KR100964402B1 (en) | 2006-12-14 | 2010-06-17 | 삼성전자주식회사 | Method and Apparatus for determining encoding mode of audio signal, and method and appartus for encoding/decoding audio signal using it |
US8098842B2 (en) | 2007-03-29 | 2012-01-17 | Microsoft Corp. | Enhanced beamforming for arrays of directional microphones |
US8005237B2 (en) | 2007-05-17 | 2011-08-23 | Microsoft Corp. | Sensor array beamformer post-processor |
DE602008002695D1 (en) | 2008-01-17 | 2010-11-04 | Harman Becker Automotive Sys | Postfilter for a beamformer in speech processing |
US9113240B2 (en) | 2008-03-18 | 2015-08-18 | Qualcomm Incorporated | Speech enhancement using multiple microphones on multiple devices |
KR20100006492A (en) | 2008-07-09 | 2010-01-19 | 삼성전자주식회사 | Method and apparatus for deciding encoding mode |
EP2146519B1 (en) | 2008-07-16 | 2012-06-06 | Nuance Communications, Inc. | Beamforming pre-processing for speaker localization |
JP2010085733A (en) | 2008-09-30 | 2010-04-15 | Equos Research Co Ltd | Speech enhancement system |
US9202456B2 (en) | 2009-04-23 | 2015-12-01 | Qualcomm Incorporated | Systems, methods, apparatus, and computer-readable media for automatic control of active noise cancellation |
US20110010172A1 (en) | 2009-07-10 | 2011-01-13 | Alon Konchitsky | Noise reduction system using a sensor based speech detector |
JP5411807B2 (en) | 2010-05-25 | 2014-02-12 | 日本電信電話株式会社 | Channel integration method, channel integration apparatus, and program |
US9037458B2 (en) | 2011-02-23 | 2015-05-19 | Qualcomm Incorporated | Systems, methods, apparatus, and computer-readable media for spatially selective audio augmentation |
CN102956230B (en) | 2011-08-19 | 2017-03-01 | 杜比实验室特许公司 | The method and apparatus that song detection is carried out to audio signal |
CN103999150B (en) | 2011-12-12 | 2016-10-19 | 杜比实验室特许公司 | Low complex degree duplicate detection in media data |
CN103325386B (en) | 2012-03-23 | 2016-12-21 | 杜比实验室特许公司 | The method and system controlled for signal transmission |
KR101318328B1 (en) | 2012-04-12 | 2013-10-15 | 경북대학교 산학협력단 | Speech enhancement method based on blind signal cancellation and device using the method |
US9768829B2 (en) | 2012-05-11 | 2017-09-19 | Intel Deutschland Gmbh | Methods for processing audio signals and circuit arrangements therefor |
TWI474317B (en) | 2012-07-06 | 2015-02-21 | Realtek Semiconductor Corp | Signal processing apparatus and signal processing method |
US10142007B2 (en) | 2012-07-19 | 2018-11-27 | Intel Deutschland Gmbh | Radio communication devices and methods for controlling a radio communication device |
DK2701145T3 (en) | 2012-08-24 | 2017-01-16 | Retune DSP ApS | Noise cancellation for use with noise reduction and echo cancellation in personal communication |
EP2747451A1 (en) | 2012-12-21 | 2014-06-25 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Filter and method for informed spatial filtering using multiple instantaneous direction-of-arrivial estimates |
US9158760B2 (en) | 2012-12-21 | 2015-10-13 | The Nielsen Company (Us), Llc | Audio decoding with supplemental semantic audio recognition and report generation |
US9183849B2 (en) | 2012-12-21 | 2015-11-10 | The Nielsen Company (Us), Llc | Audio matching with semantic audio recognition and report generation |
CN113470640B (en) | 2013-02-07 | 2022-04-26 | 苹果公司 | Voice trigger of digital assistant |
CN104078050A (en) | 2013-03-26 | 2014-10-01 | 杜比实验室特许公司 | Device and method for audio classification and audio processing |
US9043211B2 (en) | 2013-05-09 | 2015-05-26 | Dsp Group Ltd. | Low power activation of a voice activated device |
CN104217729A (en) | 2013-05-31 | 2014-12-17 | 杜比实验室特许公司 | Audio processing method, audio processing device and training method |
US9747899B2 (en) | 2013-06-27 | 2017-08-29 | Amazon Technologies, Inc. | Detecting self-generated wake expressions |
US9601130B2 (en) | 2013-07-18 | 2017-03-21 | Mitsubishi Electric Research Laboratories, Inc. | Method for processing speech signals using an ensemble of speech enhancement procedures |
US9240182B2 (en) | 2013-09-17 | 2016-01-19 | Qualcomm Incorporated | Method and apparatus for adjusting detection threshold for activating voice assistant function |
GB2518663A (en) | 2013-09-27 | 2015-04-01 | Nokia Corp | Audio analysis apparatus |
US9502028B2 (en) | 2013-10-18 | 2016-11-22 | Knowles Electronics, Llc | Acoustic activity detection apparatus and method |
US9654894B2 (en) | 2013-10-31 | 2017-05-16 | Conexant Systems, Inc. | Selective audio source enhancement |
EP2884489B1 (en) | 2013-12-16 | 2020-02-05 | Harman Becker Automotive Systems GmbH | Sound system including an engine sound synthesizer |
US9589560B1 (en) | 2013-12-19 | 2017-03-07 | Amazon Technologies, Inc. | Estimating false rejection rate in a detection system |
DK2916321T3 (en) | 2014-03-07 | 2018-01-15 | Oticon As | Processing a noisy audio signal to estimate target and noise spectral variations |
US9548065B2 (en) | 2014-05-05 | 2017-01-17 | Sensory, Incorporated | Energy post qualification for phrase spotting |
US9484022B2 (en) | 2014-05-23 | 2016-11-01 | Google Inc. | Training multiple neural networks with different accuracy |
US9369113B2 (en) | 2014-06-20 | 2016-06-14 | Steve Yang | Impedance adjusting device |
WO2016007528A1 (en) | 2014-07-10 | 2016-01-14 | Analog Devices Global | Low-complexity voice activity detection |
US9432769B1 (en) | 2014-07-30 | 2016-08-30 | Amazon Technologies, Inc. | Method and system for beam selection in microphone array beamformers |
US9953661B2 (en) | 2014-09-26 | 2018-04-24 | Cirrus Logic Inc. | Neural network voice activity detection employing running range normalization |
US9530400B2 (en) | 2014-09-29 | 2016-12-27 | Nuance Communications, Inc. | System and method for compressed domain language identification |
US10049678B2 (en) | 2014-10-06 | 2018-08-14 | Synaptics Incorporated | System and method for suppressing transient noise in a multichannel system |
JP6450139B2 (en) | 2014-10-10 | 2019-01-09 | 株式会社Nttドコモ | Speech recognition apparatus, speech recognition method, and speech recognition program |
US20160275961A1 (en) | 2015-03-18 | 2016-09-22 | Qualcomm Technologies International, Ltd. | Structure for multi-microphone speech enhancement system |
US9734822B1 (en) | 2015-06-01 | 2017-08-15 | Amazon Technologies, Inc. | Feedback based beamformed signal selection |
US10229700B2 (en) | 2015-09-24 | 2019-03-12 | Google Llc | Voice activity detection |
US9668073B2 (en) | 2015-10-07 | 2017-05-30 | Robert Bosch Gmbh | System and method for audio scene understanding of physical object sound sources |
US11631421B2 (en) | 2015-10-18 | 2023-04-18 | Solos Technology Limited | Apparatuses and methods for enhanced speech recognition in variable environments |
US10347271B2 (en) | 2015-12-04 | 2019-07-09 | Synaptics Incorporated | Semi-supervised system for multichannel source enhancement through configurable unsupervised adaptive transformations and supervised deep neural network |
US9978397B2 (en) | 2015-12-22 | 2018-05-22 | Intel Corporation | Wearer voice activity detection |
US10090005B2 (en) | 2016-03-10 | 2018-10-02 | Aspinity, Inc. | Analog voice activity detection |
EP3381033B1 (en) | 2016-03-23 | 2020-08-12 | Google LLC | Adaptive audio enhancement for multichannel speech recognition |
US9947323B2 (en) | 2016-04-01 | 2018-04-17 | Intel Corporation | Synthetic oversampling to enhance speaker identification or verification |
KR102422848B1 (en) | 2016-06-01 | 2022-07-20 | 메사추세츠 인스티튜트 오브 테크놀로지 | Low-power automatic speech recognition device |
US20180039478A1 (en) | 2016-08-02 | 2018-02-08 | Google Inc. | Voice interaction services |
WO2018061371A1 (en) | 2016-09-30 | 2018-04-05 | ソニー株式会社 | Signal processing device, signal processing method, and program |
US9741360B1 (en) | 2016-10-09 | 2017-08-22 | Spectimbre Inc. | Speech enhancement for target speakers |
US9881634B1 (en) | 2016-12-01 | 2018-01-30 | Arm Limited | Multi-microphone speech processing system |
US10755718B2 (en) | 2016-12-07 | 2020-08-25 | Interactive Intelligence Group, Inc. | System and method for neural network based speaker classification |
US10546575B2 (en) | 2016-12-14 | 2020-01-28 | International Business Machines Corporation | Using recurrent neural network for partitioning of audio data into segments that each correspond to a speech feature cluster identifier |
US10083689B2 (en) | 2016-12-23 | 2018-09-25 | Intel Corporation | Linear scoring for low power wake on voice |
US10170134B2 (en) | 2017-02-21 | 2019-01-01 | Intel IP Corporation | Method and system of acoustic dereverberation factoring the actual non-ideal acoustic environment |
JP6652519B2 (en) | 2017-02-28 | 2020-02-26 | 日本電信電話株式会社 | Steering vector estimation device, steering vector estimation method, and steering vector estimation program |
US10224053B2 (en) | 2017-03-24 | 2019-03-05 | Hyundai Motor Company | Audio signal quality enhancement based on quantitative SNR analysis and adaptive Wiener filtering |
US10269369B2 (en) | 2017-05-31 | 2019-04-23 | Apple Inc. | System and method of noise reduction for a mobile device |
US10403299B2 (en) | 2017-06-02 | 2019-09-03 | Apple Inc. | Multi-channel speech signal enhancement for robust voice trigger detection and automatic speech recognition |
US10096328B1 (en) | 2017-10-06 | 2018-10-09 | Intel Corporation | Beamformer system for tracking of speech and noise in a dynamic environment |
US10090000B1 (en) | 2017-11-01 | 2018-10-02 | GM Global Technology Operations LLC | Efficient echo cancellation using transfer function estimation |
US10504539B2 (en) | 2017-12-05 | 2019-12-10 | Synaptics Incorporated | Voice activity detection systems and methods |
US10777189B1 (en) | 2017-12-05 | 2020-09-15 | Amazon Technologies, Inc. | Dynamic wakeword detection |
US10679617B2 (en) | 2017-12-06 | 2020-06-09 | Synaptics Incorporated | Voice enhancement in audio signals through modified generalized eigenvalue beamformer |
CN111465981A (en) | 2017-12-21 | 2020-07-28 | 辛纳普蒂克斯公司 | Analog voice activity detector system and method |
US10957338B2 (en) * | 2018-05-16 | 2021-03-23 | Synaptics Incorporated | 360-degree multi-source location detection, tracking and enhancement |
JP7407580B2 (en) | 2018-12-06 | 2024-01-04 | シナプティクス インコーポレイテッド | system and method |
US11232788B2 (en) | 2018-12-10 | 2022-01-25 | Amazon Technologies, Inc. | Wakeword detection |
KR102174598B1 (en) * | 2019-01-14 | 2020-11-05 | 한국과학기술원 | System and method for localization for non-line of sight sound source using diffraction aware |
US11069353B1 (en) | 2019-05-06 | 2021-07-20 | Amazon Technologies, Inc. | Multilingual wakeword detection |
US11445294B2 (en) * | 2019-05-23 | 2022-09-13 | Shure Acquisition Holdings, Inc. | Steerable speaker array, system, and method for the same |
US11064294B1 (en) | 2020-01-10 | 2021-07-13 | Synaptics Incorporated | Multiple-source tracking and voice activity detections for planar microphone arrays |
US11308959B2 (en) | 2020-02-11 | 2022-04-19 | Spotify Ab | Dynamic adjustment of wake word acceptance tolerance thresholds in voice-controlled devices |
US11769520B2 (en) | 2020-08-17 | 2023-09-26 | EMC IP Holding Company LLC | Communication issue detection using evaluation of multiple machine learning models |
-
2020
- 2020-01-10 US US16/740,297 patent/US11064294B1/en active Active
- 2020-12-02 KR KR1020200166979A patent/KR20210091034A/en unknown
- 2020-12-22 JP JP2020212089A patent/JP2021110938A/en active Pending
-
2021
- 2021-01-08 CN CN202110023469.7A patent/CN113113034A/en active Pending
- 2021-06-16 US US17/349,589 patent/US11937054B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160057549A1 (en) * | 2013-04-09 | 2016-02-25 | Sonova Ag | Method and system for providing hearing assistance to a user |
US20190385635A1 (en) * | 2018-06-13 | 2019-12-19 | Ceva D.S.P. Ltd. | System and method for voice activity detection |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024065256A1 (en) * | 2022-09-28 | 2024-04-04 | Citrix Systems, Inc. | Positional and echo audio enhancement |
Also Published As
Publication number | Publication date |
---|---|
US11064294B1 (en) | 2021-07-13 |
KR20210091034A (en) | 2021-07-21 |
US11937054B2 (en) | 2024-03-19 |
US20210314701A1 (en) | 2021-10-07 |
JP2021110938A (en) | 2021-08-02 |
CN113113034A (en) | 2021-07-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11064294B1 (en) | Multiple-source tracking and voice activity detections for planar microphone arrays | |
US10123113B2 (en) | Selective audio source enhancement | |
Cao et al. | Acoustic vector sensor: reviews and future perspectives | |
US10957338B2 (en) | 360-degree multi-source location detection, tracking and enhancement | |
US7626889B2 (en) | Sensor array post-filter for tracking spatial distributions of signals and noise | |
US11264017B2 (en) | Robust speaker localization in presence of strong noise interference systems and methods | |
Farmani et al. | Informed sound source localization using relative transfer functions for hearing aid applications | |
Nakamura et al. | A real-time super-resolution robot audition system that improves the robustness of simultaneous speech recognition | |
CN110610718A (en) | Method and device for extracting expected sound source voice signal | |
Cho et al. | Sound source localization for robot auditory systems | |
Tourbabin et al. | Speaker localization by humanoid robots in reverberant environments | |
Wan et al. | Improved steered response power method for sound source localization based on principal eigenvector | |
Sewtz et al. | Robust MUSIC-based sound source localization in reverberant and echoic environments | |
Salvati et al. | Acoustic source localization using a geometrically sampled grid SRP-PHAT algorithm with max-pooling operation | |
Zhao et al. | A robust real-time sound source localization system for olivia robot | |
Ayllón et al. | An evolutionary algorithm to optimize the microphone array configuration for speech acquisition in vehicles | |
Dehghan Firoozabadi et al. | A novel nested circular microphone array and subband processing-based system for counting and DOA estimation of multiple simultaneous speakers | |
Doblinger | Localization and tracking of acoustical sources | |
Grondin et al. | A study of the complexity and accuracy of direction of arrival estimation methods based on GCC-PHAT for a pair of close microphones | |
Hammer et al. | FCN approach for dynamically locating multiple speakers | |
Xue et al. | Direction of arrival estimation based on subband weighting for noisy conditions | |
Hioka et al. | Estimating power spectral density for spatial audio signal separation: An effective approach for practical applications | |
Lu et al. | Separating voices from multiple sound sources using 2D microphone array | |
Garcia-Barrios et al. | Exploiting spatial diversity for increasing the robustness of sound source localization systems against reverberation | |
da Silva et al. | Acoustic source DOA tracking using deep learning and MUSIC |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: SYNAPTICS INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MASNADI-SHIRAZI, ALIREZA;NESTA, FRANCESCO;REEL/FRAME:051774/0958 Effective date: 20200210 |
|
AS | Assignment |
Owner name: WELLS FARGO BANK, NATIONAL ASSOCIATION, NORTH CAROLINA Free format text: SECURITY INTEREST;ASSIGNOR:SYNAPTICS INCORPORATED;REEL/FRAME:051936/0103 Effective date: 20200214 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |