Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0264—Noise filtering characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L15/00—Speech recognition
  • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L15/00—Speech recognition
  • G10L15/26—Speech to text systems
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L17/00—Speaker identification or verification techniques
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0216—Noise filtering characterised by the method used for estimating noise
  • G10L21/0232—Processing in the frequency domain
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0272—Voice signal separating
  • G10L21/028—Voice signal separating using properties of sound source
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0272—Voice signal separating
  • G10L21/0308—Voice signal separating characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/78—Detection of presence or absence of voice signals
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/90—Pitch determination of speech signals
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L2021/02087—Noise filtering the noise being separate speech, e.g. cocktail party
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0216—Noise filtering characterised by the method used for estimating noise
  • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
  • G10L2021/02166—Microphone arrays; Beamforming
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/90—Pitch determination of speech signals
  • G10L2025/906—Pitch tracking

Definitions

• the performance of the speech enhancement modules depends upon the ability to filter out all the interference signals leaving only the desired speech signals.
• Interference signals might be, for example, other speakers, noise from air conditions, music, motor noise (e.g. in a car or airplane) and large crowd noise also known as 'cocktail party noise'.
• the performance of speech enhancement modules is normally measured by their ability to improve the speech-to-noise-ratio (SNR) or the speech-to-interference-ratio (SIR), which reflects the ratio (often in dB scale) of the power of the desired speech signal to the total power of the noise and of other interfering signals respectively.
• SNR speech-to-noise-ratio
• SIR speech-to-interference-ratio
• the method may include: receiving or generating sound samples that represent sound signals that were received during a given time period by an array of microphones; frequency transforming the sound samples to provide frequency-transformed samples; clustering the frequency-transformed samples to speakers to provide speaker related clusters, wherein the clustering may be based on (i) spatial cues related to the received sound signals and (ii) acoustic cues related to the speakers; determining a relative transfer function for each speaker of the speakers to provide speakers related relative transfer functions; applying a multiple input multiple output (MIMO) beamforming operation on the speakers related relative transfer functions to provide heamformed signals; and inverse- frequency transforming the heamformed signals to provide speech signals.
• MIMO multiple input multiple output
• the method may include generating the acoustic cues related to the speakers.
• the generating of the acoustic cues may include searching for a keyword in the sound samples;
• the method may include extracting spatial cues related to the keyword.
• the method may include using the spatial cures related to the keyword as a clustering seed.
• the acoustic cues may include pitch frequency, pitch intensity, one or more pitch frequency harmonics, and intensity of the one or more pitch frequency harmonics.
• the method may include associating a reliability attribute to each pitch and determining that a speaker that may be associated with the pitch may be silent when a reliability of the pitch falls below a predefined threshold.
• the clustering may include processing the frequency-transformed samples to provide the acoustic cues and the spatial cues; tracking over time states of speakers using the acoustic cues; segmenting the spatial cues of each frequency component of the frequency-transformed signals to groups; and assigning to each group of frequency-transformed signals an acoustic cue related to a currently active speaker.
• the assigning may include calculating, for each group of frequency-transformed signals, a cross-correlation between elements of equal-frequency lines of a time frequency map with elements that belong to other lines of the time frequency map and and may be related to the group of frequency-transformed signals.
• the tracking may include applying an extended Kalman filter.
• the tracking may include applying multiple hypothesis tracking.
• the tracking may include applying a particle filter.
• the segmenting may include assigning a single frequency component related to a single time frame to a single speaker.
• the method may include monitoring at least one monitored acoustic feature out of speech speed, speech intensity and emotional utterances.
• the method may include feeding the at least one monitored acoustic feature to an extended Kalman filter.
• the frequency-transformed samples may be arranged in multiple vectors, one vector per each microphone of the array of microphones; wherein the method may include calculating an intermediate vector by weight averaging the multiple vectors; and searching for acoustic cue candidates by ignoring elements of the intermediate vector that have a value that may be lower than a predefined threshold.
• the method may include determining the predefined threshold to be three times a standard deviation of a noise.
• a non-transitory computer readable medium that stores instructions that once executed by a computerized system cause the computerized system to: receive or generate sound samples that represent sound signals that were received during a given time period by an array of microphones; frequency transform the sound samples to provide frequency- transformed samples; cluster the frequency-transformed samples to speakers to provide speaker related clusters, wherein the clustering may be based on (i) spatial cues related to the received sound signals and (ii) acoustic cues related to the speakers; determine a relative transfer function for each speaker of the speakers to provide speakers related relative transfer functions; apply a multiple input multiple output (MIMO) beamforming operation on the speakers related relative transfer functions to provide beamformed signals; inverse-frequency transform the beamformed signals to provide speech signals.
• MIMO multiple input multiple output
• the non-transitory computer readable medium may store instructions for generating the acoustic cues related to the speakers.
• the generating of the acoustic cues may include searching for a keyword in the sound samples.
• the generating of the acoustic cues may include searching for a keyword in the sound samples.
• the non-transitory computer readable medium may store instructions for extracting spatial cues related to the keyword.
• the non-transitory computer readable medium may store instructions for using the spatial cures related to the keyword as a clustering seed.
• the acoustic cues may include pitch frequency, pitch intensity, one or more pitch frequency harmonics, and intensity of the one or more pitch frequency harmonics.
• the non-transitory computer readable medium may store instructions for associating a reliability attribute to each pitch and determining that a speaker that may be associated with the pitch may be silent when a reliability of the pitch falls below a predefined threshold.
• the clustering may include processing the frequency-transformed samples to provide the acoustic cues and the spatial cues; tracking over time states of speakers using the acoustic cues; segmenting the spatial cues of each frequency component of the frequency-transformed signals to groups; and assigning to each group of frequency-transformed signals an acoustic cue related to a currently active speaker.
• the assigning may include calculating, for each group of frequency-transformed signals, a cross-correlation between elements of equal-frequency lines of a time frequency map with elements that belong to other lines of the time frequency map and and may be related to the group of frequency-transformed signals.
• the tracking may include applying an extended Kalman filter.
• the tracking may include applying multiple hypothesis tracking.
• the tracking may include applying a particle filter.
• the segmenting may include assigning a single frequency component related to a single time frame to a single speaker.
• the non-transitory computer readable medium may store instructions for monitoring at least one monitored acoustic feature out of speech speed, speech intensity and emotional utterances.
• the non-transitory computer readable medium may store instructions for feeding the at least one monitored acoustic feature to an extended Kalman filter.
• the frequency-transformed samples may be arranged in multiple vectors, one vector per each microphone of the array of microphones; wherein the non-transitory computer readable medium may store instructions for calculating an intermediate vector by weight averaging the multiple vectors; and searching for acoustic cue candidates by ignoring elements of the intermediate vector that have a value that may be lower than a predefined threshold.
• the non-transitory computer readable medium may store instructions for determining the predefined threshold to be three times a standard deviation of a noise.
• a computerized system may include an array of microphones, a memory unit and a processor.
• the processor may be configured to receive or generate sound samples that represent sound signals that were received during a given time period by an array of microphones; frequency transform the sound samples to provide frequency- transformed samples; cluster the frequency-transformed samples to speakers to provide speaker related clusters, wherein the clustering may be based on (i) spatial cues related to the received sound signals and (ii) acoustic cues related to the speakers; determine a relative transfer function for each speaker of the speakers to provide speakers related relative transfer functions; apply a multiple input multiple output (MIMO) beamforming operation on the speakers related relative transfer functions to provide heamformed signals; inverse-frequency transform the beamformed signals to provide speech signals; and wherein the memory unit may be configured to store at least one of the sound samples and the speech signals.
• the computerized system may not include the array of microphones but may receive signals from the array of microphones that represent the sound signals that were received during the given time
• the processor may be configured to generate the acoustic cues related to the speakers.
• the generating of the acoustic cues may include searching for a keyword in the sound samples;
• the processor may be configured to extract spatial cues related to the keyword.
• the processor may be configured to use the spatial cures related to the keyword as a clustering seed.
• the acoustic cues may include pitch frequency, pitch intensity, one or more pitch frequency harmonics, and intensity of the one or more pitch frequency harmonics.
• the processor may be configured to associate a reliability attribute to each pitch and determining that a speaker that may be associated with the pitch may be silent when a reliability of the pitch falls below a predefined threshold.
• the processor may be configured to cluster by processing the frequency-transformed samples to provide the acoustic cues and the spatial cues; track over time states of speakers using the acoustic cues; segmenting the spatial cues of each frequency component of the frequency- transformed signals to groups; and assign to each group of frequency-transformed signals an acoustic cue related to a currently active speaker.
• the processor may be configured to assign by calculating, for each group of frequency- transformed signals, a cross-correlation between elements of equal-frequency lines of a time frequency map with elements that belong to other lines of the time frequency map and and may be related to the group of frequency-transformed signals.
• the processor may be configured to track by applying an extended Kalman filter.
• the processor may be configured to track by applying multiple hypothesis tracking.
• the processor may be configured to track by applying a particle filter.
• the processor may be configured to segment by assigning a single frequency component related to a single time frame to a single speaker.
• the processor may be configured to monitor at least one monitored acoustic feature out of speech speed, speech intensity and emotional utterances. [0057] The processor may be configured to feed the at least one monitored acoustic feature to an extended Kalman filter.
• the frequency- transformed samples may be arranged in multiple vectors, one vector per each microphone of the array of microphones; wherein the processor may be configured to calculate an intermediate vector by weight averaging the multiple vectors; and search for acoustic cue candidates by ignoring elements of the intermediate vector that have a value that may be lower than a predefined threshold.
• the processor may be configured to determine the predefined threshold to be three times a standard deviation of a noise.
• FIG. 1 illustrates multipath
• FIG. 2 illustrates an example of a method
• FIG. 3 illustrates an example of a clustering step of the method of FIG. 2
• FIG. 4 illustrates an example of a pitch detection over a time-frequency map
• FIG. 5 illustrates an example of a a time -frequency-Cue map
• FIG. 6 illustrates an example of a voice recognition chain in offline training
• FIG. 7 illustrates an example of a voice recognition chain in real-time training
• FIG. 8 illustrates an example of a training mechanism
• FIG. 9 illustrates an example of a method.
• Any reference to a system should be applied, mutatis mutandis to a method that is executed by a system and/or to a non-transitory computer readable medium that stores instructions that once executed by the system will cause the system to execute the method.
• Any reference to method should be applied, mutatis mutandis to a system that is configured to execute the method and/or to a non-transitory computer readable medium that stores instructions that once executed by the system will cause the system to execute the method.
• Any reference to a non-transitory computer readable medium should be applied, mutatis mutandis to a method that is executed by a system and/or a system that is configured to execute the instructions stored in the non-transitory computer readable medium.
• system means a computerized system.
• Speech enhancement methods are focused on extracting a speech signal from a desired source (speaker) when the signal is interfered by noise and other speakers.
• a desired source for example, a desired source
• spatial filtering in the form of directional beamforming is effective.
• the speech from each source is smeared across several directions, not necessarily successive, deteriorating the advantages of the ordinary beamformers.
• TF transfer- function
• RTF relative transfer function
• the ability to estimate the RTF for each speaker, when the speech signals are captured simultaneously is yet a challenge.
• a clustering algorithm of speakers which assigns each frequency component to its original speaker especially in multi-speaker reverberant environments. This provides the necessary condition for the RTF estimator to work properly in multi-speaker reverberant environments.
• the estimate of the RTFs matrix is then used to compute the weight vector of the transfer function based linear constrained minimum variance (TF-LCMV) heamformer (see Equation (10) in the sequel) and thus satisfies the necessary condition for TF- LCMV to work. It is assumed that each human speaker is endowed with a different pitch, so that the pitch is a bijective indicator to a speaker.
• Multi-pitch detection is known to be a challenging task especially in a noisy, reverberant multi-speaker environment.
• W-DO W-Disjoint Orthogonality
• a set of spatial cues for example, signal intensity, azimuth angle and elevation angle, are used as additional features.
• the acoustical cues - pitch values - are tracked over time using extended Kalman filter (EKF) to overcome temporary inactive speakers and changes in pitch, and the spatial cues are used to segment the last L frequency components and to assign each frequency component to different sources.
• EKF extended Kalman filter
• Figure 1 describes the paths along which the frequency components of the speech signal travel from a human speaker 11 to the microhome array 12 in a reverberant environment.
• the walls 13 and other elements in the environment 14 reflect the impinging signal with attenuation and reflecting angle which depend on the material and the texture of the wall.
• Different frequency components of the human speech might take different paths. These might be a direct path 15 which reside on the shortest path between the human speaker 11 and the microphone array 12, or indirect paths 16, 17. Note that a frequency component might travel along one or more paths.
• FIG. 2 describes the algorithm.
• the microphones can be deployed in a range of constellations such as equally-spaced on a straight line, on a circle or on a sphere, or even unevenly spaced forming arbitrary shape.
• the signal from each microphone is sampled, digitized, and stored in M frames, each contains T consecutive samples 202.
• the size of the frames T may be selected to be large enough such that the short-time Fourier transform (STFT) is accurate, but short enough so that the signal is stationary along the equivalent time duration.
• STFT short-time Fourier transform
• a typical value for T is 4,096 samples for sampling rate of 16kFlz, that is, the frame is equivalent to 1/4 second.
• T may, for example, range between 0.1 Sec - 2 Sec - thereby providing 1024-32768 sampled for 16kFlz sampling rate.
• the samples are also referred to as sound samples that represent sound signals that were received by the array of microphones during period of time T.
• Each frame is transformed in 203 to the frequency domain by applying Fourier transform or a variant of Fourier transform such as short time Fourier transform (STFT), constant- Q transform (CQT), logarithmic Fourier transform (LFT), filter bank and alike.
• Fourier transform such as short time Fourier transform (STFT), constant- Q transform (CQT), logarithmic Fourier transform (LFT), filter bank and alike.
• STFT short time Fourier transform
• CQT constant- Q transform
• LFT logarithmic Fourier transform
• filter bank Several techniques such as windowing and zero-padding might be applied to control the framing effect.
• the output of step 203 may be
• the speech signals are clustered to different speakers in 204.
• the clusters may be referred to as speaker related clusters.
• 204 deals with multi-speakers in a reverberant room, so that signals from different directions can be assigned to the same speaker due to the direct paths and the indirect paths.
• the proposed solution suggests using a set of acoustic cues, for example, the pitch frequency and intensity, and its harmonics frequencies and intensities, on top of a set of spatial cues, for example the direction (azimuth and elevation) and the intensity of the signal in one of the microphones.
• the pitch and one or more of the spatial cues are served as the state vector for a tracking algorithm such as Kalman filter and its variants, multiple hypothesis tracking (MHT) or particle filter, which are used to track this state vector, and to assign each track to a different speaker.
• a tracking algorithm such as Kalman filter and its variants, multiple hypothesis tracking (MHT) or particle filter, which are used to track this state vector, and to assign each track to a different speaker.
• MHT multiple hypothesis tracking
• An RTF estimator is applied in 205 to the data in the frequency domain.
• the result of this stage is a set of RTFs each is registered to the associate speaker.
• the registration process is done using the clustering array from the clustering speakers 204.
• the set of RTFs are also referred to as speakers related relative transfer functions.
• the MIMO heamformer 206 reduces the energy of the noise and of the interfering signals with respect to the energy of the required speech signal by means of spatial filtering.
• the output of step 206 may be referred to as beamformed signals.
• the beamformed signals are then forwarded to the inverse frequency transform 207 to create a continuous speech signal in the form of a stream of samples, which is transferred, in turn, to other elements such as speech recognition, communication systems and recording devices 208.
• a keyword spotting 209 can be used to improve the performance of the clustering block 204.
• the frames from 202 are searched for a pre defined keyword (for example“hello Alexa”, or“ok Google”).
• a pre defined keyword for example“hello Alexa”, or“ok Google”.
• the acoustic cues of the speaker are extracted, such as the pitch frequency and intensity and its harmonics frequencies and intensities.
• the features of the paths over which each frequency component has arrived at the microphone array 201 are extracted. These features are used by the clustering speaker 204 as a seed for the cluster of the desired speaker. Seed is an initial guess as to the initial parameters of the cluster.
• centroid-based clustering algorithms such as K-means, PSO and 2KPM.
• centroid-based clustering algorithms such as K-means, PSO and 2KPM.
• bases of the subspace for subspace-based clustering is the bases of the subspace for subspace-based clustering.
• Fig. 3 describes the clustering algorithm of speakers. It is assumed that each speaker is endowed with a different set of acoustic cues, for example, pitch frequency and intensity and its harmonics frequencies and intensities, so that the set of acoustic cues is a bijective indicator to a speaker. Acoustic cues detection is known to be a challenging task especially in a noisy, reverberant multi-speaker environment.
• the spatial cues for example, in the form of the signal intensity, the azimuth angle and the elevation angle are used.
• the acoustical cues are tracked over time using filters such as particle filter and extended Kalman filter (EKF) to overcome temporary inactive speakers and changes in acoustic cues, and the spatial cues are used to segment the frequency components among different sources.
• EKF extended Kalman filter
• the result of the EKF and the segmentation is combined by means of cross-correlation to facilitate the clustering of the frequency components to a specific speaker with a specific pitch.
• a time-frequency map is prepared using the frequency transform of the buffers from each microphone, which are computed in 203.
• the absolute value of each of the M K-long complex-valued vectors are weight-averaged, with some weight factors which can be determined so as to diminish artifacts in some of the microphones.
• the result is a single K-long real vector. In this vector, values higher than a given threshold m are extracted, while the rest of the elements are discarded.
• the threshold m is often selected adaptively as being three times the standard deviation of the noise, but no less than a constant value which depends on the electrical parameters of the system, and especially on the number of effective bits of the sampled signal.
• Values with frequency index within the range of [k_min, k_max] are defined as candidates for pitch frequencies.
• Variable k_min and k_max are typically 85 Flz and 2550 Flz respectively, as typical adult male will have a fundamental frequency from 85 to 1800 Flz, and that of a typical adult female from 165 to 2550 Flz.
• Each pitch candidate is then verified by searching for its higher harmonics.
• the pitch of the desired speaker 32 is supplied by 210 using a keyword that was uttered by the desired speaker.
• an extended Kalman filter (EKE) is applied to the pitch from 31.
• EKE extended Kalman filter
• a Kalman filter has a state transition equation and an observation model. The state transition equation, for a discrete calculation, is:
• Time updater of the extended Kalman filter may predict the next state with prediction equations and detected pitch may update the variables by comparing the actual measurement with the predicted measurement, using the following type of equation:
• each trajectory may begin from a detected pitch, followed by a model f (x k, Uk), reflecting the temporal behavior of the pitch, which might go higher or lower because of emotions.
• the model’s inputs may be past state vectors X k (either one state vector or more), and any external inputs U k which affect the dynamics of the pitch, such as the speed of the speech, intensity of speech and emotional utterances.
• the elements of the state vector x may quantitatively describe the pitch.
• a state vector of a pitch might include, inter alia, the pitch frequency, the intensity of the 1 st order harmonics, and the frequency and intensity of higher harmonics.
• the vector function f (x k, Uk ) may be used to predict the state- vector x at some given time k+1 ahead of the current time.
• An exemplary realization of the dynamic model in the EKF may include the time update equation (a.k.a. prediction equation) as is described in the book“Lessons in Digital Estimation Theory” by Jerry M. Mendel, which is incorporated herein by reference.
• f k is the frequency of the pitch (1 st harmonic) at time k
• a k is the intensity of the pitch (1 st harmonic) at time k
• b k is the intensity of the 2 nd harmonic at time k.
• An exemplary state- vector model for the pitch may be:
• Each track is endowed with reliability field which is inversely proportional to the time over which the track evolves using the time update only.
• reliability threshold p say, representing 10 seconds of undetected pitch
• the track is defined as dead, which means that the respective speaker is not active.
• a new measurement pitch detection
• TFC time- frequency-Cue
• the spatial cues of each frequency component in the TFC are segmented.
• the idea is that along the L frames, a frequency component might originate from different speakers, and this can be observed by comparing the spatial cues. It is assumed, however, that at a single frame time l, the frequency component originates from a single speaker, owing to the W-DO assumption.
• the segmentation can be performed using any known method in the literature which is used for clustering such as K nearest neighbors (KNN). The clustering assigns an index to each
• the frequency components of the signals are grouped such that each frequency component is assigned to a specific pitch in the list of pitches which are tracked by the EKF and is active by its reliability. This is done by computing the sample-cross-correlation between the k th line of the time-frequency map (see Fig. 4), which is assigned to one of the pitches, with all the values with a specific cluster index c 0 (j,l) in other lines in the time-frequency map. This is done for every cluster index.
• the sample cross-correlation is given by:
• A is the time-frequency map
• k is the index of the line belonging to one of the pitches
• j is any other line of A
• L is the number of columns of A.
• Fig. 4 describes an example of the pitch detection over the time-frequency map.
• 41 is the time axis, which is denoted by the parameter l
• 42 is the frequency axis which is described by the parameter k .
• Each column in this 2-dimensional array is the K-long real valued vector extracted in 31 after averaging the absolute value of the M frequency transformed buffers at time l .
• the L recent vectors are saved in a 2 dimensional array of size KxL.
• two pitches are denoted by diagonal lines at different directions.
• Figure 5 describes the TFC-map, whose axes are the frame index (time) 51, the frequency component 52 and the spatial cues 53, which might be, for example, a complex value expressing the direction (azimuth and elevation) from which each frequency component arrives, and the intensity of the component.
• a vector of M complex number is received for each frequency element .
• up to M-l spatial cues are extracted.
• direction and intensity of each frequency component this might be done using any direction-finding algorithm for array processing which is known in the art such as MUSIC or ESPRIT.
• the result of this algorithm is a set of up to M-l directions in 3-dimensional space, each is expressed by two angles and the estimated intensity of the arriving signal
• the cues are arranged in the TFC-map such that a the the cell
• the performance of the speech enhancement modules depends upon the ability to filter out all the interference signals leaving only the desired speech signals.
• Interference signals might be, for example, other speakers, noise from air conditions, music, motor noise (e.g. in a car or airplane) and large crowd noise also known as 'cocktail party noise'.
• the performance of speech enhancement modules is normally measured by their ability to improve the speech-to-noise-ratio (SNR) or the speech-to-interference-ratio (SIR), which reflects the ratio (often in dB scale) of the power of the desired speech signal to the total power of the noise and of other interfering signals respectively.
• SNR speech-to-noise-ratio
• SIR speech-to-interference-ratio
• the acquisition module contains a single microphone
• the methods are termed single-microphone speech enhancement and are often based on the statistical features of the signal itself in the time-frequency domain such as single channel spectral subtraction, spectral estimation using minimum variance distortionless response (MVDR) and echo-cancelation.
• the acquisition module is often termed microphone array, and the methods - multi-microphone speech enhancement. Many of these methods exploit the differences between the signals captured simultaneously by the microphones.
• a well-established method is the beamforming which sums-up the signals from the microphones after multiplying each signal by a weighting factor. The objective of the weighting factors is to average out the interference signals so as to condition the signal of interest.
• Beamforming in other words, is a way of creating a spatial filter which algorithmically increases the power of a signal emitted from a given location in space (the desired signal from the desired speaker), and decreases the power of signals emitted from other locations in space (interfering signals from other sources), thereby increasing the SIR at the heamformer output.
• Delay-and-sum heamformer involve using weighting factors of a DSB are composed of the counter delays implied by the different ways along which the desired signal travels from its source to each of the microphones in the array.
• DSB is limited to signals which come from a single direction each, such as in free-field environments. Consequently, in reverberant environments, in which signals from the same sources travel along different ways to the microphones and arrive at the microphone from a plurality of directions, DSB performance is typically insufficient.
• heamformers may use more complicated acoustic transfer function (ATF), which represents the direction (azimuth and elevation) from which each frequency component arrives at a specific microphone from a given source.
• ATF acoustic transfer function
• DOA single direction of arrival
• the ATF in the frequency domain is a vector assigning a complex number to each frequency in the Nyquist bandwidth. The absolute value represents the gain of the path related to this frequency, and the phase indicates the phase which is added to the frequency component along the path.
• Estimating the ATF between a given point in space and a given microphone may be done by means of using a loudspeaker positioned at the given point and emitting a known signal. Taking simultaneously the signals from the input of the speaker and the output of the microphone one can readily estimate the ATF.
• the loudspeaker may be situated at one or more positions where human speakers might reside during the operation of the system.
• This method creates a map of ATFs for each point in space, or more practically, for each point on a grid. ATFs of points not included in the grid are approximated using interpolation. Nevertheless - this method suffers from major drawbacks. First, the need to calibrate the system for each installation making this method impractical.
• RTF relative transfer function
• the RTF is the difference between the ATFs between a given source to two of the microphones in the array, which, in the frequency domain takes the form of the ratio between the spectral representation of the two ATFs. Like the ATF, the RTF in the frequency domain assigns a complex number to each frequency.
• the absolute value is the gain difference between the two microphones, which is often close to unity when the microphones are close to each other, and the phase, under some conditions, reflects the incident angle of the source.
• Transfer function based linear constrained minimum variance (TF-LCMV) heamformer may reduce noise while limiting speech distortion, in multi-microphone applications, by minimizing the output energy subject to the constraint that the speech component in the output signal is equal to the speech component in one of the microphone signals.
• ATF between the n-th source and the m-th microphone is , and the noise at the m-th
• the received signal in a matrix form is given by:
• the RTF of the n-th speech source can be defined as the ratio between the n-th speech components at the m-th
• the signal in (7) can be formulated using the RTFs matrix
• the TF-LCMV is an applicable method for extracting M - 1 speech source impinging an array comprising of M sensors from different locations in a reverberant environment.
• a necessary condition for the TF-LCMV to work is that the RTFs matrix H (l, k) whose columns are the RTF vectors of all the active sources in the environment is known and available to the TF-LCMV. This needs association of each frequency component to its source speaker.
• BSS blind source separation
• BSS may be assisted by the pitch information.
• the gender of the speakers is required a-priory.
• BSS may be used in the frequency domain, while resolving the ambiguity of the estimated mixing matrix using the maximum-magnitude method, which assigns a specific column of the mixing matrix to the source corresponds to the maximal element in the vector. Nevertheless - this method depends heavily on the spectral distributions of the sources as it is assumed that the strongest component at each frequency indeed belongs to the strongest source. However, this condition is not often met, as different speakers might introduce intensity peaks at different frequencies.
• source activity detection may be used, also known as voice activity detection (VAD), such that the information on the active source at a specific time is used to resolve the ambiguity in the mixing matrix.
• VAD voice activity detection
• VAD voice-pause cannot be robustly detected, especially in a multi-speaker environment. Also, this method is effective only when no more than a single speaker at a time join to the conversation, requires a relatively long training period, and is sensitive to motion during this period.
• the TF-LCMV heamformer may be used as well as its extended version for binaural speech enhancement system, together with a binaural cues generator.
• the acoustic cues are used to segregate speech components from noise components in the input signals.
• the technique is based on the auditory scene analysis theory 1 , which suggest the use of distinctive perceptual cues to cluster signals from distinct speech sources in a“cocktail party” environment.
• Examples of primitive grouping cues that may be used for speech segregation include common onsets/offsets across frequency bands, pitch (fundamental frequency), same location in space, temporal and spectral modulation, pitch and energy continuity and smoothness.
• W-DO may be used to facilitate BSS by defining a specific class of signals which are W-DO to some extent. This may use only the first order statistics is needed, which is computationally economic. Furthermore, an arbitrary number of signal sources can be de-mixed using only two microphones, provided that the sources are W-DO and do not occupy the same spatial positions. However, this method assumes an identical underlying mixing matrix across all frequencies. This assumption is essential for using histograms of the estimated mixing coefficients across different frequencies. However, this assumption often does not hold true in a reverberant environment, but only in free-field.
• the solution may operate even without a-priory information, even without a large training process, even without constraining estimations of the attenuation and the delay of a given source at each frequency to a single point in the attenuation-delay space, even without constraining estimated values of the attenuation-delay values of a single source to create a single cluster, and even without limiting the number of mixed sounds to two.
• VUI Voice user interface
• the VUI receives the speech signal using one or more microphones and converts the speech signal into a digital signature, often by transcribing the speech signal into text, which is used to infer the intention of the speaker.
• the machine can then response to the intention of the speaker based on the application the machine is designed for.
• the key component of VUI is the automatic speech recognition engine (ASR) which converts the digitized speech signal into text.
• ASR automatic speech recognition engine
• the performance of ASRs that is, how accurately the text describes the acoustic speech signal, depends heavily on the matching of the input signal to the requirements of the ASR. Therefore, other components of VUI are designed to enhance the acquired speech signal before feeding it to the ASR. Such components may be noise suppression, echo cancellation and source separation to name but a few.
• SS source separation
• a signal, acquired by each of the microphones is a mixture of all the speech signals in the environment plus other interferences such as noise and music.
• the SS algorithm takes the mixed signals from all the microphones and decomposes them to their components. That is, the output of the source separation is a set of signals, each represents the signal of a specific source, be it a speech signal from a specific speaker, music or even the noise.
• Figure 6 illustrates an example of a voice recognition chain in offline training.
• the chain often includes an array of microphones 511, which provides a set of digitized acoustic signals.
• the number of digitized acoustic signals is equal to the number of microphones composing the array 511.
• Each digitized acoustic signal contains a mixture of all the acoustic sources in the vicinity of the array of microphones 511, be it human speakers, synthetic speakers such as TV, music and noise.
• the digitized acoustic signals are delivered to a pre-processing stage 512.
• the aim of the pre-processing stage 512 is to improve the quality of the digitized acoustic signals by removing interferences such as echo, reverberation and noise.
• the pre-processing stage 512 is usually performed using multichannel algorithms that employ statistical association between the digitized acoustic signals.
• the output of the pre-processing stage 512 is a set of processed signals, usually with the same number of signals as the number of the digitalized acoustic signals at the input to this stage.
• the set of processed signals is forwarded to the source separation (SS) stage 513 which aims at extracting the acoustic signals from each source in the vicinity of the array of microphones.
• the SS stage 513 takes a set of signals, each is a different mixture of the acoustic signals received from different sources and creates a set of signals such that each signal contains mainly a single acoustic signal from a single specific source.
• Source separation of speech signals may be performed using geometric considerations of the deployment of the sources such as beamforming or by considering the characteristics of the speech signal such as independent component analysis.
• the number of the separated signals is usually equal to the number of active sources in the vicinity of the array of microphones 511, but smaller than the number of microphones.
• the set of separated signals is forwarded to the source selector 514.
• the aim of the source selector is to select the relevant source of speech whose speech signal should be recognized.
• the source selector 514 may use a trigger word detector such that the source which pronounces a pre-defined trigger word is selected. Alternatively, the source selector 514 may consider the position of the source in the vicinity of the array of microphones 511 such as a predefined direction with respect to the array of microphones 511.
• the source selector 514 may use a predefined acoustical signature of the speech signal to select the source which matches with this signature.
• the output of the source selector 514 is a single speech signal which is forwarded to the speech recognition engine 515.
• the speech recognition engine 515 converts the digitized speech signal into text. There are many methods for speech recognitions known in the art, most of them are based on extracting features from the speech signal and comparing these features with a predefined vocabulary.
• the primary output of the speech recognition engine 515 is a text string 516 which is associated with the input speech signal.
• a predefined text 518 is pronounced to the microphones in the offline training.
• the error 519 is computed by comparing the output of the ASR 516 to this text.
• the comparison 517 can be performed using a simple word-counting or more sophisticated comparison methods which consider the meaning of the words and appropriately weights mis-detections of different word.
• the error 519 is then used by the SS 513 to modify the set of parameters to find the values which minimize the error. This can be done by any supervised estimation or optimization method such as least squares, stochastic gradient, neural network (NN) and its variants.
• Figure 7 illustrates an example of a voice recognition chain in real-time training, that is, during the normal operation of the system.
• the hue text that was pronounced by the human speaker is unknown, and a supervised error 519 is unavailable.
• An alternative is the confidence score 521 which was developed for real-time applications, when there is no reference to the real spoken text, and the application can benefit from knowing the reliability level of the ASR output. For example, when the confidence score is low, a system may go into an appropriate branch in which a more directed dialog is carried on with the user. There are many methods for estimating the confidence score, most of them target a high correlation with the error that can be computed when the spoken text is known.
• the confidence score 521 is converted to the supervised error 519 by the error estimator 522.
• the estimated error 519 may be used to train the parameters of the SS likewise the offline training.
• FIG. 3 illustrates the training mechanism of a typical SS 513.
• the source separator (SS) 513 receives a set of mixed signals from the pre-processing stage 512 and feeds the source selector 514 with separated signals.
• source separation of acoustic signals, and specifically, speech signals is done in the frequency domain.
• the mixed signals from the pre processing stage 512 are first transformed into the frequency domain 553. This is done by dividing the mixed signals into segments of identical length, with an overlap period between consequent segments. For example, when the length of the segments is determined to be 1024 samples, and the overlap period is 25%, then each of the mixed signals is divided into segments of 1024 samples each.
• the set of concurrent segments from different mixed signals is termed a batch.
• Each batch of segments starts 768 samples after the previous batch. Note, that the segments across the set of the mixed signals are synchronized, that is, the start points of all the segments belonging to the same batch are identical. The length of the segments in a batch and the overlap period are taken from the model parameters 552.
• a demixing algorithm 554 separates the batch of segments arrived from the frequency transform 553.
• source separation (SS) algorithm includes a set of mathematical models, accompanied with a set of model parameters 552.
• the mathematical models establish the method of operation, such as the way the SS copes with physical phenomena e.g. multipath.
• the set of model parameters 552 adjusts the operation of the SS to specific features of the source signals, to the architecture of the automatic speech recognition engines (ASR) which receives these signals, to the geometry of the environment, and even to the human speaker.
• ASR automatic speech recognition engines
• the demixed batch of segments is forwarded to the inverse frequency transform 555 in which it is transformed back to the time domain.
• the same set of model parameters 552 which was used in the frequency transform stage 553 is used in the inverse frequency transform stage 555.
• the overlap period is used to reconstruct the output signals in the time domain from the consequent batches. This is done for example, using the overlap-add method, in which after the inverse frequency transform, the resulting output signal is reconstructed by overlapping and adding overlapped intervals, possibly with appropriate weighting function that ranges between 0 to 1 across the overlap region, so that the total energy is conserved.
• the overlap segment from the former batch fades out while the overlap segment from the latter batch fades in.
• the output of the Inverse frequency transform block is forwarded to the source selector 514.
• the model parameters 552 is a set of parameters which are used by the frequency transform block 553, the demixing block 554 and the inverse frequency transform block 555.
• a clocking mechanism such as real-time clock.
• each of the frequency transform block 553, the demixing block 554 and the inverse frequency transform block 555 extract the parameters from the model parameters 552. These parameters are then substituted in the mathematical models which are executed in the frequency transform block 553, the demixing block 554 and the inverse frequency transform block 555.
• the corrector 551 optimizes the set of model parameters 552, aiming at reducing the error 519 from the error estimator.
• the corrector 551 receives the error 519 and the current set of model parameters 552 and outputs a corrected set of model parameters 552.
• the correction of the set of parameters can be done a-priori (offline) or during the operation of the VUI (real-time).
• offline training the error 519, which is used to correct the set of model parameters 552, is extracted using predefined text which is pronounced to the microphones and comparing the output of the ASR to this text.
• the error 519 is extracted from the confidence score of the ASR.
• the error is then used to modify the set of parameters to find the values which minimize the error.
• This can be done by any supervised estimation or optimization method, preferably derivative free methods, such as golden section search, grid search and Nelder-Mead.
• the Nelder-Mead method (also downhill simplex method, amoeba method, or polytope method) is a commonly applied numerical method used to find the minimum or maximum of an objective function in a multidimensional space. It is a direct search
• Nelder-Mead iteratively finds a local minimum of the error 519 as a function of several parameters.
• the method starts with a set of values which determines a simplex (a generalized triangle in N dimensions). There is an assumption that a local minimum exists within the simplex.
• the errors at the vertices of the simplex are computed.
• the vertex with the maximal error is replaced with a new vertex so that the volume of the simplex is reduced. This repeats until the simplex volume is smaller than a predefined volume, and the optimal value is one of the vertices.
• the process is performed by the corrector 551.
• Golden section search finds the minimum of the error 519 by successively narrowing the range of values inside which the minimum is known to exist.
• the golden section search requires a strictly unimodal error as a function of the parameter.
• the operation of narrowing the range is done by the corrector 551.
• the golden-section search is a technique for finding the extremum (minimum or maximum) of a strictly unimodal function by successively narrowing the range of values inside which the extremum is known to exist. (www.wikipedia.org).
• Grid search iterates through a set of values associated with one or more of the parameters to be optimized.
• each value in the set is a vector whose length is equal to the number of parameters.
• the error 519 is computed, and the value corresponding to the minimal error is selected. The iteration through the set of values is performed by the corrector 551.
• Grid Search the traditional way of performing hyperparameter optimization has been grid search, or a parameter sweep, which is simply an exhaustive searching through a manually specified subset of the hyperparameter space of a learning algorithm.
• a grid search algorithm must be guided by some performance metric, typically measured by cross- validation on the training set or evaluation on a held-out validation set. Since the parameter space of a machine learner may include real- valued or unbounded value spaces for certain parameters, manually set bounds and discretization may be necessary before applying grid search.
• One approach is to operate the optimization in parallel to the normal operation of the system using parallel threads or multi cores. That is, there is one or more parallel tasks which perform blocks 513, 514, 515, 522 in parallel to the task of the normal operation of the system.
• the parallel tasks a batch of mixed signals with length of 1-2 seconds is taken from the pre processing 512, and repeatedly separated 513 and interpreted 514, 515 with different sets of model parameters 552.
• the error 519 is computed is computed for each such cycle.
• the set of model parameters is selected by the corrector 551 according to the optimization method.
• Second approach is to operate the optimization when there is no speech in the room. Periods with absence of human speech can be detected using voice activity detection (VAD) algorithms. These periods are used to optimize the model parameters 552 the same way as in the first approach, saving the need for parallel threads or multi cores.
• VAD voice activity detection
• a suitable optimization method should be selected for each parameter in 552. Some of the methods are applied to a single parameter, and some are applied to a group of parameters.
• the length of segment parameter is related to FFT/IFFT .
• ASRs which use features of separated phonemes require short segments of the order of 20mSec
• ASRs which use features of series of consequent phonemes use segments of the order of 100-200 mSec.
• the length of the segment is affected by the scenario, such as the reverberation time of the room.
• the segment length should be of the order of the reverberation time which may be as much as 200-500 mSec. As there is no sweet point for the length of the segment, this value should be optimized to the scenario and the ASR.
• Typical value is 100-500 mSec, in terms of samples. For example, sampling rate of 8 kHz, implies segment length of 800- 4000 samples. This is a continuous parameter.
• the optimization of this parameter can be done using various optimization methods such as a golden section search or Nelder - Mead together with the overlap period.
• the input to the algorithm is the minimum and maximum possible length, for example 10 mSec to 500 mSec, and the error function 519.
• the output is the length of the segment which minimizes the error function 519.
• the input is a set of three two-tuples of the segment length and the overlap period, for example (10 mSec, 0%), (500 mSec, 10%) and (500 mSec, 80%) and the error function 519, and the output is an optimal length of segment and an optimal overlap period.
• the overlap period parameter is related to FFT/IFFT .
• the overlap period is used to avoid overlooking of phonemes because of the segmentation. That is, phonemes which are divided between consequent segments.
• the overlap period depends on the features the ASR employs. Typical range - 0-90% of the length of the segment. This is a continuous parameter.
• the optimization of this parameter can be done using various optimization methods such as a Golden section search or Nelder-mead with the length of segment.
• a Golden section search the input to the algorithm is the minimum and maximum possible overlap period, for example 0% to 90%, and the error function 519.
• the output is the overlap period which minimizes the error function 519.
• the window parameter is related to FFT/IFFT.
• the frequency transform 553 often uses windowing to alleviate the effect of segmentation.
• Some windows such as Kaiser and Chebyshev are parameterized. This means that the effect of the window can be controlled by changing the parameter of the window. The typical range depends on the type of the window. This is a continuous parameter.
• the optimization of this parameter can be done using various optimization methods such as a Golden section search. When using the golden section search, the input to the algorithm is the minimum and maximum values of the parameter of the window, which depend on the window type, and the error function 519. For example, for Kaiser window, the minimum and maximum values are (0,30). The output is an optimal window parameter.
• the sampling rate parameter is related to FFT/IFFT.
• Sampling rate is one of the critical parameters which affect the performance of the speech recognition. For example, there are ASRs which demonstrate poor results for sampling rate which is lower than 16kHz. Other can work well even with 4 or 8 kHz. Typically, this parameter is optimized once when the ASR is selected. The typical range is 4, 8, 16, 44.1, 48 kHz. This parameter is a discrete parameter.
• the optimization of this parameter can be done using various optimization methods such as a grid search.
• the input to the algorithm is the values over which the grid search is performed - for example sampling rate of (4, 8, 16, 44.1, 48) kHz, and the error function 519.
• the output is the optimal sampling rate.
• the filtering parameter is related to the demixing.
• Some ASRs use features which represent limited frequency range. Therefore, filtering the separated signals after the source separation 513, may emphasize specific features which are used by the ASR, thereby improving its performance. Moreover, filtering out spectral components which are not used by the ASR may improve the signal to noise ratio (SNR) of the separated signals, which in turn may improve the performance of the ASR.
• SNR signal to noise ratio
• the typical range is 4-8 kHz.
• the optimization of this parameter can be done using various optimization methods such as a Golden section search. This parameter is continuous. When applying the golden section search the input to the algorithm is the error function 519 and the initial guess of the sections of the cut-off frequency, for example, 1000 Hz and 0.5 X sampling rate. The output is the optimal filtering parameter.
• Weighting factors for each microphone is related to demixing. Theoretically, the sensitivity of different microphones on a specific array should be similar up to 3 dB. Practically, however, the span of the sensitivity of different microphones may be greater. Furthermore, the sensitivity of microphones may change in time due to dust and humidity. The typical range is 0-10 dB. This is a continuous parameter. The optimization of this parameter can be done using various optimization methods such as Nelder- mead with or without the Weighting factors for each microphone. When applying the Nelder- mead method the input to the algorithm is the error function 519 and the initial guess of the vertices of the simplex.
• each n-tuple is the number of microphones - N: (1,0, ..,0,0), (0,0,...,0,1) and (1/N,1/N,.. ,1/N).
• the output is the optimal weight per microphone.
• the Number of microphones is related to demixing.
• the number of microphones affects the number of sources which can be separated on the one hand, and the complexity and numerical precision on the other hand. Practical experiments also shown that too many microphones may cause a decrease in output SNR.
• the typical range is 4-8.
• the optimization of this parameter can be done using various optimization methods such as a Grid search or Nelder-mead with the Weighting factors for each microphone. When applying a grid search the input to the algorithm is the error function 519 and the number of microphones over which the search is performed. For example, 4, 5, 6, 7, 8 microphones.
• the output is the optimal number of microphones.
• Method 600 may start by step 610 of receiving or calculating an error related to a speech recognition process that was applied on a previous output of a source selection process.
• Step 610 may be followed by step 620 of amending at least one parameter of the source separation process based on the error.
• Step 620 may be followed by step 630 of receiving signals that represent audio signals that originated from multiple sources and are detected by an array of microphones.
• Step 630 may be followed by step 640 of performing a source separation process for separating audio signals that originated from different sources of the multiple sources to provide source separated signals; and sending the source separated signals to the source selection process.
• Step 640 may be followed by step 630.
• steps 630 and 640 may be followed by step 610 (not shown) - where the output of step 640 may be fed to the source selection process and the ASR to provide the previous output of the ASR.
• steps 630 and 640 may be executed without receiving the error.
• Step 640 may include applying a frequency conversion (such as but not limited to FFT), demixing and applying an inverse frequency conversion (such as but not limited to IFFT).
• a frequency conversion such as but not limited to FFT
• demixing such as but not limited to IFFT
• an inverse frequency conversion such as but not limited to IFFT
• Step 620 may include at least one of the following:
• j Amending a number of microphones of the array of microphones.
• k Determining an amended value of at least one parameter using a golden section search.
• n Determining an amended value of a parameter of at least one parameter based on a predefined mapping between the error and the at least one parameter.
• any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved.
• any two components herein combined to achieve a particular functionality may be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components.
• any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality.
• any method may include at least the steps included in the figures and/or in the specification, only the steps included in the figures and/or the specification. The same applies to the system.
• the system may include an array of microphones, a memory unit and one or more hardware processors such as digital signals processors, FPGAs, ASICs, a general-purpose processor programmed to execute any of the mentioned above method and the like.
• the system may not include the array of microphones but may be fed from sound signals generated by the array of microphones.
• any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved.
• any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components.
• any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality.
• the illustrated examples may be implemented as circuitry located on a single integrated circuit or within a same device.
• the examples may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in a suitable manner.
• the examples, or portions thereof may implemented as soft or code representations of physical circuitry or of logical representations convertible into physical circuitry, such as in a hardware description language of any appropriate type.
• the invention is not limited to physical devices or units implemented in non programmable hardware but can also be applied in programmable devices or units able to perform the desired device functions by operating in accordance with suitable program code, such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automotive and other embedded systems, cell phones and various other wireless devices, commonly denoted in this application as‘computer systems’.
• suitable program code such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automotive and other embedded systems, cell phones and various other wireless devices, commonly denoted in this application as‘computer systems’.
• computer systems various other wireless devices
• any reference signs placed between parentheses shall not be construed as limiting the claim.
• the word‘comprising’ does not exclude the presence of other elements or steps then those listed in a claim.
• the terms “a” or “an,” as used herein, are defined as one as or more than one.
• the use of introductory phrases such as “at least one " and “one or more " in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles "a " or “an " limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases “one or more " or “at least one " and indefinite articles such as "a " or “an.
• the invention may also be implemented in a computer program for running on a computer system, at least including code portions for performing steps of a method according to the invention when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the invention.
• the computer program may cause the storage system to allocate disk drives to disk drive groups.
• a computer program is a list of instructions such as a particular application program and/or an operating system.
• the computer program may for instance include one or more of: a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, a shared library/dynamic load library and/or other sequence of instructions designed for execution on a computer system.
• the computer program may be stored internally on a non-transitory computer readable medium. All or some of the computer program may be provided on computer readable media permanently, removably or remotely coupled to an information processing system.
• the computer readable media may include, for example and without limitation, any number of the following: magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.
• a computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process.
• An operating system is the software that manages the sharing of the resources of a computer and provides programmers with an interface used to access those resources.
• An operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as a service to users and programs of the system.
• the computer system may for instance include at least one processing unit, associated memory and a number of input/output (I/O) devices.
• I/O input/output
• Any system referred to this patent application includes at least one hardware component.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Multimedia (AREA)
• Computational Linguistics (AREA)
• Signal Processing (AREA)
• Quality & Reliability (AREA)
• Circuit For Audible Band Transducer (AREA)

Priority Applications (9)

Applications Claiming Priority (1)

Publications (1)

Family

ID=72427785

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (7)

Citations (6)

Family Cites Families (91)

• 2019
  • 2019-03-10 JP JP2021553756A patent/JP7564117B2/ja active Active
  • 2019-03-10 KR KR1020217032319A patent/KR102789155B1/ko active Active
  • 2019-03-10 EP EP19918690.9A patent/EP3939035A4/en active Pending
  • 2019-03-10 KR KR1020257009801A patent/KR20250044808A/ko active Pending
  • 2019-03-10 WO PCT/IB2019/051933 patent/WO2020183219A1/en not_active Ceased
  • 2019-03-10 US US17/437,748 patent/US12148441B2/en active Active
  • 2019-03-10 CN CN202510264269.9A patent/CN120089153A/zh active Pending
  • 2019-03-10 CN CN201980096208.9A patent/CN113795881B/zh active Active
• 2024
  • 2024-09-26 JP JP2024167615A patent/JP2025000790A/ja active Pending

Patent Citations (6)

Also Published As

Similar Documents

Legal Events