WO2014147442A1 - Spatial audio apparatus - Google Patents

Spatial audio apparatus Download PDF

Info

Publication number
WO2014147442A1
WO2014147442A1 PCT/IB2013/052212 IB2013052212W WO2014147442A1 WO 2014147442 A1 WO2014147442 A1 WO 2014147442A1 IB 2013052212 W IB2013052212 W IB 2013052212W WO 2014147442 A1 WO2014147442 A1 WO 2014147442A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
model
spatial covariance
spatial
covariance matrix
Prior art date
Application number
PCT/IB2013/052212
Other languages
French (fr)
Inventor
Miikka Vilermo
Mikko Tammi
Joonas Nikunen
Tuomas Virtanen
Original Assignee
Nokia Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Corporation filed Critical Nokia Corporation
Priority to EP13878649.6A priority Critical patent/EP2976893A4/en
Priority to PCT/IB2013/052212 priority patent/WO2014147442A1/en
Priority to CN201380076395.7A priority patent/CN105230044A/en
Priority to US14/777,825 priority patent/US9788119B2/en
Publication of WO2014147442A1 publication Critical patent/WO2014147442A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/027—Spatial or constructional arrangements of microphones, e.g. in dummy heads
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0272—Voice signal separating
    • G10L21/028—Voice signal separating using properties of sound source
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00—Details of transducers, loudspeakers or microphones
    • H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
    • H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
    • H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
    • H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers, loudspeakers or microphones
    • H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2201/00—Details of transducers, loudspeakers or microphones covered by H04R1/00 but not provided for in any of its subgroups
    • H04R2201/40—Details of arrangements for obtaining desired directional characteristic by combining a number of identical transducers covered by H04R1/40 but not provided for in any of its subgroups
    • H04R2201/401—2D or 3D arrays of transducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2430/00—Signal processing covered by H04R, not provided for in its groups
    • H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
    • H04R2430/23—Direction finding using a sum-delay beam-former

Definitions

  • the present application relates to apparatus for spatial audio signal processing.
  • the invention further relates to, but is not limited to, apparatus for spatial audio signal processing within mobile devices.
  • a stereo or multi-channel recording can be passed from the recording or capture apparatus to a listening apparatus and replayed using a suitable multi-channel output such as a multi-channel loudspeaker arrangement and with virtual surround processing a pair of stereo headphones or headset.
  • aspects of this application thus provide a spatial audio capture and processing whereby listening orientation or video and audio capture orientation differences can be compensated for.
  • a method for separating audio sources comprising: receiving at least two audio signals; transforming the at least two audio signals into a frequency domain representation of the at least two signals; generating an observed spatial covariance matrix from the frequency domain representations of the at ieast two audio signals; generating a spatial covariance matrix model comprising at Ieast one beamformer kernel; generating a linear magnitude model of audio objects; combining the spatial covariance matrix model and the linear magnitude model; determining at Ieast one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and clustering the audio objects based on the at Ieast one combination parameter to create separated audio sources.
  • Receiving the at Ieast two audio signals may comprise at Ieast one of: the audio signals being generated from microphones receiving acoustic waves from at Ieast one audio source; and the audio signals being received from a memory, the audio signals describing acoustic waves from at Ieast one audio source.
  • Transforming the at Ieast two audio signals into a frequency domain representation of the at Ieast two signals may comprise performing a time to frequency transform on the at Ieast two audio signals.
  • Generating the observed spatial covariance matrix from the frequency domain representations of the at Ieast two audio signals may comprise generating an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
  • the method may further comprise generating at Ieast one beamformer kernel by generating time delays in a frequency domain of a set of took directions sampling the spatial space around the microphone array.
  • Generating at Ieast one beamformer kernel may comprise generating an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • Generating a linear model of audio objects may comprise performing a non-negative matrix factorization for audio object magnitudes.
  • Performing a non-negative matrix factorization for audio object magnitudes may comprise initialising linear model parameters and 3 ⁇ 4 ⁇ with random values uniformly distributed between zero and one;
  • Determining the spatial covariance model parameters may comprise performing an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at Ieast one audio object are iteratively updated until either the linear magnitude modei parameters do not change significantly between updates or a defined number of iterations are processed,
  • Performing the iterative optimization based on the comp ex value non-negative factorization may comprise: updating linear model parameters z k0 , t ik and 3 ⁇ 4? according to an optimization framework of complex non-negative matrix factorization; and updating the spatial covariance matrix modei magnitudes.
  • Updating the spatial covariance matrix model magnitudes may comprise updating the relative magnitude difference of estimates of the at Ieast two audio signals.
  • Clustering the audio objects based on the at Ieast one combination model parameter to create separated audio sources may comprise generating at Ieast one audio source from the at Ieast one audio object and from the estimated linear magnitude model parameters.
  • the method may further comprise synthesising at least one audio signal based on the at least one combination model parameters associated with the at least one audio source.
  • an apparatus comprising: means for receiving at least two audio signals; means for transforming the at least two audio signals into a frequency domain representation of the at least two signals; means for generating an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals: means for generating a spatial covariance matrix model comprising at least one beamformer kernel; means for generating a linear magnitude model of audio objects; means for combining the spatial covariance matrix model and the linear magnitude model; means for determining at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and means for clustering the audio objects based on the at least one combination parameter to create separated audio sources.
  • the means for receiving the at least two audio signals may comprise at least one of: the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and the audio signals being received from a memory, the audio signals describing acoustic waves from at least one audio source.
  • the means for transforming the at least two audio signals into a frequency domain representation of the at least two signals may comprise means for performing a time to frequency transform on the at least two audio signals.
  • the means for generating the observed spatial covariance matrix from the frequency domain representations of the at least two audio signals may comprise generating an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
  • the apparatus may further comprise means for generating at least one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • the means for generating at least one beamformer kernel may comprise generating an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • the means for generating a linear model of audio objects may comprise performing a non-negative matrix factorization for audio object magnitudes.
  • the means for performing a non-negative matrix factorization for audio object magnitudes may comprise means for initialising linear model parameters and v ki with random values uniformly distributed between zero and one;
  • the means for combining spatial covariance matrix model and the linear magnitude model may comprise means for multiplying the spatial covariance matrix model by a magnitude envelope represented by the linear model,
  • the means for determining at least one combination parameter may comprise determining spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model.
  • the means for determining the spatial covariance model parameters may comprise performing an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at least one audio object are iteratively updated until either the linear magnitude model parameters do not change significantly between updates or a defined number of iterations are processed.
  • the means for performing the iterative optimization based on the complex value non- negative factorization may comprise: means for updating linear model parameters zko. k and 3 ⁇ 4 3 ⁇ 4 according to an optimization framework of complex non-negative matrix factorization; and means for updating the spatial covariance matrix model magnitudes.
  • the means for updating the spatial covariance matrix model magnitudes may comprise means for updating the relative magnitude difference of estimates of the at least two audio signals,
  • the means for clustering the audio objects based on the at least one combination model parameter to create separated audio sources may comprise means for generating at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters.
  • the apparatus may further comprise means for synthesising at least one audio signal based on the at least one combination model parameters associated with the at least one audio source.
  • an apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least: receive at least two audio signals; transform the at least two audio signals into a frequency domain representation of the at least two signals; generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; generate a spatial covariance matrix model comprising at least one beamformer kernel; generate a linear magnitude model of audio objects; combine the spatial covariance matrix model and the linear magnitude model; determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and clustering the audio objects based on the at least one combination parameter to create separated audio sources.
  • Receiving the at least two audio signals may cause the apparatus perform at least one of; receive the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and receive the audio signals from a memory, the audio signals describing acoustic waves from at teast one audio source.
  • Transforming the at least two audio signals into a frequency domain representation of the at least two signals may cause the apparatus to perform a time to frequency transform on the at least two audio signals.
  • Generating the observed spatial covariance matrix from the frequency domain representations of the at least two audio signals may cause the apparatus to generate an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
  • the apparatus may further be causes to generate at teast one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • Generating at least one beamformer kernel may cause the apparatus to generate an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • Generating a linear model of audio objects may cause the apparatus to perform a non-negative matrix factorization for audio object magnitudes.
  • Performing a non-negative matrix factorization for audio object magnitudes may cause the apparatus to initialise linear model parameters and %? with random values uniformly distributed between zero and one;
  • Performing the iterative optimization based on the complex value non-negative factorization may cause the apparatus to: update linear model parameters 3 ⁇ 4 0; 3 ⁇ 4 and v kj according to an optimization framework of complex non-negative matrix factorization; and update the spatial covariance matrix model magnitudes.
  • Updating the spatial covariance matrix model magnitudes may cause the apparatus to update the relative magnitude difference of estimates of the at least two audio signals.
  • Clustering the audio objects based on the at least one combination model parameter to create separated audio sources may cause the apparatus to generate at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters,
  • an apparatus comprising: an input configured to receive at least two audio signals; a frequency domain transformer configured to transform the at least two audio signals into a frequency domain representation of the at least two signals; a spatial covariance processor configured to generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; a beamformer configured to generate a spatial covariance matrix model comprising at least one beamformer kernel; a matrix factorizer configured to generate a linear magnitude model of audio objects; to combine the spatial covariance matrix model and the linear magnitude model; and further configured to determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and a separator configured to cluster the audio objects based on the at least one combination parameter to create separated audio sources.
  • the input may comprise at least one of: at least one microphone input configured to receive the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and at least one memory input configured to receive the audio signals from a memory, the audio signals describing acoustic waves from at least one audio source.
  • the frequency domain transformer may be configured to perform a time to frequency transform on the at least two audio signals.
  • the spatial covariance processor may be configured to generate an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
  • the beamformer may be configured to generate at least one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • the beamformer may be configured to generate an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
  • the matrix factorizer may be configured to perform a non-negative matrix factorization for audio object magnitudes.
  • the matrix factorizer may be configured to initialise linear model parameters and 3 ⁇ 4j with random values uniformly distributed between zero and one;
  • the matrix factorizer configured to combine the spatiai covariance matrix model and the linear magnitude model may be configured to multiply the spatial covariance matrix model by a magnitude envelope represented by the linear modei.
  • the matrix factorizer configured to determining at least one combination parameter may be configured to determine spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model.
  • the matrix factorizer configured to determine the spatial covariance model parameters may be configured to perform an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at ieast one audio object are iteratively updated until either the linear magnitude model parameters do not change significantly between updates or a defined number of iterations are processed.
  • the matrix factorizer configured to perform the iterative optimization based on the complex value non-negative factorization may be configured to: update linear model parameters 3 ⁇ 4> % and %/ according to an optimization framework of complex non- negative matrix factorization; and update the spatial covariance matrix model magnitudes.
  • the matrix factorizer configured to update the spatial covariance matrix model magnitudes may be configured to update the relative magnitude difference of estimates of the at Ieast two audio signals.
  • the separator configured to cluster the audio objects based on the at Ieast one combination model parameter to create separated audio sources may be configured to generate at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters.
  • the apparatus may further comprise a spatial synthesizer configured to synthesise at least one audio signal based on the at least one combination mode! parameters associated with the at least one audio source.
  • a computer program product stored on a medium may cause an apparatus to perform the method as described herein,
  • An electronic device may comprise apparatus as described herein,
  • a chipset may comprise apparatus as described herein.
  • Embodiments of the present application aim to address problems associated with the state of the art.
  • Figure 1 shows schematically an apparatus suitable for being employed in some embodiments
  • FIG. 2 shows schematically an example audio signal processing apparatus according to some embodiments
  • Figure 3 shows schematically a flow diagram of the operation of the audio signal processing apparatus shown in Figure 2 according to some embodiments
  • Figure 4 shows schematically example microphone configurations on an apparatus according to some embodiments
  • Figure 5 shows example array geometry of an apparatus comprising two microphones
  • Figure 6 shows example beam directions approximating uniform sampling of the unit sphere around the array
  • Figure 7 shows schematically a fiow diagram of the operation of parameter estimation according to some embodiments
  • Figure 8 shows schematically the generai principle of the clustering of CNMF components based on spatial weights according to some embodiments
  • Figure 9 shows schematically the spatial synthesiser as shown in Figure 2 configured to generate a binaural output according to some embodiments
  • Figure 10 shows schematically a flow diagram showing the operation of the spatial synthesiser as shown in Figure 9 configured to generate a binaural output according to some embodiments;
  • Figure 11 shows graphical plots of spatial weights analysed by an example of some embodiments as pictured from above the array where original source positioning was on the lateral plane at azimuths 0 (left) and 90 (bottom) degrees;
  • Figure 12 shows a graphicai plot of an example capturing array of microphones used in some embodiments where microphones are denoted by circles and the zero angle references axis by an arrow;
  • Figure 13 shows a graphical plot of source positioning in embodiment exampie 1 ;
  • Figure 14 shows a graphical plot of source positioning in embodiment example 2;
  • Figure 15 shows a graphical plot of source positioning in embodiment example 3;
  • Figures 18 and 17 show graphical plots of example outputs generated by source processing according to some embodiments.
  • mobile devices or apparatus are more commonly being equipped with multiple microphone configurations or microphone arrays suitable for recording or capturing the audio environment or audio scene surrounding the mobile device or apparatus.
  • a multiple microphone configuration enables the recording of stereo or surround sound signals and the known location and orientation of the microphones further enables the apparatus to process the captured or recorded audio signals from the microphones to perform spatial processing to emphasise or focus on the audio signals from a defined direction relative to other directions.
  • One way to perform spatial processing is to initially extract and manipulate the direction or sound source dependent information so to use this information in subsequent applications.
  • These applications can include for example, spatial audio coding (SAC), 3D soundfield analysis and synthesis, sound source separation and speakers extraction for further processing such as speech recognition.
  • SAC spatial audio coding
  • 3D soundfield analysis and synthesis sound source separation and speakers extraction for further processing such as speech recognition.
  • BSS blind source separation
  • a classic example of such a case is known as the cocktail party problem enabling the separation of each individual speaker from the party recorded using a microphone array.
  • the field of BSS has been intensively studied, but is still categorized as an unsolved problem.
  • the capturing or recording apparatus or device usually consists of a small hand held device having multiple microphones. The multiple channels and their information correlation and relationship can then be utilized for source separation and direction of arrival estimation.
  • Beamforming for source direction of arrival detection and related problems and more recently spherical array beamforming techniques have been successfully used in sound field capture and analysis and also developed into final products such as Eigenmike.
  • the problem with spherical array processing is the array structure and the sheer size of the actual arrays used prevent it to be incorporated into a single mobile device.
  • pure beamforming does not address the problem of source separation but analyses the spatial space around the device with beams as narrow as possible.
  • the sidelobe cancellation for decreasing the beam width generally requires increasing the microphone count of the array, which is as discussed costly in volume, device complexity and cost of manufacture.
  • ICA independent component analysis
  • !CA independent component analysis
  • ICA independent component analysis
  • the frequency-domain ICA leads to an arbitrary source ordering at each frequency.
  • This permutation ambiguity has been solved by different means over the years, mixing filter frequency response smoothness, temporal structure of the source signals, and time-difference of arrival (TDoA) and direction of arrival (DoA), and interpretation of ICA mixing parameters.
  • TDoA time-difference of arrival
  • DoA direction of arrival
  • ICA-based methods that avoid the permutation problem by unifying the source independencies across frequencies.
  • ICA based separation is one of the methods which is sensitive to problems caused by spatial aliasing in permutation alignment and in unifying the source independencies over frequency.
  • non-negative matrix factorization based separation in multichannel cases.
  • These include for example multichannel NMF for convoluted mixtures, however the E -a!gorithm used for parameter estimation is inefficient without oracle initialization, (in other words knowing source characteristics for initializing the algorithm).
  • Complex multichannel NMF (CNMF) with multiplicative updates has been proposed with promising separation results.
  • the proposed CNMF algorithms estimate the source spatial covariance properties and the magnitude model. However the spatial covariance matrices are estimated and updated individually for each frequency bin making the algorithm prone for estimation errors at high frequencies with spatial aliasing. Also the estimated covariance properties are not connected to the spatial locations of the sources.
  • the problem includes solving and executing 3D sound synthesis of the separated sources. It would further be understood that where spatial processing is performed with respect to spatial audio synthesis, such as 3D audio synthesis, the 3D synthesis of the separated sources or parts of the sources require pairing the separation algorithm with DoA analysis making the system potentially discontinuous and less efficient for the 3D sound scene analysis-synthesis loop. As such an enclosed microphone array with an unknown directivity pattern of each capsule requires a machine learning based algorithm for learning and compensating the unknown properties of the array.
  • the audio recording system provides apparatus and/or methods for separating audio sources 18 using several microphones in one device.
  • sounds from different audio sources arrive at different times to the different microphones.
  • each audio source has individual spectral features.
  • the concepts thus discuss herein that the apparatus and methods analyse microphone signals so as to determine a combination of the time-differences and spectral features.
  • the effect of such embodiments is the sound sources that are separated from a mixture of audio sources.
  • Figure 1 shows a schematic block diagram of an exemplary apparatus or electronic device 10, which may be used to record (or operate as a capture apparatus).
  • the electronic device 10 may for example be a mobile terminal or user equipment of a wireless communication system when functioning as the recording apparatus or listening apparatus.
  • the apparatus can be an audio player or audio recorder, such as an MPS player, a media recorder/player (also known as an MP4 player), or any suitable portable apparatus suitable for recording audio or audio/ ideo camcorder/memory audio or video recorder.
  • the apparatus 10 can in some embodiments comprise an audio-video subsystem.
  • the audio-video subsystem for example can comprise in some embodiments a microphone or array of microphones 11 for audio signal capture.
  • the microphone or array of microphones can be a solid state microphone, in other words capable of capturing audio signals and outputting a suitable digital format signal in other words not requiring an analogue-to-digital converter
  • the microphone or array of microphones 11 can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, Electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or micro electrical-mechanical system (MEMS) microphone.
  • the microphone 11 or array of microphones can in some embodiments output the audio captured signal to an analogue-to-digital converter (ADC) 14.
  • ADC analogue-to-digital converter
  • the apparatus can further comprise an analogue-to-digital converter (ADC) 14 configured to receive the analogue captured audio signal from the microphones and outputting the audio captured signal in a suitable digital form.
  • ADC analogue-to-digital converter
  • the analogue-to-digitai converter 14 can be any suitable analogue-to-digital conversion or processing means.
  • the apparatus 10 audio-video subsystem further comprises a digital-to-analogue converter 32 for converting digital audio signals from a processor 21 to a suitable analogue format.
  • the digital-to-analogue converter (DAC) or signal processing means 32 can in some embodiments be any suitable DAC technology.
  • the audio-video subsystem can comprise in some embodiments a speaker 33.
  • the speaker 33 can in some embodiments receive the output from the digital-to-analogue converter 32 and present the analogue audio signal to the user.
  • the speaker 33 can be representative of multi-speaker arrangement, a headset, for example a set of headphones, or cordless headphones.
  • the apparatus audio-video subsystem comprises a camera 51 or image capturing means configured to supply to the processor 21 image data.
  • the camera can be configured to supply multiple images over time to provide a video stream.
  • the apparatus audio-video subsystem comprises a display 52
  • the display or image display means can be configured to output visual images which can be viewed by the user of the apparatus.
  • the display can be a touch screen display suitable for supplying input data to the apparatus.
  • the display can be any suitable display technology, for example the display can be implemented by a flat panel comprising cells of LCD, LED. OLED, or 'plasma' display implementations.
  • the apparatus 10 is shown having both audio/video capture and audio/video presentation components, it would be understood that in some embodiments the apparatus 10 can comprise only the audio capture and audio presentation parts of the audio subsystem such that in some embodiments of the apparatus the microphone (for audio capture) or the speaker (for audio presentation) are present. Similarly in some embodiments the apparatus 10 can comprise one or the other of the video capture and video presentation parts of the video subsystem such that in some embodiments the camera 51 (for video capture) or the display 52 (for video presentation) is present.
  • the apparatus 10 comprises a processor 21.
  • the processor 21 is coupled to the audio-video subsystem and specifically in some examples the analogue-to-digital converter 14 for receiving digital signals representing audio signals from the microphone 11 , the digital-to-analogue converter (DAG) 12 configured to output processed digital audio signals, the camera 51 for receiving digital signals representing video signals, and the display 52 configured to output processed digital video signals from the processor 21.
  • DAG digital-to-analogue converter
  • the processor 21 can be configured to execute various program codes.
  • the implemented program codes can comprise for example audio-video recording and audio-video presentation routines.
  • the program codes can be configured to perform audio signal processing.
  • the apparatus further comprises a memory 22.
  • the processor is coupled to memory 22.
  • the memory can be any suitable storage means.
  • the memory 22 comprises a program code section 23 for storing program codes implementabie upon the processor 21.
  • the memory 22 can further comprise a stored data section 24 for storing data, for example data that has been encoded in accordance with the application or data to be encoded via the application embodiments as described later.
  • the implemented program code stored within the program code section 23, and the data stored within the stored data section 24 can be retrieved by the processor 21 whenever needed via the memory-processor coupling.
  • the apparatus 10 can comprise a user interface 15,
  • the user interface 15 can be coupled in some embodiments to the processor 21.
  • the processor can control the operation of the user interface and receive inputs from the user interface 15.
  • the user interface 15 can enable a user to input commands to the electronic device or apparatus 10, for example via a keypad, and/or to obtain information from the apparatus 10, for example via a display which is part of the user interface 15.
  • the user interface 15 can in some embodiments as described herein comprise a touch screen or touch interface capable of both enabling information to be entered to the apparatus 10 and further displaying information to the user of the apparatus 10.
  • the apparatus further comprises a transceiver 13, the transceiver in such embodiments can be coupled to the processor and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network.
  • the transceiver 13 or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.
  • the transceiver 13 can communicate with further apparatus by any suitable known communications protocol, for example in some embodiments the transceiver 13 or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).
  • UMTS universal mobile telecommunications system
  • WLAN wireless local area network
  • IRDA infrared data communication pathway
  • the apparatus comprises a position sensor 18 configured to estimate the position of the apparatus 10.
  • the position sensor 16 can in some embodiments be a satellite positioning sensor such as a GPS (Global Positioning System), GLONASS or Galileo receiver. In some embodiments the positioning sensor can be a ce!iular ID system assisted GPS system.
  • the apparatus 10 further comprises a direction or orientation sensor.
  • the orientation/direction sensor can in some embodiments be an electronic compass, accelerometer, and a gyroscope or be determined by the motion of the apparatus using the positioning estimate.
  • the apparatus 10 comprises a front face 301 comprising a camera 51 , a rear face 303 and a top edge or face 305.
  • the apparatus comprises four microphones a first (front right) microphone 11 3 located at the front right side of the apparatus where right is towards the top edge of the front face of the apparatus, a front left microphone 113 located at the front left side of the apparatus, a right high microphone 112 located at the top edge or face side of the apparatus, and a left rear microphone 11 located at the left rear side of the apparatus.
  • FIG. 2 an example audio signal processing apparatus according to some embodiments is shown. Furthermore with respect to Figure 3 a flow diagram of the operation of the audio signal processing apparatus as shown in Figure 2 is shown.
  • the apparatus comprises the microphone or array of microphones configured to capture or record the acoustic waves and generate an audio signal for each microphone which is passed or input to the audio signal processing apparatus.
  • the microphones 11 are configured to output an analogue signal which is converted into a digital format by the analogue to digital converter (ADC) 14.
  • ADC analogue to digital converter
  • the microphones shown in the example herein are integrated microphones configured to output a digital format signal directly to a short time Fourier transformer (STFT) 101.
  • STFT short time Fourier transformer
  • the microphones are part of the apparatus it would be understood that in some embodiments the microphone array is physically separate from the apparatus, for example the microphone array can be located on a headset (where the headset also has an associated video camera capturing the video images which can also be passed to the apparatus and processed in a manner to generate an encoded video signal which can incorporate the processed audio signals as described herein) which wireiessly or otherwise passes the audio signals to the apparatus for processing.
  • the headset also has an associated video camera capturing the video images which can also be passed to the apparatus and processed in a manner to generate an encoded video signal which can incorporate the processed audio signals as described herein
  • the apparatus comprises a short time Fourier transformer (STFT) 101 .
  • STFT short time Fourier transformer
  • the short time Fourier transformer 1 01 is configured to perform a short time Fourier transform on the microphone input audio signais.
  • the 'mixing' model can be approximated in the STFT-domain as,
  • STFT short-time Fourier transform
  • the spatial response filter ⁇ (r) has a length of severai hundreds of milliseconds depending on the reverberation time of the capturing environment, but approximation of it by ik in the frequency domain using an analysis window size of tens of milliseconds has been found to work well in practice due to the negligible energy after the main reverberant part of the source spatial response.
  • the STFT is configured to output the transformed domain signals to a spatial covariance processor 1 03.
  • the operation of applying a SIFT to the time domain signals is shown in Figure 3 by step 301 .
  • the apparatus comprises a spatial covariance processor 103.
  • the spatial covariance processor In order to avoid estimating and modelling the absolute phase of each channel the spatial covariance processor generates a spatial covariance matrix for each time- frequency point as the signal representation. In such embodiments the spatial covariance calculation translates the absolute phase to a phase delay between each microphone pair which is employed as a signal representation input for BSS algorithm described herein. In some embodiments the calculation of the SCMs determines a magnitude square-rooted version of the array capture. The determination of the magnitude square rooted version of the array capture ensures that the nonnegative part in the diagonal of the spatial covariance matrix (SCM), contains the STFT magnitudes, in some embodiments an unmodified STFT of the capture array output can be used to produce a power spectrum on the diagonal of the SCM.
  • SCM spatial covariance matrix
  • the magnitude square-rooted STFT is used as a starting point the SCMs for each time-frequency point derived by the following method.
  • the spatial covariance processor 103 can be configured to determine the SCM for a single time-frequency point from the array capture vector * ::: ⁇ * ⁇ ⁇
  • the spatial covariance processor 103 can be configured to calculate the SCM for each time frequency point *, i and which can be denoted as the matrix with the following subindexing x y e c M*M ,
  • the complex-valued monoaural source spectrogram in the SCM domain is equal to a real valued power spectrum where c denotes complex conjugate of complex number c, In some embodiments because of the square-rooted STFT used to calculate SCMs hereafter the sources are denoted using their magnitude spectrum
  • the spatial covariance signal representation replaces the mixing vector h ik in the original model by mixing matrix 3 ⁇ 4& e c. Mx M which denotes the source spatial response expressed in the form of a covariance matrix >3 ⁇ 4 ii
  • the mixing matrix is still defined for each STFT frequency bin % individually.
  • the SCM of each source s w at each frequency can be modelled using a Hermitian positive semidefinite matrix 3 ⁇ 4 approximating the true but unknown SCM.
  • the mixture SCM and source magnitudes are only perceived and are yet to be estimated. Estimating the source magnitude details 1 ⁇ 2* and the corresponding mixing denoted by 3 ⁇ 4* by turns in such embodiments could provide the desired BSS properties, however estimating 3 ⁇ 4 in such a way that it corresponds to a single source over all frequencies is difficult in general.
  • the spatial properties being individually estimated for each STFT frequency bin i does not utilize the fact that the covariance properties are connected by Time Difference of Arrival (TDoA) of the direct path and early reflections.
  • TDoA Time Difference of Arrival
  • the model in such embodiments relies on the fact that the Non-negative matrix factorization (MF) model described herein in more detail enforces to correspond to a single acoustical source and the phase delay evidence at each frequency % averaged over time yields to correct estimate of the true spatial properties.
  • MF Non-negative matrix factorization
  • the spatial covariance processor is configured to output the transformed SCM signals to a complex non-negative matrix factorizer 105.
  • the apparatus comprises a beamformer generator 107.
  • the beamforrner is configured in some embodiments to generate at least one beamforming matrix for application to the spatial covariance matrix by the non- negative matrix factorizer 105.
  • ICA independent component analysis
  • CNMF complex valued non- negative matrix factorization
  • the source parameters are estimated and optimized for each frequency bin independently.
  • DoA Direction of Arrival
  • DoA estimation approaches a single source position is linked to a single Time Difference of Arrival (TDoA) between the microphones assuming anechoic propagation or in case of reverberant conditions considering only its direct path propagation.
  • TDoA Time Difference of Arrival
  • the single TDoA unifies the phase delay over frequency making the DoA methods able to integrate evidence over all frequencies to obtain the estimate of DoA over the whole frequency spectrum making the source estimation more robust.
  • 8SS Blind Source Separation
  • the TDoAs of certain spatial position are used for aligning the received signal and for searching for beam direction, in other words the source DoA, that produces the highest beamformer output energy.
  • This is equivalent to making a hypothesis of all possible source positions and searching for TDoAs that give the time alignment of the microphone signals that generate the highest signal power when the aligned signals are summed together.
  • a similar method can be used to unify STFT bin dependencies for estimating the source spatial response in any generic BSS method suffering from said problem of optimizing the model parameters individually regarding frequencies. The difference is merely that in BSS the optimization algorithm is set to fit a collection of TDoA kernels (which we call beamforming kernels in the following embodiments) to the observed data and that way finding the most likely DoA of the source in question,
  • TDoA delay and sum beamforming
  • Figure 5 shows an example microphone array where one pair of microphones n 401 and m 403 He on the xy-plane at locations n and m, respectively, and an unit vector k 405 is pointing towards the source from the geometrical center p 407 of the array.
  • the direction vector k 405 is defined in Cartesian coordinates.
  • P [O, o, oj.
  • the reference axis of the array, zero elevation ⁇ - o° and zero azimuth ⁇ can be arbitrarily set to point towards any direction in the spatial0 space.
  • the TDoA ⁇ x rm ⁇ can be converted to a phase delay in the frequency domain as,
  • the beamforming kernels can in some embodiments be output by the beamformer 107 to the complex non-negative matrix factorizer.
  • the beamformer 107 can be configured to define a set of look directions vectors, k memori where o ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . , . ⁇ , which spatially sample the surface of a unit sphere set around the geometrical center of the array.
  • look direction vectors 501 sample the 5 spatial space around the microphone array.
  • the beamforming kernels are denoted in each look direction at each frequency ⁇ ⁇ ⁇ . , ⁇ using w io e C M * M .
  • entries [w jo ] nm denote a TDoA in terms of phase difference expressed as complex number for a microphone pair (n, m).
  • the kernels are obtained from the equation above as
  • F S is the sampling frequency and i is the STFT frequency bin index in question and N is the STFT window length.
  • the generating of the beamforming kernels is shown in Figure 3 by step 305.
  • the apparatus comprises a complex non-negative matrix factorizer 105.
  • the complex non-negative matrix factorizer 105 is configured to receive the beamforming kernels from the beamformer and apply these to the spatial co variance matrix form of the received audio signals.
  • non-negative matrix factorization (CNMF) model employed in some embodiments as described herein comprises a non-negative matrix factorization (NMF) magnitude model core for modelling the source STFT magnitude details denoted by 3 ⁇ 43 ⁇ 4.
  • NMF non-negative matrix factorization
  • several non-negative matrix factorization (NMF) components are used for representing one acoustical sound source however in the examples described herein
  • one NMF component represents one sound source. This simplification enables a more intuitive explanation of the spatial covariance matrix (SCM) model of the audio signals.
  • SCM spatial covariance matrix
  • the complex non-negative matrix factorizer 105 is configured to estimate 3 ⁇ 4 in such way that they correspond to a single acoustical source, this is achieved in the embodiments described herein by estimating spatial weights 3 ⁇ 4 0 that are independent of frequency.
  • the definition of the DSBF kernels by the beamformer takes into account the frequency dependencies that the DoA of a certain source in form of TDoA causes.
  • the spatial weights z ko are furthermore restricted to be positive.
  • the complex non-negative matrix factorizer 105 given the SCM model attempts to minimize the squared Euclidean distance
  • the complex non-negative matrix factorizer 105 therefore given the SCM model uses a source model for covariance observations as
  • each source 3 ⁇ 43 ⁇ 4 can be modelled such that each source has a SCM defined according to,
  • N F non-negative matrix factorization
  • Each column vector contains the DFT spectrum of a source, and the corresponding row of v ⁇ represents its time-varying gain in each SIFT frame.
  • the scaling ensures that the SCM part is only responsible of modelling the phase delay and relative magnitude differences between input channels (diagonal values). Substituting the NMF model into the covariance model and rearranging the parameters generates the whole CNfvlF model,
  • w io are the beamforming kernels
  • 3 ⁇ 4 are the kernel weights for source k, i.e. directional weights.
  • the CN F modei can be generated using the source SCM by incorporating the single channel source magnitudes which generates the following model,
  • the delay and sum beamformer (DSBF) steering matrices generated by the beamformer produce unit magnitude for all values.
  • the complex non-negative matrix factorizer 105 is configured to estimate and update magnitudes of w tt , accordingly. This is because the sources are captured with varying gain with respect to each microphone, which is further caused by microphones having a different distance from the source and the possibie acoustic shade of the device producing directional dependent gain even if omnidirectionai microphones are used. While the spatial covariance matrix magnitudes are subject to optimization, the complex non-negative matrix factorizer 105 in some embodiments keeps the original beamforming phase difference the same, in other words the original phase shift caused by a certain direction of the source. In this way the dependency between different frequency components is retained when modelling the phase difference by only estimating the frequency independent spatial weights 3 ⁇ 4 «.
  • the complex non-negative matrix factorizer 105 is configured to employ multiplicative updates for iteratively deriving the NMF decomposition parameters minimize a given cost function, for example squared Euclidean distance or Kullback-Leibler divergence.
  • the multiplicative updates can be obtained from the partial derivatives of the cost function with respect to each estimated parameter.
  • the complex non-negative matrix factorizer 105 is configured to obtain the updates via auxiliary functions and Expectation Maximization (EM)-algorithm structure.
  • EM Expectation Maximization
  • A" O where denotes complex Gaussian distribution.
  • the Gaussian distribution matrix notation ⁇ -' denotes the distribution mean using ⁇ and the variance for each matrix entry is denoted using ⁇ 2 .
  • This statistical interpretation converts squared a Frobenius norm H x y used for measuring the modelling error into a maximum likelihood estimation of the mean of the Gaussian distribution with a fixed variance ⁇ 2 .
  • the complex non-negative matrix factorizer 105 can determine the negative log-likelihood of ⁇ / ⁇ ) which results in the following,
  • the complex non-negative matrix factorizer 105 can be configured to introduce unknown underlying components assumed to be mutually independent and modelled by individual Gaussian distributions.
  • the latent components denote the spatial covariance of an unknown gaussian variable. These latent components are defined such that,
  • the variance of each of these latent variables is 1 / K where ⁇ is the total number of sources. Furthermore the total likelihood with the latent components can be written as i j K a
  • the complex non-negative matrix factorizer 105 can define the auxiliary function as:
  • the complex non-negative matrix factorizer 105 additionally introduces the following constraints for numerical stability and to fix the scaling between parameters,
  • the complex non-negative matrix factorizer 105 applies a partial derivation of £ + (W,Z,T,VC) w ith respect to each model parameter and setting the derivative to zero,
  • the complex non-negative matrix factorizer 105 can redefine the likelihood function £ + (W,Z,T,VC) expanding the Frobenius form into the following form i,3A° rijk °
  • the complex non-negative matrix factorizer 105 can further be configured to update or optimize the DSBF kernels. This updating needs a different update scheme to that derived above, since the phase differences of the predefined beamforming kernels are to be retained.
  • the complex non- negative matrix factorizer 105 estimates the DSBF kerne! magnitudes by first deriving the update for complex w ift) but restrict the update to the magnitude only. Thus the model can be seen as magnitude dependent delay and sum beamformer. After the update each spatial covariance matrix is scaled into a unity Frobenius norm by applying w « ⁇ - w io /]
  • the update rule for the spatial covariance matrices w fo is obtained via partial derivation of £ ⁇ (W, Z, T, v c) resulting in
  • w io is a preliminary update with a modified phase delay compared to the actual desired update of w in . Due to the scaling defined in
  • the divisor in the above update can be disregarded and substituting with its definition k, the above update can be modified into the following multiplicative update
  • the complex non-negative matrix factorizer 105 can be configured to generate the update of w io by applying
  • the complex non-negative matrix factorizer 105 is configured to receive the spatial covariance matrices. Furthermore the complex non-negative matrix factorizer 105 is configured to initialize 3 ⁇ 4 0; 3 ⁇ 4 and v kj with random values uniformly distributed between zero and one.
  • the beamformer 107 is configured to initialize w iu according to
  • the complex non-negative matrix factorizer 105 is further configured to recalculate gnitude model 3 ⁇ 4 according to Xij ::: z—
  • r 105 is further configured to update t ik
  • the complex non-negative matrix factorizer 105 is further configured to recalculate
  • the complex non-negative matrix factorizer 105 is further configured to update v kj
  • the complex non-negative matrix factorizer 105 is further configured scale v kj to unity -norm and compensate by rescaling t ik as specified in
  • the operation of scaling 3 ⁇ 4 and ⁇ * is shown in Figure 7 by step 607.
  • the operation of recalculating 3 ⁇ 4 is shown in Figure 7 by step 608.
  • the complex non-negative matrix factorizer 105 is further configured to update z ko
  • the operation of updating z ko is shown in Figure 7 by step 609
  • the complex non-negative matrix factorizer 105 is further configured to scale 3 ⁇ 4 to i 2 - norm and compensate by rescaling t ik as specified in
  • the operation of scaling 3 ⁇ 4 victim and 3 ⁇ 4 is shown in Figure 7 by step 610.
  • the complex non-negative matrix factorizer 105 is further configured to recalculate
  • the complex non-negative matrix factorizer 105 is further configured to update w io
  • the complex non-negative matrix factorizer 105 is further configured to enforce w io to be positive semidefinite by w io VDV-" and apply w io ⁇ ; ⁇ w « ,/
  • the complex non-negative matrix factorizer 105 is further configured to repeat steps 803 to 613 for a determined number of iterations or until the parameter updates converge and do not change significantly between updates.
  • step 814 The operation of performing the looping back to step 603 is shown in Figure 7 by step 814,
  • the complex non-negative matrix factorizer 105 is further configured to output the model parameters.
  • model parameters k 3 ⁇ 4 3 ⁇ 4 0 w lo is shown in Figure 7 by step 815.
  • the model parameters are output to the separator 109.
  • the apparatus comprises a separator 109.
  • the separator 109 in some embodiments is configured to receive the CNMF components and the array input and to separate the audio signals into individual sources.
  • the separation to real acoustic sources therefore requires clustering the complex value non-negative matrix factorized (CNMF) components that were earlier interpreted as individual sources.
  • CNMF complex value non-negative matrix factorized
  • the CNMF components originating from the same source share very similar spatial covariance properties determined by the spatial weights This is illustrated in Figure 8 where it is shown that the target sources 1 :Q 701 spatial responses 702 are reflected in the array 703 which as described herein is factorized into NMF components 705 which are estimated and clustered 708 to generate the estimated or reconstructed sources 1 :Q 707.
  • the clustering is based on the spatial weight similarity and separate clustering algorithms can be used for solving the CNMF component associations to actual acoustic sources mapping.
  • k-means clustering applied on the spatial weights 3 ⁇ 4o are used for testing and evaluating the separation quality.
  • the separator 109 is configured so that each ⁇ . acts as a feature vector and k-means clustering is applied with the desired cluster count being equal to the total number of original sound sources Q.
  • a cluster decision is reached that component k belongs to a source g and the cluster decision is denoted by binary parameter b ⁇ .
  • the CNMF magnitude model for a single source magnitude spectrogram can be then defined as h
  • any other clustering algorithm or CNMF component to source linking strategy can be used to estimate either a binary or a soft decision h ⁇ and the choice to use the k-means clustering applied on the spatial weights is an example to demonstrate the DoA analysis and modelling performance of the proposed embodiments.
  • other features from the CN F parameters such as spectral similarity and gain behaviour over time, can be used in parallel for associating the CNMF components to the sources.
  • the separator 109 in some embodiments is configured to output the clustered sources to a spatial synthesizer 111.
  • the apparatus comprises a spatial synthesizer 111.
  • the spatial synthesizer 111 is configured to receive the clustered source parameters to regenerate the source signals.
  • the reconstruction of the source signals employs a Wiener filter.
  • the source reconstruction filtering is given with the CNMF component clustering decision b kk as,
  • the time-domain sources are obtained from inverse STFT (iSTFT).
  • FIG. 9 an example spatial synthesizer 111 according to some embodiments is shown. Furthermore with respect to Figure 10 the operation of the spatial synthesizer shown in Figure 9 is shown.
  • the spatial synthesizer 111 comprises an inverse short time Fourier transformer (iSTFT) 801 configured to apply an inverse short time Fourier transform to the clustered CNMF parameters output by the separator 109.
  • the inverse short time Fourier transformed parameters are in some embodiments output to a head related Impulse response (HRIR) filter 803,
  • HRIR head related Impulse response
  • step 901 The operation of applying an iSTFT to the clustered CNMF parameters is shown in Figure 10 by step 901 .
  • the apparatus comprises a HRIR filter 803 configured to receive the iSTFT clustered CNMF parameters. Furthermore in some embodiments the HRIR filter 803 is configured to receive original STFT observations K n and construct a Wiener-filter from the CNMF magnitude model parameters to fiiter out the source spectral details in STFT domain.
  • the source reconstruction filtering is given in some embodiments as,
  • the time-domain signals are obtained in some embodiments from the ISTFT 801 .
  • the HRIR filter 803 is configured to chosen as the dominant source direction the direction determined by the beams with the highest power and a single HRTF filter associated with this direction used.
  • other binaural spatial synthesis schemes can be employed , for example using several dominant directions in HRIR filtering stages for each source.
  • the spatial synthesizer 111 comprises a HRTF database configured to store the retrieved HRIR filter components.
  • the individual spatial covariance for each CNMF component is interpreted as the direction of a single source and the CNMF component can be directly used for the spatial synthesis, in case where only 3D sound synthesis is desired, no separation stage is needed but each CNMF component is synthesized individually and they are positioned to their analysed spatial location with one of the above defined spatial sound field synthesis approaches.
  • the associated spatial weights are used to determine which HRTFs are used in form of head related impulse response (HRIR) to filter the separated sources. This simulates source positioning by binaural cues such as time- difference of arrival and direction dependent frequency filter caused by the pinna.
  • HRIR head related impulse response
  • the HRIR filter is configured to output the filtered components to a combiner 805.
  • the spatial synthesizer 111 comprises a combiner 805 configured to sum or combine the HRIR filter sources. In other words in some embodiments after each source is filtered with its associated direction dependent binaural filter (HRIR) all the individual sources are summed together to form a binaural synthesis signal.
  • HRIR direction dependent binaural filter
  • step 905 The operation of combining (sum) the filter source is shown in Figure 10 by step 905. It would be understood that any other spatial sound field synthesis technique based on knowing source signals and their multidimensional source positioning can be used, For example in some embodiments the spatial sound field synthesis can be performed by vector base amplitude panning.
  • step 313 The operation of outputting an output (for example a binaural stereo output) is shown in Figures 3 (and 9) by step 313.
  • sparsity can be enforced to the directional weights 2& 0 which can be used to guide the algorithm to find spatial covariances for sources that have point-like nature and do not include ambient content in form of reverberation, in some embodiments this can be employed to decrease the cross-talk between the separated sources.
  • the CN F component association to sources can be estimated directly.
  • the clustering decision is a non- negative parameter that can be included in the model and an update rule based on the partial derivation can be obtained.
  • the SCMs are estimated with respect to each real acoustic sound source and similarly to the presented model the frequency dependency using DSBFs is maintained. In the following the separation performance of some embodiments is demonstrated by comparing the results of embodiments against conventional BSS methods suitable for the particular case of small microphone array captures with a reasonable amount of reverberation.
  • the separation performance is determined by the separation quality measures, the signal-to-distortion ratio (SDR), image-to-spatial distortion ratio (ISR), signal-to-interference ratio (SIR) and signal-to-artifact ratio (SAR) such as provided by a BSS Eval toolbox
  • SDR signal-to-distortion ratio
  • ISR image-to-spatial distortion ratio
  • SIR signal-to-interference ratio
  • SAR signal-to-artifact ratio
  • the sampling frequency used was 48kHz, the room dimensions were approximately 5 x 12 x 2.59 m and the reverberation time (T60) was approximately 0.35s.
  • the capture array in the following examples consists of four DPA 4080-BM prepoiarized omnidirectional miniature condensator microphones. The array dimensions are given by the following table x (sum)
  • the spatial aliasing frequency for the given array is 1563 Hz.
  • the signal length for the embodiments following examples to analyze was set to 10 seconds in each test case, the STFT window length was 2048 samples with 50% window overlap.
  • the determined number of iterations for non-negative matrix factorization in the following examples was 200 iterations.
  • the impulse responses were captured from following angles [0, 40, 90. 135, 180, 305] 0 f which a several combinations with different source types (speech, music and noise) were chosen for the evaluation. After convolving the impulse responses the signals were downsampled to 24kHz.
  • the spatial sampling defined by all look direction vectors for different DSBFs are illustrated in Figure 6 which comprises 110 beam directions which approximately uniformly sample the unit sphere surface around the array.
  • the lateral resolution at zero elevation in other words at the flat surface of the array, is 10 degrees and the different elevations are at 22.5 degrees spacing.
  • the azimuth resolution in such embodiments is decreased close to the poles of the unit sphere.
  • the separation performance was compared to the CNMF method proposed herein.
  • test cases a total of five 10 second signals were tested and separation measures were averaged over ail test signals and all sources within the test case.
  • the achieved separation performance indicates that the embodiments as shown in the example beamforming based spatial covariance matrix estimation for source separation systematically increases the separation compared to the reference that is also based on spatial covariance estimation. Furthermore it would be understood that the separation performance improvement is greatest with speech sources with significant azimuth difference and decreases with difficult broadband noises such as artificial pink noise.
  • the apparatus and for example the spatial synthesizer can be configured to process the modelled audio sources according to any suitable means.
  • Figures 18 and 17 an example processing is shown wherein in Figure 16 an example of a male speaker and two other speakers are present, in this exampie the microphones are omnidirectional and ali speakers are present in both channels. However the male speaker is to the right of the microphone array and therefore his voice is dominant in the right channel (right channel-the lower of the two channels).
  • the graph shows where the male speaker is removed by removing the audio source estimated according to the embodiments described herein.
  • the male speaker is attenuated by approximately 12 dB in the right channel without affecting the sound from other speakers that dominate the left channel.
  • user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers, as well as wearable devices.
  • the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.
  • some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be impiemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • the embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware, Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions.
  • the software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
  • the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.
  • the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate. Programs, such as those provided by Synopsys, inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.
  • a standardized electronic format e.g., Opus, GDSII, or the like

Landscapes

  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Otolaryngology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Circuit For Audible Band Transducer (AREA)

Abstract

An apparatus comprising: an input configured to receive at least two audio signals; a frequency domain transformer configured to transform the at least two audio signals into a frequency domain representation of the at least two signals; a spatial covariance processor configured to generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; a beamformer configured to generate a spatial covariance matrix model comprising at least one beamformer kernel; a matrix factorizer configured to generate a linear magnitude mode! of audio objects; to combine the spatial covariance matrix model and the linear magnitude model; and further configured to determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and a separator configured to cluster the audio objects based on the at least one combination parameter to create separated audio sources.

Description

SPATIAL AUDIO APPARATUS Field The present application relates to apparatus for spatial audio signal processing. The invention further relates to, but is not limited to, apparatus for spatial audio signal processing within mobile devices.
Background
Spatial audio signals are being used in greater frequency to produce a more immersive audio experience. A stereo or multi-channel recording can be passed from the recording or capture apparatus to a listening apparatus and replayed using a suitable multi-channel output such as a multi-channel loudspeaker arrangement and with virtual surround processing a pair of stereo headphones or headset.
It would be understood that in the near future it will be possible for mobile apparatus such as mobile phone to have more than two microphones. This offers the possibility to record real multichannel audio. With advanced signal processing it is further possible to beamform or directionaliy process the audio signal from the microphones from a specific or desired direction by determining parameters such as direction associated with audio sources and processing the audio sources based on their directions. Summary
Aspects of this application thus provide a spatial audio capture and processing whereby listening orientation or video and audio capture orientation differences can be compensated for.
According to a first aspect there is provided a method for separating audio sources comprising: receiving at least two audio signals; transforming the at least two audio signals into a frequency domain representation of the at least two signals; generating an observed spatial covariance matrix from the frequency domain representations of the at ieast two audio signals; generating a spatial covariance matrix model comprising at Ieast one beamformer kernel; generating a linear magnitude model of audio objects; combining the spatial covariance matrix model and the linear magnitude model; determining at Ieast one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and clustering the audio objects based on the at Ieast one combination parameter to create separated audio sources. Receiving the at Ieast two audio signals may comprise at Ieast one of: the audio signals being generated from microphones receiving acoustic waves from at Ieast one audio source; and the audio signals being received from a memory, the audio signals describing acoustic waves from at Ieast one audio source. Transforming the at Ieast two audio signals into a frequency domain representation of the at Ieast two signals may comprise performing a time to frequency transform on the at Ieast two audio signals.
Generating the observed spatial covariance matrix from the frequency domain representations of the at Ieast two audio signals may comprise generating an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
The method may further comprise generating at Ieast one beamformer kernel by generating time delays in a frequency domain of a set of took directions sampling the spatial space around the microphone array.
Generating at Ieast one beamformer kernel may comprise generating an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
Generating a linear model of audio objects may comprise performing a non-negative matrix factorization for audio object magnitudes. Performing a non-negative matrix factorization for audio object magnitudes may comprise initialising linear model parameters and ¾· with random values uniformly distributed between zero and one;
Combining spatial covariance matrix model and the linear magnitude mode!, may comprise multiplying the spatial covariance matrix model by a magnitude envelope represented by the linear modei. Determining at Ieast one combination parameter may comprise determining spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model.
Determining the spatial covariance model parameters may comprise performing an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at Ieast one audio object are iteratively updated until either the linear magnitude modei parameters do not change significantly between updates or a defined number of iterations are processed,
Performing the iterative optimization based on the comp ex value non-negative factorization may comprise: updating linear model parameters zk0, tik and ¾? according to an optimization framework of complex non-negative matrix factorization; and updating the spatial covariance matrix modei magnitudes.
Updating the spatial covariance matrix model magnitudes may comprise updating the relative magnitude difference of estimates of the at Ieast two audio signals.
Clustering the audio objects based on the at Ieast one combination model parameter to create separated audio sources may comprise generating at Ieast one audio source from the at Ieast one audio object and from the estimated linear magnitude model parameters. The method may further comprise synthesising at least one audio signal based on the at least one combination model parameters associated with the at least one audio source. According to a second aspect there is provided an apparatus comprising: means for receiving at least two audio signals; means for transforming the at least two audio signals into a frequency domain representation of the at least two signals; means for generating an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals: means for generating a spatial covariance matrix model comprising at least one beamformer kernel; means for generating a linear magnitude model of audio objects; means for combining the spatial covariance matrix model and the linear magnitude model; means for determining at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and means for clustering the audio objects based on the at least one combination parameter to create separated audio sources.
The means for receiving the at least two audio signals may comprise at least one of: the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and the audio signals being received from a memory, the audio signals describing acoustic waves from at least one audio source.
The means for transforming the at least two audio signals into a frequency domain representation of the at least two signals may comprise means for performing a time to frequency transform on the at least two audio signals.
The means for generating the observed spatial covariance matrix from the frequency domain representations of the at least two audio signals may comprise generating an observed spatial covariance matrix from an autocovariance of the two frequency domain representations. The apparatus may further comprise means for generating at least one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array. The means for generating at least one beamformer kernel may comprise generating an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
The means for generating a linear model of audio objects may comprise performing a non-negative matrix factorization for audio object magnitudes.
The means for performing a non-negative matrix factorization for audio object magnitudes may comprise means for initialising linear model parameters and vki with random values uniformly distributed between zero and one;
The means for combining spatial covariance matrix model and the linear magnitude model may comprise means for multiplying the spatial covariance matrix model by a magnitude envelope represented by the linear model, The means for determining at least one combination parameter may comprise determining spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model.
The means for determining the spatial covariance model parameters may comprise performing an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at least one audio object are iteratively updated until either the linear magnitude model parameters do not change significantly between updates or a defined number of iterations are processed.
The means for performing the iterative optimization based on the complex value non- negative factorization may comprise: means for updating linear model parameters zko. k and ¾¾ according to an optimization framework of complex non-negative matrix factorization; and means for updating the spatial covariance matrix model magnitudes.
The means for updating the spatial covariance matrix model magnitudes may comprise means for updating the relative magnitude difference of estimates of the at least two audio signals,
The means for clustering the audio objects based on the at least one combination model parameter to create separated audio sources may comprise means for generating at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters.
The apparatus may further comprise means for synthesising at least one audio signal based on the at least one combination model parameters associated with the at least one audio source.
According to a third aspect there is provided an apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least: receive at least two audio signals; transform the at least two audio signals into a frequency domain representation of the at least two signals; generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; generate a spatial covariance matrix model comprising at least one beamformer kernel; generate a linear magnitude model of audio objects; combine the spatial covariance matrix model and the linear magnitude model; determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and clustering the audio objects based on the at least one combination parameter to create separated audio sources.
Receiving the at least two audio signals may cause the apparatus perform at least one of; receive the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and receive the audio signals from a memory, the audio signals describing acoustic waves from at teast one audio source.
Transforming the at least two audio signals into a frequency domain representation of the at least two signals may cause the apparatus to perform a time to frequency transform on the at least two audio signals.
Generating the observed spatial covariance matrix from the frequency domain representations of the at least two audio signals may cause the apparatus to generate an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
The apparatus may further be causes to generate at teast one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array.
Generating at least one beamformer kernel may cause the apparatus to generate an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
Generating a linear model of audio objects may cause the apparatus to perform a non-negative matrix factorization for audio object magnitudes.
Performing a non-negative matrix factorization for audio object magnitudes may cause the apparatus to initialise linear model parameters and %? with random values uniformly distributed between zero and one;
Combining spatial covariance matrix model and the linear magnitude model may cause the apparatus to multiply the spatial covariance matrix model by a magnitude envelope represented by the linear model. Determining at least one combination parameter may cause the apparatus to determine spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model. Determining the spatial covariance model parameters may cause the apparatus to perform an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at least one audio object are iteratively updated until either the linear magnitude model parameters do not change significantly between updates or a defined number of iterations are processed.
Performing the iterative optimization based on the complex value non-negative factorization may cause the apparatus to: update linear model parameters ¾0;¾ and vkj according to an optimization framework of complex non-negative matrix factorization; and update the spatial covariance matrix model magnitudes.
Updating the spatial covariance matrix model magnitudes may cause the apparatus to update the relative magnitude difference of estimates of the at least two audio signals.
Clustering the audio objects based on the at least one combination model parameter to create separated audio sources may cause the apparatus to generate at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters,
The apparatus may further be caused to synthesise at least one audio signal based on the at least one combination model parameters associated with the at least one audio source. According to a fourth aspect there is provided an apparatus comprising: an input configured to receive at least two audio signals; a frequency domain transformer configured to transform the at least two audio signals into a frequency domain representation of the at least two signals; a spatial covariance processor configured to generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; a beamformer configured to generate a spatial covariance matrix model comprising at least one beamformer kernel; a matrix factorizer configured to generate a linear magnitude model of audio objects; to combine the spatial covariance matrix model and the linear magnitude model; and further configured to determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and a separator configured to cluster the audio objects based on the at least one combination parameter to create separated audio sources.
The input may comprise at least one of: at least one microphone input configured to receive the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and at least one memory input configured to receive the audio signals from a memory, the audio signals describing acoustic waves from at least one audio source.
The frequency domain transformer may be configured to perform a time to frequency transform on the at least two audio signals. The spatial covariance processor may be configured to generate an observed spatial covariance matrix from an autocovariance of the two frequency domain representations.
The beamformer may be configured to generate at least one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array.
The beamformer may be configured to generate an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array.
The matrix factorizer may be configured to perform a non-negative matrix factorization for audio object magnitudes. The matrix factorizer may be configured to initialise linear model parameters
Figure imgf000011_0001
and ¾j with random values uniformly distributed between zero and one; The matrix factorizer configured to combine the spatiai covariance matrix model and the linear magnitude model may be configured to multiply the spatial covariance matrix model by a magnitude envelope represented by the linear modei.
The matrix factorizer configured to determining at least one combination parameter may be configured to determine spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model.
The matrix factorizer configured to determine the spatial covariance model parameters may be configured to perform an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude model representing the at ieast one audio object are iteratively updated until either the linear magnitude model parameters do not change significantly between updates or a defined number of iterations are processed.
The matrix factorizer configured to perform the iterative optimization based on the complex value non-negative factorization may be configured to: update linear model parameters ¾> % and %/ according to an optimization framework of complex non- negative matrix factorization; and update the spatial covariance matrix model magnitudes.
The matrix factorizer configured to update the spatial covariance matrix model magnitudes may be configured to update the relative magnitude difference of estimates of the at Ieast two audio signals.
The separator configured to cluster the audio objects based on the at Ieast one combination model parameter to create separated audio sources may be configured to generate at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters.
The apparatus may further comprise a spatial synthesizer configured to synthesise at least one audio signal based on the at least one combination mode! parameters associated with the at least one audio source.
A computer program product stored on a medium may cause an apparatus to perform the method as described herein,
An electronic device may comprise apparatus as described herein,
A chipset may comprise apparatus as described herein. Embodiments of the present application aim to address problems associated with the state of the art.
Summary of the Figures For better understanding of the present application, reference will now be made by way of example to the accompanying drawings in which:
Figure 1 shows schematically an apparatus suitable for being employed in some embodiments;
Figure 2 shows schematically an example audio signal processing apparatus according to some embodiments;
Figure 3 shows schematically a flow diagram of the operation of the audio signal processing apparatus shown in Figure 2 according to some embodiments;
Figure 4 shows schematically example microphone configurations on an apparatus according to some embodiments;
Figure 5 shows example array geometry of an apparatus comprising two microphones;
Figure 6 shows example beam directions approximating uniform sampling of the unit sphere around the array; Figure 7 shows schematically a fiow diagram of the operation of parameter estimation according to some embodiments;
Figure 8 shows schematically the generai principle of the clustering of CNMF components based on spatial weights according to some embodiments;
Figure 9 shows schematically the spatial synthesiser as shown in Figure 2 configured to generate a binaural output according to some embodiments;
Figure 10 shows schematically a flow diagram showing the operation of the spatial synthesiser as shown in Figure 9 configured to generate a binaural output according to some embodiments;
Figure 11 shows graphical plots of spatial weights analysed by an example of some embodiments as pictured from above the array where original source positioning was on the lateral plane at azimuths 0 (left) and 90 (bottom) degrees;
Figure 12 shows a graphicai plot of an example capturing array of microphones used in some embodiments where microphones are denoted by circles and the zero angle references axis by an arrow;
Figure 13 shows a graphical plot of source positioning in embodiment exampie 1 ; Figure 14 shows a graphical plot of source positioning in embodiment example 2; Figure 15 shows a graphical plot of source positioning in embodiment example 3; and Figures 18 and 17 show graphical plots of example outputs generated by source processing according to some embodiments.
Embodiments
The following describes in further detail suitable apparatus and possible mechanisms for the provision of effective sound-field directional processing of audio recording for example within audio-video capture apparatus. In the following examples audio signals and processing is described. However it would be appreciated that in some embodiments the audio signal/audio capture and processing is a part of an audio- video system.
As described herein mobile devices or apparatus are more commonly being equipped with multiple microphone configurations or microphone arrays suitable for recording or capturing the audio environment or audio scene surrounding the mobile device or apparatus. A multiple microphone configuration enables the recording of stereo or surround sound signals and the known location and orientation of the microphones further enables the apparatus to process the captured or recorded audio signals from the microphones to perform spatial processing to emphasise or focus on the audio signals from a defined direction relative to other directions.
One way to perform spatial processing is to initially extract and manipulate the direction or sound source dependent information so to use this information in subsequent applications. These applications can include for example, spatial audio coding (SAC), 3D soundfield analysis and synthesis, sound source separation and speakers extraction for further processing such as speech recognition. in general the field that studies such spatial sound processing is known as blind source separation (BSS) for simultaneously emitting sound sources. A classic example of such a case is known as the cocktail party problem enabling the separation of each individual speaker from the party recorded using a microphone array. The field of BSS has been intensively studied, but is still categorized as an unsolved problem. The capturing or recording apparatus or device usually consists of a small hand held device having multiple microphones. The multiple channels and their information correlation and relationship can then be utilized for source separation and direction of arrival estimation.
Furthermore applications employing such analysis, such as 3D soundfield analysis and synthesis can employ the accurate and detailed directional information of the separated sources when rendering the captured field by positioning the source using either binaural synthesis by head related transfer function (HRTF) filtering or source positioning in multichannel and multidimensional loudspeaker arrays using source positioning techniques such as vector base amplitude panning (VBAP). Blind sound separation (BSS) of audio captures recorded using a small and enclosed microphone array such as conventionally found on a mobile device or apparatus can include the following problems and difficulties that are addressed herein by embodiments as described herein. Firstly the number of microphones is typically small, approximately 2-5 capsules, because of design volume and cost arrangements making the source direction of arrival (DoA) estimation difficult and pure beamforming based separation inefficient. Beamforming for source direction of arrival detection and related problems and more recently spherical array beamforming techniques have been successfully used in sound field capture and analysis and also developed into final products such as Eigenmike. However the problem with spherical array processing is the array structure and the sheer size of the actual arrays used prevent it to be incorporated into a single mobile device. Furthermore pure beamforming does not address the problem of source separation but analyses the spatial space around the device with beams as narrow as possible. The sidelobe cancellation for decreasing the beam width generally requires increasing the microphone count of the array, which is as discussed costly in volume, device complexity and cost of manufacture.
Furthermore the small geometrical distance between capsules reduces the time- delay between microphones which require capturing using high sampling rate in order to observe the small time instance differences. When high sampling frequency is used there are problems with frequency domain based BSS methods in form of spatial aliasing. In other words audio frequencies with wavelength less than two times the distance of the microphone separation can cause ambiguity in resolving the time delays in form of a phase delay after a short time Fourier transform (STFT).
For example independent component analysis (!CA) can be applied in frequency domain to estimate statistically independent components at each frequency. The frequency-domain ICA leads to an arbitrary source ordering at each frequency. This permutation ambiguity has been solved by different means over the years, mixing filter frequency response smoothness, temporal structure of the source signals, and time-difference of arrival (TDoA) and direction of arrival (DoA), and interpretation of ICA mixing parameters. Furthermore there also exist ICA-based methods that avoid the permutation problem by unifying the source independencies across frequencies. However ICA based separation is one of the methods which is sensitive to problems caused by spatial aliasing in permutation alignment and in unifying the source independencies over frequency.
Furthermore non-negative matrix factorization (NMF) based separation in multichannel cases has been proposed. These include for example multichannel NMF for convoluted mixtures, however the E -a!gorithm used for parameter estimation is inefficient without oracle initialization, (in other words knowing source characteristics for initializing the algorithm). Complex multichannel NMF (CNMF) with multiplicative updates has been proposed with promising separation results. The proposed CNMF algorithms estimate the source spatial covariance properties and the magnitude model. However the spatial covariance matrices are estimated and updated individually for each frequency bin making the algorithm prone for estimation errors at high frequencies with spatial aliasing. Also the estimated covariance properties are not connected to the spatial locations of the sources.
In addition direct source magnitude envelope and spatial covariance matrix estimation has been proposed. The spatial properties are estimated frequency bin wise leading again into permutation ambiguity and with a separate algorithm for solving the component ordering, making it inefficient with high sampling rate captures.
Additionally the problem includes solving and executing 3D sound synthesis of the separated sources. It would further be understood that where spatial processing is performed with respect to spatial audio synthesis, such as 3D audio synthesis, the 3D synthesis of the separated sources or parts of the sources require pairing the separation algorithm with DoA analysis making the system potentially discontinuous and less efficient for the 3D sound scene analysis-synthesis loop. As such an enclosed microphone array with an unknown directivity pattern of each capsule requires a machine learning based algorithm for learning and compensating the unknown properties of the array.
Thus the concept as described herein in further detail is one which the audio recording system provides apparatus and/or methods for separating audio sources 18 using several microphones in one device. In such embodiments sounds from different audio sources arrive at different times to the different microphones. Furthermore each audio source has individual spectral features. The concepts thus discuss herein that the apparatus and methods analyse microphone signals so as to determine a combination of the time-differences and spectral features. The effect of such embodiments is the sound sources that are separated from a mixture of audio sources.
In this regard reference is first made to Figure 1 which shows a schematic block diagram of an exemplary apparatus or electronic device 10, which may be used to record (or operate as a capture apparatus).
The electronic device 10 may for example be a mobile terminal or user equipment of a wireless communication system when functioning as the recording apparatus or listening apparatus. In some embodiments the apparatus can be an audio player or audio recorder, such as an MPS player, a media recorder/player (also known as an MP4 player), or any suitable portable apparatus suitable for recording audio or audio/ ideo camcorder/memory audio or video recorder. The apparatus 10 can in some embodiments comprise an audio-video subsystem. The audio-video subsystem for example can comprise in some embodiments a microphone or array of microphones 11 for audio signal capture. In some embodiments the microphone or array of microphones can be a solid state microphone, in other words capable of capturing audio signals and outputting a suitable digital format signal in other words not requiring an analogue-to-digital converter, in some other embodiments the microphone or array of microphones 11 can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, Electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or micro electrical-mechanical system (MEMS) microphone. The microphone 11 or array of microphones can in some embodiments output the audio captured signal to an analogue-to-digital converter (ADC) 14. In some embodiments the apparatus can further comprise an analogue-to-digital converter (ADC) 14 configured to receive the analogue captured audio signal from the microphones and outputting the audio captured signal in a suitable digital form. The analogue-to-digitai converter 14 can be any suitable analogue-to-digital conversion or processing means. In some embodiments where the microphones are 'integrated' microphones the microphones contain both audio signal generating and analogue-to-digital conversion capability, In some embodiments the apparatus 10 audio-video subsystem further comprises a digital-to-analogue converter 32 for converting digital audio signals from a processor 21 to a suitable analogue format. The digital-to-analogue converter (DAC) or signal processing means 32 can in some embodiments be any suitable DAC technology. Furthermore the audio-video subsystem can comprise in some embodiments a speaker 33. The speaker 33 can in some embodiments receive the output from the digital-to-analogue converter 32 and present the analogue audio signal to the user. In some embodiments the speaker 33 can be representative of multi-speaker arrangement, a headset, for example a set of headphones, or cordless headphones.
In some embodiments the apparatus audio-video subsystem comprises a camera 51 or image capturing means configured to supply to the processor 21 image data. In some embodiments the camera can be configured to supply multiple images over time to provide a video stream.
In some embodiments the apparatus audio-video subsystem comprises a display 52, The display or image display means can be configured to output visual images which can be viewed by the user of the apparatus. In some embodiments the display can be a touch screen display suitable for supplying input data to the apparatus. The display can be any suitable display technology, for example the display can be implemented by a flat panel comprising cells of LCD, LED. OLED, or 'plasma' display implementations. Although the apparatus 10 is shown having both audio/video capture and audio/video presentation components, it would be understood that in some embodiments the apparatus 10 can comprise only the audio capture and audio presentation parts of the audio subsystem such that in some embodiments of the apparatus the microphone (for audio capture) or the speaker (for audio presentation) are present. Similarly in some embodiments the apparatus 10 can comprise one or the other of the video capture and video presentation parts of the video subsystem such that in some embodiments the camera 51 (for video capture) or the display 52 (for video presentation) is present.
In some embodiments the apparatus 10 comprises a processor 21. The processor 21 is coupled to the audio-video subsystem and specifically in some examples the analogue-to-digital converter 14 for receiving digital signals representing audio signals from the microphone 11 , the digital-to-analogue converter (DAG) 12 configured to output processed digital audio signals, the camera 51 for receiving digital signals representing video signals, and the display 52 configured to output processed digital video signals from the processor 21.
The processor 21 can be configured to execute various program codes. The implemented program codes can comprise for example audio-video recording and audio-video presentation routines. In some embodiments the program codes can be configured to perform audio signal processing.
In some embodiments the apparatus further comprises a memory 22. In some embodiments the processor is coupled to memory 22. The memory can be any suitable storage means. In some embodiments the memory 22 comprises a program code section 23 for storing program codes implementabie upon the processor 21. Furthermore in some embodiments the memory 22 can further comprise a stored data section 24 for storing data, for example data that has been encoded in accordance with the application or data to be encoded via the application embodiments as described later. The implemented program code stored within the program code section 23, and the data stored within the stored data section 24 can be retrieved by the processor 21 whenever needed via the memory-processor coupling.
In some further embodiments the apparatus 10 can comprise a user interface 15, The user interface 15 can be coupled in some embodiments to the processor 21. In some embodiments the processor can control the operation of the user interface and receive inputs from the user interface 15. In some embodiments the user interface 15 can enable a user to input commands to the electronic device or apparatus 10, for example via a keypad, and/or to obtain information from the apparatus 10, for example via a display which is part of the user interface 15. The user interface 15 can in some embodiments as described herein comprise a touch screen or touch interface capable of both enabling information to be entered to the apparatus 10 and further displaying information to the user of the apparatus 10. In some embodiments the apparatus further comprises a transceiver 13, the transceiver in such embodiments can be coupled to the processor and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver 13 or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.
The transceiver 13 can communicate with further apparatus by any suitable known communications protocol, for example in some embodiments the transceiver 13 or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).
In some embodiments the apparatus comprises a position sensor 18 configured to estimate the position of the apparatus 10. The position sensor 16 can in some embodiments be a satellite positioning sensor such as a GPS (Global Positioning System), GLONASS or Galileo receiver. In some embodiments the positioning sensor can be a ce!iular ID system assisted GPS system.
In some embodiments the apparatus 10 further comprises a direction or orientation sensor. The orientation/direction sensor can in some embodiments be an electronic compass, accelerometer, and a gyroscope or be determined by the motion of the apparatus using the positioning estimate.
It is to be understood again that the structure of the electronic device 10 could be supplemented and varied in many ways.
With respect to Figure 4 an example apparatus implementation is shown wherein the apparatus 10 comprises a front face 301 comprising a camera 51 , a rear face 303 and a top edge or face 305. In the example shown in Figure 4 the apparatus comprises four microphones a first (front right) microphone 11 3 located at the front right side of the apparatus where right is towards the top edge of the front face of the apparatus, a front left microphone 113 located at the front left side of the apparatus, a right high microphone 112 located at the top edge or face side of the apparatus, and a left rear microphone 11 located at the left rear side of the apparatus.
With respect to Figure 2. an example audio signal processing apparatus according to some embodiments is shown. Furthermore with respect to Figure 3 a flow diagram of the operation of the audio signal processing apparatus as shown in Figure 2 is shown.
In some embodiments the apparatus comprises the microphone or array of microphones configured to capture or record the acoustic waves and generate an audio signal for each microphone which is passed or input to the audio signal processing apparatus. As described herein in some embodiments the microphones 11 are configured to output an analogue signal which is converted into a digital format by the analogue to digital converter (ADC) 14. However the microphones shown in the example herein are integrated microphones configured to output a digital format signal directly to a short time Fourier transformer (STFT) 101. In the example shown herein there are four microphones these microphones can be arranged in some embodiments in a manner similar to that shown in Figure 4. it would be understood that in some embodiments there can be more than or fewer than four microphones and the microphones can be arranged or located on the apparatus in any suitable manner.
Furthermore although as shown in Figure 3 the microphones are part of the apparatus it would be understood that in some embodiments the microphone array is physically separate from the apparatus, for example the microphone array can be located on a headset (where the headset also has an associated video camera capturing the video images which can also be passed to the apparatus and processed in a manner to generate an encoded video signal which can incorporate the processed audio signals as described herein) which wireiessly or otherwise passes the audio signals to the apparatus for processing.
The microphone array can be considered to capture in the time domain the sound or audio sources which have been convolved with their spatial responses. This can be mathematically modelled or be described as:
Figure imgf000022_0001
where xm{t) is the mixture of k = 1...K sources captured by microphones m = L.. (in other words the microphone m receives the audio signal 3c,,, , The sample index is denoted by t.
In this 'mixing' model the spatial response from the source k to the microphone m is denoted by hmk(r) and the source signals are given as sk(t).
The operation of receiving the microphone input audio signals is shown in Figure 3 by step 201 . In some embodiments the apparatus comprises a short time Fourier transformer (STFT) 101 . The short time Fourier transformer 1 01 is configured to perform a short time Fourier transform on the microphone input audio signais. The 'mixing' model can be approximated in the STFT-domain as,
Figure imgf000023_0001
where xy = { i xM }T e€l x M is the short-time Fourier transform (STFT) of the array capture xm(t), ί = is the frequency index and j = i..J is the frame index. This STFT of a time domain signal can in some embodiments be obtained as follows,
Xii where t ~ (j - 1)1 + 1, . . . , jl
Figure imgf000023_0002
Figure imgf000023_0003
where the operator FFT[. . .]< denotes ith frequency element of short time Fourier transformed input signal.
The operation of the STFT 1 01 on the array microphone signals and the model effectively replaced the spatial response ¾mfe(r) with the frequency response denoted by =:: [¾! , ·■■ ) ¾Λί ]Γ€€Λί χ 1 and the STFTs of the sources k = i . . . K are denoted by 3«fe. The single source signals as seen by the array, in other words the source signals convolved with their spatial impulse response are denoted by yw ~ *»i*%fc. The spatial response filter < (r) has a length of severai hundreds of milliseconds depending on the reverberation time of the capturing environment, but approximation of it by ik in the frequency domain using an analysis window size of tens of milliseconds has been found to work well in practice due to the negligible energy after the main reverberant part of the source spatial response.
In some embodiments the STFT is configured to output the transformed domain signals to a spatial covariance processor 1 03. The operation of applying a SIFT to the time domain signals is shown in Figure 3 by step 301 .
In some embodiments the apparatus comprises a spatial covariance processor 103. The spatial covariance processor 1 03 is configured to receive the frequency domain form of the audio signals ¾ = |n iM|T e Cl xM and perform a spatial covariance processing on them,
In order to avoid estimating and modelling the absolute phase of each channel the spatial covariance processor generates a spatial covariance matrix for each time- frequency point as the signal representation. In such embodiments the spatial covariance calculation translates the absolute phase to a phase delay between each microphone pair which is employed as a signal representation input for BSS algorithm described herein. In some embodiments the calculation of the SCMs determines a magnitude square-rooted version of the array capture. The determination of the magnitude square rooted version of the array capture ensures that the nonnegative part in the diagonal of the spatial covariance matrix (SCM), contains the STFT magnitudes, in some embodiments an unmodified STFT of the capture array output can be used to produce a power spectrum on the diagonal of the SCM.
In the following example embodiments the magnitude square-rooted STFT is used as a starting point the SCMs for each time-frequency point derived by the following method.
In some embodiments the spatial covariance processor determines the magnitude square-rooted version of the capture array representation x*j = K, · · ·¾■] for a time- frequency point (i- ) as
= (ixi |1/ 2 exp(. arg(ici)), \xM \ l > 2 exp(i -g(xM))}T
Figure imgf000024_0001
sign(xf,i)3T,
where si n(*) z/ \ z \ s-s ^e S gnum function for complex numbers. Furthermore in some embodiments the spatial covariance processor 103 can be configured to determine the SCM for a single time-frequency point from the array capture vector * ::: \* ·■ · | as the outer product where n stands for Hermitian transpose.
The determined SCM x cMx M for a one time-frequency point can be considered to consist of an observation magnitude M = [ ι ί! · · · . Κν ί]τ on the diagonal ίχ] and the off-diagonal values [x]nm, n≠m represent the magnitude and phase difference ia; na:,„|1/2 si n(ar„a:*l) between microphone pair .
In some embodiments the spatial covariance processor 103 can be configured to calculate the SCM for each time frequency point *, i and which can be denoted as the matrix with the following subindexing xy e cM*M,
The complex-valued monoaural source spectrogram in the SCM domain is equal to a real valued power spectrum where c denotes complex conjugate of complex number c, In some embodiments because of the square-rooted STFT used to calculate SCMs hereafter the sources are denoted using their magnitude spectrum
It would be understood that by using the SCM representation of the microphone array audio signals defined by equations as defined above the absolute phase of the sources is not significant from the parameter estimation point of view and only the phase delay between all microphone pairs is modelled. Therefore the spatial covariance signal representation replaces the mixing vector hik in the original model by mixing matrix ¾& e c.Mx M which denotes the source spatial response expressed in the form of a covariance matrix >¾ ii In this representation the mixing matrix is still defined for each STFT frequency bin % individually. In some embodiments and employing the above definitions the mixing model defined earlier can be expressed for the covariance observations as
K K Y V"1 S3 "" ώ -
Je-l fc=l where ¾■* ==== ¾&½¾ is the source spatial covariance incorporating source magnitude details and the mixing.
In some embodiments the SCM of each source sw at each frequency can be modelled using a Hermitian positive semidefinite matrix ¾ approximating the true but unknown SCM. In such embodiments the mixture SCM and source magnitudes are only perceived and are yet to be estimated. Estimating the source magnitude details ½* and the corresponding mixing denoted by ¾* by turns in such embodiments could provide the desired BSS properties, however estimating ¾ in such a way that it corresponds to a single source over all frequencies is difficult in general. The spatial properties being individually estimated for each STFT frequency bin i does not utilize the fact that the covariance properties are connected by Time Difference of Arrival (TDoA) of the direct path and early reflections. The model in such embodiments relies on the fact that the Non-negative matrix factorization ( MF) model described herein in more detail enforces to correspond to a single acoustical source and the phase delay evidence at each frequency % averaged over time yields to correct estimate of the true spatial properties.
In some embodiments the spatial covariance processor is configured to output the transformed SCM signals to a complex non-negative matrix factorizer 105.
The operation of generating a spatial covariance matrix form of the STFT is shown in Figure 3 by step 303. In some embodiments the apparatus comprises a beamformer generator 107. The beamforrner is configured in some embodiments to generate at least one beamforming matrix for application to the spatial covariance matrix by the non- negative matrix factorizer 105. In the source searching methods described herein modelling the source spatial response, such as independent component analysis (ICA) and complex valued non- negative matrix factorization (CNMF), the source parameters are estimated and optimized for each frequency bin independently. However in some embodiments as described herein in Direction of Arrival (DoA) estimation methods evidence across the whole frequency band is considered as whole. In DoA estimation approaches a single source position is linked to a single Time Difference of Arrival (TDoA) between the microphones assuming anechoic propagation or in case of reverberant conditions considering only its direct path propagation. The single TDoA unifies the phase delay over frequency making the DoA methods able to integrate evidence over all frequencies to obtain the estimate of DoA over the whole frequency spectrum making the source estimation more robust. However the application of the DoA concept has not been widely utilized in Blind Source Separation (8SS) parameter estimation due to difficulties in formulating the estimation algorithm and spatial aliasing that makes the process ambiguous.
In DoA analysis by beamforming, the TDoAs of certain spatial position are used for aligning the received signal and for searching for beam direction, in other words the source DoA, that produces the highest beamformer output energy. This is equivalent to making a hypothesis of all possible source positions and searching for TDoAs that give the time alignment of the microphone signals that generate the highest signal power when the aligned signals are summed together, A similar method can be used to unify STFT bin dependencies for estimating the source spatial response in any generic BSS method suffering from said problem of optimizing the model parameters individually regarding frequencies. The difference is merely that in BSS the optimization algorithm is set to fit a collection of TDoA kernels (which we call beamforming kernels in the following embodiments) to the observed data and that way finding the most likely DoA of the source in question,
Beamforming in its simplest form, delay and sum beamforming (DSBF), translates a given wavefront arrival direction into a set of TDoA values between each microphone pair. The TDoA values are derived from the geometric properties of the array and the process is described in further detail herein.
For example Figure 5 shows an example microphone array where one pair of microphones n 401 and m 403 He on the xy-plane at locations n and m, respectively, and an unit vector k 405 is pointing towards the source from the geometrical center p 407 of the array. With respect to the following equations the direction vector k 405 is defined in Cartesian coordinates. For simplicity and in order to use a spherical coordinate system for denoting the source direction we define that the geometrical center of the array is in the origin of the Cartesian coordinate system, i.e, P = [O, o, oj. 5 The source direction of arrival is hereafter given in the spherical coordinate system using elevation 9
Figure imgf000028_0001
{ο, 2ΐΐ} ancj radius r = i. If the DoA is given in degrees we define the following ranges - 90° < ® < ¾)° and o° < φ < 360° for elevation and azimuth respectively. The reference axis of the array, zero elevation β - o° and zero azimuth Φ = can be arbitrarily set to point towards any direction in the spatial0 space.
By assuming a far field model for the source propagation and the wavefront being planar when arriving to the array we can write the TDoA of the microphone pair n,m as
Figure imgf000028_0002
where v is the speed of sound. Each different source location denoted by k translates into one set of TDoAs which further translates into a phase delay proportional to the frequency in STFT domain. However the phase difference is unambiguous only up to0 frequency / = ¾ where d 409 is the smallest distance between any two microphones in the array. Hereafter this point is referred to as the spatial aliasing frequency. The limitation originates from the fact that frequencies with half the wavelength shorter than the smallest microphone spacing d 409 cannot be discriminated by their phase difference.
5
The TDoA {xrm} can be converted to a phase delay in the frequency domain as,
Wnm (f, k) = exp (i2ir/rn,m(k))
where k is a unit vector pointing to the desired look direction. Evaluating Wmi for every pair n ~ 1 . . . and m = \ . . . M gives the beamforming kernels ¾ι( ) into the0 look direction k we are interested in. The beamforming kernels can in some embodiments be output by the beamformer 107 to the complex non-negative matrix factorizer. In other words the beamformer 107 can be configured to define a set of look directions vectors, k„ where o■■■■■■■■ ι . , . ο, which spatially sample the surface of a unit sphere set around the geometrical center of the array. An example of the look direction vectors 501 is illustrated in Figure 6. These look directions 501 sample the 5 spatial space around the microphone array. In some embodiments the beamforming kernels are denoted in each look direction at each frequency ί ^ ι . , Λ using wio e CM*M. In other words entries [wjo]nm denote a TDoA in terms of phase difference expressed as complex number for a microphone pair (n, m). The kernels are obtained from the equation above as
-jQ [Wioj„m ::= Wnm(f,k0) :=== β3 (j2irfrnm (k„)) , / === ~
where FS is the sampling frequency and i is the STFT frequency bin index in question and N is the STFT window length.
The generating of the beamforming kernels is shown in Figure 3 by step 305.
15
In some embodiments the apparatus comprises a complex non-negative matrix factorizer 105. The complex non-negative matrix factorizer 105 is configured to receive the beamforming kernels from the beamformer and apply these to the spatial co variance matrix form of the received audio signals. The proposed complex valued
20 non-negative matrix factorization (CNMF) model employed in some embodiments as described herein comprises a non-negative matrix factorization (NMF) magnitude model core for modelling the source STFT magnitude details denoted by ¾¾. In practice several non-negative matrix factorization (NMF) components are used for representing one acoustical sound source however in the examples described herein
25 one NMF component represents one sound source. This simplification enables a more intuitive explanation of the spatial covariance matrix (SCM) model of the audio signals.
The source spatial covariance defined as sijk = Miksijk and the SCM model that 30 comprises of weighted superposition of multiple DSBFs can be defined as,
o where ¾0 are the directional weights corresponding to the bearnforming kernels into each look direction and ¾ models the source spatial response hik expressed in the form of covarSance matrix i hg, in the embodiments described herein the complex non-negative matrix factorizer 105 is configured to estimate ¾ in such way that they correspond to a single acoustical source, this is achieved in the embodiments described herein by estimating spatial weights ¾0 that are independent of frequency. The definition of the DSBF kernels by the beamformer takes into account the frequency dependencies that the DoA of a certain source in form of TDoA causes. The spatial weights zko are furthermore restricted to be positive.
In some embodiments the complex non-negative matrix factorizer 105, given the SCM model attempts to minimize the squared Euclidean distance ||χϋ - f:,|i , between the observations xi; and the signal model ^. In some embodiments the complex non-negative matrix factorizer 105 therefore given the SCM model uses a source model for covariance observations as
Figure imgf000030_0001
The CNMF parameter structure for estimating and factorizing ¾* is explained further herein. However given a DoA based interpretation of Hii£ with direction dependent weights ¾ it is possible to formulate the model in such a way that magnitudes ½* correspond to a single real acoustical source.
For example each source ¾¾ can be modelled such that each source has a SCM defined according to,
o which determines its spatial position.
The non-negative matrix factorization (N F) model for the magnitude spectrogram of single source can thus be written as ½* :::: and single channel mixture of the sources is defined as κ κ
k k
where k is the source index and κ is the totai number of sources used for modelling and approximating the observed magnitude spectrogram matrix. Each column vector contains the DFT spectrum of a source, and the corresponding row of v^ represents its time-varying gain in each SIFT frame.
In some embodiments the scaling of DSBF kernels is restricted to ===: i, where i i · denotes frobenius norm for matrices. The scaling ensures that the SCM part is only responsible of modelling the phase delay and relative magnitude differences between input channels (diagonal values). Substituting the NMF model into the covariance model and rearranging the parameters generates the whole CNfvlF model,
if o
~ .ij '■'■"· X¾ ^.?- ^ :::: ^ ^ ^ho^ik^'kj where wio are the beamforming kernels, ¾ are the kernel weights for source k, i.e. directional weights.
In some embodiments the CN F modei can be generated using the source SCM by incorporating the single channel source magnitudes which generates the following model,
K
k i
from where it is understood that real value entries in the diagonal of !¾.%! are responsible for modelling the absolute source magnitude level with respect to each channel and the off-diagonal values model the cross-channel magnitude and phase difference properties. The spatial covariance matrix obtained further means that the magnitudes lw»i combined with the non-negative weights ¾« eventually determine the multichannel magnitude model.
The delay and sum beamformer (DSBF) steering matrices generated by the beamformer produce unit magnitude for all values. To model the magnitude differences between each channel the complex non-negative matrix factorizer 105 is configured to estimate and update magnitudes of wtt, accordingly. This is because the sources are captured with varying gain with respect to each microphone, which is further caused by microphones having a different distance from the source and the possibie acoustic shade of the device producing directional dependent gain even if omnidirectionai microphones are used. While the spatial covariance matrix magnitudes are subject to optimization, the complex non-negative matrix factorizer 105 in some embodiments keeps the original beamforming phase difference the same, in other words the original phase shift caused by a certain direction of the source. In this way the dependency between different frequency components is retained when modelling the phase difference by only estimating the frequency independent spatial weights ¾«.
In some embodiments the complex non-negative matrix factorizer 105 is configured to employ multiplicative updates for iteratively deriving the NMF decomposition parameters minimize a given cost function, for example squared Euclidean distance or Kullback-Leibler divergence. The multiplicative updates can be obtained from the partial derivatives of the cost function with respect to each estimated parameter.
However in some embodiments the complex non-negative matrix factorizer 105 is configured to obtain the updates via auxiliary functions and Expectation Maximization (EM)-algorithm structure. The statistical interpretation of different NMF cost functions has been discussed in the art. It has been shown that the multiplicative algorithm updates for optimizing the algorithm parameters is equivalent to the maximum likelihood estimation of mean or variance of a suitable probability distribution. In case of a squared Euclidean distance the parameter optimization equals to the maximum likelihood estimation of the mean of a Gaussian distribution with a fixed variance. in order to use the EM-aigorithm structure in the maximum likelihood estimation of model parameters we redefine the model with the following statistical interpretation
A" O where denotes complex Gaussian distribution. The Gaussian distribution matrix notation σ-') denotes the distribution mean using γ and the variance for each matrix entry is denoted using σ2. This statistical interpretation converts squared a Frobenius norm Hxy used for measuring the modelling error into a maximum likelihood estimation of the mean of the Gaussian distribution with a fixed variance σ2. The CN F model for the observed spatial covariance matrix x. in a single time- frequency point (*> J) can be given as following posterior distribution piXijlff) =
Figure imgf000033_0001
K O
!x e pi-ll y WiaZkotikVkj j IF) where # = {W. Z, T, V} represents the parameters of the proposed CNMF model, c denotes complex Gaussian distribution and I I— II denotes Frobenius norm for matrices, in some embodiments the complex non-negative matrix factorizer 105 derivation of the update rules write the total likelihood over all time-frequency points as,
I J K G CX ] [ ] ex H j y™ Y Y WioZkotikVkj l lf-)
In order to estimate the posterior distribution parameters and optimize the model parameters the complex non-negative matrix factorizer 105 can determine the negative log-likelihood of ρ{Χ/θ) which results in the following,
/ J K o
£(W , Z, T, V) - T I |Xy - ∑ \%! inzkntikvkj j \ .
The complex non-negative matrix factorizer 105, by minimizing the above defined negative log-likelihood obtains the model parameters 0 = {W, Z, T, V} t at maximize the likelihood of the posterior distribution given observations x.
In some embodiments the complex non-negative matrix factorizer 105 can be configured to introduce unknown underlying components assumed to be mutually independent and modelled by individual Gaussian distributions. The latent components denote the spatial covariance of an unknown gaussian variable. These latent components are defined such that,
Figure imgf000033_0002
Based on the summation property of Gaussian variables the variance of each of these latent variables is 1/K where κ is the total number of sources. Furthermore the total likelihood with the latent components can be written as i j K a
c+ ( w, z, T, v c) - ∑ I - wifJ¾Afc¾i j f where the variance of the Gaussian variables are now addressed using parameter ri.iko which satisfies ^ 1 and > °. The totai likelihood of £+ can be used for an indirect optimization of £(W, Z, T, V), j is is due the likelihood of the auxiliary function having the following properties
£(W, z, T, V) < £+(W, z. T; v c)
£(W, Z, T, V) = nun £+{W, Z, T. V C)
which indicates that minimizing ÷ with respect to C, T and v corresponds to the minimization of c and the maximum likelihood estimation of ρ(Χ/θ). in some embodiments the complex non-negative matrix factorizer 105 can define the auxiliary function as:
which substituted into equation £+(W, Z, T, V C) makes it equal to £(W, Z) T) V) and again allows indirect minimization of the whole model using the auxiliary variables and their likelihood. For the variance parameters Sawada et. al. derives following definitions
Figure imgf000034_0001
which equals to assigning the variance to the latent components in proportion to the magnitude of the component divided by the whole model magnitude level.
In some embodiments the complex non-negative matrix factorizer 105 additionally introduces the following constraints for numerical stability and to fix the scaling between parameters,
Figure imgf000034_0002
The scaling of ¾ to unity iJ-norm along DSBF kernel direction dimension is compensated by multiplying tik by the same norm, Similarly enforcing unity z2-norm to ¾.? is compensated by scaling of U . The beam kernel covariance matrices are scaled to unity Frobenius norm to prevent them changing the overall magnitude level of the model. The proposed scaling is achieved by applying the following, 2
Figure imgf000035_0001
W4o:|, In some embodiments the complex non-negative matrix factorizer 105 applies a partial derivation of £+(W,Z,T,VC) with respect to each model parameter and setting the derivative to zero, For the derivation of ¾Α¾ and ¾¾ the complex non-negative matrix factorizer 105 can redefine the likelihood function £+(W,Z,T,VC) expanding the Frobenius form into the following form i,3A°rijk°
using the equality I|A|| = tr(A" A). AISO BASED ON THE SCA|ING THE SECOND TERM SLMPWIES
. ~2 f.2...2
tO ~ka*ikvkj.
The partial derivatives of with respect to parameters and can be given as,
ø£+(0)
Zkoti'kvlj ~ tikVkjteC ijkoWic}]
. . Tii a
zt' kVkj " ¾o«fei r(C<jfc<>Wi0)
Figure imgf000035_0002
where 0 = {W,Z,T,VC}. Setting the derivatives to zero, substituting r o with its definition by Sawada et al and solving the equation with respect to the parameter to be updated determines the following update rules k'i'kj'
Figure imgf000035_0003
It would be understood that the above updates are not in a multiplicative form that is desired for fast and reliable algorithm convergence. However the above updates can be brought into a multiplicative form with following equality, where = Xy -∑Λ(β WfeZfaAkViy jS j-ne error 0| ^e mo el. Placing *ytr(Cyfcowio) = zkotikVk} {xii -i- tr(Ey wio)) baci^ in o the update rules determined above and after few manipulations generates the following multiplicative updates.
∑,·,„ gfes¾jtr(EjjWjo)
tit ÷~ til... +
Figure imgf000036_0001
which are the updates to be used for the maximum likelihood estimation of the algorithm parameters by the complex non-negative matrix factorizer 105 in some embodiments and which by definition minimize the model error defined by the squared Frobenius norm given as,
K o
£>Euc -■■ i|Xy - WjoSfcoiifeWjyll'p
Figure imgf000036_0002
In some embodiments the complex non-negative matrix factorizer 105 can further be configured to update or optimize the DSBF kernels. This updating needs a different update scheme to that derived above, since the phase differences of the predefined beamforming kernels are to be retained. In some embodiments the complex non- negative matrix factorizer 105 estimates the DSBF kerne! magnitudes by first deriving the update for complex wift) but restrict the update to the magnitude only. Thus the model can be seen as magnitude dependent delay and sum beamformer. After the update each spatial covariance matrix is scaled into a unity Frobenius norm by applying w« <- wio/]| wio|[
The update rule for the spatial covariance matrices wfo is obtained via partial derivation of £÷(W, Z, T, v c) resulting in
dC+{0)
(Ci k. ~ WioZkotikVkj){-HkotikVkj)- dWi,
j,k ' ijko Setting the above derivative to zero, substituting rw° with its definition from Sawada et al results in the following update
Figure imgf000037_0001
where the wio is a preliminary update with a modified phase delay compared to the actual desired update of win. Due to the scaling defined in
Figure imgf000037_0002
the divisor in the above update can be disregarded and substituting with its definition k, the above update can be modified into the following multiplicative update
Figure imgf000037_0003
In some embodiments the complex non-negative matrix factorizer 105 can be configured to generate the update of wio by applying
Wio <- |Wio| exp(i arg(W<0)), Wj0 <- W^/|iWic|!.
which only updates the magnitude part of the beamforming kernels and thus the magnitudes of the spatial covariance matrices for sources.
The update of wi(>) in particular at the highest frequencies, may produce matrices that are not positive semidefinite, and negative values at the diagonal equals to a subtractive magnitude model, which is not desired. Therefore in some embodiments the complex non-negative matrix factorizer 105 can be configured to enforce positive semidefinite beamkerne! matrices by applying an eigenvalue decomposition wi0 = VDVh and setting eigencomponents with negative eigenvalues to zero, represented by D.
The final stage of updating the beamkemels is given as
The summary of the operation of the beamformer and complex non-negative matrix factorizer 105 can be shown with respect to Figure 7. Firstly the complex non-negative matrix factorizer 105 is configured to receive the spatial covariance matrices. Furthermore the complex non-negative matrix factorizer 105 is configured to initialize ¾0; ¾ and vkj with random values uniformly distributed between zero and one.
The initialization of zk , tik and ¾· with random values uniformly distributed between zero and one is shown in Figure 7 by step 801.
The beamformer 107 is configured to initialize wiu according to
iFs
|Wj„],im ==· H^nm(/,k„) - exp (j2nfrnm(ko)} t f η~
and apply w„ <- wfe/|jwioji. The initialization of wio is shown in Figure 7 by step 602.
The complex non-negative matrix factorizer 105 is further configured to recalculate gnitude model ¾ according to Xij ::: z—
The operation of evaluating the magnitude model i»j is shown in Figure 7 by step 603. r 105 is further configured to update tik
Figure imgf000038_0001
The operation of updating tik is shown in Figure 7 by step 804.
The complex non-negative matrix factorizer 105 is further configured to recalculate
X:
magnitude model ¾· according to The operation of recalculating the magnitude model ¾ is shown in Figure 7 by step
The complex non-negative matrix factorizer 105 is further configured to update vkj
1^)0 iifctrfEyWi )'
Vkj +- Vkj 1 +
according to ^.o ^-i^y
The operation of updating ¾? is shown in Figure 7 by step 606.
The complex non-negative matrix factorizer 105 is further configured scale vkj to unity -norm and compensate by rescaling tik as specified in
Figure imgf000039_0001
The operation of scaling ¾ and <<* is shown in Figure 7 by step 607. The complex non-negative matrix factorizer 105 is further configured to recalculate magnitude model ¾■ according to *=,»
The operation of recalculating ¾ is shown in Figure 7 by step 608. The complex non-negative matrix factorizer 105 is further configured to update zko
Ei j tikVkjtv EijWi } -
1 +
according to 2-,¾· ¾¾¾
The operation of updating zko is shown in Figure 7 by step 609 The complex non-negative matrix factorizer 105 is further configured to scale ¾ to i2- norm and compensate by rescaling tik as specified in
The operation of scaling ¾„ and ¾ is shown in Figure 7 by step 610. The complex non-negative matrix factorizer 105 is further configured to recalculate
ί ij ~ 5 ZkotikVkj
magnitude model ¾■ according to *,» The operation of recalculating ¾ is shown in Figure 7 by step 611 ,
The complex non-negative matrix factorizer 105 is further configured to update wio
W<0 <···· Wio [ 53 Zko kVkjXij + 53
Figure imgf000040_0001
according to ,¾ J,*
and w« +~ !Wiui exp(i arg(Wto)), Wio <- Wto/j jWio[|.
The operation of updating wio is shown in Figure 7 by step 612.
The complex non-negative matrix factorizer 105 is further configured to enforce wio to be positive semidefinite by wio VDV-" and apply wio ·;■■■ w«,/||wioji.
The operation of enforcing enforces wio to be positive semidefinite is shown in Figure 7 by step 613.
The complex non-negative matrix factorizer 105 is further configured to repeat steps 803 to 613 for a determined number of iterations or until the parameter updates converge and do not change significantly between updates.
The operation of performing the looping back to step 603 is shown in Figure 7 by step 814,
Furthermore in some embodiments the complex non-negative matrix factorizer 105 is further configured to output the model parameters.
The generation or outputting of the model parameters k ¾ ¾0 wlo is shown in Figure 7 by step 815. in some embodiments the model parameters are output to the separator 109.
The operation of performing complex non-negative matrix factorization with beam kernels is shown in Figure 3 by step 307.
In some embodiments the apparatus comprises a separator 109. The separator 109 in some embodiments is configured to receive the CNMF components and the array input and to separate the audio signals into individual sources. The separation to real acoustic sources therefore requires clustering the complex value non-negative matrix factorized (CNMF) components that were earlier interpreted as individual sources. In other word take the fixed spectral bases of CNMF components and restrict them to model only parts of the real sources. The CNMF components originating from the same source share very similar spatial covariance properties determined by the spatial weights This is illustrated in Figure 8 where it is shown that the target sources 1 :Q 701 spatial responses 702 are reflected in the array 703 which as described herein is factorized into NMF components 705 which are estimated and clustered 708 to generate the estimated or reconstructed sources 1 :Q 707. In some embodiments the clustering is based on the spatial weight similarity and separate clustering algorithms can be used for solving the CNMF component associations to actual acoustic sources mapping.
In the following examples k-means clustering applied on the spatial weights ¾o are used for testing and evaluating the separation quality. In such embodiments the separator 109 is configured so that each ζ . acts as a feature vector and k-means clustering is applied with the desired cluster count being equal to the total number of original sound sources Q. In such embodiments from the k-means clustering a cluster decision is reached that component k belongs to a source g and the cluster decision is denoted by binary parameter b≠. The CNMF magnitude model for a single source magnitude spectrogram can be then defined as h
It would be understood that in some embodiments any other clustering algorithm or CNMF component to source linking strategy can be used to estimate either a binary or a soft decision h≠ and the choice to use the k-means clustering applied on the spatial weights is an example to demonstrate the DoA analysis and modelling performance of the proposed embodiments. In some embodiments other features from the CN F parameters, such as spectral similarity and gain behaviour over time, can be used in parallel for associating the CNMF components to the sources.
The separator 109 in some embodiments is configured to output the clustered sources to a spatial synthesizer 111.
The operation of separation of the sources into clusters is shown in Figure 3 by step 309, In some embodiments the apparatus comprises a spatial synthesizer 111. The spatial synthesizer 111 is configured to receive the clustered source parameters to regenerate the source signals. In some embodiments the reconstruction of the source signals employs a Wiener filter. In such embodiments the source reconstruction filtering is given with the CNMF component clustering decision bkk as,
J ijq— xi_f v1 , .
L,fc„ afcoiifc¾j
where «¾ are now in the role of the real acoustic sources. The time-domain sources are obtained from inverse STFT (iSTFT).
With respect to Figure 9 an example spatial synthesizer 111 according to some embodiments is shown. Furthermore with respect to Figure 10 the operation of the spatial synthesizer shown in Figure 9 is shown.
In some embodiments the spatial synthesizer 111 comprises an inverse short time Fourier transformer (iSTFT) 801 configured to apply an inverse short time Fourier transform to the clustered CNMF parameters output by the separator 109. The inverse short time Fourier transformed parameters are in some embodiments output to a head related Impulse response (HRIR) filter 803,
The operation of applying an iSTFT to the clustered CNMF parameters is shown in Figure 10 by step 901 .
In some embodiments the apparatus comprises a HRIR filter 803 configured to receive the iSTFT clustered CNMF parameters. Furthermore in some embodiments the HRIR filter 803 is configured to receive original STFT observations Kn and construct a Wiener-filter from the CNMF magnitude model parameters to fiiter out the source spectral details in STFT domain. The source reconstruction filtering is given in some embodiments as,
yijk— X-ij <··-. where are the estimated source signals as seen by the array (in other words convolved with their spatial impulse response and x«? is the STFT of the mixture signals captured by the array).
The time-domain signals are obtained in some embodiments from the ISTFT 801 .
In some embodiments the HRIR filter 803 is configured to chosen as the dominant source direction the direction determined by the beams with the highest power and a single HRTF filter associated with this direction used. In some embodiments other binaural spatial synthesis schemes can be employed , for example using several dominant directions in HRIR filtering stages for each source.
In some embodiments the spatial synthesizer 111 comprises a HRTF database configured to store the retrieved HRIR filter components.
The operation of retrieving the HRIR filter components from the HRTF database is shown in Figure 10 by step 902, In the following examples the binaural synthesis accuracy was determined in informal testing as source positioning was determined to resemble the actual positioning of the sources. One example of the spatial weights analysed by the model is given in Figure 11. The original positions are 0 and 90 degrees with zero elevation which can be clearly seen from the analysed weights which produced both good separation and accurate spatial synthesis using the binaural synthesis scheme. in some embodiments the individual spatial covariance for each CNMF component is interpreted as the direction of a single source and the CNMF component can be directly used for the spatial synthesis, in case where only 3D sound synthesis is desired, no separation stage is needed but each CNMF component is synthesized individually and they are positioned to their analysed spatial location with one of the above defined spatial sound field synthesis approaches.
In some embodiments the associated spatial weights are used to determine which HRTFs are used in form of head related impulse response (HRIR) to filter the separated sources. This simulates source positioning by binaural cues such as time- difference of arrival and direction dependent frequency filter caused by the pinna.
The operation of HRIR filtering is show in Figure 10 by 903.
In some embodiments the HRIR filter is configured to output the filtered components to a combiner 805.
In some embodiments the spatial synthesizer 111 comprises a combiner 805 configured to sum or combine the HRIR filter sources. In other words in some embodiments after each source is filtered with its associated direction dependent binaural filter (HRIR) all the individual sources are summed together to form a binaural synthesis signal.
The operation of combining (sum) the filter source is shown in Figure 10 by step 905. It would be understood that any other spatial sound field synthesis technique based on knowing source signals and their multidimensional source positioning can be used, For example in some embodiments the spatial sound field synthesis can be performed by vector base amplitude panning.
The operation of outputting an output (for example a binaural stereo output) is shown in Figures 3 (and 9) by step 313. In some embodiments sparsity can be enforced to the directional weights 2&0 which can be used to guide the algorithm to find spatial covariances for sources that have point-like nature and do not include ambient content in form of reverberation, in some embodiments this can be employed to decrease the cross-talk between the separated sources. In such embodiments the i2- norm scaling of ¾0 specified in
O J
*«* J=3 is defined in such way that a sparsity cost ^v Zko can be used.
Furthermore in some embodiments the CN F component association to sources can be estimated directly. In such embodiments the clustering decision is a non- negative parameter that can be included in the model and an update rule based on the partial derivation can be obtained. In these embodiments the SCMs are estimated with respect to each real acoustic sound source and similarly to the presented model the frequency dependency using DSBFs is maintained. In the following the separation performance of some embodiments is demonstrated by comparing the results of embodiments against conventional BSS methods suitable for the particular case of small microphone array captures with a reasonable amount of reverberation. The separation performance is determined by the separation quality measures, the signal-to-distortion ratio (SDR), image-to-spatial distortion ratio (ISR), signal-to-interference ratio (SIR) and signal-to-artifact ratio (SAR) such as provided by a BSS Eval toolbox In the following examples the separation evaluation is performed using impulse responses recorded in a regular meeting room from various directions. The impulse responses of each source location to the microphone array were obtained using maximum length sequence (MLS) method and using Genelec 1029A speakers for the MLS excitation. The anechoic test samples were convolved with the impulse responses and summed together to simulate sources that are captured simultaneously. The sampling frequency used was 48kHz, the room dimensions were approximately 5 x 12 x 2.59 m and the reverberation time (T60) was approximately 0.35s. The capture array in the following examples consists of four DPA 4080-BM prepoiarized omnidirectional miniature condensator microphones. The array dimensions are given by the following table x (sum)
1 0 -46 8
2 -22 ~8 8
3 22 -8 6
4 0 61 -18
and the array geometry with reference axis is illustrated in Figure 12.
The spatial aliasing frequency for the given array is 1563 Hz.
The signal length for the embodiments following examples to analyze was set to 10 seconds in each test case, the STFT window length was 2048 samples with 50% window overlap. The determined number of iterations for non-negative matrix factorization in the following examples was 200 iterations. Furthermore for the following examples the impulse responses were captured from following angles [0, 40, 90. 135, 180, 305] 0f which a several combinations with different source types (speech, music and noise) were chosen for the evaluation. After convolving the impulse responses the signals were downsampled to 24kHz. The spatial sampling defined by all look direction vectors for different DSBFs are illustrated in Figure 6 which comprises 110 beam directions which approximately uniformly sample the unit sphere surface around the array. In this example the lateral resolution at zero elevation, in other words at the flat surface of the array, is 10 degrees and the different elevations are at 22.5 degrees spacing. The azimuth resolution in such embodiments is decreased close to the poles of the unit sphere. The separation performance was compared to the CNMF method proposed herein.
In all the following test cases a total of five 10 second signals were tested and separation measures were averaged over ail test signals and all sources within the test case.
In the first test case two speakers with 90 degrees spacing were tested. The source positioning is illustrated in Figure 13 (speaker 1 at 0 degrees and speaker 2 at 90 degrees) and the separation results are given by the following table
Reference Embodiment
Example 1
SDR (dB) 320 7Λ8
ISR (dB) 7.69 12.45
SIR (dB) 4.19 11.11
SAR (dB) 10.70 11.09
in a second test case a combination of a speaker and an artificial pink noise source with 90 degrees spacing was tested from different angles (speaker at 45 degrees and pink noise from 135 degrees). The source positioning is illustrated in Figure 14 and the separation results are given in the following table
Reference Embodiment
Example 2
SDR (dB) 3J5 5J06
ISR (dB) 7.95 9.30
SIR (dB) 4.03 6.57 10.52 9.38
In the third test case one speaker and a television broadcast with a 145 degree spacing was tested. The source positioning is illustrated in Figure 15 (TV source at 305 degrees and speaker at 90 degrees) and the separation results are given in the following table
Figure imgf000048_0001
SDR (cJB) 3.50 4,69
ISR (dB) 7.30 10.54
SI (dB) 3.45 8.51
SAR (dB) 10.28 9.55
It would be understood that the achieved separation performance indicates that the embodiments as shown in the example beamforming based spatial covariance matrix estimation for source separation systematically increases the separation compared to the reference that is also based on spatial covariance estimation. Furthermore it would be understood that the separation performance improvement is greatest with speech sources with significant azimuth difference and decreases with difficult broadband noises such as artificial pink noise.
It would be understood that although the following examples show the operations capturing audio signals, analysis of the audio signal to generate modelled audio sources, and regeneration of the audio signals from the modelled audio sources in some embodiments the apparatus and for example the spatial synthesizer can be configured to process the modelled audio sources according to any suitable means. For example as shown in Figures 18 and 17 an example processing is shown wherein in Figure 16 an example of a male speaker and two other speakers are present, in this exampie the microphones are omnidirectional and ali speakers are present in both channels. However the male speaker is to the right of the microphone array and therefore his voice is dominant in the right channel (right channel-the lower of the two channels). In Figure 17, the graph shows where the male speaker is removed by removing the audio source estimated according to the embodiments described herein. In such embodiments as can be seen the male speaker is attenuated by approximately 12 dB in the right channel without affecting the sound from other speakers that dominate the left channel. It shall be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers, as well as wearable devices.
In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be impiemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
The embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware, Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.
Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate. Programs, such as those provided by Synopsys, inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.
The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention as defined in the appended claims.

Claims

CLAIMS:
1. A method for separating audio sources comprising:
receiving at least two audio signals;
transforming the at least two audio signals into a frequency domain representation of the at least two signals;
generating an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals;
generating a spatial covariance matrix model comprising at least one beamformer kernel;
generating a linear magnitude model of audio objects;
combining the spatial covariance matrix model and the linear magnitude model;
determining at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and
clustering the audio objects based on the at least one combination parameter to create separated audio sources. 2. The method as claimed in claim 1 , wherein receiving the at least two audio signals comprises at least one of:
the audio signals being generated from microphones receiving acoustic waves from at least one audio source; and
the audio signals being received from a memory, the audio signals describing acoustic waves from at least one audio source.
3. The method as claimed in claims 1 to 2, wherein transforming the at least two audio signals into a frequency domain representation of the at least two signals comprises performing a time to frequency transform on the at least two audio signals.
4. The method as claimed in claims 1 to 3, wherein generating the observed spatial covariance matrix from the frequency domain representations of the at least two audio signals comprises generating an observed spatiai covariance matrix from an autocovariance of the two frequency domain representations.
5. The method as claimed in claims 1 to 4, further comprising generating at least one beamformer kernel by generating time delays in a frequency domain of a set of look directions sampling the spatial space around the microphone array,
8. The method as claimed in claim 5, wherein generating at least one beamformer kernel comprises generating an autocovariance of the time delays in the frequency domain of a set of look directions sampling the spatial space around the microphone array,
7, The method as claimed in claims 1 to 6, wherein generating a linear model of audio objects comprises performing a non-negative matrix factorization for audio object magnitudes.
8, The method as claimed in claim 7, wherein performing a non-negative matrix factorization for audio object magnitudes comprises initialising linear model parameters and %? with random values uniformly distributed between zero and one;
9, The method as claimed in claims 1 to 8, wherein combining spatial covariance matrix model and the linear magnitude model, comprises multiplying the spatial covariance matrix model by a magnitude envelope represented by the linear model.
10, The method as claimed in claims 1 to 9 wherein determining at least one combination parameter comprises determining spatial covariance model parameters to minimize the difference between the observed spatial covariance matrix and the spatial covariance model,
11 , The method as claimed in claim 10 wherein determining the spatial covariance model parameters comprises performing an iterative optimization based on a complex-valued non-negative matrix factorization, such that the parameters of the linear magnitude mode! representing the at least one audio object are iteratively updated until either the linear magnitude mode! parameters do not change significantly between updates or a defined number of iterations are processed. 12. The method as claimed in claim 11 , wherein performing the iterative optimization based on the complex value non-negative factorization comprises:
updating linear model parameters and l¾ according to an optimization framework of complex non-negative matrix factorization; and
updating the spatial covariance matrix model magnitudes.
13. The method as claimed in claim 12 wherein updating the spatial covariance matrix model magnitudes comprises updating the relative magnitude difference of estimates of the at least two audio signals, 14. The method as claimed in claims 8 and 11 to 13, wherein clustering the audio objects based on the at least one combination model parameter to create separated audio sources comprises generating at least one audio source from the at least one audio object and from the estimated linear magnitude model parameters. 15, The method as claimed in claims 1 to 14, further comprising synthesising at least one audio signal based on the at least one combination model parameters associated with the at least one audio source.
18. An apparatus comprising:
means for receiving at least two audio signals;
means for transforming the at least two audio signals into a frequency domain representation of the at least two signals;
means for generating an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals;
means for generating a spatial covariance matrix model comprising at least one beamformer kernel;
means for generating a linear magnitude model of audio objects; means for combining the spatiai covariance matrix mode! and the linear magnitude model;
means for determining at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and means for clustering the audio objects based on the at least one combination parameter to create separated audio sources.
17. An apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least:
receive at least two audio signals;
transform the at least two audio signals into a frequency domain representation of the at least two signals;
generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals;
generate a spatial covariance matrix model comprising at least one beamformer kernel;
generate a linear magnitude model of audio objects;
combine the spatial covariance matrix model and the linear magnitude model; determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and
clustering the audio objects based on the at least one combination parameter to create separated audio sources.
18. An apparatus comprising:
an input configured to receive at least two audio signals;
a frequency domain transformer configured to transform the at least two audio signals into a frequency domain representation of the at least two signals;
a spatial covariance processor configured to generate an observed spatial covariance matrix from the frequency domain representations of the at least two audio signals; a beamformer configured to generate a spatial covariance matrix model comprising at least one beamformer kernel;
a matrix factorizer configured to generate a linear magnitude model of audio objects; to combine the spatial covariance matrix model and the linear magnitude model; and further configured to determine at least one combination parameter, such that the at least one parameter for the combination attempts to optimise the combination; and
a separator configured to cluster the audio objects based on the at least one combination parameter to create separated audio sources,
PCT/IB2013/052212 2013-03-20 2013-03-20 Spatial audio apparatus WO2014147442A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP13878649.6A EP2976893A4 (en) 2013-03-20 2013-03-20 Spatial audio apparatus
PCT/IB2013/052212 WO2014147442A1 (en) 2013-03-20 2013-03-20 Spatial audio apparatus
CN201380076395.7A CN105230044A (en) 2013-03-20 2013-03-20 Space audio device
US14/777,825 US9788119B2 (en) 2013-03-20 2013-03-20 Spatial audio apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IB2013/052212 WO2014147442A1 (en) 2013-03-20 2013-03-20 Spatial audio apparatus

Publications (1)

Publication Number Publication Date
WO2014147442A1 true WO2014147442A1 (en) 2014-09-25

Family

ID=51579360

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2013/052212 WO2014147442A1 (en) 2013-03-20 2013-03-20 Spatial audio apparatus

Country Status (4)

Country Link
US (1) US9788119B2 (en)
EP (1) EP2976893A4 (en)
CN (1) CN105230044A (en)
WO (1) WO2014147442A1 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016050780A1 (en) * 2014-10-02 2016-04-07 Sony Corporation Method, apparatus and system
WO2016102752A1 (en) 2014-12-22 2016-06-30 Nokia Technologies Oy Audio processing based upon camera selection
CN105989851A (en) * 2015-02-15 2016-10-05 杜比实验室特许公司 Audio source separation
WO2017044208A1 (en) * 2015-09-09 2017-03-16 Microsoft Technology Licensing, Llc Microphone placement for sound source direction estimation
WO2017108097A1 (en) * 2015-12-22 2017-06-29 Huawei Technologies Duesseldorf Gmbh Localization algorithm for sound sources with known statistics
US20170245082A1 (en) * 2016-02-18 2017-08-24 Google Inc. Signal processing methods and systems for rendering audio on virtual loudspeaker arrays
WO2017143003A1 (en) * 2016-02-18 2017-08-24 Dolby Laboratories Licensing Corporation Processing of microphone signals for spatial playback
EP3239981A1 (en) * 2016-04-26 2017-11-01 Nokia Technologies Oy Methods, apparatuses and computer programs relating to modification of a characteristic associated with a separated audio signal
US9875080B2 (en) 2014-07-17 2018-01-23 Nokia Technologies Oy Method and apparatus for an interactive user interface
US9949052B2 (en) 2016-03-22 2018-04-17 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
EP3343559A1 (en) * 2016-12-29 2018-07-04 Beijing Xiaoniao Tingting Technology Co., Ltd De-reverberation control method and device of sound producing equipment
US10176826B2 (en) 2015-02-16 2019-01-08 Dolby Laboratories Licensing Corporation Separating audio sources
RU2678650C2 (en) * 2014-12-11 2019-01-30 Долби Лэборетериз Лайсенсинг Корпорейшн Clustering of audio objects with metadata preservation
EP3503592A1 (en) 2017-12-19 2019-06-26 Nokia Technologies Oy Methods, apparatuses and computer programs relating to spatial audio
CN110088835A (en) * 2016-12-28 2019-08-02 谷歌有限责任公司 Use the blind source separating of similarity measure
US10959032B2 (en) 2016-02-09 2021-03-23 Dolby Laboratories Licensing Corporation System and method for spatial processing of soundfield signals
US11152014B2 (en) 2016-04-08 2021-10-19 Dolby Laboratories Licensing Corporation Audio source parameterization
US11234072B2 (en) 2016-02-18 2022-01-25 Dolby Laboratories Licensing Corporation Processing of microphone signals for spatial playback
RU2773512C2 (en) * 2014-12-11 2022-06-06 Долби Лэборетериз Лайсенсинг Корпорейшн Clustering audio objects with preserving metadata
CN114639398A (en) * 2022-03-10 2022-06-17 电子科技大学 Broadband DOA estimation method based on microphone array

Families Citing this family (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10860683B2 (en) 2012-10-25 2020-12-08 The Research Foundation For The State University Of New York Pattern change discovery between high dimensional data sets
US20170178664A1 (en) * 2014-04-11 2017-06-22 Analog Devices, Inc. Apparatus, systems and methods for providing cloud based blind source separation services
US9825693B1 (en) * 2014-08-26 2017-11-21 Northrop Grumman Systems Corporation Method and apparatus for symbol measurement and combining
US20160071526A1 (en) * 2014-09-09 2016-03-10 Analog Devices, Inc. Acoustic source tracking and selection
US9554207B2 (en) 2015-04-30 2017-01-24 Shure Acquisition Holdings, Inc. Offset cartridge microphones
US9565493B2 (en) 2015-04-30 2017-02-07 Shure Acquisition Holdings, Inc. Array microphone system and method of assembling the same
US10079028B2 (en) * 2015-12-08 2018-09-18 Adobe Systems Incorporated Sound enhancement through reverberation matching
US9721582B1 (en) * 2016-02-03 2017-08-01 Google Inc. Globally optimized least-squares post-filtering for speech enhancement
WO2017176941A1 (en) * 2016-04-08 2017-10-12 Dolby Laboratories Licensing Corporation Audio source parameterization
US9837064B1 (en) 2016-07-08 2017-12-05 Cisco Technology, Inc. Generating spectrally shaped sound signal based on sensitivity of human hearing and background noise level
EP3507993B1 (en) 2016-08-31 2020-11-25 Dolby Laboratories Licensing Corporation Source separation for reverberant environment
GB2556093A (en) * 2016-11-18 2018-05-23 Nokia Technologies Oy Analysis of spatial metadata from multi-microphones having asymmetric geometry in devices
US10573291B2 (en) 2016-12-09 2020-02-25 The Research Foundation For The State University Of New York Acoustic metamaterial
US10367948B2 (en) 2017-01-13 2019-07-30 Shure Acquisition Holdings, Inc. Post-mixing acoustic echo cancellation systems and methods
US10460727B2 (en) * 2017-03-03 2019-10-29 Microsoft Technology Licensing, Llc Multi-talker speech recognizer
US11468884B2 (en) * 2017-05-08 2022-10-11 Sony Corporation Method, apparatus and computer program for detecting voice uttered from a particular position
CN107484070B (en) * 2017-08-22 2019-07-19 广州山水比德设计股份有限公司 Pond face music landscape system
US10334357B2 (en) 2017-09-29 2019-06-25 Apple Inc. Machine learning based sound field analysis
GB201718341D0 (en) 2017-11-06 2017-12-20 Nokia Technologies Oy Determination of targeted spatial audio parameters and associated spatial audio playback
US20190297422A1 (en) * 2018-03-20 2019-09-26 3Dio, Llc Binaural recording device with directional enhancement
CN108766457B (en) * 2018-05-30 2020-09-18 北京小米移动软件有限公司 Audio signal processing method, audio signal processing device, electronic equipment and storage medium
CN112205006B (en) * 2018-06-01 2022-08-26 索尼公司 Adaptive remixing of audio content
EP3804356A1 (en) 2018-06-01 2021-04-14 Shure Acquisition Holdings, Inc. Pattern-forming microphone array
CN108769874B (en) * 2018-06-13 2020-10-20 广州国音科技有限公司 Method and device for separating audio in real time
US11297423B2 (en) 2018-06-15 2022-04-05 Shure Acquisition Holdings, Inc. Endfire linear array microphone
CN109119092B (en) * 2018-08-31 2021-08-20 广东美的制冷设备有限公司 Beam direction switching method and device based on microphone array
US11227621B2 (en) 2018-09-17 2022-01-18 Dolby International Ab Separating desired audio content from undesired content
WO2020061353A1 (en) 2018-09-20 2020-03-26 Shure Acquisition Holdings, Inc. Adjustable lobe shape for array microphones
US11438691B2 (en) 2019-03-21 2022-09-06 Shure Acquisition Holdings, Inc. Auto focus, auto focus within regions, and auto placement of beamformed microphone lobes with inhibition functionality
CN113841419A (en) 2019-03-21 2021-12-24 舒尔获得控股公司 Housing and associated design features for ceiling array microphone
US11558693B2 (en) 2019-03-21 2023-01-17 Shure Acquisition Holdings, Inc. Auto focus, auto focus within regions, and auto placement of beamformed microphone lobes with inhibition and voice activity detection functionality
EP3973716A1 (en) 2019-05-23 2022-03-30 Shure Acquisition Holdings, Inc. Steerable speaker array, system, and method for the same
US11302347B2 (en) 2019-05-31 2022-04-12 Shure Acquisition Holdings, Inc. Low latency automixer integrated with voice and noise activity detection
CN110400575B (en) * 2019-07-24 2024-03-29 腾讯科技（深圳）有限公司 Inter-channel feature extraction method, audio separation method and device and computing equipment
JP7243840B2 (en) * 2019-08-21 2023-03-22 日本電信電話株式会社 Estimation device, estimation method and estimation program
EP4018680A1 (en) 2019-08-23 2022-06-29 Shure Acquisition Holdings, Inc. Two-dimensional microphone array with improved directivity
CN110517702B (en) * 2019-09-06 2022-10-04 腾讯科技（深圳）有限公司 Signal generation method, and voice recognition method and device based on artificial intelligence
BR112022006331A2 (en) * 2019-10-04 2022-06-28 Fraunhofer Ges Forschung APPARATUS AND METHOD TO OBTAIN A PLURALITY OF OUTPUT SIGNALS, TELECONFERENCE DEVICE, BINAURAL SYSTEM, OPTIMIZER, OPTIMIZATION METHOD AND NON-TRANSITORY STORAGE UNIT
US11638111B2 (en) 2019-11-01 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for classifying beamformed signals for binaural audio playback
US12028678B2 (en) 2019-11-01 2024-07-02 Shure Acquisition Holdings, Inc. Proximity microphone
US11552611B2 (en) 2020-02-07 2023-01-10 Shure Acquisition Holdings, Inc. System and method for automatic adjustment of reference gain
US11706562B2 (en) 2020-05-29 2023-07-18 Shure Acquisition Holdings, Inc. Transducer steering and configuration systems and methods using a local positioning system
US11678111B1 (en) * 2020-07-22 2023-06-13 Apple Inc. Deep-learning based beam forming synthesis for spatial audio
JP7387565B2 (en) * 2020-09-16 2023-11-28 株式会社東芝 Signal processing device, trained neural network, signal processing method, and signal processing program
JP2024505068A (en) 2021-01-28 2024-02-02 シュアー アクイジッション ホールディングス インコーポレイテッド Hybrid audio beamforming system
CN113035209B (en) * 2021-02-25 2023-07-04 北京达佳互联信息技术有限公司 Three-dimensional audio acquisition method and three-dimensional audio acquisition device
CN114678037B (en) * 2022-04-13 2022-10-25 北京远鉴信息技术有限公司 Overlapped voice detection method and device, electronic equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1473964A2 (en) * 2003-05-02 2004-11-03 Samsung Electronics Co., Ltd. Microphone array, method to process signals from this microphone array and speech recognition method and system using the same
US20090012779A1 (en) * 2007-03-05 2009-01-08 Yohei Ikeda Sound source separation apparatus and sound source separation method
US20090055170A1 (en) * 2005-08-11 2009-02-26 Katsumasa Nagahama Sound Source Separation Device, Speech Recognition Device, Mobile Telephone, Sound Source Separation Method, and Program
US20090164212A1 (en) * 2007-12-19 2009-06-25 Qualcomm Incorporated Systems, methods, and apparatus for multi-microphone based speech enhancement
US20110096915A1 (en) * 2009-10-23 2011-04-28 Broadcom Corporation Audio spatialization for conference calls with multiple and moving talkers
US20110123046A1 (en) * 2009-11-20 2011-05-26 Atsuo Hiroe Signal processing apparatus, signal processing method, and program therefor
US20110142252A1 (en) * 2009-12-11 2011-06-16 Oki Electric Industry Co., Ltd. Source sound separator with spectrum analysis through linear combination and method therefor
WO2012061149A1 (en) * 2010-10-25 2012-05-10 Qualcomm Incorporated Three-dimensional sound capturing and reproducing with multi-microphones

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102006050068B4 (en) * 2006-10-24 2010-11-11 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method for generating an environmental signal from an audio signal, apparatus and method for deriving a multi-channel audio signal from an audio signal and computer program
US8036601B2 (en) * 2007-03-13 2011-10-11 Nec Laboratories America, Inc. Group LMMSE demodulation using noise and interference covariance matrix for reception on a cellular downlink
US8015003B2 (en) * 2007-11-19 2011-09-06 Mitsubishi Electric Research Laboratories, Inc. Denoising acoustic signals using constrained non-negative matrix factorization
JP5195652B2 (en) * 2008-06-11 2013-05-08 ソニー株式会社 Signal processing apparatus, signal processing method, and program
US10726861B2 (en) * 2010-11-15 2020-07-28 Microsoft Technology Licensing, Llc Semi-private communication in open environments
JP2012163918A (en) * 2011-02-09 2012-08-30 Sony Corp Voice signal processing apparatus, voice signal processing method, and program
KR101243897B1 (en) * 2011-06-24 2013-03-20 서강대학교산학협력단 Blind Source separation method in reverberant environments based on estimation of time delay and attenuation of the signals

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1473964A2 (en) * 2003-05-02 2004-11-03 Samsung Electronics Co., Ltd. Microphone array, method to process signals from this microphone array and speech recognition method and system using the same
US20090055170A1 (en) * 2005-08-11 2009-02-26 Katsumasa Nagahama Sound Source Separation Device, Speech Recognition Device, Mobile Telephone, Sound Source Separation Method, and Program
US20090012779A1 (en) * 2007-03-05 2009-01-08 Yohei Ikeda Sound source separation apparatus and sound source separation method
US20090164212A1 (en) * 2007-12-19 2009-06-25 Qualcomm Incorporated Systems, methods, and apparatus for multi-microphone based speech enhancement
US20110096915A1 (en) * 2009-10-23 2011-04-28 Broadcom Corporation Audio spatialization for conference calls with multiple and moving talkers
US20110123046A1 (en) * 2009-11-20 2011-05-26 Atsuo Hiroe Signal processing apparatus, signal processing method, and program therefor
US20110142252A1 (en) * 2009-12-11 2011-06-16 Oki Electric Industry Co., Ltd. Source sound separator with spectrum analysis through linear combination and method therefor
WO2012061149A1 (en) * 2010-10-25 2012-05-10 Qualcomm Incorporated Three-dimensional sound capturing and reproducing with multi-microphones

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
HAOHAI SUN; ET AL.: "OPTIMAL HIGHER ORDER AMBISONICS ENCODING WITH PREDEFINED CONSTRAINTS", IEEE TRANSACTIONS ON AUDIO, SPEECH AND LANGUAGE PROCESSING, vol. 20, no. 3, 1 March 2012 (2012-03-01), NEW YORK, NY, USA, pages 742 - 754, XP011391644, DOI: 10.1109/TASL.2011.2164532 *
See also references of EP2976893A4 *
STEVEN J RENNIE; ET AL.: "Variational Probabilistic Speech Separation Using Microphone Arrays", IEEE TRANSACTIONS ON AUDIO, SPEECH AND LANGUAGE PROCESSING, vol. 15, no. 1, 1 January 2007 (2007-01-01), pages 135 - 149, XP011151932, DOI: 10.1109/TASL.2006.876865 *

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9875080B2 (en) 2014-07-17 2018-01-23 Nokia Technologies Oy Method and apparatus for an interactive user interface
US11550541B2 (en) 2014-07-17 2023-01-10 Nokia Technologies Oy Method and apparatus for an interactive user interface
US10789042B2 (en) 2014-07-17 2020-09-29 Nokia Technologies Oy Method and apparatus for an interactive user interface
WO2016050780A1 (en) * 2014-10-02 2016-04-07 Sony Corporation Method, apparatus and system
US10657973B2 (en) 2014-10-02 2020-05-19 Sony Corporation Method, apparatus and system
US11937064B2 (en) 2014-12-11 2024-03-19 Dolby Laboratories Licensing Corporation Metadata-preserved audio object clustering
RU2773512C2 (en) * 2014-12-11 2022-06-06 Долби Лэборетериз Лайсенсинг Корпорейшн Clustering audio objects with preserving metadata
US11363398B2 (en) 2014-12-11 2022-06-14 Dolby Laboratories Licensing Corporation Metadata-preserved audio object clustering
US20220272474A1 (en) * 2014-12-11 2022-08-25 Dolby Laboratories Licensing Corporation Metadata-preserved audio object clustering
RU2678650C2 (en) * 2014-12-11 2019-01-30 Долби Лэборетериз Лайсенсинг Корпорейшн Clustering of audio objects with metadata preservation
EP3503580A1 (en) 2014-12-22 2019-06-26 Nokia Technologies Oy Audio processing based upon camera selection
WO2016102752A1 (en) 2014-12-22 2016-06-30 Nokia Technologies Oy Audio processing based upon camera selection
US10192568B2 (en) 2015-02-15 2019-01-29 Dolby Laboratories Licensing Corporation Audio source separation with linear combination and orthogonality characteristics for spatial parameters
CN105989851B (en) * 2015-02-15 2021-05-07 杜比实验室特许公司 Audio source separation
CN105989851A (en) * 2015-02-15 2016-10-05 杜比实验室特许公司 Audio source separation
US10176826B2 (en) 2015-02-16 2019-01-08 Dolby Laboratories Licensing Corporation Separating audio sources
US9788109B2 (en) 2015-09-09 2017-10-10 Microsoft Technology Licensing, Llc Microphone placement for sound source direction estimation
CN108028977A (en) * 2015-09-09 2018-05-11 微软技术许可有限责任公司 Microphone for Sounnd source direction estimation is placed
CN108028977B (en) * 2015-09-09 2020-03-03 微软技术许可有限责任公司 Microphone placement for sound source direction estimation
WO2017044208A1 (en) * 2015-09-09 2017-03-16 Microsoft Technology Licensing, Llc Microphone placement for sound source direction estimation
US10901063B2 (en) 2015-12-22 2021-01-26 Huawei Technologies Duesseldorf Gmbh Localization algorithm for sound sources with known statistics
WO2017108097A1 (en) * 2015-12-22 2017-06-29 Huawei Technologies Duesseldorf Gmbh Localization algorithm for sound sources with known statistics
US10959032B2 (en) 2016-02-09 2021-03-23 Dolby Laboratories Licensing Corporation System and method for spatial processing of soundfield signals
US10142755B2 (en) * 2016-02-18 2018-11-27 Google Llc Signal processing methods and systems for rendering audio on virtual loudspeaker arrays
US11706564B2 (en) 2016-02-18 2023-07-18 Dolby Laboratories Licensing Corporation Processing of microphone signals for spatial playback
WO2017143003A1 (en) * 2016-02-18 2017-08-24 Dolby Laboratories Licensing Corporation Processing of microphone signals for spatial playback
US11234072B2 (en) 2016-02-18 2022-01-25 Dolby Laboratories Licensing Corporation Processing of microphone signals for spatial playback
US20170245082A1 (en) * 2016-02-18 2017-08-24 Google Inc. Signal processing methods and systems for rendering audio on virtual loudspeaker arrays
US9949052B2 (en) 2016-03-22 2018-04-17 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
US11356787B2 (en) 2016-03-22 2022-06-07 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
US11843930B2 (en) 2016-03-22 2023-12-12 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
US10897682B2 (en) 2016-03-22 2021-01-19 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
US10405120B2 (en) 2016-03-22 2019-09-03 Dolby Laboratories Licensing Corporation Adaptive panner of audio objects
US11152014B2 (en) 2016-04-08 2021-10-19 Dolby Laboratories Licensing Corporation Audio source parameterization
CN107316650A (en) * 2016-04-26 2017-11-03 诺基亚技术有限公司 Method, device and the computer program of the modification of the feature associated on the audio signal with separating
CN107316650B (en) * 2016-04-26 2020-12-18 诺基亚技术有限公司 Method, apparatus and computer program product for modifying features associated with separate audio signals
EP3239981A1 (en) * 2016-04-26 2017-11-01 Nokia Technologies Oy Methods, apparatuses and computer programs relating to modification of a characteristic associated with a separated audio signal
CN110088835A (en) * 2016-12-28 2019-08-02 谷歌有限责任公司 Use the blind source separating of similarity measure
CN110088835B (en) * 2016-12-28 2024-03-26 谷歌有限责任公司 Blind source separation using similarity measures
EP3343559A1 (en) * 2016-12-29 2018-07-04 Beijing Xiaoniao Tingting Technology Co., Ltd De-reverberation control method and device of sound producing equipment
US10410651B2 (en) 2016-12-29 2019-09-10 Beijing Xiaoniao Tingting Technology Co., LTD. De-reverberation control method and device of sound producing equipment
WO2019123060A1 (en) 2017-12-19 2019-06-27 Nokia Technologies Oy Methods, apparatuses and computer programs relating to spatial audio
EP3503592A1 (en) 2017-12-19 2019-06-26 Nokia Technologies Oy Methods, apparatuses and computer programs relating to spatial audio
CN114639398A (en) * 2022-03-10 2022-06-17 电子科技大学 Broadband DOA estimation method based on microphone array
CN114639398B (en) * 2022-03-10 2023-05-26 电子科技大学 Broadband DOA estimation method based on microphone array

Also Published As

Publication number Publication date
EP2976893A4 (en) 2016-12-14
EP2976893A1 (en) 2016-01-27
US9788119B2 (en) 2017-10-10
US20160073198A1 (en) 2016-03-10
CN105230044A (en) 2016-01-06

Similar Documents

Publication Publication Date Title
US9788119B2 (en) Spatial audio apparatus
EP3320692B1 (en) Spatial audio processing apparatus
Nikunen et al. Direction of arrival based spatial covariance model for blind sound source separation
JP5814476B2 (en) Microphone positioning apparatus and method based on spatial power density
TWI530201B (en) Sound acquisition via the extraction of geometrical information from direction of arrival estimates
US10873814B2 (en) Analysis of spatial metadata from multi-microphones having asymmetric geometry in devices
Politis et al. Sector-based parametric sound field reproduction in the spherical harmonic domain
JP5229053B2 (en) Signal processing apparatus, signal processing method, and program
WO2018060550A1 (en) Spatial audio signal format generation from a microphone array using adaptive capture
CN109804559A (en) Gain control in spatial audio systems
EP3409025A1 (en) System and apparatus for tracking moving audio sources
US11350213B2 (en) Spatial audio capture
McCormack et al. Parametric acoustic camera for real-time sound capture, analysis and tracking
Smaragdis et al. Position and trajectory learning for microphone arrays
Yang et al. Geometrically constrained source extraction and dereverberation based on joint optimization
Çöteli et al. Multiple sound source localization with rigid spherical microphone arrays via residual energy test
Gonzalez et al. Spherical decomposition of arbitrary scattering geometries for virtual acoustic environments
Mathews Development and evaluation of spherical microphone array-enabled systems for immersive multi-user environments
Nikunen et al. Source separation and reconstruction of spatial audio using spectrogram factorization
Chern et al. Voice Direction-Of-Arrival Conversion
Sun et al. Optimal 3-D hoa encoding with applications in improving close-spaced source localization
Rettberg et al. Practical aspects of the calibration of spherical microphone arrays
Jin et al. SUPER-RESOLUTION SOUND FIELD ANALYSES
Kornycky Acoustic source separation for static and moving targets

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201380076395.7

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13878649

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2013878649

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 14777825

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE