EP2760021B1 - Sound field spatial stabilizer - Google Patents
Sound field spatial stabilizer Download PDFInfo
- Publication number
- EP2760021B1 EP2760021B1 EP13153065.1A EP13153065A EP2760021B1 EP 2760021 B1 EP2760021 B1 EP 2760021B1 EP 13153065 A EP13153065 A EP 13153065A EP 2760021 B1 EP2760021 B1 EP 2760021B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- signal
- microphone signals
- gains
- implemented method
- computer implemented
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L2021/02082—Noise filtering the noise being echo, reverberation of the speech
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
- G10L2021/02165—Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
Definitions
- the present disclosure relates to the field of processing sound fields.
- a system and method for maintaining the spatial stability of a sound field are known in the art.
- Stereo and multichannel microphone configurations may be used for processing a sound field that is a spatial representation of an audible environment associated with the microphones.
- the audio received from the microphones may be used to reproduce the sound field using audio transducers.
- Many computing devices may have multiple integrated microphones used for recording an audible environment associated with the computing device and communicating with other users.
- Computing devices typically use multiple microphones to improve noise performance with noise suppression processes.
- the noise suppression processes may result in the reduction or loss of spatial information.
- the noise suppression processing may result in a single, or mono, output signal that has no spatial information.
- WO 2011/112382 relates to a method for filtering a multi-channel audio signal.
- US 7117145 relates to an adaptive feature for speech enhancement.
- EP 1830348 relates to a hands free system for speech signal acquisition in a vehicular cabin.
- WO 2002/19768 relates to deriving audio channels from audio signals.
- balance gains may be calculated for each of two or more microphone signals.
- the balance gains may be associated with a spatial image in the sound field.
- Signal values may be calculated for each of the received microphone signals.
- the signal values may be signal estimates or signal gains calculated to improve a characteristic of the microphone signals.
- the differences between the signal values associated with each microphone signal are limited to mitigate audible distortions in the spatial image. Some difference between signal values may be allowable in order to improve the audible characteristics of the received microphone signals.
- One or more microphone signals are adjusted responsive to the two or more balance gains and the signal gains to maintain the spatial stability of the sound field.
- the adjustments of one or more microphone signals may include mixing of two or more microphone signals. Further adjustments to the signal gains may be made responsive to the mixing process.
- the signal gains are applied respectively to each of the two or more microphone signals.
- Figure 1 is a schematic representation of a system for maintaining the spatial stability of a sound field 100.
- Two or more microphones 102 receive the sound field.
- Stereo and multichannel microphone configurations may be utilized for processing the sound field that is a spatial representation of an audible environment associated with the microphones 102.
- Many audible environments associated with the microphones 102 may include undesirable content that may be mitigated by processing the received sound field.
- Microphones 102 that are arranged in a far field configuration typically receive more undesirable content, noise, than microphones 102 in a near field configuration.
- Far field configurations may include, for example, a hands free phone, a conference phone and microphones embedded into an automobile.
- Far field configurations are capable of receiving a sound field that represents the spatial environment associated with the microphones 102.
- Near field configurations typically place the microphone 102 in close proximity to a user. Undesirable content may be mitigated in both near and far field configurations by processing the received sound field.
- Processing that may mitigate undesirable content received in the sound field may include echo cancellation and noise reduction processes.
- Echo cancellation, noise reduction and other audio processing processes may calculate one or more suppression, or signal, gains utilizing a suppression gain calculator 106.
- An echo cancellation process and a noise reduction process may each calculate one or more signal gains. Each respective signal gain may be applied individually or a composite signal gain may be applied to process the sound field using a gain filter 114.
- Echo cancellation processing mitigates echoes caused by signal feedback between two or more communication devices. Signal feedback occurs when an audio transducer on a first communication device reproduces the signal received from a second communication device and subsequently the microphones on the first communication device recapture the reproduced signal.
- the recaptured signal may be transmitted to the second communication device where the recaptured signal may be perceived as an echo of the previously transmitted signal.
- Echo cancellation processes may detect when the signal has been recaptured and attempt to suppress the recaptured signal.
- Many different echo cancellation processes may mitigate echoes by calculating one or more signal gains that, when applied to the signals received by the microphones 102, suppress the echoes.
- the echo suppression gain may be calculated using coherence calculation between the predicted echo and the microphone disclosed in U.S. Patent No. 8,036,879 .
- the echo cancellation process may determine a large amount of suppression, or calculate large signal gains, as a result of the signal produced by the audio transducer dominating, or coupling with, the microphone 102.
- the echo cancellation process may determine that a large amount of suppression may mitigate the signal produced by the audio transducer from dominating, or coupling with, the microphone 102.
- the echo cancellation process may calculate large signal gains to mitigate the coupling.
- the large signal gains may result in a gating effect where the communication device effectively supports only half duplex communication.
- Half duplex communication may occur when the communication channel allows for reliable communication from alternatively either the far side or near side but not both simultaneously.
- the large signal gains may suppress the coupling but may also suppress all content, including desired voice content resulting in half duplex communication.
- Background noise is another type of undesirable signal content that may be mitigated by processing the received sound field.
- Many different types of noise reduction processing techniques may mitigate background noise.
- An exemplary noise reduction method is a recursive Wiener filter.
- N ⁇ i,k is a background noise estimate.
- the background noise estimate, or signal values may be calculated using the background noise estimation techniques disclosed in U.S. Patent No. 7,844,453 , which is incorporated herein by reference, except that in the event of any inconsistent disclosure or definition from the present specification, the disclosure or definition herein shall be deemed to prevail.
- alternative background noise estimation techniques may be used, such as, for example, a noise power estimation technique based on minimum statistics.
- Additional noise reduction processing may mitigate specific types of undesirable noise characteristics including, for example, wind noise, transient noise, rain noise and engine noise. Mitigation of some specific types of undesirable noise may be referred to as signature noise reduction processes.
- Signature noise reduction processes detect signature noise and generate signal gains that may be used to suppress a detected signature noise.
- wind noise suppression gains a.k.a. signal gains
- the sound field received by the two or more microphones 102 may contain a spatial representation, or a spatial image, of an audible environment.
- Balance gains may be calculated responsive to the spatial image in the sound field.
- the balance gains may be calculated with a balance calculator 108.
- the balance calculator 108 may calculate the balance gains by measuring an energy level in a signal from each microphone 102. The energy level differences may represent the approximate balance of the spatial image.
- One or more energy levels may be calculated for each microphone 102 generating one or more balance gains.
- a single balance gain may be utilized in a two microphone configuration where the single balance gain may be the ratio of energy levels between the two microphone signals 118.
- a subband filter may process the received microphone signal 118 to extract frequency information.
- the subband filter may be accomplished by various methods, such as a Fast Fourier Transform (FFT), critical filter bank, octave filter band, or one-third octave filter bank.
- the subband analysis may include a time-based filter bank.
- the time-based filter bank may be composed of a bank of overlapping bandpass filters, where the center frequencies have non-linear spacing such as octave, 3 rd octave, bark, mel, or other spacing techniques.
- the one or more energy levels may be calculated for each frequency bin or band of the subband filter.
- the resulting balance gains may be filtered, or smoothed, over time and/or frequency.
- the balance calculator 108 may update the balance gains responsive to desired signal content.
- the balance gains may be updated when, for example, the energy level exceeds a threshold, the signal to noise ratio (SNR) exceeds a threshold, a voice activity detector detects voice content or any combination thereof.
- SNR signal to noise ratio
- the background noise estimator 104 may calculate a background noise estimate, or signal value, for each microphone signal 118. When the microphones 102 are spaced apart, the background noise estimator 104 may calculate different signal values responsive to the received sound value. Some difference in the calculated background noise estimate may be acceptable but relatively large differences may indicate a potential corruption or misrepresentation of one or more of the signals. For example, a user may be blocking one microphone 102 with a finger resulting in a relatively large difference in the background noise estimate.
- the background noise estimate may be utilized for many subsequent calculations including signal-to-noise ratios, echo cancellers and noise reduction calculators. When the subsequent calculations utilize background noise estimates that contain relatively large differences the subsequent calculations may yield corrupted or misrepresentative results. For example, large differences in suppression gains between microphones 102 may result in audible distortions in the spatial image of the sound field.
- a difference limiter 110 may limit the difference in the background noise estimates, or signal values, and/or the adaption rates utilized in the background noise estimator 104.
- the difference limiter 110 may mitigate audio distortions in the spatial image when reproduced in the output sound field. For example, a difference between corresponding signal values in the calculated background noise estimates may be acceptable when the difference is 2 dB (decibels) to 4 dB but noticeable when the difference exceeds 6 dB.
- the difference limiter 110 may, for example, limit the difference between signal values to 6 dB or may allow a difference proportional to the signal value when the difference is greater than 6 dB.
- the difference limiter 110 may utilize a coherence and/or correlation calculation between microphones to limit a difference between the signal values. Two signals that are correlated may indicate that the difference between signal values should be limited.
- the difference limiter 110 may smooth, or filter, the amount of limiting over time and frequency.
- the difference limiter 110 may be applied to other signal values including suppression gains, or signal gains, calculated using the suppression gain calculator 106.
- the suppression gain calculator 106 may calculate signal gains for the echo cancellation and noise reduction processes described above.
- Signature noise reduction processes may calculate signal gains that have large differences between microphone signals 118.
- a first microphone 102 may receive significant wind noise and the second microphone 102 may receive negligible wind noise.
- An example portable computing device may have two microphones 102 placed several inches apart where the first microphone 102 may be located on the bottom surface and the second microphone 102 may be located on the top surface. The first microphone 102 and the second microphone 102 may be relatively close in position although they may not be close enough to process phase differences to utilize, for example, a beam forming combining process.
- the suppression gain calculator 106 may calculate signal gains that may contain relatively large differences.
- the difference limiter 110 may allow some of the wind noise to be suppressed while mitigating audio distortions in the spatial image of the sound field. For example, a difference between corresponding signal gains generated by the suppression gain calculators 106 may be acceptable when the difference is 2 dB to 4 dB but noticeable when the difference exceeds 6 dB.
- the difference limiter 110 may limit the difference between signal values to 6 dB or may allow a difference proportional to the signal value when the difference is greater than 6 dB.
- the difference limiter 110 may smooth, or filter, the amount of limiting over time and frequency.
- the difference limiter 110 may mitigate some distortion in the spatial image when reproduced in the output sound field although it may be possible that the combination of one or more of the signal values calculated utilizing the background noise estimator 104 and suppression gain calculator 106 may still distort the spatial image. Additionally, in some cases the suppression gain calculator 106 may not utilize the difference limiter 110. For example, when the microphone 102 and audio transducer are coupled as described above resulting in a gating effect, the difference limiter 110 may not be utilized because the audible artifacts associated with the coupling are perceptibly more distracting than distorting the spatial image. In this case, the echo cancellation process may be allowed to gate the microphone signal 118 without applying the difference limiter 110.
- a balance adjuster 112 may maintain the spatial stability when reproduced in the output sound field.
- the balance adjuster 112 may mitigate distortions in the spatial image that may not be mitigated with the difference limiter 110. Additionally, the balance adjuster 112 may mitigate audio distortions in the spatial image where the difference limiter 110 may not be applied.
- the balance adjuster 112 may adjust the signal gains using the balance gains calculated with the balance calculator 108 and the signal gains.
- the balance gains may represent the approximate balance of the spatial image.
- the balance adjuster 112 may adjust the signal gains responsive to the balance gains. Additionally, the balance adjuster 112 may mix, or borrow, between two or more microphone signals 118 to maintain the spatial stability and to more closely track the balance gains.
- the echo-gating triggered half-duplex use case described above may have a first microphone signal 118 that may be gated.
- the balance adjuster 112 may mitigate audio distortions in the spatial image by borrowing audio from a second microphone signal 118 responsive to the balance gain.
- the second microphone signal 118 may have associated signal gains that may be adjusted responsive to the balance gain.
- the second microphone signal 118 that is burrowed may be mixed into the first microphone signal 118.
- the balance adjuster 112 may adjust the signal gains and the borrowing of microphone signals 118 may be filtered, or smoothed, over time and frequency. The adjustments may be performed on a frequency bin and/or band using the subband filter described above.
- a gain filter 114 applies the signal gains to the two or more microphone signals 118.
- the signal gains may be a combination of signal gains associated with one or more suppression gain calculators 106.
- the gain filter 114 may utilize the subband filter described above.
- FIG. 2 is a further schematic representation of a system for maintaining the spatial stability when reproduced in the output sound field.
- the system 200 comprises a processor 202, memory 204 (the contents of which are accessible by the processor 202), two or more microphones 102 and an I/O interface 206.
- the two or more microphones 102 may be either internal or external to the system 200 or a combination of internal and external.
- the memory 204 may store instructions which when executed using the processor 202 may cause the system 200 to render the functionality associated with the background noise estimator module 104, the suppression gain calculator module 106, the balance calculator module 108, the difference limiter module 110, the balance adjuster module 112 and the gain filter module 114 described herein.
- data structures, temporary variables and other information may store data in data storage 208.
- the processor 202 may comprise a single processor or multiple processors than may be disposed on a single chip, on multiple devices or distributed over more that one system.
- the processor 202 may be hardware that executes computer executable instructions or computer code embodied in the memory 204 or in other memory to perform one or more features of the system.
- the processor 202 may include a general purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a digital circuit, an analog circuit, a microcontroller, any other type of processor, or any combination thereof.
- the memory 204 may comprise a device for storing and retrieving data, processor executable instructions, or any combination thereof.
- the memory 204 may include non-volatile and/or volatile memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory.
- RAM random access memory
- ROM read-only memory
- EPROM erasable programmable read-only memory
- flash memory a flash memory.
- the memory 204 may comprise a single device or multiple devices that may be disposed on one or more dedicated memory devices or on a processor or other similar device.
- the memory 204 may include an optical, magnetic (hard-drive) or any other form of data storage device.
- the memory 204 may store computer code, such as the background noise estimator module 104, the suppression gain calculator module 106, the balance calculator module 108, the difference limiter module 110, the balance adjuster module 112 and the gain filter module 114 described herein.
- the computer code may include instructions executable with the processor 202.
- the computer code may be written in any computer language, such as C, C++, assembly language, channel program code, and/or any combination of computer languages.
- the memory 204 may store information in data structures in the data storage 208.
- the I/O interface 206 may be used to connect devices such as, for example, microphones 102, and to other components internal or external to the system 200.
- FIG. 3 is flow diagram representing a method for maintaining a spatial stability of a sound field.
- the method 300 may be, for example, implemented using either of the systems 100 and 200 described herein with reference to Figures 1 and 2 .
- the method 300 may include the following acts. Calculating a balance gain for each of two or more microphone signals 302.
- the balance gain may be associated with a spatial image in the sound field.
- the signal values may be the background noise estimate or signal gains associated with echo cancellation and noise reduction processes. Limiting the difference between the two or more signal values 306.
- Limiting the difference between signal values may be limited to mitigate distortions in the spatial image of the sound field.
- One or more microphone signals may be mixed, or borrowed, with another microphone signal responsive to the balance gains and signal gains. Applying the signal gains to the two or more microphone signals 310.
- the systems 100 and 200 may include more, fewer, or different components than illustrated in Figures 1 and 2 . Furthermore, each one of the components of systems 100 and 200 may include more, fewer, or different elements than is illustrated in Figures 1 and 2 .
- Flags, data, databases, tables, entities, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be distributed, or may be logically and physically organized in many different ways.
- the components may operate independently or be part of a same program or hardware.
- the components may be resident on separate hardware, such as separate removable circuit boards, or share common hardware, such as a same memory and processor for implementing instructions from the memory. Programs may be parts of a single program, separate programs, or distributed across several memories and processors.
- the functions, acts or tasks illustrated in the figures or described may be executed in response to one or more sets of logic or instructions stored in or on computer readable media.
- the functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro code and the like, operating alone or in combination.
- processing strategies may include multiprocessing, multitasking, parallel processing, distributed processing, and/or any other type of processing.
- the instructions are stored on a removable media device for reading by local or remote systems.
- the logic or instructions are stored in a remote location for transfer through a computer network or over telephone lines.
- the logic or instructions may be stored within a given computer such as, for example, a CPU.
Description
- The present disclosure relates to the field of processing sound fields. In particular, to a system and method for maintaining the spatial stability of a sound field.
- Stereo and multichannel microphone configurations may be used for processing a sound field that is a spatial representation of an audible environment associated with the microphones. The audio received from the microphones may be used to reproduce the sound field using audio transducers.
- Many computing devices may have multiple integrated microphones used for recording an audible environment associated with the computing device and communicating with other users. Computing devices typically use multiple microphones to improve noise performance with noise suppression processes. The noise suppression processes may result in the reduction or loss of spatial information. In many cases the noise suppression processing may result in a single, or mono, output signal that has no spatial information.
-
WO 2011/112382 relates to a method for filtering a multi-channel audio signal. -
US 7117145 relates to an adaptive feature for speech enhancement. -
EP 1830348 relates to a hands free system for speech signal acquisition in a vehicular cabin. -
WO 2002/19768 - The system may be better understood with reference to the following drawings and description. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.
- Moreover, in the figures, like referenced numerals designate corresponding parts throughout the different views.
- Other systems, methods, features and advantages will be, or will become, apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included with this description, be within the scope of the invention, and be protected by the following claims.
-
Fig. 1 is a schematic representation of a system for maintaining the spatial stability of a sound field. -
Fig. 2 is a further schematic representation of a system for maintaining the spatial stability of the sound field. -
Fig. 3 is flow diagram representing a method for maintaining the spatial stability of the sound field. - In a system and method for maintaining the spatial stability of a sound field balance gains may be calculated for each of two or more microphone signals. The balance gains may be associated with a spatial image in the sound field. Signal values may be calculated for each of the received microphone signals. The signal values may be signal estimates or signal gains calculated to improve a characteristic of the microphone signals. The differences between the signal values associated with each microphone signal are limited to mitigate audible distortions in the spatial image. Some difference between signal values may be allowable in order to improve the audible characteristics of the received microphone signals. One or more microphone signals are adjusted responsive to the two or more balance gains and the signal gains to maintain the spatial stability of the sound field. The adjustments of one or more microphone signals may include mixing of two or more microphone signals. Further adjustments to the signal gains may be made responsive to the mixing process. The signal gains are applied respectively to each of the two or more microphone signals. The invention is defined by the independent claims. Preferred embodiments are defined in the dependent claims.
-
Figure 1 is a schematic representation of a system for maintaining the spatial stability of asound field 100. Two ormore microphones 102 receive the sound field. Stereo and multichannel microphone configurations may be utilized for processing the sound field that is a spatial representation of an audible environment associated with themicrophones 102. Many audible environments associated with themicrophones 102 may include undesirable content that may be mitigated by processing the received sound field.Microphones 102 that are arranged in a far field configuration typically receive more undesirable content, noise, thanmicrophones 102 in a near field configuration. Far field configurations may include, for example, a hands free phone, a conference phone and microphones embedded into an automobile. Far field configurations are capable of receiving a sound field that represents the spatial environment associated with themicrophones 102. Near field configurations typically place themicrophone 102 in close proximity to a user. Undesirable content may be mitigated in both near and far field configurations by processing the received sound field. - Processing that may mitigate undesirable content received in the sound field may include echo cancellation and noise reduction processes. Echo cancellation, noise reduction and other audio processing processes may calculate one or more suppression, or signal, gains utilizing a
suppression gain calculator 106. An echo cancellation process and a noise reduction process may each calculate one or more signal gains. Each respective signal gain may be applied individually or a composite signal gain may be applied to process the sound field using again filter 114. Echo cancellation processing mitigates echoes caused by signal feedback between two or more communication devices. Signal feedback occurs when an audio transducer on a first communication device reproduces the signal received from a second communication device and subsequently the microphones on the first communication device recapture the reproduced signal. The recaptured signal may be transmitted to the second communication device where the recaptured signal may be perceived as an echo of the previously transmitted signal. Echo cancellation processes may detect when the signal has been recaptured and attempt to suppress the recaptured signal. Many different echo cancellation processes may mitigate echoes by calculating one or more signal gains that, when applied to the signals received by themicrophones 102, suppress the echoes. In one example implementation, the echo suppression gain may be calculated using coherence calculation between the predicted echo and the microphone disclosed inU.S. Patent No. 8,036,879 . - When the
microphone 102 and an audio transducer are close in proximity, the echo cancellation process may determine a large amount of suppression, or calculate large signal gains, as a result of the signal produced by the audio transducer dominating, or coupling with, themicrophone 102. - When one of the
microphones 102 and an audio transducer are in close proximity, the echo cancellation process may determine that a large amount of suppression may mitigate the signal produced by the audio transducer from dominating, or coupling with, themicrophone 102. The echo cancellation process may calculate large signal gains to mitigate the coupling. The large signal gains may result in a gating effect where the communication device effectively supports only half duplex communication. Half duplex communication may occur when the communication channel allows for reliable communication from alternatively either the far side or near side but not both simultaneously. The large signal gains may suppress the coupling but may also suppress all content, including desired voice content resulting in half duplex communication. - Background noise is another type of undesirable signal content that may be mitigated by processing the received sound field. Many different types of noise reduction processing techniques may mitigate background noise. An exemplary noise reduction method is a recursive Wiener filter. The Wiener suppression gain Gi,k , or signal gain, is defined as
-
-
- Here |N̂ i,k | is a background noise estimate. In one example implementation, the background noise estimate, or signal values, may be calculated using the background noise estimation techniques disclosed in
U.S. Patent No. 7,844,453 , which is incorporated herein by reference, except that in the event of any inconsistent disclosure or definition from the present specification, the disclosure or definition herein shall be deemed to prevail. In other implementations, alternative background noise estimation techniques may be used, such as, for example, a noise power estimation technique based on minimum statistics. - Additional noise reduction processing may mitigate specific types of undesirable noise characteristics including, for example, wind noise, transient noise, rain noise and engine noise. Mitigation of some specific types of undesirable noise may be referred to as signature noise reduction processes. Signature noise reduction processes detect signature noise and generate signal gains that may be used to suppress a detected signature noise. In one implementation, wind noise suppression gains (a.k.a. signal gains) may be calculated using the system for suppressing wind noise disclosed in
U.S. Patent No. 7,885,420 . - The sound field received by the two or
more microphones 102 may contain a spatial representation, or a spatial image, of an audible environment. Balance gains may be calculated responsive to the spatial image in the sound field. The balance gains may be calculated with abalance calculator 108. Thebalance calculator 108 may calculate the balance gains by measuring an energy level in a signal from eachmicrophone 102. The energy level differences may represent the approximate balance of the spatial image. One or more energy levels may be calculated for eachmicrophone 102 generating one or more balance gains. A single balance gain may be utilized in a two microphone configuration where the single balance gain may be the ratio of energy levels between the two microphone signals 118. - A subband filter may process the received
microphone signal 118 to extract frequency information. The subband filter may be accomplished by various methods, such as a Fast Fourier Transform (FFT), critical filter bank, octave filter band, or one-third octave filter bank. Alternatively, the subband analysis may include a time-based filter bank. The time-based filter bank may be composed of a bank of overlapping bandpass filters, where the center frequencies have non-linear spacing such as octave, 3rd octave, bark, mel, or other spacing techniques. The one or more energy levels may be calculated for each frequency bin or band of the subband filter. The resulting balance gains may be filtered, or smoothed, over time and/or frequency. Thebalance calculator 108 may update the balance gains responsive to desired signal content. For example, the balance gains may be updated when, for example, the energy level exceeds a threshold, the signal to noise ratio (SNR) exceeds a threshold, a voice activity detector detects voice content or any combination thereof. - The
background noise estimator 104 may calculate a background noise estimate, or signal value, for eachmicrophone signal 118. When themicrophones 102 are spaced apart, thebackground noise estimator 104 may calculate different signal values responsive to the received sound value. Some difference in the calculated background noise estimate may be acceptable but relatively large differences may indicate a potential corruption or misrepresentation of one or more of the signals. For example, a user may be blocking onemicrophone 102 with a finger resulting in a relatively large difference in the background noise estimate. The background noise estimate may be utilized for many subsequent calculations including signal-to-noise ratios, echo cancellers and noise reduction calculators. When the subsequent calculations utilize background noise estimates that contain relatively large differences the subsequent calculations may yield corrupted or misrepresentative results. For example, large differences in suppression gains betweenmicrophones 102 may result in audible distortions in the spatial image of the sound field. - A
difference limiter 110 may limit the difference in the background noise estimates, or signal values, and/or the adaption rates utilized in thebackground noise estimator 104. Thedifference limiter 110 may mitigate audio distortions in the spatial image when reproduced in the output sound field. For example, a difference between corresponding signal values in the calculated background noise estimates may be acceptable when the difference is 2 dB (decibels) to 4 dB but noticeable when the difference exceeds 6 dB. Thedifference limiter 110 may, for example, limit the difference between signal values to 6 dB or may allow a difference proportional to the signal value when the difference is greater than 6 dB. Thedifference limiter 110 may utilize a coherence and/or correlation calculation between microphones to limit a difference between the signal values. Two signals that are correlated may indicate that the difference between signal values should be limited. Thedifference limiter 110 may smooth, or filter, the amount of limiting over time and frequency. - The
difference limiter 110 may be applied to other signal values including suppression gains, or signal gains, calculated using thesuppression gain calculator 106. Thesuppression gain calculator 106 may calculate signal gains for the echo cancellation and noise reduction processes described above. Signature noise reduction processes may calculate signal gains that have large differences between microphone signals 118. For example, in the case of wind noise reduction, afirst microphone 102 may receive significant wind noise and thesecond microphone 102 may receive negligible wind noise. An example portable computing device may have twomicrophones 102 placed several inches apart where thefirst microphone 102 may be located on the bottom surface and thesecond microphone 102 may be located on the top surface. Thefirst microphone 102 and thesecond microphone 102 may be relatively close in position although they may not be close enough to process phase differences to utilize, for example, a beam forming combining process. Even though themicrophones 102 are relatively close in position on the example portable computing device, onemicrophone 102 may receive significant wind noise. Thesuppression gain calculator 106 may calculate signal gains that may contain relatively large differences. Thedifference limiter 110 may allow some of the wind noise to be suppressed while mitigating audio distortions in the spatial image of the sound field. For example, a difference between corresponding signal gains generated by thesuppression gain calculators 106 may be acceptable when the difference is 2 dB to 4 dB but noticeable when the difference exceeds 6 dB. Thedifference limiter 110 may limit the difference between signal values to 6 dB or may allow a difference proportional to the signal value when the difference is greater than 6 dB. Thedifference limiter 110 may smooth, or filter, the amount of limiting over time and frequency. - The
difference limiter 110 may mitigate some distortion in the spatial image when reproduced in the output sound field although it may be possible that the combination of one or more of the signal values calculated utilizing thebackground noise estimator 104 andsuppression gain calculator 106 may still distort the spatial image. Additionally, in some cases thesuppression gain calculator 106 may not utilize thedifference limiter 110. For example, when themicrophone 102 and audio transducer are coupled as described above resulting in a gating effect, thedifference limiter 110 may not be utilized because the audible artifacts associated with the coupling are perceptibly more distracting than distorting the spatial image. In this case, the echo cancellation process may be allowed to gate themicrophone signal 118 without applying thedifference limiter 110. - A
balance adjuster 112 may maintain the spatial stability when reproduced in the output sound field. Thebalance adjuster 112 may mitigate distortions in the spatial image that may not be mitigated with thedifference limiter 110. Additionally, thebalance adjuster 112 may mitigate audio distortions in the spatial image where thedifference limiter 110 may not be applied. Thebalance adjuster 112 may adjust the signal gains using the balance gains calculated with thebalance calculator 108 and the signal gains. The balance gains may represent the approximate balance of the spatial image. Thebalance adjuster 112 may adjust the signal gains responsive to the balance gains. Additionally, thebalance adjuster 112 may mix, or borrow, between two or more microphone signals 118 to maintain the spatial stability and to more closely track the balance gains. In one example, the echo-gating triggered half-duplex use case described above may have afirst microphone signal 118 that may be gated. Thebalance adjuster 112 may mitigate audio distortions in the spatial image by borrowing audio from asecond microphone signal 118 responsive to the balance gain. Thesecond microphone signal 118 may have associated signal gains that may be adjusted responsive to the balance gain. Thesecond microphone signal 118 that is burrowed may be mixed into thefirst microphone signal 118. Thebalance adjuster 112 may adjust the signal gains and the borrowing of microphone signals 118 may be filtered, or smoothed, over time and frequency. The adjustments may be performed on a frequency bin and/or band using the subband filter described above. - A
gain filter 114 applies the signal gains to the two or more microphone signals 118. The signal gains may be a combination of signal gains associated with one or moresuppression gain calculators 106. Thegain filter 114 may utilize the subband filter described above. -
Figure 2 is a further schematic representation of a system for maintaining the spatial stability when reproduced in the output sound field. Thesystem 200 comprises aprocessor 202, memory 204 (the contents of which are accessible by the processor 202), two ormore microphones 102 and an I/O interface 206. The two ormore microphones 102 may be either internal or external to thesystem 200 or a combination of internal and external. Thememory 204 may store instructions which when executed using theprocessor 202 may cause thesystem 200 to render the functionality associated with the backgroundnoise estimator module 104, the suppressiongain calculator module 106, thebalance calculator module 108, thedifference limiter module 110, thebalance adjuster module 112 and thegain filter module 114 described herein. In addition, data structures, temporary variables and other information may store data indata storage 208. - The
processor 202 may comprise a single processor or multiple processors than may be disposed on a single chip, on multiple devices or distributed over more that one system. Theprocessor 202 may be hardware that executes computer executable instructions or computer code embodied in thememory 204 or in other memory to perform one or more features of the system. Theprocessor 202 may include a general purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a digital circuit, an analog circuit, a microcontroller, any other type of processor, or any combination thereof. - The
memory 204 may comprise a device for storing and retrieving data, processor executable instructions, or any combination thereof. Thememory 204 may include non-volatile and/or volatile memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory. Thememory 204 may comprise a single device or multiple devices that may be disposed on one or more dedicated memory devices or on a processor or other similar device. Alternatively or in addition, thememory 204 may include an optical, magnetic (hard-drive) or any other form of data storage device. - The
memory 204 may store computer code, such as the backgroundnoise estimator module 104, the suppressiongain calculator module 106, thebalance calculator module 108, thedifference limiter module 110, thebalance adjuster module 112 and thegain filter module 114 described herein. The computer code may include instructions executable with theprocessor 202. The computer code may be written in any computer language, such as C, C++, assembly language, channel program code, and/or any combination of computer languages. Thememory 204 may store information in data structures in thedata storage 208. - The I/
O interface 206 may be used to connect devices such as, for example,microphones 102, and to other components internal or external to thesystem 200. -
Figure 3 is flow diagram representing a method for maintaining a spatial stability of a sound field. Themethod 300 may be, for example, implemented using either of thesystems Figures 1 and2 . Themethod 300 may include the following acts. Calculating a balance gain for each of two or more microphone signals 302. The balance gain may be associated with a spatial image in the sound field. Calculating one or more signal values for each of two or more microphone signals 304. The signal values may be the background noise estimate or signal gains associated with echo cancellation and noise reduction processes. Limiting the difference between the two or more signal values 306. The difference between signal values may be limited to mitigate distortions in the spatial image of the sound field. Adjusting one or more microphone signals responsive to the two or more balance gains and the signal gains 308. One or more microphone signals may be mixed, or borrowed, with another microphone signal responsive to the balance gains and signal gains. Applying the signal gains to the two or more microphone signals 310. - All of the disclosure, regardless of the particular implementation described, is exemplary in nature, rather than limiting. The
systems Figures 1 and2 . Furthermore, each one of the components ofsystems Figures 1 and2 . Flags, data, databases, tables, entities, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be distributed, or may be logically and physically organized in many different ways. The components may operate independently or be part of a same program or hardware. The components may be resident on separate hardware, such as separate removable circuit boards, or share common hardware, such as a same memory and processor for implementing instructions from the memory. Programs may be parts of a single program, separate programs, or distributed across several memories and processors. - The functions, acts or tasks illustrated in the figures or described may be executed in response to one or more sets of logic or instructions stored in or on computer readable media. The functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro code and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, distributed processing, and/or any other type of processing. In one embodiment, the instructions are stored on a removable media device for reading by local or remote systems. In other embodiments, the logic or instructions are stored in a remote location for transfer through a computer network or over telephone lines. In yet other embodiments, the logic or instructions may be stored within a given computer such as, for example, a CPU.
- While various embodiments of the system and method for maintaining the spatial stability of a sound field have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the present invention. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.
Claims (11)
- A computer implemented method for maintaining spatial stability of a sound field comprising:calculating (302) balance gains for two or more microphone signals 118, where each of the two or more microphone signals is from a corresponding one of two or more microphones (102), the balance gains representing a difference in energy level of two or more microphones and hence an approximate balance of a spatial image of a sound field received by the microphones (102), and each of the balance gains is for a corresponding one of the microphone signals (118);calculating (304) one or more signal values for each of the two or more microphone signals (118), where the one or more signal values comprises one or more of an estimated background noise and a calculated suppression gain;limiting (306) differences between corresponding one or more signal values for each of the two or more microphone signals (118); andmaintaining a spatial stability of the sound field when reproduced in the output sound field, by gain adjusting (308) the two or more microphone signals (118) responsive to the calculated balance gains and the limited one or more signal values for each of the two or more microphone signals (118).
- The computer implemented method of claim 1 where the suppression gain comprises one or more of a noise reduction calculation and an echo cancellation calculation.
- The computer implemented method of claims 1 to 2, where calculating each of the one or more signal values further comprises calculating one or more of a background noise estimate and a background noise adaptation rate.
- The computer implemented method of claims 1 to 3, where limiting the difference between each of the one or more signal value that comprise suppression gains further comprises limiting the suppression gains when a threshold is exceeded.
- The computer implemented method of claims 3 to 4, where the noise reduction calculation comprises any one or more of a wind noise reduction calculation, transients noise reduction calculation, road noise reduction calculation, repetitive noise reduction calculation and engine noise reduction calculation.
- The computer implemented method of claims 1 to 5, where calculating the balance gains for each of two or more microphone signals (118) further comprises an energy measurement.
- The computer implemented method of claims 1 to 6, where calculating the balance gains for each of two or more microphone signals (118) further comprises selectively updating the balance gains responsive to one or more of the signal values exceeding a threshold and a signal-to-noise ratio exceeding a threshold.
- The computer implemented method of claims 1 to 7, where gain adjusting the two or more microphone signals (118) further comprises mixing a first microphone signal (118) with a second microphone signal (118).
- The computer implemented method of claims 1 to 8, further comprising generating a set of sub-bands for each of the two or more microphone signals (118) using a subband filter or a Fast Fourier Transform.
- The computer implemented method of claims 1 to 9, further comprising generating a set of sub-bands for each of the two or more (118) signals according to a critical, octave, mel, or bark band spacing technique.
- A system for maintaining spatial stability of a sound field, the system comprising:a processor (202); anda memory (204) coupled to the processor (202) containing instructions, executable by the processor (202), for performing the instructions executing the steps of any of method claims 1 to 10.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP13153065.1A EP2760021B1 (en) | 2013-01-29 | 2013-01-29 | Sound field spatial stabilizer |
CA2835991A CA2835991C (en) | 2013-01-29 | 2014-01-24 | Sound field spatial stabilizer |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP13153065.1A EP2760021B1 (en) | 2013-01-29 | 2013-01-29 | Sound field spatial stabilizer |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2760021A1 EP2760021A1 (en) | 2014-07-30 |
EP2760021B1 true EP2760021B1 (en) | 2018-01-17 |
Family
ID=47713883
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP13153065.1A Active EP2760021B1 (en) | 2013-01-29 | 2013-01-29 | Sound field spatial stabilizer |
Country Status (1)
Country | Link |
---|---|
EP (1) | EP2760021B1 (en) |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7117145B1 (en) * | 2000-10-19 | 2006-10-03 | Lear Corporation | Adaptive filter for speech enhancement in a noisy environment |
US7885420B2 (en) | 2003-02-21 | 2011-02-08 | Qnx Software Systems Co. | Wind noise suppression system |
EP1830348B1 (en) * | 2006-03-01 | 2016-09-28 | Nuance Communications, Inc. | Hands-free system for speech signal acquisition |
US7844453B2 (en) | 2006-05-12 | 2010-11-30 | Qnx Software Systems Co. | Robust noise estimation |
US8036879B2 (en) | 2007-05-07 | 2011-10-11 | Qnx Software Systems Co. | Fast acoustic cancellation |
TWI459828B (en) * | 2010-03-08 | 2014-11-01 | Dolby Lab Licensing Corp | Method and system for scaling ducking of speech-relevant channels in multi-channel audio |
-
2013
- 2013-01-29 EP EP13153065.1A patent/EP2760021B1/en active Active
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
EP2760021A1 (en) | 2014-07-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9106196B2 (en) | Sound field spatial stabilizer with echo spectral coherence compensation | |
US9437180B2 (en) | Adaptive noise reduction using level cues | |
US9949034B2 (en) | Sound field spatial stabilizer | |
US10049678B2 (en) | System and method for suppressing transient noise in a multichannel system | |
US9756440B2 (en) | Maintaining spatial stability utilizing common gain coefficient | |
US9854358B2 (en) | System and method for mitigating audio feedback | |
CN104158990A (en) | Method for processing an audio signal and audio receiving circuit | |
EP3316256A1 (en) | Voice activity modification frame acquiring method, and voice activity detection method and apparatus | |
US9626987B2 (en) | Speech enhancement apparatus and speech enhancement method | |
US9143107B2 (en) | System and method for dynamically mixing audio signals | |
US9099973B2 (en) | Sound field spatial stabilizer with structured noise compensation | |
US9743179B2 (en) | Sound field spatial stabilizer with structured noise compensation | |
US9210507B2 (en) | Microphone hiss mitigation | |
EP2816818B1 (en) | Sound field spatial stabilizer with echo spectral coherence compensation | |
US9349383B2 (en) | Audio bandwidth dependent noise suppression | |
CA2840730C (en) | Maintaining spatial stability utilizing common gain coefficient | |
EP2760021B1 (en) | Sound field spatial stabilizer | |
EP2816817B1 (en) | Sound field spatial stabilizer with spectral coherence compensation | |
CA2835991C (en) | Sound field spatial stabilizer | |
EP2816816B1 (en) | Sound field spatial stabilizer with structured noise compensation | |
EP2760020B1 (en) | Maintaining spatial stability utilizing common gain coefficient | |
EP2760221A1 (en) | Microphone hiss mitigation | |
EP2760022A1 (en) | Audio bandwidth dependent noise suppression |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20130129 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: 2236008 ONTARIO INC. |
|
17Q | First examination report despatched |
Effective date: 20160421 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20170810 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: PARANJPE, SHREYAS Inventor name: HETHERINGTON, PHILLIP ALAN |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 6 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 964929 Country of ref document: AT Kind code of ref document: T Effective date: 20180215 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602013032259 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20180117 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 964929 Country of ref document: AT Kind code of ref document: T Effective date: 20180117 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180417 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180517 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180418 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180417 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602013032259 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180129 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20180131 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180131 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180131 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180131 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
26N | No opposition filed |
Effective date: 20181018 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180129 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180129 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20130129 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180117 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602013032259 Country of ref document: DE Representative=s name: MERH-IP MATIAS ERNY REICHL HOFFMANN PATENTANWA, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602013032259 Country of ref document: DE Owner name: BLACKBERRY LIMITED, WATERLOO, CA Free format text: FORMER OWNER: 2236008 ONTARIO INC., WATERLOO, ONTARIO, CA |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180117 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20200730 AND 20200805 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230125 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230127 Year of fee payment: 11 Ref country code: DE Payment date: 20230127 Year of fee payment: 11 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230518 |