CN116320851A - Microphone array noise reduction method, device and system, electronic equipment and storage medium - Google Patents
Microphone array noise reduction method, device and system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116320851A CN116320851A CN202310080879.4A CN202310080879A CN116320851A CN 116320851 A CN116320851 A CN 116320851A CN 202310080879 A CN202310080879 A CN 202310080879A CN 116320851 A CN116320851 A CN 116320851A
- Authority
- CN
- China
- Prior art keywords
- frequency domain
- sub
- domain signal
- noise
- reduced
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000009467 reduction Effects 0.000 title claims abstract description 132
- 238000000034 method Methods 0.000 title claims abstract description 67
- 238000003860 storage Methods 0.000 title claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 76
- 230000005236 sound signal Effects 0.000 claims abstract description 65
- 239000000654 additive Substances 0.000 claims abstract description 39
- 230000000996 additive effect Effects 0.000 claims abstract description 39
- 238000004590 computer program Methods 0.000 claims description 12
- 238000011946 reduction process Methods 0.000 claims description 11
- 230000003111 delayed effect Effects 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 230000002596 correlated effect Effects 0.000 claims description 4
- 230000008901 benefit Effects 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 12
- 238000003491 array Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 5
- 230000000875 corresponding effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/08—Mouthpieces; Microphones; Attachments therefor
- H04R1/083—Special constructions of mouthpieces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2410/00—Microphones
- H04R2410/01—Noise reduction using microphones having different directional characteristics
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Circuit For Audible Band Transducer (AREA)
Abstract
The disclosure provides a microphone array noise reduction method, a device, a system, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a sound signal to be noise reduced; performing frequency domain processing on the sound signal to obtain a frequency domain signal; responding to a first sub-frequency domain signal with the frequency smaller than a preset threshold value in the frequency domain signal, and performing first noise reduction processing based on a preset subtractive microphone array to obtain a first sub-frequency domain signal after noise reduction; responding to a second sub-frequency domain signal with the frequency greater than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing based on a preset additive microphone array to obtain a second sub-frequency domain signal after noise reduction; and obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal. The comprehensive noise reduction microphone array and the additive microphone array have the advantages that the excellent characteristics of the comprehensive noise reduction microphone array and the additive microphone array respectively conduct targeted noise reduction processing on low-frequency signals and high-frequency signals, and the noise reduction performance is better.
Description
Technical Field
The disclosure relates to the technical field of signal processing, in particular to a microphone array noise reduction method, a device, a system, electronic equipment and a storage medium.
Background
Along with the development of modern society science and technology, people are increasingly widely used for headphones. In order to ensure the conversation quality when a user wears the earphone, a plurality of microphones are often arranged in the earphone to form a microphone array, and the microphone array is utilized to carry out wave beam formation to reduce noise of collected sound signals.
The related art mainly relates to noise reduction methods of two microphone arrays, wherein typical representative of additive microphone arrays are generalized sidelobe canceller (General sidelobe canceller, GSC), and typical representative of subtractive microphone arrays are differential microphone arrays (Differential Microphone Array, DMA).
However, as a portable mobile device, especially a real wireless stereo (True Wireless Stereo, TWS) headset, the miniaturization feature is remarkable, the microphone pitch is small, and the above two noise reduction methods alone cannot achieve a good noise reduction effect.
Disclosure of Invention
The embodiment of the disclosure at least provides a microphone array noise reduction method, device, system, electronic equipment and storage medium, so as to improve noise reduction performance of a microphone array.
In a first aspect, an embodiment of the present disclosure provides a microphone array noise reduction method, including:
acquiring a sound signal to be noise reduced;
performing frequency domain processing on the sound signal to obtain a frequency domain signal;
responding to a first sub-frequency domain signal with the frequency smaller than a preset threshold value in the frequency domain signal, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a noise-reduced first sub-frequency domain signal;
responding to a second sub-frequency domain signal with the frequency greater than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a second sub-frequency domain signal after noise reduction;
and obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
In one possible implementation, the subtractive microphone array is a differential microphone array DMA and the additive microphone array is a generalized sidelobe canceller GSC.
In a possible implementation manner, the performing, based on a preset subtractive microphone array, a first noise reduction process on the first sub-frequency domain signal to obtain a noise reduced first sub-frequency domain signal, where the noise reduced first sub-frequency domain signal includes:
Aiming at a first frequency point where the first sub-frequency domain signal is located, filtering the first sub-frequency domain signal based on a preset low-pass filter to obtain a filtered first sub-frequency domain signal;
and performing first noise reduction processing on the filtered first sub-frequency domain signal based on a preset DMA (direct memory access), and obtaining a noise-reduced first sub-frequency domain signal.
In one possible implementation, the passband bandwidth of the low pass filter is positively correlated with the frequency of the first sub-frequency domain signal.
In a possible implementation manner, the obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal includes:
aligning the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain an aligned first sub-frequency domain signal and second sub-frequency domain signal;
and obtaining the noise-reduced sound signal based on the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal.
In a possible implementation manner, the aligning the first denoised sub-frequency domain signal and the second denoised sub-frequency domain signal includes:
Performing delay processing on the noise-reduced second sub-frequency domain signal based on the number of taps of the low-pass filter, and determining a delayed second sub-frequency domain signal;
and obtaining the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal based on the noise-reduced first sub-frequency domain signal and the delayed second sub-frequency domain signal.
In a possible implementation manner, the obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal includes:
summing the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain a noise-reduced frequency domain signal;
and performing time domain processing on the frequency domain signal after noise reduction to obtain a sound signal after noise reduction.
In a possible implementation manner, the acquiring the sound signal to be noise reduced includes:
and responding to an earphone communication instruction of the real wireless stereo TWS earphone, and acquiring a sound signal to be reduced in noise.
In a second aspect, the present disclosure also provides a microphone array noise reduction apparatus, including:
the acquisition module is used for acquiring the sound signal to be noise reduced;
the frequency domain processing module is used for carrying out frequency domain processing on the sound signals to obtain frequency domain signals;
The first noise reduction module is used for responding to a first sub-frequency domain signal with the frequency smaller than a preset threshold value in the frequency domain signals, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a first sub-frequency domain signal after noise reduction;
the second noise reduction module is used for responding to a second sub-frequency domain signal with the frequency larger than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a noise-reduced second sub-frequency domain signal;
the determining module is configured to obtain a noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
In a third aspect, the present disclosure also provides a microphone array noise reduction system, comprising: a subtractive microphone array, an additive microphone array, and a processor; the processor is respectively connected with the subtractive microphone array and the additive microphone array;
the processor is used for acquiring a sound signal to be noise reduced; performing frequency domain processing on the sound signal to obtain a frequency domain signal; the frequency domain signals comprise first sub-frequency domain signals with the frequency smaller than a preset threshold value and second sub-frequency domain signals with the frequency larger than or equal to the preset threshold value; obtaining a noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal of the subtractive microphone array and the noise-reduced second sub-frequency domain signal of the additive microphone array;
The subtractive microphone array is configured to perform a first noise reduction process on the first sub-frequency domain signal to obtain a noise-reduced first sub-frequency domain signal;
and the additive microphone array is used for performing second noise reduction processing on the second sub-frequency domain signals to obtain noise-reduced second sub-frequency domain signals.
In a fourth aspect, the present disclosure also provides an electronic device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the microphone array noise reduction method of the first aspect and any of its various embodiments.
In a fifth aspect, the present disclosure also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the microphone array noise reduction method as in the first aspect and any of its various embodiments.
By adopting the microphone array noise reduction method, the device, the system, the electronic equipment and the storage medium, under the condition that the sound signal to be noise reduced is acquired, frequency domain processing can be firstly performed, then, first noise reduction processing can be performed on the basis of the subtractive microphone array for the first sub-frequency domain signal with smaller frequency after the frequency domain processing, the first noise reduction processing can be obtained, second noise reduction processing can be performed on the basis of the additive microphone array for the second sub-frequency domain signal with larger frequency after the frequency domain processing, the second sub-frequency domain signal after the noise reduction can be obtained, and finally, the noise signal after the noise reduction can be obtained on the basis of the first sub-frequency domain signal after the noise reduction and the second sub-frequency domain signal after the noise reduction. The comprehensive noise reduction microphone array and the additive microphone array have the advantages that the low-frequency signals and the high-frequency signals are subjected to targeted noise reduction processing respectively, so that noise in the sound signals can be eliminated to a greater extent by the obtained noise signals after noise reduction, and the noise reduction performance is better.
Other advantages of the present disclosure will be explained in more detail in conjunction with the following description and accompanying drawings.
It should be understood that the foregoing description is only an overview of the technical solutions of the present disclosure so that the technical means of the present disclosure may be more clearly understood and may be implemented in accordance with the content of the specification. The following specific examples illustrate the present disclosure in order to make the above and other objects, features and advantages of the present disclosure more comprehensible.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 illustrates a flow chart of a method of microphone array noise reduction provided by an embodiment of the present disclosure;
fig. 2 is a flowchart of a specific method for GSC noise reduction in the microphone array noise reduction method provided in the embodiment of the present disclosure;
FIG. 3 illustrates a schematic diagram of a microphone array noise reduction system provided by an embodiment of the disclosure;
FIG. 4 illustrates a schematic diagram of a microphone array noise reduction device provided by an embodiment of the disclosure;
fig. 5 shows a schematic diagram of an electronic device provided by an embodiment of the disclosure.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In the description of the embodiments of the present disclosure, it should be understood that terms such as "comprises" or "comprising" are intended to indicate the presence of features, numbers, steps, acts, components, portions or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility of the presence of one or more other features, numbers, steps, acts, components, portions or combinations thereof.
Unless otherwise indicated, "/" means or, e.g., A/B may represent A or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone.
The terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", etc. may explicitly or implicitly include one or more such feature. In the description of the embodiments of the present disclosure, unless otherwise indicated, the meaning of "a plurality" is two or more.
It has been found that methods of noise reduction using additive microphone arrays are widely used. However, additive microphone arrays such as GSC are relatively complex to calculate and perform best when the microphone size is required to be close to half a wavelength due to the inherent characteristics of additive microphones. Due to the characteristics of the microphone array, the GSC cannot well meet the noise reduction requirement in practical applications of directly applying to headphones, especially TWS headphones with advantages such as small size and portability.
To at least partially address one or more of the above-mentioned problems, as well as other potential problems, the present disclosure provides at least one solution for joint noise reduction in conjunction with multiple microphone arrays to improve noise reduction performance.
For the sake of understanding the present embodiment, first, a detailed description will be given of a microphone array noise reduction method disclosed in an embodiment of the present disclosure, where an execution body of the microphone array noise reduction method provided in the embodiment of the present disclosure is generally an electronic device with a certain computing capability, and the electronic device includes, for example: the terminal device or other processing device may be a User Equipment (UE), a mobile device, a User terminal, a vehicle device, a wearable device, etc. In practical applications, the user equipment may be, for example, a TWS headset. Furthermore, in some possible implementations, the microphone array noise reduction method may be implemented by way of a processor invoking computer readable instructions stored in a memory.
Referring to fig. 1, a flowchart of a microphone array noise reduction method according to an embodiment of the disclosure is shown, where the method includes steps S101 to S105, where:
s101: acquiring a sound signal to be noise reduced;
S102: performing frequency domain processing on the sound signal to obtain a frequency domain signal;
s103: responding to a first sub-frequency domain signal with frequency smaller than a preset threshold value in the frequency domain signal, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a noise-reduced first sub-frequency domain signal;
s104: responding to a second sub-frequency domain signal with the frequency greater than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a noise-reduced second sub-frequency domain signal;
s105: and obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
In order to facilitate understanding of the microphone array noise reduction method provided by the embodiment of the present disclosure, an application scenario of the method will be specifically described next. The microphone array noise reduction method in the embodiment of the present disclosure may be mainly applied to any scene suitable for microphone array noise reduction, for example, a traffic noise scene, an abnormal noise detection scene, etc., and may also be suitable for a service product that needs noise reduction, such as a TWS earphone, etc., and in consideration of the wide application of the TWS earphone, the following description will be given by taking the TWS earphone as an execution body.
The sound signal to be noise-reduced here may be a sound signal acquired in various application scenarios of the above-described examples, or may be a sound signal of a user acquired in response to an earphone communication instruction of the TWS earphone. In consideration of the complexity of the environment in which the user is located, the sound signal may be accompanied with noise of various sources, and based on this, the microphone array noise reduction method provided by the embodiment of the disclosure may perform frequency domain processing on the sound signal to obtain a frequency domain signal, which mainly considers that in the process of frequency domain analysis, the change of the signal spectrum after the signal passes through the system can be intuitively reflected, so that the system performance can be conveniently analyzed. The frequency domain signal obtained here may refer to a signal in which the argument is frequency, i.e. the horizontal axis is frequency and the vertical axis is the amplitude of the frequency signal.
In consideration of performance differences of different microphone arrays at different frequency points, different frequency points can be distinguished, noise reduction processing can be performed by adopting a subtractive microphone array at a frequency point with lower frequency, noise reduction processing can be performed by adopting an additive microphone array at a frequency point with higher frequency, and then final noise-reduced sound signals can be obtained through fusion of noise reduction signals and conversion from a frequency domain to a time domain.
In addition, the frequency domain signal further includes a second sub-frequency domain signal with a frequency greater than or equal to a preset threshold, and the second sub-frequency domain signal is subjected to a second noise reduction process by using an additive microphone array, so that the noise reduction performance in a scene that the microphone size is close to half wavelength is ensured according to the direction self-adaptive adjustment of the signal.
The preset threshold value of the frequency domain may be 1.5Hz, for example, the second noise reduction processing is performed on the second sub-frequency domain signal higher than 1.5Hz and smaller than the maximum frequency value, and for example, the first noise reduction processing is performed on the first sub-frequency domain signal higher than the minimum frequency value and smaller than 1.5 Hz.
The subtractive microphone array here mainly refers to the differential microphone array DMA, and the additive microphone array mainly refers to the generalized sidelobe canceller GSC.
In practical application, the result of the low-frequency part can use the coefficient of DMA to replace GSC operation, thus reducing the operation complexity in the low-frequency processing process and further improving the noise reduction performance.
In the embodiment of the disclosure, the denoised first sub-frequency domain signal obtained based on DMA processing and the denoised second sub-frequency domain signal obtained based on GSC processing may be summed first to obtain a denoised frequency domain signal, and then the denoised frequency domain signal is subjected to time domain processing to obtain the denoised sound signal. In the frequency domain processing process, the low-frequency result and the high-frequency result are subjected to more proper noise reduction respectively, so that noise interference is eliminated to a greater extent in the finally determined noise-reduced sound signal, and the practical use experience of electronic equipment such as TWS headphones is ensured.
In the actual noise reduction process of the Low-frequency signal, i.e. the first sub-frequency domain signal, through the DMA, since the DMA may amplify noise in the Low-frequency portion, a Low-Pass Filter (LPF) may be added to average the Low-frequency portion between different frames, so as to remove the influence of white noise, which may be specifically implemented by the following steps:
step one, aiming at a first frequency point where a first sub-frequency domain signal is located, filtering the first sub-frequency domain signal based on a preset low-pass filter to obtain a filtered first sub-frequency domain signal;
And step two, performing first noise reduction processing on the filtered first sub-frequency domain signal based on a preset DMA to obtain a noise-reduced first sub-frequency domain signal.
Here, the mth antenna of DMA, the t frame, and the signal on the w frequency is Z m (t,e jw ) Signal Z can be applied m (t,e jw ) Through a low pass filter LPF (Z m (t,e jw ) Filtering out the influence of white noise at the frequency point. In the process of noise reduction based on DMA, the ratio of the transmission equations of different antennas of the microphone array can be obtained first. Assuming that the target direction is 0 degree, H H (e jw ,0)h(w)＝1。
Then, the ratio of the transmission equations of the antennas with different microphone blocking directions theta, H, is obtained H (e jw θ) h (w) =0, N-1 blocking directions can be set for an array of N antennas. And obtaining coefficients of the differential microphone array by solving an equation set formed by the two equations, thereby obtaining a first sub-frequency domain signal after noise reduction.
In an application of a practical TWS headset, there may be a microphone array of two microphones in each headset, such that the corresponding n=2.
The passband bandwidth of the low pass filter provided herein may be positively correlated with the frequency of the first sub-frequency domain signal, on the basis of which the passband bandwidth of the LPF is narrower for lower frequencies and vice versa, considering that white noise is amplified when the frequency is particularly low and consequently unstable. Lpf=fir (n), n being the number of taps.
Considering that the low-pass filter can bring time delay to a great extent, the alignment of the low-frequency signal and the high-frequency signal can be performed first, and then the two aligned sub-frequency domain signals are combined to obtain a final noise reduction result. In the actual alignment process, the alignment may be performed as follows:
step one, carrying out delay processing on a second sub-frequency domain signal after noise reduction based on the number of taps of a low-pass filter, and determining a delayed second sub-frequency domain signal;
and step two, obtaining the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal based on the noise-reduced first sub-frequency domain signal and the delayed second sub-frequency domain signal.
Here, since LPFs of different frequencies are different, it is possible to set the delay to n/2 number of taps and to align the signals by correspondingly delaying the result of the high frequency. For example, for the current 0 th second low frequency signal, the final noise reduction result may be obtained by combining the high frequency signal delayed forward for 2 seconds.
To facilitate a further understanding of the noise reduction process with respect to the high frequency signal, an example is illustrated below in connection with fig. 2.
As shown in FIG. 2, the target signal is s (t), and the signal received by the mth microphone is z m (t) the noise signal received by the mth microphone is n m (t) the transfer function TF from the sound source to the mth microphone is a m (t) thus z m (t)＝a m (t)*s(t)+n m (t), m=1,.. by Short-time fourier transform (Short-Term Fourier Transform, STFT) to the frequency domain to obtain Z m (t，e jw )＝a m (t，e jw )*S(t，e jw )+N m (t，e jw )，m＝1，...，M。
Calculating the ratio of other antenna transmission equations to the antenna by taking the main antenna as a reference antenna, H m (e jω )＝A m (e jω )/A 1 (e jω )，H T (e jω )＝[1 H 1 (e jω )...H m (e jω )]。
The transmission equation may use a preset sweep result.
For H T (e jω ) Normalizing W 0 (e jω )＝H(e jω )/||H(e jω )|| 2 。
Such as algorithm flow blocksThe upper half shown in FIG. 2
The noise is constructed so that the noise is not transmitted,
here, E { ||y is minimized FBF (t，e jω )-G(t，e jω ) + U(e jω )|| 2 This function constructs G (t, e) as the target j ω )。
In practical cases, it is necessary to obtain G (t, e using iterative methods jω )。
P est (t，e jω )＝ρP est (t-1，e jω )+(1-ρ)v|Z m (t，e jω )| 2 。
In an application of a practical TWS headset, there may be a microphone array of two microphones in each headset, such that the corresponding m=2.
It is known that compared with DMA, the method is simpler in calculation and more complex in calculation of GSC, but GSC can be adaptively adjusted according to the direction of signals, and the performance is best when the microphone size is close to half wavelength due to the inherent characteristics of the additive microphone, so that the advantage of noise reduction processing of high-frequency signals is more outstanding.
The microphone noise reduction method provided by the embodiment of the disclosure combines the DMA and the GSC, comprehensively utilizes the advantages of the DMA and the GSC to achieve better noise reduction performance, and considers the characteristic that the DMA is unstable at low frequency, and reduces noise by using the same frequency point and an inter-frame smoothing method, thereby further improving the noise reduction performance and having better practicability.
Based on the microphone array noise reduction method provided in the foregoing embodiment, the embodiment of the present disclosure further provides a microphone array noise reduction system, as shown in fig. 3, where the system mainly includes: a subtractive microphone array 301, an additive microphone array 302, and a processor 303; the processor 303 is connected to the subtractive microphone array 301 and the additive microphone array 302, respectively;
a processor 303 for acquiring a sound signal to be noise reduced; performing frequency domain processing on the sound signal to obtain a frequency domain signal; the frequency domain signals comprise first sub-frequency domain signals with frequencies smaller than a preset threshold value and second sub-frequency domain signals with frequencies larger than or equal to the preset threshold value; obtaining a noise-reduced sound signal based on the first sub-frequency domain signal after noise reduction of the subtractive microphone array 301 and the second sub-frequency domain signal after noise reduction of the additive microphone array 302;
a subtractive microphone array 301, configured to perform a first noise reduction process on the first sub-frequency domain signal, to obtain a noise-reduced first sub-frequency domain signal;
And the additive microphone array 302 is configured to perform a second noise reduction process on the second sub-frequency domain signal, so as to obtain a noise-reduced second sub-frequency domain signal.
Here, in cooperation with the two microphone arrays (i.e., the subtractive microphone array 301 and the additive microphone array 302), the processor 303 performs targeted noise reduction processing on the low-frequency signal and the high-frequency signal, respectively, so that the noise reduction performance is better.
For the specific noise reduction schemes of the subtractive microphone array 301 and the additive microphone array 302, reference is made to the description of the above embodiments, and the details are not repeated here.
In the description of the present specification, reference to the terms "some possible embodiments," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiments or examples is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the various embodiments or examples described in this specification and the features of the various embodiments or examples may be combined and combined by those skilled in the art without contradiction.
With respect to the method flow diagrams of the disclosed embodiments, certain operations are described as distinct steps performed in a certain order. Such a flowchart is illustrative and not limiting. Some steps described herein may be grouped together and performed in a single operation, may be partitioned into multiple sub-steps, and may be performed in an order different than that shown herein. The various steps illustrated in the flowcharts may be implemented in any manner by any circuit structure and/or tangible mechanism (e.g., by software running on a computer device, hardware (e.g., processor or chip implemented logic functions), etc., and/or any combination thereof).
It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.
Based on the same inventive concept, the embodiments of the present disclosure further provide a microphone array noise reduction device corresponding to the microphone array noise reduction method, and since the principle of solving the problem of the device in the embodiments of the present disclosure is similar to that of the microphone array noise reduction method in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.
Referring to fig. 4, a schematic diagram of a microphone array noise reduction device according to an embodiment of the disclosure is shown, where the device includes: an acquisition module 401, a frequency domain processing module 402, a first noise reduction module 403, a second noise reduction module 404, and a determination module 405; wherein, the liquid crystal display device comprises a liquid crystal display device,
an acquisition module 401, configured to acquire a sound signal to be noise reduced;
the frequency domain processing module 402 is configured to perform frequency domain processing on the sound signal to obtain a frequency domain signal;
a first noise reduction module 403, configured to respond to a first sub-frequency domain signal with a frequency smaller than a preset threshold value in the frequency domain signal, and perform a first noise reduction process on the first sub-frequency domain signal based on a preset subtractive microphone array, so as to obtain a noise-reduced first sub-frequency domain signal;
the second noise reduction module 404 is configured to respond to a second sub-frequency domain signal with a frequency greater than or equal to a preset threshold value in the frequency domain signal, and perform second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array, so as to obtain a noise-reduced second sub-frequency domain signal;
the determining module 405 is configured to obtain a noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
According to the embodiment of the disclosure, under the condition that a sound signal to be denoised is obtained, frequency domain processing can be performed first, then, first denoising processing can be performed on a first sub-frequency domain signal with smaller frequency after the frequency domain processing based on the subtractive microphone array, a first sub-frequency domain signal after denoise is obtained, second denoising processing can be performed on a second sub-frequency domain signal with larger frequency after the frequency domain processing based on the additive microphone array, a second sub-frequency domain signal after denoise is obtained, and finally, a sound signal after denoise can be obtained based on the first sub-frequency domain signal after denoise and the second sub-frequency domain signal after denoise. The comprehensive noise reduction microphone array and the additive microphone array have the advantages that the low-frequency signals and the high-frequency signals are subjected to targeted noise reduction processing respectively, so that noise in the sound signals can be eliminated to a greater extent by the obtained noise signals after noise reduction, and the noise reduction performance is better.
In one possible implementation, the subtractive microphone array is a differential microphone array DMA and the additive microphone array is a generalized sidelobe canceller GSC.
In a possible implementation manner, the first noise reduction module 403 is configured to perform a first noise reduction process on the first sub-frequency domain signal based on a preset subtractive microphone array according to the following steps, to obtain a noise reduced first sub-frequency domain signal:
aiming at a first frequency point where the first sub-frequency domain signal is located, filtering the first sub-frequency domain signal based on a preset low-pass filter to obtain a filtered first sub-frequency domain signal;
and performing first noise reduction processing on the filtered first sub-frequency domain signal based on a preset DMA (direct memory access), and obtaining a noise-reduced first sub-frequency domain signal.
In one possible implementation, the passband bandwidth of the low pass filter is positively correlated with the frequency of the first sub-frequency domain signal.
In a possible implementation manner, the determining module 405 is configured to obtain the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal according to the following steps:
aligning the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain an aligned first sub-frequency domain signal and second sub-frequency domain signal;
And obtaining the noise-reduced sound signal based on the aligned first sub-frequency domain signal and the second sub-frequency domain signal.
In a possible implementation manner, the determining module 405 is configured to align the denoised first sub-frequency domain signal and the denoised second sub-frequency domain signal according to the following steps:
delaying the second sub-frequency domain signal after noise reduction based on the number of taps of the low-pass filter, and determining the delayed second sub-frequency domain signal;
and obtaining the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal based on the noise-reduced first sub-frequency domain signal and the delayed second sub-frequency domain signal.
In a possible implementation manner, the determining module 405 is configured to obtain the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal according to the following steps:
summing the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain a noise-reduced frequency domain signal;
and performing time domain processing on the frequency domain signal after noise reduction to obtain a sound signal after noise reduction.
In a possible implementation manner, the obtaining module 401 is configured to obtain a sound signal to be noise reduced according to the following steps:
And responding to an earphone communication instruction of the real wireless stereo TWS earphone, and acquiring a sound signal to be reduced in noise.
It should be noted that, the apparatus in the embodiments of the present disclosure may implement each process of the foregoing embodiments of the method, and achieve the same effects and functions, which are not described herein again.
The embodiment of the disclosure further provides an electronic device, as shown in fig. 5, which is a schematic structural diagram of the electronic device provided by the embodiment of the disclosure, including: a processor 501, a memory 502, and a bus 503. The memory 502 stores machine-readable instructions executable by the processor 501 (e.g., the acquisition module 401, the frequency domain processing module 402, the first noise reduction module 403, the second noise reduction module 404, the execution instructions corresponding to the determination module 405, etc. in the apparatus of fig. 4), when the electronic device is running, the processor 501 communicates with the memory 502 through the bus 503, and when the machine-readable instructions are executed by the processor 501, the following processing is performed:
acquiring a sound signal to be noise reduced;
performing frequency domain processing on the sound signal to obtain a frequency domain signal;
responding to a first sub-frequency domain signal with frequency smaller than a preset threshold value in the frequency domain signal, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a noise-reduced first sub-frequency domain signal;
Responding to a second sub-frequency domain signal with the frequency greater than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a noise-reduced second sub-frequency domain signal;
and obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the microphone array noise reduction method described in the method embodiments above. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The embodiments of the present disclosure further provide a computer program product, where the computer program product carries program code, and instructions included in the program code may be used to perform the steps of the microphone array noise reduction method described in the foregoing method embodiments, and specifically reference may be made to the foregoing method embodiments, which are not described herein.
Wherein the above-mentioned computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
The various embodiments in this disclosure are described in a progressive manner, and identical and similar parts of the various embodiments are all referred to each other, and each embodiment is mainly described as different from other embodiments. In particular, for apparatus, devices and computer readable storage medium embodiments, the description thereof is simplified as it is substantially similar to the method embodiments, as relevant points may be found in part in the description of the method embodiments.
The apparatus, the device, and the computer readable storage medium provided in the embodiments of the present disclosure are in one-to-one correspondence with the methods, and therefore, the apparatus, the device, and the computer readable storage medium also have similar advantageous technical effects as the corresponding methods, and since the advantageous technical effects of the methods have been described in detail above, the advantageous technical effects of the apparatus, the device, and the computer readable storage medium are not repeated herein.
It will be apparent to those skilled in the art that embodiments of the present disclosure may be provided as a method, apparatus (device or system), or computer readable storage medium. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer-readable storage medium embodied in one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices or systems) and computer-readable storage media according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that this disclosure is not limited to the particular embodiments disclosed nor does it imply that features in these aspects are not to be combined to benefit from this division, which is done for convenience of description only. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (12)
1. A method of microphone array noise reduction, comprising:
acquiring a sound signal to be noise reduced;
performing frequency domain processing on the sound signal to obtain a frequency domain signal;
responding to a first sub-frequency domain signal with the frequency smaller than a preset threshold value in the frequency domain signal, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a noise-reduced first sub-frequency domain signal;
responding to a second sub-frequency domain signal with the frequency greater than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a second sub-frequency domain signal after noise reduction;
and obtaining the noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
2. The method of claim 1, wherein the subtractive microphone array is a differential microphone array DMA and the additive microphone array is a generalized sidelobe canceller GSC.
3. The method of claim 2, wherein the performing, based on the preset subtractive microphone array, a first noise reduction process on the first sub-frequency domain signal to obtain a noise reduced first sub-frequency domain signal includes:
aiming at a first frequency point where the first sub-frequency domain signal is located, filtering the first sub-frequency domain signal based on a preset low-pass filter to obtain a filtered first sub-frequency domain signal;
and performing first noise reduction processing on the filtered first sub-frequency domain signal based on a preset DMA (direct memory access), and obtaining a noise-reduced first sub-frequency domain signal.
4. A method according to claim 3, wherein the passband bandwidth of the low pass filter is positively correlated with the frequency of the first sub-frequency domain signal.
5. The method according to claim 3 or 4, wherein the obtaining the denoised sound signal based on the denoised first sub-frequency domain signal and the denoised second sub-frequency domain signal comprises:
Aligning the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain an aligned first sub-frequency domain signal and second sub-frequency domain signal;
and obtaining the noise-reduced sound signal based on the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal.
6. The method of claim 5, wherein aligning the denoised first sub-frequency domain signal and the denoised second sub-frequency domain signal comprises:
performing delay processing on the noise-reduced second sub-frequency domain signal based on the number of taps of the low-pass filter, and determining a delayed second sub-frequency domain signal;
and obtaining the aligned first sub-frequency domain signal and the aligned second sub-frequency domain signal based on the noise-reduced first sub-frequency domain signal and the delayed second sub-frequency domain signal.
7. The method of claim 1, wherein the obtaining the denoised sound signal based on the denoised first sub-frequency domain signal and the denoised second sub-frequency domain signal comprises:
summing the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal to obtain a noise-reduced frequency domain signal;
And performing time domain processing on the frequency domain signal after noise reduction to obtain a sound signal after noise reduction.
8. The method of claim 1, wherein the acquiring the sound signal to be denoised comprises:
and responding to an earphone communication instruction of the real wireless stereo TWS earphone, and acquiring a sound signal to be reduced in noise.
9. A microphone array noise reduction device, comprising:
the acquisition module is used for acquiring the sound signal to be noise reduced;
the frequency domain processing module is used for carrying out frequency domain processing on the sound signals to obtain frequency domain signals;
the first noise reduction module is used for responding to a first sub-frequency domain signal with the frequency smaller than a preset threshold value in the frequency domain signals, and performing first noise reduction processing on the first sub-frequency domain signal based on a preset subtractive microphone array to obtain a first sub-frequency domain signal after noise reduction;
the second noise reduction module is used for responding to a second sub-frequency domain signal with the frequency larger than or equal to a preset threshold value in the frequency domain signal, and performing second noise reduction processing on the second sub-frequency domain signal based on a preset additive microphone array to obtain a noise-reduced second sub-frequency domain signal;
the determining module is configured to obtain a noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal and the noise-reduced second sub-frequency domain signal.
10. A microphone array noise reduction system, comprising: a subtractive microphone array, an additive microphone array, and a processor; the processor is respectively connected with the subtractive microphone array and the additive microphone array;
the processor is used for acquiring a sound signal to be noise reduced; performing frequency domain processing on the sound signal to obtain a frequency domain signal; the frequency domain signals comprise first sub-frequency domain signals with the frequency smaller than a preset threshold value and second sub-frequency domain signals with the frequency larger than or equal to the preset threshold value; obtaining a noise-reduced sound signal based on the noise-reduced first sub-frequency domain signal of the subtractive microphone array and the noise-reduced second sub-frequency domain signal of the additive microphone array;
the subtractive microphone array is configured to perform a first noise reduction process on the first sub-frequency domain signal to obtain a noise-reduced first sub-frequency domain signal;
and the additive microphone array is used for performing second noise reduction processing on the second sub-frequency domain signals to obtain noise-reduced second sub-frequency domain signals.
11. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the microphone array noise reduction method of any of claims 1-8.
12. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, performs the microphone array noise reduction method according to any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310080879.4A CN116320851A (en) | 2023-01-17 | 2023-01-17 | Microphone array noise reduction method, device and system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310080879.4A CN116320851A (en) | 2023-01-17 | 2023-01-17 | Microphone array noise reduction method, device and system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116320851A true CN116320851A (en) | 2023-06-23 |
Family
ID=86789674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310080879.4A Pending CN116320851A (en) | 2023-01-17 | 2023-01-17 | Microphone array noise reduction method, device and system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116320851A (en) |
-
2023
- 2023-01-17 CN CN202310080879.4A patent/CN116320851A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9922663B2 (en) | Voice signal processing method and apparatus | |
| US9094496B2 (en) | System and method for stereophonic acoustic echo cancellation | |
| JP6703525B2 (en) | Method and device for enhancing sound source | |
| CN111341336B (en) | Echo cancellation method, device, terminal equipment and medium | |
| JP4286637B2 (en) | Microphone device and playback device | |
| US20170103774A1 (en) | Audio Signal Processing | |
| US20130301840A1 (en) | Methods for processing audio signals and circuit arrangements therefor | |
| US20140016794A1 (en) | Echo cancellation system and method with multiple microphones and multiple speakers | |
| CN110176244B (en) | Echo cancellation method, device, storage medium and computer equipment | |
| US11457310B2 (en) | Apparatus, method and computer program for audio signal processing | |
| JP4827675B2 (en) | Low frequency band audio restoration device, audio signal processing device and recording equipment | |
| US20160182770A1 (en) | Display apparatus and echo cancellation method thereof | |
| WO2020020247A1 (en) | Signal processing method and device, and computer storage medium | |
| TWI690220B (en) | Spectral defect compensation for crosstalk processing of spatial audio signals | |
| CN111885276B (en) | Method and system for eliminating echo | |
| CN109215672B (en) | Method, device and equipment for processing sound information | |
| JP5762479B2 (en) | Voice switch device, voice switch method, and program thereof | |
| TWI747252B (en) | Systems, methods, and devices for audio processing | |
| CN111756906B (en) | Echo suppression method and device for voice signal and computer readable medium | |
| US10771896B2 (en) | Crosstalk cancellation for speaker-based spatial rendering | |
| CN116320851A (en) | Microphone array noise reduction method, device and system, electronic equipment and storage medium | |
| CN112929506A (en) | Audio signal processing method and apparatus, computer storage medium, and electronic device | |
| US11640830B2 (en) | Multi-microphone signal enhancement | |
| CN110661510B (en) | Beam former forming method, beam forming device and electronic equipment | |
| WO2017171864A1 (en) | Acoustic environment understanding in machine-human speech communication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |