JP2009282536A - Method and device for removing known acoustic signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a known acoustic signal removal device which receives an acoustic signal as a mixture of a plurality of acoustic signals and can remove a known acoustic signal when the known acoustic signal similar to one of the acoustic signals is given. <P>SOLUTION: The known acoustic signal removal device converts an input mixed acoustic signal m(t) and a known acoustic signal b' (t) into an amplitude spectra M(ω, t) and B' (ω, t) in the respective time frequency areas and subtracts a component corresponding to the B' (ω, t) in the M(ω, t) to be removed, thereby obtaining an amplitude spectrum S(ω, t) after the removal. Here, the component corresponding to the B' (ω, t) in the M(ω, t) is deformed by various factors such as a positional shift by time, temporal change of frequency characteristics, and temporal change of the sound volume. Accordingly, they are corrected to obtain B(ω, t), which is subtracted. Lastly, by using the phase of the m(t) and S(ω, t), reverse conversion is performed in the temporal region to obtain an acoustic signal s(t) after a desired removal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a method and apparatus for removing a component of a known acoustic signal from a mixed acoustic signal obtained by mixing a plurality of acoustic signals, an interface used for the apparatus, and a program.

  Conventionally, a method called a spectral subtraction method (Non-Patent Document 1) is known. The conventional spectral subtraction method is based on an acoustic signal (mixed sound) in which stationary noise (noise whose spectrum does not change with time and whose frequency characteristics and volume are almost constant) and a desired sound (target sound) are mixed. This is a method for obtaining a target sound by removing stationary noise. In this method, the stationary noise spectrum is learned in advance by a simple method such as calculating the average of the stationary spectrum in advance, and the process of subtracting the stationary noise spectrum from the input mixed sound spectrum is performed (that is, noise). Process to subtract the average). In general, a number of methods using input from a plurality of microphones have been proposed for acoustic signal removal. Various improvements have been made to the spectral subtraction method (Patent Literature).

Steven Boll, “Suppression of Acoustic Noise in Speech Using Spectral Subtraction”, IEEE Transactions on Acoustics, Speech, and SignalProceed. ASSP-27, no. 2, April 1979.

JP 2002-175099 “Noise Suppression Method and Noise Suppression Device” Japanese Patent Laid-Open No. 2002-014694 “Voice Recognition Device” Japanese Patent Laid-Open No. 2001-228892 “Noise Removal Device, Noise Removal Method, and Recording Medium” Japanese Patent Laid-Open No. 2001-215992 “Voice Recognition Device” Japanese Patent Laid-Open No. 11-003094 “Noise Removal Device” Japanese Patent Application Laid-Open No. 10-240294 “Noise Reduction Method and Noise Reduction Device” Japanese Patent Application Laid-Open No. 08-2221092 “Noise Reduction System Using Spectral Subtraction”

  However, the conventional spectral subtraction method is based on stationary noise and cannot be applied to non-stationary noise (noise in which the spectrum changes greatly with time and the frequency characteristics, volume, etc. also change). In particular, it has been impossible to remove non-stationary noise such as music that varies greatly with time. This is because the spectrum of non-stationary noise is too large to learn. Furthermore, even if the conventional method is used to handle conditions in which non-stationary noise is given in advance, the frequency characteristics, volume, amplitude spectrum expansion and contraction in the time axis direction and expansion and contraction in the frequency axis direction of the non-stationary noise are changed. Due to the influence, the removal process could not be performed properly. Further, the method using inputs from a plurality of microphones cannot be applied to a monaural sound signal. Also, any of the improved conventional spectral subtraction methods is primarily intended for preprocessing of speech recognition. Therefore, non-stationary noise is given in advance and cannot be used for the purpose of removing the non-stationary noise.

  An object of the present invention is to remove a component of a known acoustic signal (which may be non-stationary or stationary) from a mixed acoustic signal obtained by mixing a plurality of acoustic signals using the known acoustic signal of the corresponding original sound source. It is an object to provide a method and apparatus for removing a known acoustic signal that can be performed, and a program used for the apparatus.

  Another object of the present invention is that the known acoustic signal is music, and the musical acoustic signal corresponds to a known acoustic signal from a mixed sound that is used as background music (BGM) for human speech or sound. Provided is a known acoustic signal removal method and apparatus capable of removing BGM using a known acoustic signal of an original sound source (a separately obtained acoustic signal of the same music from a CD or a record), and a program used for the apparatus There is.

  Another object of the present invention is to accurately detect a known acoustic signal in a mixed sound when removing a component of the known acoustic signal from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. An object of the present invention is to provide a known acoustic signal removal method and apparatus capable of automatically estimating a position and removing a known acoustic signal at the position, and a program used for the apparatus.

  Another object of the present invention is to accurately detect a known acoustic signal in a mixed sound when removing a component of the known acoustic signal from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. An object of the present invention is to provide a known acoustic signal removing device having an interface that allows a human to designate a position.

  Another object of the present invention is to remove a known acoustic signal component from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. It is an object of the present invention to provide a known acoustic signal removal method and apparatus capable of automatically estimating and correcting the change when the volume changes with time, and a program used for the apparatus.

  Another object of the present invention is to remove a known acoustic signal component from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. An object of the present invention is to provide a known acoustic signal removing device having an interface that allows a human to specify a change in volume when the volume changes with time.

  Another object of the present invention is to remove a known acoustic signal component from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. An object of the present invention is to provide a known acoustic signal removal method and apparatus that can be removed while automatically estimating and correcting the expansion and contraction when expanding and contracting in the frequency axis direction, and a program used for the apparatus.

  Another object of the present invention is to remove a known acoustic signal component from an acoustic signal (mixed sound) in which a plurality of acoustic signals are mixed. It is an object of the present invention to provide a known acoustic signal removing device having an interface that allows a human to specify expansion and contraction when expanding and contracting in the frequency axis direction.

  Another object of the present invention is to repeatedly remove known acoustic signals one by one when removing a plurality of known acoustic signal components from an acoustic signal in which a plurality of acoustic signals are mixed. It is an object to provide a method and apparatus for removing a known acoustic signal and a program used for the apparatus.

  The present invention is a known acoustic signal for removing a component of a known acoustic signal (which may be non-stationary or stationary) from a mixed acoustic signal obtained by mixing a plurality of acoustic signals using a known acoustic signal of a corresponding original sound source. Target removal method. In the method of the present invention, first, the mixed sound signal is converted into a time-frequency representation to obtain the amplitude spectrum of the mixed sound signal and the phase of the mixed sound signal (mixed sound signal conversion step). As a method for converting an acoustic signal into a time-frequency representation, a known conversion method such as Fourier transform or wavelet transform can be used. Next, a known sound signal corresponding to (similar to) a known sound signal included in the mixed sound signal (separately obtained sound signal of the same music from a CD or a record) is converted into a time-frequency representation. Then, the amplitude spectrum of the known acoustic signal is obtained (known acoustic signal conversion step). Then, based on the amplitude spectrum of the mixed acoustic signal, the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the temporal change of the frequency characteristics, the temporal change of the volume, the expansion and contraction in the time axis direction, and the frequency A corrected amplitude spectrum of the known acoustic signal obtained by correcting at least one of axial expansion and contraction is obtained (correction step). Next, the corrected amplitude spectrum of the known acoustic signal is removed from the amplitude spectrum of the mixed acoustic signal (removal step). Based on the post-removal amplitude spectrum obtained in this removal step and the phase of the mixed acoustic signal, inverse transformation is performed on the time representation to obtain a unit waveform (inverse transformation step). Finally, a unit waveform is synthesized using various synthesis methods such as an overlap-add method to obtain an acoustic signal from which a known acoustic signal component is removed (synthesis step).

  In the present invention, by the correction step, the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the temporal change of the frequency characteristic, the temporal change of the volume, the expansion / contraction in the time axis direction, and the frequency axis direction In order to obtain a corrected amplitude spectrum of a known acoustic signal in which at least one of the expansion and contraction has been corrected, and to remove the corrected amplitude spectrum from the amplitude spectrum of the mixed acoustic signal, the known acoustic signal included as non-stationary noise in the mixed acoustic signal Can be removed with high accuracy. Ideally, the mixed acoustic signal is actually mixed among the time position shift of the amplitude spectrum of the known acoustic signal, the time change of the frequency characteristic, the time change of the volume, the time axis direction expansion and contraction, and the frequency axis direction expansion and contraction. It is preferable to correct all the phenomena or changes that have occurred. However, it is possible to increase the accuracy of removing known acoustic signals by correcting even one of the phenomena or changes actually occurring in the mixed acoustic signal, compared with the case where nothing is corrected. It is not necessary to perform. Of course, all necessary corrections may be performed.

  In the correction step, the temporal position of the known acoustic signal included in the mixed acoustic signal is estimated, and the temporal position shift of the amplitude spectrum of the known acoustic signal is corrected based on the estimated temporal position. it can. For example, the estimation method obtains a distance (similarity) between a predetermined section of the amplitude spectrum of the mixed acoustic signal and a predetermined section of the amplitude spectrum of the known acoustic signal, and a known section included in the mixed acoustic signal includes a section having the closest distance. It can be estimated as the temporal position of the acoustic signal. The estimation method is arbitrary.

  In the correction step, a change in the frequency characteristic of the known acoustic signal included in the mixed acoustic signal is estimated, and the temporal change in the frequency characteristic of the amplitude spectrum of the known acoustic signal is corrected based on the temporal change in the estimated frequency characteristic. Can do. The estimation of the change in the frequency characteristic is performed by, for example, identifying a section including only the known acoustic signal in the mixed acoustic signal, and determining the frequency characteristic of this section and the frequency characteristic of the known acoustic signal corresponding to this section. From the comparison, it is possible to estimate a change in frequency characteristics of a known acoustic signal included in the mixed acoustic signal. This estimation method is arbitrary.

  In the correction step, the temporal change in the volume of the known acoustic signal included in the mixed acoustic signal can be estimated, and the temporal change in the volume of the amplitude spectrum of the known acoustic signal can be corrected based on the estimated temporal change in the volume. . The estimation of the temporal change in volume is performed after the frequency characteristics are corrected, for example, by specifying a frequency band having an amplitude corresponding to the known acoustic signal included in the mixed acoustic signal at each time, and the mixed acoustic signal in that frequency band. It can be estimated from a comparison between the amplitude of the known sound signal and the amplitude of the known acoustic signal. This estimation method is arbitrary.

  In the correction step, the expansion / contraction in the time axis direction of the known acoustic signal included in the mixed acoustic signal is estimated, and the expansion / contraction in the time axis direction of the amplitude spectrum of the known acoustic signal is corrected based on the estimated expansion / contraction in the time axis direction. be able to. In the estimation of the expansion and contraction in the time axis direction, for example, a section containing only known acoustic signals in the mixed acoustic signal is specified, and by comparing the time axis with the section of the known acoustic signal corresponding to this section, Expansion and contraction in the time axis direction can be estimated. Alternatively, the time axis may be estimated by comparing all the sections divided into short sections. This estimation method is arbitrary.

  Further, in the correction step, the expansion and contraction in the frequency axis direction of the known acoustic signal included in the mixed acoustic signal is estimated, and the expansion and contraction in the frequency axis direction of the amplitude spectrum of the known acoustic signal is corrected based on the estimated expansion and contraction in the frequency axis direction. be able to. In the estimation of the expansion and contraction in the frequency axis direction, for example, a section containing only known acoustic signals in the mixed acoustic signal is specified, and by comparing the frequency axis with the section of the known acoustic signal corresponding to this section, Expansion and contraction in the frequency axis direction can be estimated. This estimation method is arbitrary.

  The method of the present invention may further include an image display step of displaying an image so that the amplitude spectrum of the mixed acoustic signal and the amplitude spectrum of the known acoustic signal can be visually recognized. In this case, a section in which a known acoustic signal is included in the mixed acoustic signal is determined by a human based on the image display, and a correction step, a removal step, an inverse conversion step, or a synthesis step is executed for this section.

  The method of the present invention may further include a sound reproduction step of reproducing the mixed sound signal, the known sound signal, and the output signal of the synthesis step as sound. In this case, based on the reproduced sound from the sound reproduction step, a human defines a section in which a known sound signal is included in the mixed sound signal, and a correction step, a removal step, an inverse conversion step, and a synthesis step are performed for this section. Execute.

  In addition, a section including a known acoustic signal in the mixed acoustic signal is automatically estimated based on the amplitude spectrum of the mixed acoustic signal, and a correction step, a removal step, an inverse conversion step, and a synthesis step are executed for this section. it can. If the mixed sound signal contains a known sound signal relatively clearly (for example, if there is a section where a known sound signal is sounding alone in the mixed sound signal), the section is automatically estimated. It is possible to specify. If automatic estimation can be used, a known acoustic signal can be removed quickly. Of course, if the existence of a known acoustic signal included in the mixed acoustic signal is not so clear, a human may designate a section.

  Further, when there are a plurality of types of known acoustic signals corresponding to the acoustic signals included in the mixed acoustic signal, the known acoustic signal conversion step and the correction step are executed for all of the plurality of known acoustic signals, and the mixed acoustic signal is executed. The inverse conversion step and the synthesis step may be executed using the post-removal amplitude spectrum obtained by executing the removal step of removing all the corrected amplitude spectra of the plurality of known acoustic signals from the signal amplitude spectrum. In this way, it is possible to remove all of a plurality of types of known acoustic signals from the mixed acoustic signal.

  In addition, when executing the correction step, humans manually specify at least one correction of time position shift, frequency characteristic time change, volume time change, time axis direction expansion and contraction, and frequency axis direction expansion and contraction. An interface can be used that allows

  This interface allows a human to specify the exact position of a known acoustic signal in a mixed acoustic signal when removing a component of the known acoustic signal from a mixed acoustic signal in which a plurality of acoustic signals are mixed. It can be constituted as follows.

  In addition, this interface can be configured so that when the frequency characteristics of known acoustic signals in the mixed acoustic signal are temporally changing, those changes can be specified by a human. In addition, this interface can be configured such that when the volume of a known acoustic signal in the mixed acoustic signal changes with time, the change can be specified by a human.

  Further, this interface can be configured such that when a known acoustic signal in the mixed acoustic signal is expanded or contracted in the time axis or frequency axis direction, the human can specify the expansion and contraction.

  This interface can also be configured so that a human can specify the corresponding sections of the mixed acoustic signal and the known acoustic signal.

  The known acoustic signal removing device of the present invention includes a mixed acoustic signal converting means for converting a mixed acoustic signal into a time-frequency representation to obtain an amplitude spectrum of the mixed acoustic signal and a phase of the mixed acoustic signal, and is included in the mixed acoustic signal. A known acoustic signal conversion means for converting the known acoustic signal corresponding to the existing acoustic signal into a time-frequency representation to obtain the amplitude spectrum of the known acoustic signal, and the amplitude spectrum of the mixed acoustic signal based on the amplitude spectrum of the mixed acoustic signal. A corrected amplitude spectrum of a known acoustic signal in which at least one of a temporal position shift of the amplitude spectrum of the known acoustic signal, a temporal change in frequency characteristics, a temporal change in volume, expansion and contraction in the time axis direction, and expansion and contraction in the frequency axis direction is corrected. Correction means for obtaining the correction, removal means for removing the corrected amplitude spectrum of the known acoustic signal from the amplitude spectrum of the mixed acoustic signal, and removal Based on the post-removal amplitude spectrum obtained by the stage and the phase of the mixed acoustic signal, the inverse transform means for performing the inverse transform to the time expression to obtain the unit waveform, and synthesizing the unit waveform to remove the components of the known acoustic signal And synthesizing means for obtaining an acoustic signal.

  The correction means includes at least a shift in time position of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, a temporal change in frequency characteristics, a temporal change in volume, expansion and contraction in the time axis direction, and expansion and contraction in the frequency axis direction. An interface can be provided that allows one person to specify one correction manually. This interface reproduces the mixed sound signal, the known sound signal, and the output signal of the synthesizing means as sound. The image display unit displays an image so that the amplitude spectrum of the mixed sound signal and the amplitude spectrum of the known sound signal can be visually compared. It is preferable to include a sound reproducing unit. When this interface is used, the amplitude spectrum of the mixed acoustic signal displayed on the image display unit and the image display of the amplitude spectrum of the known acoustic signal and / or the reproduced sound from the acoustic reproduction unit are included in the mixed acoustic signal. In addition to being able to specify a known acoustic signal section by human beings, the human being can manually specify the time position shift of the amplitude spectrum of the known acoustic signal, the time variation of the frequency characteristics, the time variation of the volume, and the time. At least one correction of expansion and contraction in the axial direction and expansion and contraction in the frequency axis direction can be designated. As a result, even if the aspect of the known acoustic signal included in the mixed acoustic signal is somewhat complicated, the known acoustic signal can be removed with high removal accuracy.

  In addition, the image display unit is configured such that the amplitude spectrum of the section in the mixed acoustic signal including the known acoustic signal and the temporal position shift of the amplitude spectrum of the corresponding section of the known acoustic signal, the time change of the frequency characteristics, the volume It is preferable that the corrected amplitude spectrum obtained by correcting at least one of the time change, the expansion / contraction in the time axis direction, and the expansion / contraction in the frequency axis direction can be displayed in a position aligned on the time axis. In this way, since the state of the corrected amplitude spectrum can be visually confirmed, it can be estimated while viewing the image how the corrected spectrum can be used to improve the removal accuracy. Become.

  The image display unit is preferably configured to display an image of the amplitude spectrum of the acoustic signal obtained by removing the corrected amplitude spectrum from the amplitude spectrum of the mixed acoustic signal. In this way, since the effect of the correction can be confirmed on the image, the known acoustic signal can be removed from the mixed acoustic signal to the maximum while performing the correction by the cut-and-try method.

  Further, the program of the present invention is a computer for use in a known acoustic signal removal device, a mixed acoustic signal conversion step for converting a mixed acoustic signal into a time-frequency representation to obtain an amplitude spectrum of the mixed acoustic signal and a phase of the mixed acoustic signal; Based on the known acoustic signal conversion step for obtaining the amplitude spectrum of the known acoustic signal by converting the known acoustic signal corresponding to the acoustic signal included in the mixed acoustic signal into a time-frequency representation, and the amplitude spectrum of the mixed acoustic signal, Corrected at least one of the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the temporal change of the frequency characteristics, the temporal change of the volume, the expansion and contraction in the time axis direction, and the expansion and contraction in the frequency axis direction. A correction step for obtaining a corrected amplitude spectrum of the known acoustic signal and a known amplitude spectrum of the mixed acoustic signal A removing step for removing the corrected amplitude spectrum of the reverberant signal, an inverse converting step for obtaining a unit waveform by performing inverse transformation to the time expression based on the amplitude spectrum after removal obtained by the removing step and the phase of the mixed acoustic signal, and a unit And a synthesizing step for obtaining an acoustic signal obtained by synthesizing a waveform and removing a component of a known acoustic signal.

It is a block diagram which shows the structure of an example of embodiment of the known acoustic signal removal apparatus of this invention. It is a figure which shows the step in the case of implementing the known acoustic signal removal method of this invention. It is a flowchart which shows an example of the algorithm of the program used when implement | achieving the principal part of the known acoustic signal removal apparatus of this invention using a computer. It is a flowchart which shows the detailed step in step ST103. It is a flowchart which shows the detail of the step in the case of performing estimation operation | movement by both the estimation which a human is concerned, and automatic estimation. It is a figure which shows the screen structure of the interface of an editor. It is a figure which shows the time change of the power of a mixed acoustic signal. It is a figure which shows the time change of the amplitude spectrum of a mixed acoustic signal. It is a figure which shows the time change of the power of the known acoustic signal of the sound source used as the origin of BGM. It is a figure which shows the time change of the amplitude spectrum of the known acoustic signal of the sound source used as the origin of BGM. It is a figure which shows the time change of the power of the desired acoustic signal after a known acoustic signal removal. It is a figure which shows the time change of the amplitude spectrum of the desired acoustic signal after a known acoustic signal removal.

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an embodiment of a known acoustic signal removal apparatus of the present invention that implements the known acoustic signal removal method of the present invention. This known acoustic signal removing device includes a mixed acoustic signal converting means 1, a known acoustic signal converting means 2, a correcting means 3, an interface 4, a removing means 5, an inverse converting means 6, and a synthesizing means 7. Is done. The mixed acoustic signal conversion means 1 outputs a mixed acoustic signal m (t) in which an acoustic signal b (t) such as BGM is mixed with an acoustic signal s (t) (t is a time axis) such as a desired voice or a physical sound. (At this time, s (t) and b (t) are unknown, and only m (t) is input), converted into a time-frequency representation and the amplitude spectrum M (ω, t) of the mixed acoustic signal and the mixed acoustic The signal phase θm (ω, t) is obtained. Further, the known acoustic signal conversion means 2 converts the known acoustic signal b ′ (t) of the sound source that is the source of the acoustic signal b (t) to be removed into a time-frequency representation to convert the amplitude spectrum B ′ (ω of the known acoustic signal. , T). Then, the correcting means 3 is based on the amplitude spectrum M (ω, t) of the mixed acoustic signal, and the time of the amplitude spectrum B ′ (ω, t) of the known acoustic signal with respect to the amplitude spectrum M (ω, t) of the mixed acoustic signal. A corrected amplitude spectrum B (ω, t) of a known acoustic signal in which a positional shift, a time change in frequency characteristics, a time change in volume, a time axis direction expansion and contraction, and a frequency axis direction expansion and contraction are corrected is obtained. For the automation, the correction means 3 is configured to automatically estimate and correct all of the position shift, the time change of the frequency characteristic, the time change of the volume, the expansion and contraction in the time axis direction, and the expansion and contraction in the frequency axis direction. Can be configured. However, in this embodiment, the correction means 3 performs all corrections for the positional displacement, the time change of the frequency characteristics, the time change of the volume, the expansion / contraction in the time axis direction, and the expansion / contraction in the frequency axis direction. It is configured so that humans can specify manually. As will be described in detail later, the interface 4 includes an image display unit that displays an image so that the amplitude spectrum of the mixed acoustic signal and the amplitude spectrum of the known acoustic signal can be visually compared. The interface 4 allows a human to specify a section of a known acoustic signal included in the mixed acoustic signal based on the amplitude spectrum of the mixed acoustic signal and the amplitude spectrum of the known acoustic signal, and can specify the above-described correction. It is configured. The removing unit 5 removes the corrected amplitude spectrum B (ω, t) of the known acoustic signal from the amplitude spectrum M (ω, t) of the mixed acoustic signal. Then, the inverse conversion means 6 performs an inverse conversion to the time expression based on the amplitude spectrum S (ω, t) after removal obtained by the removal means 5 and the phase θm (ω, t) of the mixed acoustic signal, and the unit waveform s. '(T) is obtained. Finally, the synthesizing unit 7 synthesizes the unit waveform s ′ (t) output from the inverse transform unit 6 to obtain an acoustic signal s (t) from which a known acoustic signal component is removed. The interface 4 displays the post-removal amplitude spectrum S (ω, t) output from the removing unit 5 on the image display unit (see FIG. 6). The interface 4 has a built-in sound reproduction unit, and reproduces the mixed sound signal, the known sound signal, and the synthesized sound signal output from the synthesizing unit 7. According to this configuration, the effect of the correction can be confirmed visually by the image display unit, and can also be confirmed by the auditory sense by the sound reproduction unit. Therefore, while performing correction by the cut-and-try method, By specifying the necessary correction, it is possible to remove as much known acoustic signals as possible from the mixed acoustic signals.

  Next, an example of a more detailed embodiment of the present invention will be described with reference to FIGS. FIG. 2 shows steps when the known acoustic signal removal method of the present invention is implemented, and FIG. 3 shows an algorithm of a program used when the main part of the known acoustic signal removal apparatus of the present invention is realized using a computer. It is a flowchart which shows an example.

  FIG. 4 is a flowchart showing detailed steps in step ST103. FIG. 5 is a flowchart showing the details of the steps in the case where the estimation operation is performed for both estimation involving human beings and automatic estimation. Hereinafter, the signal removal operation in the method and apparatus of the present invention will be described with reference to FIGS.

First, in the following description, a mixed acoustic signal m (t) in which an acoustic signal b (t) such as BGM is mixed with an acoustic signal s (t) (t is a time axis) such as a desired voice or a physical sound is observed. Shall be.

  Here, the problem of obtaining an unknown s (t) is solved when m (t) is given under the condition that the acoustic signal b '(t) of the sound source that is the source of b (t) is known. For example, when an acoustic signal m (t) of a TV program or the like in which a BGM is sounding together with a human voice or a sound is input, the music of the BGM is known and the acoustic signal b ′ (t) can be prepared separately. The BGM in the program is removed using the BGM music sound signal, and the process of obtaining the sound signal s (t) of only human voice or sound is realized.

Here, since b (t) and b ′ (t) do not completely match,

In the process corresponding to the subtraction, it is necessary to estimate the component corresponding to b (t) from b '(t) and obtain s (t). Specifically, since the known acoustic signal b ′ (t) is often accompanied by the following deformation in the mixed sound m (t), a component corresponding to b (t) is estimated by correction. To do. The correction target is mainly the following temporal position shift, time change of frequency characteristics, time change of volume, and expansion or contraction in the time axis or frequency axis direction.

(Time shift)
The position where the known acoustic signal b ′ (t) is sounding in the mixed sound m (t) is not always from the beginning. Therefore, it is necessary to shift the known acoustic signal b ′ (t) in the time axis direction, match both relative positions, and subtract the known acoustic signal from the mixed sound.

(Change in frequency characteristics over time)
When the known acoustic signal b ′ (t) is produced in the mixed sound m (t), the frequency characteristic often changes due to the influence of a graphic equalizer or the like. For example, the low range and high range may be emphasized and attenuated. Therefore, it is necessary to correct by changing the frequency characteristic of b ′ (t) in the same manner, and to subtract a known acoustic signal from the mixed sound.

(Volume change over time)
When a known acoustic signal b ′ (t) is generated in the mixed sound m (t), the mixing ratio is changed by operation of a mixer fader or the like when the mixed sound is generated, and the volume often changes with time. Therefore, it is necessary to correct the volume of b ′ (t) by changing the time similarly, and to subtract a known acoustic signal from the mixed sound.

(Expansion / contraction in time axis or frequency axis direction)
When the known acoustic signal b ′ (t) sounds in the mixed sound m (t), it may be expanded or contracted in the time axis or frequency axis direction due to the difference in the rotational speed of the record or the like. Therefore, it is necessary to correct b ′ (t) by expanding and contracting in the time axis or frequency axis direction, and subtracting a known acoustic signal from the mixed sound.

  In the method of the present invention, as shown in FIG. 2, in step ST1, first, the mixed acoustic signal is Fourier transformed to obtain the phase of the mixed acoustic signal (step ST2) and the amplitude spectrum of the mixed acoustic signal (step ST3). (Mixed acoustic signal conversion step). Next, in step ST4, the known acoustic signal corresponding to the acoustic signal included in the mixed acoustic signal is Fourier transformed to obtain the amplitude spectrum (step ST5) of the known acoustic signal (known acoustic signal converting step). Then, at step ST6, based on the amplitude spectrum of the mixed acoustic signal, the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the time change of the frequency characteristic, the time change of the volume, the time axis direction A corrected amplitude spectrum (step ST7) of a known acoustic signal obtained by correcting at least one of the expansion and contraction of frequency and the expansion and contraction in the frequency axis direction is obtained (correction step). Next, in step ST8, the corrected amplitude spectrum of the known acoustic signal is removed from the amplitude spectrum of the mixed acoustic signal to obtain a removed amplitude spectrum (step ST9) (removal step). In step ST10, a unit waveform is obtained by performing inverse Fourier transform based on the amplitude spectrum after removal obtained in the removal step and the phase of the mixed acoustic signal (inverse transform step). Finally, in step ST11, a unit waveform is synthesized by an overlap-add method to obtain an acoustic signal from which a known acoustic signal component is removed (synthesis step).

  In the algorithm of FIG. 3, in step ST101, the mixed acoustic signal is Fourier transformed to obtain the amplitude spectrum of the mixed acoustic signal and the phase of the mixed acoustic signal. Next, in step ST102, a known acoustic signal corresponding to the acoustic signal included in the mixed acoustic signal is Fourier transformed to obtain an amplitude spectrum of the known acoustic signal. Next, in step ST103, based on the amplitude spectrum of the mixed acoustic signal, the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the time change of the frequency characteristic, the time change of the volume, the time axis A corrected amplitude spectrum of a known acoustic signal in which at least one of direction expansion and contraction in the frequency axis direction is corrected is obtained. Thereafter, in step ST104, the corrected amplitude spectrum of the known acoustic signal is removed from the amplitude spectrum of the mixed acoustic signal to obtain the removed amplitude spectrum. Next, in step ST105, an inverse Fourier transform is performed based on the amplitude spectrum after removal obtained in step ST104 and the phase of the mixed acoustic signal to obtain a unit waveform, and in step ST106, the unit waveform is synthesized by the overlap-add method. Thus, an acoustic signal from which a known acoustic signal component is removed is obtained. Thereafter, in step ST107, a determination is made as to whether or not the user has evaluated the acoustic signal after removal. If the determination result is unsatisfactory, the process returns to step ST103 and correction is performed again. Step ST103 to step ST107 are repeated until the user is satisfied.

The contents executed at each step will be described in detail below. In the method according to the embodiment of the present invention, the subtraction process is performed on the amplitude spectrum in the time frequency domain without performing the subtraction process on the waveform in the time domain. Short-time Fourier transform (STFT) Xm (ω, t), Xb ′ (ω, t) at time t using the window function h (t) for the acoustic signals m (t) and b ′ (t)

And their amplitude spectra M (ω, t), B ′ (ω, t) are

It is obtained by

  In the current implementation, the acoustic signal is A / D converted with a sampling frequency of 44.1 kHz and a quantization bit number of 16 bits, and a short-time Fourier transform (STFT) using a Hanning window with a window width of 8192 points as the window function h (t). ) By fast Fourier transform (FFT). At that time, since the frame of the fast Fourier transform (FFT) is shifted by 441 points, the frame shift time (1 frame shift) is 10 ms. This frame shift is used as a processing time unit.

The amplitude spectrum S (ω, t) of the desired acoustic signal s (t) after removal of the known acoustic signal is obtained from the amplitude spectra M (ω, t) and B ′ (ω, t) by the following formula. Here, B (ω, t) is an amplitude spectrum after correcting B ′ (ω, t).

  Various parameter functions a (t), g (ω, t), p (ω), q (t), r (t), c (ω, t) in the above formula will be described in order.

  a (t) is a function of an arbitrary shape for finally adjusting the amount by which the component corresponding to the amplitude spectrum of the known acoustic signal is subtracted from the amplitude spectrum of the mixed sound. Usually, a (t) ≧ 1 And The larger the value, the larger the subtraction amount.

g (ω, t) is a function for correcting the time change of the frequency characteristic and the time change of the volume,

Define as follows. Here, gω (ω, t) represents a time change of the frequency characteristic, and gω (ω, t) = 1 when there is no change of the frequency characteristic. On the other hand, gt (t) represents a temporal change in volume, and is a constant when there is no change in volume. The volume difference between M (ω, t) and B ′ (ω, t) is basically corrected with gt (t). gr (t) is a function mainly for raising the value of g (ω, t) as a whole, and is used for fine adjustment during correction. If not used, gr (t) = 0.

  p (ω) is a function for correcting expansion / contraction in the frequency axis direction, and linear / nonlinear expansion / contraction in the frequency axis direction is obtained by converting the frequency axis ω of the amplitude spectrum B ′ (ω, t). enable. B ′ (ω, t) is 0 outside the original definition range of ω, and is interpolated as appropriate when discretized and implemented.

  q (t) is a function for correcting expansion / contraction in the time axis direction, and linear / nonlinear expansion / contraction in the time axis direction is obtained by converting the time axis t of the amplitude spectrum B ′ (ω, t). enable. Note that B ′ (ω, t) takes 0 outside the original definition range of t, and is appropriately interpolated when discretized and mounted.

  r (t) is a function for correcting a positional shift in time, and usually a constant shift width is corrected by setting a constant. When the shift width changes with time, a function for correcting the width at each time is set. Note that B ′ (ω, t) takes 0 outside the original definition range of t, and is appropriately interpolated when discretized and mounted. Although it is possible to express q (t) and r (t) by a single function, q (t) is set for the purpose of continuous expansion and contraction, and r (t) is discontinuous. It is set for the purpose of representing the position shift.

  c (ω, t) is a function of an arbitrary shape for the equalizing process and the fader operation process for the amplitude spectrum. Depending on the shape in the ω direction, the frequency characteristic after removal of the known acoustic signal can be adjusted like a graphic equalizer. Further, the change in volume after removal of the known acoustic signal can be adjusted by the shape in the t direction, like the volume fader operation of the mixer. When not used, c (ω, t) = 1.

Xs (ω, t) is obtained by using the amplitude spectrum S (ω, t) thus obtained and the phase θm (ω, t) of the mixed sound m (t), and is subjected to inverse Fourier transform (IFFT). A unit waveform s ′ (t) is obtained.

  The unit waveform s ′ (t) is arranged by an overlap add method, thereby synthesizing a desired acoustic signal s (t) after removal of the known acoustic signal.

In the above, the case where one kind of known acoustic signal b ′ (t) is included in the mixed acoustic signal m (t) has been described, but b′1 (t), b′2 (t),. . . , B′N (t), a plurality of amplitude spectra B′1 (ω, t), B′2 (ω, t),. . . , B′N (ω, t), B1 (ω, t), B2 (ω, t),. . . , BN (ω, t)

Thus, the process can be expanded to obtain S (ω, t). In that case, various parameter functions of Bn (ω, t) are set in order, or various parameter functions of a plurality of Bn (ω, t) are set in parallel while maintaining the overall balance.

  In the above description, the monaural signal has been described. However, the stereo signal may be applied by mixing the left and right to convert to a monaural signal, or may be applied to the left and right signals of the stereo signal. Good. Moreover, you may apply using the direction of a sound source in a stereo signal.

  The setting of the various parameter functions will be described. When the method of the present invention is applied, various parameter functions a (t), g (ω, t) (gω (ω, t), gt (t) of [Equation 11], [Equation 12], and [Equation 13]. ), Gr (t)), p (ω), q (t), r (t), c (ω, t) may be automatically estimated or manually set by a human. Good. Alternatively, a human may correct after automatic estimation. In the following, a specific automatic estimation method and a case of using the interface 4 on the known acoustic signal removal editor that enables manual correction by humans will be described.

  First, various parameter functions g (ω, t) (gω (ω, t), gt (t)), p (ω), q (t) of [Equation 11], [Equation 12], and [Equation 13]. , R (t) will be described below with reference to FIG. First, in step ST201, a set Ψ of BGM sections ψ is designated and automatically estimated, p (ω) and q (t) are automatically estimated in step ST202, and gω (ω, t) and gt (t) in step ST203. , R (t) is automatically estimated. These steps are continued until the parameter function of the estimation result converges (step ST204). After step ST205, the correction operation is performed using the interface 4.

  In the estimation of g (ω, t), first, the time change gω (ω, t) of the frequency characteristic is estimated, and then the time change gt (t) of the volume is estimated. However, prior to estimating g (ω, t), p (ω), q (t), and r (t) need to be determined. Here, for convenience, B ′ (p (ω), q (t) + r (t)) is described as B ′ (ω, t).

In the estimation of the time variation gω (ω, t) of the frequency characteristic, as a rule, a section (hereinafter referred to as a BGM section) in which the acoustic signal s (t) of only human voice or sound is not included is used. A plurality of BGM sections may be used. In the BGM section, the amplitude spectrum M (ω, t) of the mixed sound m (t) is almost entirely derived from the amplitude spectrum B ′ (ω, t) corresponding to BGM by the known acoustic signal b ′ (t). Become. Therefore, when the frequency characteristic is steady without changing over time, that is, when gω (ω, t) = g′ω (ω) can be assumed, g′ω (ω) is

Estimated by However, ψ represents one BGM section (region on the time axis), and ψ is a set of ψ. On the other hand, when the frequency characteristic changes with time, from the BGM section ψ near time t of gω (ω, t).

Gω (ω, t) is estimated by interpolation (interpolation or extrapolation) (when there are BGM sections on both sides, interpolation is performed from both sides). Finally, gω (ω, t) is smoothed in the frequency axis direction. Note that the smoothing width can be arbitrarily set, and smoothing may not be performed.

In the estimation of the sound volume change gt (t), the amplitudes of M (ω, t) and gω (ω, t) B ′ (ω, t) after frequency characteristic correction at each time are compared. However, M (ω, t) includes components derived from s (t) in addition to components derived from B ′ (ω, t). Therefore, the frequency axis ω is divided into a plurality of frequency bands Φ, and for each band φ (φ∈Φ)

(Φ represents a set of φ). Arbitrary division can be applied as Φ. For example, the division may be performed for every octave of equal temperament used in music (divided at equal intervals on the logarithmic frequency axis). And gt (t) is min (g't (φ, t)) or

Estimated by In the case of min (g′t (φ, t)), the amplitude is compared in the frequency band where M (ω, t) and gω (ω, t) B ′ (ω, t) are closest. become. Finally, gt (t) is smoothed in the time axis direction. Note that the smoothing width can be arbitrarily set, and smoothing may not be performed.

In the estimation of p (ω) and q (t), p (ω) and q are set so that the distance (for example, logarithmic spectral distance) between M (ω, t) and B (ω, t) is minimized. Change (t). At that time, B (ω, t) = a (t) g (ω, t) B ′ (p (ω), q (t) + r (t)) out of the right side, a (t) = 1,
1. 1. Estimate g (ω, t) and r (t) after temporarily fixing p (ω) and q (t) (during estimation) After g (ω, t) and r (t) (during estimation) are temporarily fixed, p (ω) and q (t) are repeatedly iterated to obtain an appropriate p (ω) , Q (t). This may be performed in a segmented manner by dividing the time axis, rather than being performed at once for all sections of the acoustic signal. The initial value is determined in consideration of the continuity of the preceding and following sections. Further, using the set Ψ of BGM sections ψ, p (ω), q () so that the time axis of the correspondence relationship between M (ω, t) and B (ω, t) in the plurality of sections is matched. t) may be estimated.

  In the estimation of r (t), in principle, using the set Ψ of BGM intervals ψ, the time axis of the correspondence relationship between M (ω, t) and B (ω, t) in those intervals is matched. Find r (t). In many cases, r (t) is a constant, but when a part of the known acoustic signal b ′ (t) is not used but is mixed while being used in a jump, the section is skipped. r (t) is a discontinuous function.

In the above estimation of g (ω, t), r (t), etc., a set Ψ of BGM sections ψ is used. This may be specified manually by a human. Or you may add to the set of BGM section designated manually by automatic estimation. FIG. 5 is a flowchart showing a software algorithm of a program corresponding to either a case where a human designates manually or an automatic estimation. In the case of automatic estimation, steps ST302 to ST313 in FIG. 5 are executed. In the automatic estimation of Ψ, basically, a set of remaining BGM sections is obtained using one BGM section ψ1 as a clue. First, the first ψ1 is determined manually by a human or by dividing the time axis of the acoustic signal finely and determining the correspondence between these short divided sections. When a human does not designate it manually, B (ω, t) is temporarily calculated (step ST302), and the distance between the amplitude spectra of time windows obtained by finely dividing M (ω, t) and B (ω, t) ( (Corresponding to similarity) is calculated (step ST303). Then, the correspondence relationship of the time window of the minimum distance is checked (step ST304), and the section including the result is set as ψ1 to be an initial ψ (step ST305). Next, various parameter functions of B (ω, t) are estimated based on ψ including ψ1 (step ST306 to step ST309), and B (ω, t) is calculated (step ST310). It is checked whether the estimated values of the parameters have converged. If the estimated values have not converged, the distance between the amplitude spectra of M (ω, t) and B (ω, t) (for all the sections of Ψ ( Equivalent). Here, a constant multiple of the maximum value (or average value) is set as a BGM section determination threshold (step ST312). Then, a section having a distance less than or equal to the BGM section determination threshold is detected and newly added to Ψ (step ST313). However, an upper limit can be set for addition. By repeating this estimation and addition, Ψ is updated and various parameter functions are obtained appropriately. Here, as the distance between M (ω, t) and B (ω, t), for example, the root mean square spectral distance

Is effective.

  Next, adjustment of various parameter functions by the interface on the known acoustic signal removal editor will be described.

  All parameter functions a (t), g (ω, t) (gω (ω, t), gt (t), gr (t)), p (ω), q of [Equation 11] to [Equation 13] A known acoustic signal removal editor in which the shapes of (t), r (t), and c (ω, t) can be manually set by a human will be described below. The user of the editor may draw and specify an arbitrary function shape from the beginning, or may first perform automatic estimation and modify the result.

  The screen configuration of the editor interface 4 is shown in FIG. The editor is roughly divided into a sub-window W1 for operating the mixed sound signal m (t), a sub-window W2 for operating the known sound signal b ′ (t), and a desired sound signal s (t) operation after removing the known sound signal. The sub-window W3 is composed of three. When there are a plurality of types of known acoustic signals b ′ (t), the known acoustic signal b ′ (t) operated in the sub-window W2 can be switched by the changeover switch W2S. In this interface, steps ST205 to ST219 shown in FIG. 4 are executed.

  First, the functions common to all subwindows are described. The operation range slider P1 indicates where in the acoustic signal is currently displayed. The cursor P2 represents the position on the time axis of the current operation target. When the iconized (folding) button P3 is pressed, the subwindow to which the button belongs is temporarily folded to become smaller. Narrow sub-screens can be effectively used by hiding unused sub-windows that are not currently operated. When the float (enlarge) button P4 is pressed, the subwindow to which the button belongs is temporarily separated from the parent window (float), and further enlarged to facilitate operation / editing. When only the float (enlarge) button P4 is drawn, when this button is pressed, the subwindow associated therewith is floated and newly appears.

  In the sub window W1, a graph E1 of the power of the mixed acoustic signal m (t) and a graph E2 of its amplitude spectrum M (ω, t) are displayed. In the sub-window W2, a graph E3 of the power of the known acoustic signal b ′ (t) and a graph E4 of its amplitude spectrum B ′ (ω, t) are displayed. In the sub window W3, a graph E5 of the power of the acoustic signal s (t) after removal of the known acoustic signal and a graph E6 of its amplitude spectrum S (ω, t) are displayed. In each amplitude spectrum, the amplitude is drawn with shading on the left side (the horizontal axis is the time axis, the vertical axis is the frequency axis), and the amplitude at the cursor position is drawn on the right side (the horizontal axis is power, the vertical axis is the frequency axis) ).

  In addition, the reproduction control operation panel P51 has a group of buttons that can reproduce, stop, fast-forward, and fast-reverse the mixed sound signal so that a human can hear and confirm. By operating the reproduction control operation panel P51, the interface 4 reproduces the mixed sound signal by the built-in sound reproduction unit.

  The sub-window W2 for operation of the known acoustic signal b ′ (t) is the center of operation, and all parameter functions a (t), g (ω, t) (gω) of [Equation 12] and [Equation 13]. The shapes of (ω, t), gt (t), gr (t)), p (ω), q (t), r (t) can be freely set. Hereinafter, description of each operation panel will be described.

1. Operation panel C1 for correcting the time variation of the frequency characteristics (right side of E7)
On the panel for displaying and operating gω (ω, t), gω (ω, t) at time t at the cursor position is drawn (the horizontal axis is the size and the vertical axis is the frequency axis). The setting operation result is immediately reflected on the display panel E7 of g (ω, t) (steps ST205 and ST206). In E7, the magnitude of the value of g (ω, t) is depicted in shading (the horizontal axis is the time axis, and the vertical axis is the frequency axis).

2. Operation panel C2 for correcting changes in volume over time (lower side of E7)
The setting operation result is immediately reflected on the display panel E7 for g (ω, t) on the panel for displaying and operating gt (t) (steps ST207 and ST208).

3. Operation panel C3 (lower side of E7) for raising the value of g (ω, t) as a whole
The setting operation result is immediately reflected on the display panel E7 for g (ω, t) on the panel for displaying and operating gr (t) (steps ST209 and ST210).

4). Operation panel C4 for finally adjusting the amount of subtraction of the component corresponding to the amplitude spectrum of the known acoustic signal from the amplitude spectrum of the mixed sound
It is a panel for displaying and operating a (t). When this panel is operated, the change of a (t) is immediately reflected in the display (steps ST211 and ST212).

5. Operation panel C5 for correcting expansion and contraction in the frequency axis direction
This is a panel for displaying and operating p (ω). When this panel is operated, the change of p (t) is immediately reflected in the display (steps ST213 and ST214).

6). Operation panel C6 for correcting expansion and contraction in the time axis direction
This is a panel for displaying and operating q (t). When this panel is operated, the change of q (t) is immediately reflected in the display (steps ST215 and ST216).

7). Operation panel C7 for correcting a positional shift in time
It is a panel for displaying and operating r (t). When this panel is operated, the change in r (t) is immediately reflected in the display (steps ST217 and ST218).

  The reproduction control operation panel P52 includes a group of buttons that can reproduce, stop, fast-forward, and fast-reverse known acoustic signals for human confirmation by listening. By operating the reproduction control operation panel P52, the interface 4 reproduces a known acoustic signal by a built-in acoustic reproduction unit.

  Next, in the sub-window W3 for operating the acoustic signal s (t) after removing the known acoustic signal, the shape of the parameter function c (ω, t) of [Equation 11] can be freely set. Hereinafter, description of each operation panel will be described.

1. Graphic equalizer (GEQ) operation panel C8 (right side of E8)
A panel for displaying / manipulating the shape of c (ω, t) in the ω direction, where c (ω, t) at the time t at the cursor position is drawn (the horizontal axis is the size and the vertical axis is the frequency) axis). The setting operation result is immediately reflected on the display panel E8 of c (ω, t). In E8, the magnitude of the value of c (ω, t) is depicted in shading (the horizontal axis is the time axis, and the vertical axis is the frequency axis).

2. Volume fader operation panel C9 (below E8)
This is a panel for displaying / manipulating the shape of c (ω, t) in the t direction, and the setting operation result is immediately reflected on the display panel E8 of c (ω, t).

  The reproduction control operation panel P53 has a group of buttons that can reproduce, stop, fast forward, and fast reverse the synthesized sound signal (output of the synthesizing means 7) for human confirmation by listening. By operating the reproduction control operation panel P53, the interface 4 reproduces the sound signal synthesized by the built-in sound reproduction unit.

  Next, implementation of the present embodiment will be described. First, when a mixed acoustic signal m (t) in which an acoustic signal b (t) such as BGM is added to an acoustic signal s (t) such as a sound or a sound of sound is observed, a sound source that is the source of b (t) Programs that can determine the unknown s (t) under the condition that the acoustic signal b ′ (t) of the computer is known, various operating systems (Linux 2.4, SGI IRIX 6.5, Microsoft Windows XP: registered trademark) Implemented above. When an audio file containing m (t) and b '(t) is given to this program, an audio file of s (t) can be obtained.

  As a result of experiments on various mixed sounds in which background music (BGM) is added to human voice and sound, the BGM in the mixed sound is removed using the sound signal of the original music of the BGM. It was confirmed that voice and sound were obtained. Even if songs of various genres such as songs with and without drums, popular music and classical music are included as BGM, they can be removed.

  As an example of the experimental result, FIGS. 7 to 12 show results of actually processing a mixed sound in which classical music is played in the BGM of a dialogue between two men and women. The mixed acoustic signal m (t) shown in FIG. 7 and FIG. 8 is used as an input, and the result of removing the BGM component using the known acoustic signal b ′ (t) of the original sound source shown in FIG. 9 and FIG. The acoustic signal s (t) after removal of the known acoustic signal shown in FIG. The mixed sound of the example of this processing result is obtained by adding the acoustic signal of classical music extracted from the “music database for RWC research” to the acoustic signal of the conversation between two men and women extracted from the “RWCP audio dialogue database” It is.

  According to the present invention, the correction step shifts the time position of the amplitude spectrum of the known acoustic signal relative to the amplitude spectrum of the mixed acoustic signal, the time change of the frequency characteristic, the time change of the volume, the expansion and contraction in the time axis direction, and the frequency axis In order to obtain a corrected amplitude spectrum of a known acoustic signal corrected for at least one of the direction expansion and contraction, and to remove the corrected amplitude spectrum from the amplitude spectrum of the mixed acoustic signal, it is included in the mixed acoustic signal as non-stationary noise. There is an advantage that the known acoustic signal can be removed with high accuracy.

  Further, according to the present invention, when an audio signal such as a TV program or a movie with BGM sounding in the background of a human voice or sound is input, BGM in the program is removed using a BGM music audio signal prepared separately. In addition, it is possible to obtain an acoustic signal only of a human voice or sound.

  Furthermore, by assigning another music as BGM to the acoustic signal after BGM removal, it becomes possible to reuse music such as TV programs and movies.

  Since the known acoustic signal may be an arbitrary acoustic signal, it can be applied regardless of the genre of music, with or without vocals, and with or without accompaniment. Further, the present invention is not limited to music, and can be applied to any known noise including stationary noise and non-stationary noise.

  Further, by using the interface on the known acoustic signal removal editor and manually correcting by a human, a higher quality removal operation can be realized at a practical site.

W1, W2, W3 Subwindow P1 Operation range slider P2 Cursor P3, P4 Button P51-P53 Playback control operation panel E1-E6 Graph E7, E8 Display panel C1-C9 Operation panel

Claims (1)

A known acoustic signal removal method for removing a component of a known acoustic signal from a mixed acoustic signal obtained by mixing a plurality of acoustic signals,
A mixed acoustic signal converting step of converting the mixed acoustic signal into a time-frequency representation to obtain an amplitude spectrum of the mixed acoustic signal and a phase of the mixed acoustic signal;
A known acoustic signal converting step of converting a known acoustic signal corresponding to a known acoustic signal included in the mixed acoustic signal into a time-frequency representation to obtain an amplitude spectrum of the known acoustic signal;
Based on the amplitude spectrum of the mixed acoustic signal, the temporal position shift of the amplitude spectrum of the known acoustic signal with respect to the amplitude spectrum of the mixed acoustic signal, the time change of the frequency characteristics, the time change of the volume, and the expansion and contraction in the time axis direction And a correction step for obtaining a corrected amplitude spectrum of the known acoustic signal in which at least one of expansion and contraction in the frequency axis direction is corrected,
A removing step of removing the corrected amplitude spectrum of the known acoustic signal from the amplitude spectrum of the mixed acoustic signal;
An inverse transformation step for obtaining a unit waveform by performing an inverse transformation on the time representation based on the amplitude spectrum after removal obtained by the removal step and the phase of the mixed acoustic signal;
A known acoustic signal removal method comprising: a synthesis step of obtaining an acoustic signal obtained by synthesizing the unit waveforms to remove the components of the known acoustic signal.