JP6482880B2 - Mixing apparatus, signal mixing method, and mixing program - Google Patents

Description

  The present invention relates to a technique for mixing two or more input signals.

  The basic configuration of mixing is the addition of a plurality of input signals. In the field of recording and broadcasting, equalizers are placed before and after the adder in order to listen to a variety of mixed instruments and audio in a balanced manner. By using the equalizer, processing such as emphasizing a desired frequency band for each input signal or lowering the volume of an insignificant frequency band in the background to enhance the priority sound in the mixed sound is performed.

  On the other hand, a “smart mixing” method has been proposed in which the input signal is developed on the time-frequency plane, and after amplitude adjustment and phase adjustment are performed for each point on the time-frequency plane, they are added and returned to the signal on the time axis. (For example, refer to Patent Document 1). Voices and musical instrument sounds have a fine structure on the time-frequency plane. By performing various kinds of input signals in accordance with the structure on the time-frequency plane, it is possible to perform more detailed mixing processing. In Patent Document 1, the signal characteristic of a processing target point is determined using input data of another point having a predetermined relationship with the processing target point on the time-frequency plane, and the priority sound is clarified according to the signal characteristic. The degree is determined.

Patent No. 5057535

  In Patent Document 1, a rational method for optimizing a gain given to an input signal is not defined. When the mixing method of Patent Document 1 is actually applied, the gain for each frequency is determined by trial and error in accordance with the sound source, and an appropriate gain may not be set. In addition, conventional general mixing techniques rely on experience and intuition, and no reasonable standards have been established for gain setting and equalizer characteristic setting.

  The following problems arise if the gain for each frequency of the sound to be preferentially clarified by mixing (hereinafter referred to as “priority sound”) is not set appropriately. First, if the change in the gain of the priority sound is too strong, it will feel unnatural as the priority sound in the output (mixed sound), and even if it is heard as a sound, the content cannot be heard as a sound. In addition, the volume change or sound quality change of the priority sound may be too strong and cause discomfort. Conversely, if the gain change is too weak, the priority sound cannot be heard sufficiently.

  The following problems arise when the gains of sounds other than the priority sounds (hereinafter referred to as “non-priority sounds”) cannot be set appropriately. If the gain change is too strong, the lack of non-priority sounds and sudden changes in sound quality will be noticeable in the output (mixed sound), and you will feel uncomfortable. Listening to the priority sound is hindered by being distracted. Conversely, if the gain change is too weak, the priority sound cannot be fully enhanced.

  Therefore, it is an object to establish a rational gain setting method at the time of audio data mixing, and to improve and stabilize the operation of the mixing apparatus.

In order to solve the above problems, the present invention uses the following two principles as basic principles.
(1) The log intensity of the output signal is limited to a range that does not exceed the sum of the log intensity of the input signal. This is referred to as “the principle of sum of logarithmic intensities”. According to the “principle of sum of logarithmic intensities”, it is suppressed that the priority sound is excessively enhanced and the mixed sound is uncomfortable.
(2) Limiting the power reduction of the non-priority sound to a range that does not exceed the power increase of the priority sound. This is called “the principle of hole filling”. By the “principle of hole filling”, it is possible to suppress the occurrence of a sense of incongruity due to excessive suppression of non-priority sounds in the mixed sound.

Specifically, in one aspect of the present invention, the mixing device includes:
A frequency analysis unit that develops the first input signal and the second input signal in the time domain into a first signal and a second signal on a time frequency plane, respectively;
A signal processing unit for generating a mixed signal obtained by mixing the first signal and the second signal;
A frequency time conversion unit for converting the mixed signal into a time domain signal;
A signal output unit for outputting the converted signal;
Have
The signal processing unit
A first power for adjusting the power of the first signal for each point on the time-frequency plane within a range in which the log intensity of the output signal does not exceed the sum of the log intensity of the first signal and the log intensity of the second signal. A gain determination unit that determines a gain and a second gain that decreases the power of the second signal within a range not exceeding an increase in power of the first signal;
An adder for adding the first signal adjusted by the first gain and the second signal adjusted by the second gain;
Have

  An appropriate gain is set when the input signal is mixed, so that the operation of the mixing apparatus can be improved and stabilized.

It is a figure explaining the basic principle of this invention. It is a schematic block diagram of the mixing apparatus of embodiment. It is a figure which shows the power of a priority sound and a non-priority sound with a listener's audible level. It is the figure which subtracted the audible level from each of the priority sound and non-priority sound of FIG. 3, and plotted as the sound intensity (audibility correction power) which a listener feels. It is a figure which shows the sum by the logarithm scale of the priority sound and non-priority sound of FIG. It is a figure which shows the result of having multiplied the simple addition sum of the priority sound of FIG. It is a figure which makes the range which satisfy | fills both FIG. 5 (logarithmic intensity sum) and FIG. 6 (constant multiple of simple addition sum) as an upper limit at the time of addition (mixing). It is a figure which shows the priority sound and non-priority sound which were gain-adjusted within the range of FIG. It is a figure which shows the example of the loudness curve used for selection of the minimum audible power. It is a figure which shows the operation example of the mixing apparatus of embodiment. It is a figure which shows the operation state of the mixing apparatus at the time of smoothing a gain. It is a figure which shows the example in the case of setting correction coefficient B [k] directly. It is a flowchart which shows an example of the mixing signal processing method of embodiment. It is a flowchart which shows an example of the mixing signal processing method of embodiment, and is a flowchart following FIG. 13A. It is a flowchart which shows another example of the mixing signal processing method of embodiment.

  FIG. 1 is a diagram for explaining the basic principle of the present invention. In the following description, the case where the input signals x1 [n] and x2 [n] are mixed is taken as an example. The input signal x1 [n] is a priority signal such as voice. The input signal x2 [n] is a non-priority signal such as a background sound.

  The input signals x1 [n] and x2 [n] are each developed on a time-frequency plane (denoted as “tf plane” in the figure) by frequency analysis. For the frequency analysis, any method such as a short-time FFT (Fast Fourier Transform), a short-time Fourier transform, a wavelet transform, a transformation by a filter bank, or a transformation to a time-frequency distribution such as a Wigner distribution can be used. Let each signal developed on the time-frequency plane be X1 [i, k], X2 [i, k]. X1 [i, k] and X2 [i, k] are points of the input signal on the time frequency plane represented by the coordinate value i in the time direction and the coordinate value k in the frequency direction.

  Based on the power of each point of the input signal developed on the time-frequency plane, the gain of the priority sound and the non-priority sound at each point is determined using the "logarithmic intensity sum principle" and the "fill hole principle". . “The principle of the sum of logarithmic intensities” is a process for limiting the power of an output signal to a range not exceeding the sum of the logarithmic intensities of input signals as described above. “Principle of hole filling” is a process for limiting the power reduction of non-priority sounds to a range that does not exceed the power increase of the priority sound. Specific processing methods of these principles will be described later.

In the embodiment, in order to determine an optimum gain, in addition to the process (1) based on the principle of the sum of logarithmic intensity and the process (2) based on the principle of hole filling, the following process (3) is optionally added: (5) is introduced.
(3) When determining the gain, (a) an upper limit is set so that the level of power increase determined by the principle of the sum of logarithmic intensity does not exceed a level obtained by multiplying the simple addition value of the input sound by a certain amount. At least one of the following three conditions is added: (1) a fixed upper limit is set for the gain of the priority sound, and (c) a fixed lower limit is set for the gain of the non-priority sound. As a result, the mixed sound can be made more natural and gentle.
(4) In the time interval in which the signal-to-noise ratio is extremely lowered, the upper limit and the lower limit of (3) are relaxed. This makes the priority sound stand out even in a time interval where the signal-to-noise ratio is low, making it easy for the listener to hear.
(5) Parameters in the mixing process are configured to be updated sequentially rather than being calculated as a solution to the optimization problem. By adopting sequential update, “solving equations” can be replaced with “judgment of inequality true / false”, eliminating operations such as exponential function, logarithmic function, multiplication, etc., and a high-speed algorithm with only multiplication and addition / subtraction Can be configured. This facilitates mounting on a programmable logic device such as a field-programmable gate array (FPGA) or a plug-in for a digital audio workstation (DAW), thereby realizing real-time processing.

  At each point on the time frequency plane, the gain corresponding to X1 [i, k] and X2 [i, k] is multiplied. The signals after gain multiplication are assumed to be M1 [i, k] and M2 [i, k]. The gain-adjusted signals M1 [i, k] and M2 [i, k] are added and the two signals are superimposed on the time-frequency plane. Thereafter, the mixed sound is output by returning to the time domain signal.

  In this way, a natural mixed sound can be generated by determining and multiplying the gain for the input signal at each point on the time-frequency plane.

  FIG. 2 is a schematic diagram of the mixing device 1 according to the embodiment. The mixing apparatus 1 includes a signal input unit 11, a frequency analysis unit 12, a signal processing unit 15, a frequency time conversion unit 16, and a signal output unit 17. The signal input unit 11 inputs a plurality of input signals to be mixed. The input signal is an audio signal, for example, and includes a priority signal such as voice and a non-priority signal such as background sound.

  For example, the frequency analysis unit 12 develops an input signal on a time-frequency plane by a short-time FFT. The signal processing unit 15 calculates the power of the input signal at each point on the time-frequency plane, smoothes the power, and then calculates the gain for the priority signal and the non-priority signal by the gain determination unit 151. And after multiplying each gain calculated with respect to the priority signal and the non-priority signal, it adds and outputs the addition result. The frequency time conversion unit 16 converts the output signal from the signal processing unit 15 into a time domain signal. The signal output unit 17 outputs a signal restored in the time domain.

  The basic processing in the signal processing unit 15 will be described with reference to FIGS. Tables 1 and 2 show symbols used in the following description. Table 1 is a list of constant symbols, and Table 2 is a list of variable symbols.

<Principle of sum of log intensity>
FIG. 3 schematically shows the power [dB] of the priority sound (thick line) and the non-priority sound (solid line) at a certain time on the time-frequency plane as a function of frequency. This power is a power value E smoothed by the signal processing unit 15. The dotted line indicates the hearing limit at which the listener can detect sound if it is above this level.

  It is said that human hearing feels the strength of sound as a logarithm of power. Based on this idea, sound components that are 10 dB higher than the audible limit of the dotted line feel almost the same strength, and sound components that are 20 dB higher than the hearing limit feel almost the same strength. Further, the volume difference between the sound component that is 10 dB higher than the audible limit and the sound component that is 20 dB higher than the audible limit can be regarded as the same as the volume difference between the sound component that is 20 dB higher than the audible limit and the sound line segment that is 30 dB higher.

  FIG. 4 is a plot in which the dotted-line hearing limit audible level is subtracted from the priority sound and the non-priority sound of FIG. The power values of the priority signal and the non-priority signal obtained in FIG. 4 are “auditory correction power P” corrected with the human hearing limit set to zero dB. The size of the vertical axis in FIG. 4 is the sound intensity felt by the listener.

FIG. 5 shows the sum of logarithmic scales of the priority sound and the non-priority sound whose auditory sense is corrected in FIG. According to the above-described knowledge about human hearing, the principle is reached that it is appropriate for the listener to feel the sum of the priority sound and the non-priority sound (logarithmic intensity sum) corrected for auditory sense as a mixed sound. That is, the power of the mixed sound is indicated by a dotted line A in FIG. This is the “principle of sum of logarithmic intensity”. If the volume of a sound equal to the human hearing limit is multiplied by 1, 20 dB is 10 times and its logarithm is 1. 40 dB is 100 times and its logarithm is 2. Expressed on a logarithmic scale, the sum of 10 and 100 times the human hearing limit is 10 ³ or 1000 times the power.

However, if this principle is used as it is, the power of the mixed sound represented by the sum of the power P1 of the priority sound and the power P2 of the non-priority sound after auditory correction (FIG. 4) becomes P1 × P2, and depending on the case Is too big. For example, if P1 = P2 = 10 ⁵ , the power of the mixed sound becomes 10 ¹⁰ , and overflow may occur in many processing systems. Therefore, the power of the mixed sound derived by the “principle of sum of logarithmic intensity” is used as the upper limit value of mixing.
<Additional conditions>
At least one of the three conditions (a) to (c) of the process (3) is added in order to solve the problem that the volume of the mixed sound obtained by the principle of the logarithmic intensity becomes too strong in some cases.

  FIG. 6 is a diagram showing the condition (a). Condition (a) limits the power increase rate of the mixed sound to a fixed magnification (ratio) of the simple addition value of the powers of the two input sounds. In nature, when a person listens to the addition (mixed sound) of two sounds, he / she listens to the mixture by simple addition. For example, the simple addition value of the input sound 10 times the hearing limit and the input sound 100 times is 110 times. On the other hand, the addition value on the logarithmic scale is 1000 times.

Therefore, the amplification limit _TG of the mixed power is set. The amplification limit _TG of the mixed power is represented by the amplitude ratio of the simple addition value, and is set to T _G = 4.0, for example. In this case, the amplification limit of the mixed power is 4.0 times the amplitude (for example, 110) obtained by simple addition. A dotted line B in FIG. 6 is an amplification limit that limits the power of the sum (mixed) of the priority sound and the non-priority sound to a predetermined magnification _TG of the simple addition value.

FIG. 7 shows a level that satisfies both the upper limit of the mixed sound power obtained by the principle of the sum of logarithmic in FIG. 5 and the amplification limit set by a fixed multiple of the simple addition in FIG. 6, that is, the lower power The process which sets a level as an upper limit of signal addition is shown. In FIG. 7, the solid line in which the lower power of the dotted line A (upper limit determined by the principle of the sum of logarithmic intensity) and the dotted line B (amplification limit based on simple addition) is selected is the upper limit at the time of signal addition.
<Principle of hole filling>
FIG. 8 is a diagram showing the gain setting within the upper limit range of the signal addition of FIG. In order to improve listening to the priority sound, it is necessary to suppress the non-priority sound in a necessary part on the time frequency plane. The greater the amount of suppression, the better. If the non-priority sound is suppressed unconditionally, the volume change of the non-priority sound becomes too stimulating, which not only increases the sense of incongruity, but also prevents listening to the priority sound. Therefore, a reasonable standard is required for suppressing non-priority sounds.

  In the embodiment, the power of the non-priority sound is reduced within a range that does not exceed the amount of increase in power by increasing the gain of the priority sound. That is, the hole created by suppressing the non-priority sound is filled by increasing the priority sound. By this process, it is possible to avoid a sense of incongruity with the non-priority sound.

  In FIG. 8, the priority sound is amplified within the upper limit of signal addition in FIG. The dotted line C is the power of the priority sound after gain adjustment. On the other hand, the non-priority sound is reduced within a range that does not exceed the change amount of the priority sound (that is, the power increase). The dotted line D is the power of the non-priority sound after gain adjustment.

  Thus, the signal processing unit 15 determines the respective gains based on the restriction on the increase in the priority sound power and the restriction on the decrease in the non-priority sound power. The gain mask determines the increase of the priority sound and the decrease (unevenness) of the non-priority sound at each point on the time-frequency plane.

In addition to condition (a) in process (3), or in addition to condition (a), condition (b) for setting a fixed upper limit on the gain of priority sound, or fixed to the gain of non-priority sound A condition (c) for setting a lower limit may be added. By adding these conditions to the principle of the sum of logarithmic intensities and the principle of filling in holes, a natural mixed sound can be generated.
<Fast-time FFT>
Next, details of the processing of the frequency analysis unit 12 will be described. In the embodiment, the frequency analysis unit 12 performs a short-time FFT of about 256 as the number of FFT points. This short-time FFT is a process of developing a one-dimensional input signal on a two-dimensional time frequency (tf) plane.

The sampling frequency _F signal taken at _s x1 [n] and x2 [n], the respective priority sound and the non-priority sound. Both signals xj [n] a (j = 1, 2) to short-time Fourier transform of the _{N F} point _{N d} point shift. Assuming that the conversion result at block number i and frequency bin number k is Xj [i, k], Xj [i, k] is expressed by equation (1).

Here, h [n] is a window function. N _h is a parameter for determining the width of the window function, and h [n] = 0 for n where | n | ≧ N _h . As the window function, an arbitrary window function such as Hann window, Hanning window, Gaussian window or the like can be used. In the embodiment, a Gaussian window of Equation (2) is used.

Here, σ is a parameter for adjusting the width of the window function.

The FFT result of the real signal does not need to handle negative frequencies because the positive and negative frequency outputs are in a complex conjugate relationship with each other. As N _H = N _F / 2, only frequency bins in the range of 0 ≦ k ≦ N _H need be handled. Further, in order to perform the inverse FFT when N _d = 1 only by addition and a constant multiple after the addition, the conversion of Expression (3) is performed. The amount of calculation is reduced by operations of only addition and multiplication.

The inverse FFT is performed by the frequency time conversion unit 16 of the mixing apparatus 1. The signal processing unit 15 of the embodiment does not perform phase processing and mixes input signals only by amplitude processing. This is because the number of FFT points N _F is small. As an example, N _F = 256 and sampling frequency F _S = 44.1 kHz. Under these conditions, it is insufficient to decompose the line spectral structure of speech, and a plurality of harmonic components are mixed in one frequency bin, making it difficult to use the phase.

  In the embodiment, since only amplitude processing is performed, the mixing output Y [i, k] has gains α1 [i, k] and α2 [i, k] on X1 [i, k] and X2 [i, k], respectively. Is generated by multiplying and adding.

The output y in the time domain is obtained by performing an inverse FFT on Y [i, k] in Equation (4).

Here, in the case of one sample shift (N _d = 1), y can be generated even if n is fixed to zero, so that the processing is simple as shown in Equation (6).

Further, by converting X [i, k] according to Equation (3), the number of frequency bins to be added can be reduced to almost half as shown in Equation (7).

<Calculation of smoothing power>
Next, the calculation of the smoothing power by the signal processing unit 15 will be described. Prior to power smoothing, the square of the absolute value of the signal Xj [i, k] in the time frequency domain (| Xj [i, k] | ² ) is calculated and smoothed. As the smoothing, for example, exponential smoothing represented by Expression (8) is used. The exponential smoothing method is suitable for FPGA implementation because it requires a small amount of calculation and a small amount of memory.

Here, μ is a coefficient of the exponential smoothing method, and is derived from the smoothing time constant τ _s by Equation (9).

When Expression (8) is regarded as an IIR (Infinite Impulse Response) type digital filter, the time for the impulse response to decay to 1 / e of the peak value is τ _s . In the embodiment, exponential smoothing is used for smoothing, but any smoothing method such as an FIR (Finite Impulse Response) filter or an IIR filter can be used.
<Calculation of minimum audible power>
In mixing the input signal, it is necessary to determine whether the component at each point on the time-frequency plane is a audible component or an inaudible component. Therefore, for each frequency bin k of each sound source j, a minimum power A [k] for defining the audible component is defined.

FIG. 9A shows a sample extracted by extracting a main part from the curves of 20 phon and 70 phon among the equal loudness curves defined in the international standard ISO 226: 2003. These are referred to as C ₂₀ [k] and C ₇₀ [k], respectively.

Originally, the 0 phon curve is the minimum audible power. However, since the volume at which the sound is presented to the listener varies depending on the volume setting of the electroacoustic device, the signal processing unit 15 of the mixing device 1 according to the embodiment has a specified value for the loudness level. Judgment is audible when It may be designed so that the user of the mixing apparatus 1 can select the L _p phon curve from the equal loudness curve as the minimum audible power. The curve of L _p phon can be obtained by Equation (10) as an approximate value obtained by interpolating or extrapolating C ₂₀ [k] and C ₇₀ [k].

Note that when determining whether the smoothed power level Ej [i, k] is audible, Ej [i, k] cannot be compared with C _Lp [k], and the signal xj [n ] is the need to account for the maximum value x _max and window function h of the absolute value [n]. Therefore, C _Lp [k] is converted as shown in Equation (11) to derive the minimum audible power A [k].

Here, the constant Lf, when xj [n] is a full-scale signal, how much dB of the sound pressure level (SPL: Sound Pressure Level) on the vertical axis in FIG. Is a constant for setting

From the viewpoint of freely setting the operation of the mixing apparatus 1, there is no necessity to make C ₂₀ [k] and C ₇₀ [k] conform to ISO 226: 2003, and an equal loudness curve as shown in FIG. May be generated. If the curve in FIG. 9B is used, a sound with a high frequency of about 8 kHz is likely to be audible, so that the components near this band are respected in the priority sound. As a result, since the priority sound is sharp, it is easy to obtain high evaluation as a feeling of actual listening. The experimental results to be described later use C ₂₀ [k] and C ₇₀ [k] in FIG. 9B.
<Calculation of auditory correction power>
The calculation of the auditory correction power for determining the gain will be described. The calculation of the auditory correction power corresponds to the processing of FIG. The smoothed power Ej [i, k] divided by the minimum audible power A [k] is audible if the result is greater than 1, and the audible level is Ej [i, k] / A [k]. Expressed. For example, if Ej [i, k] / A [k] = 100, the power is 100 times that of the minimum audible sound.

  Although division occurs in this evaluation method, FPGA is not good at division. Therefore, since the minimum audible power A [k] is determined in advance, division is avoided by making the reciprocal B [k] in advance.

Using this correction coefficient B [k], auditory correction power Pj [i, k] is generated from the smoothing power Ej [i, k] by multiplication of Expression (14).

The auditory correction power Pj [i, k] is an amount whose value is determined for each point on the time-frequency plane. From the auditory correction power Pj [i, k] at each point, the total auditory correction power Qj [i] defined by the equation (15) is calculated.

The auditory sensation correction total power Qj [i] is an amount obtained by integrating the power of each point in the frequency direction, and is a simplified estimated value of sound energy that the listener can feel. The auditory correction total power Qj [i] is used for attribute determination of a time interval described below.
<Time zone attribute determination calculation>
The signal processing unit 15 performs sound determination, low SNR (Signal to Noise Ratio) determination, and boost determination in each time interval when performing the mixing processing. These determinations are related to the process (4) described above.

  First, the sound determination will be described. When the mixing process is performed in a non-sound part, a slight sound included in the priority signal, for example, a wind sound between narrations, is enhanced, and an undesired mixed sound is generated. In order to prevent this, a part of the priority sound that should not be listened to during this time interval is set in advance as a sound part.

  For the determination of the sound part, a function e [i] that becomes 1 when sound is present is defined by Expression (16).

Here, Te is a parameter for sound determination. For example, if Te = 1.0, it is determined to be sound when all bins are at the audible limit.

  Next, the low SNR determination will be described. As described with reference to FIGS. 5 to 7, the mixing device 1 sets an upper limit on the gain of the priority sound. For this reason, when the priority sound is extremely low compared to the non-priority sound, it may be difficult to listen to the priority sound even if the upper limit value of the gain is used. In order to prevent this, it is determined whether or not the SNR is low, and the upper limit is increased in the time interval determined to be low SNR.

  For the determination of the low SNR, the function l [i] that becomes 1 at the time of the low SNR can be defined by Expression (17).

Here, _TSN is a parameter for low SNR determination. For example, if T _SN = 10.0, the audibility correction total power is determined to be low SNR when there is a difference of 10 times in amplitude ratio (100 times in power ratio) between the priority sound and the non-priority sound. Is done.

  Finally, boost determination will be described. The boost determination is performed when the priority sound is sound and has a low SNR. B [i] which becomes 1 at the time of boosting is defined by Expression (18).

In order to perform a boost operation without division when the boost determination becomes true, the numerator b _n and the denominator b _d when the boost ratio is displayed as a fraction are obtained by the equations (19) and (20), respectively. deep. These are used to boost b _n / b _d for various evaluation criteria.

<Generation of gain>
The generation of the gain is the core of the mixing process of the embodiment. A gain α1 [i, k] for the priority sound and a gain α2 [i, k] for the non-priority sound are generated. Both gains are initialized to 1 when the operation of the mixing apparatus 1 is started. For all k, α1 [0, k] = α2 [0, k] = 1.

  It is assumed that processing related to the time block i has just started. At this time, α1 [i−1, k] and α2 [i−1, k] have already been determined for all k. α1 [i, k] is updated by increasing or decreasing α1 [i−1, k] using Δ1. α2 [i, k] is updated by increasing or decreasing α2 [i−1, k] using Δ2.

The increase / decrease of α1 [i, k] is realized by multiplying α1 [i−1, k] by (1 + Δ1) or (1 + Δ1) ⁻¹ . On the other hand, α2 [i, k] is increased or decreased by adding ± Δ2 to α2 [i−1, k].

  The reason for adopting such a different update method will be described. The gain α1 [i, k] for the priority sound may be 10 or more depending on conditions. In particular, when α1 [i, k] is large, it is necessary to increase the difference in change, and multiplicative updating is suitable. On the other hand, the gain α2 [i, k] for the non-priority sound is limited to a range from 0 to 1, so that a constant increment is sufficient, and the signal when the constant increment becomes lower is obtained. Suppression can be performed sharply.

  The reason why the gains α1 [i, k] and α2 [i, k] are updated only by addition / subtraction and multiplication is to reduce the calculation as described in the process (5). In the method of determining the next gain by solving the equation, in many cases, division, square root, or the like occurs. There is also a concern that the output waveform may be discontinuous due to a large fluctuation in gain.

On the other hand, in the embodiment, by limiting the increase / decrease to a minute amount, it is possible to prevent the gain from changing smoothly and causing a step in the output.
(A) Calculation of auditory correction power of gain adjustment signal If gain αj [i−1, k] of the previous frame is used as it is without increasing or decreasing gain, that is, αj [i, k] = When αj [i−1, k] is set, the audibility correction power of the priority sound and the non-priority sound related to the sound source j is expressed by Expression (21) and Expression (22), respectively.

At this time, the perceptual correction power L [i, k] of the mixing output is expressed by the equation (23) as the sum of contributions of both sound sources.

The audibility correction power when the gain of the priority sound is increased is defined as L _1p [i, k].

_Let L _p [i, k] be the auditory correction power of the mixing output at the time of increase.

The auditory correction power when the gain of the non-priority sound is increased by decreasing by Δ2 is defined as L _2m [i, k].

The auditory correction power related to the priority sound when the adjusted gain α1 [i, k] is used is defined as L _1α [i, k].

(B) Restriction of band to be operated Next, restriction of a band for gain adjustment will be described. Manipulating the signal gain of the frequency bin corresponding to 0 Hz may impair the natural feeling of the sound. Further, when a high-frequency signal gain is manipulated, the demerit of adding a harsh sound may be greater than the merit of improving the ease of listening.

Therefore, for the priority sound, α1 [i, k] is updated only at the frequency f in the range of f _1L ≦ f ≦ f _1H . This range is a range of frequency bin k and corresponds to a range of k _1L ≦ k ≦ k _1H . However,
k _1L = rd (N _F f _1L / F _s )
k _1H = rd (N _F f _1H / F _s )
It is. Here, “rd ()” means a rounding function (rounding function) to the nearest integer.

Similarly, for non-priority sounds, gain adjustment is performed only in the range of f _2L ≦ f ≦ f _2H , and only α2 [i, k] that satisfies k _2L ≦ k ≦ k _2H is increased or decreased.
(C) Conditions for increasing α1 Increase of α1, that is, the calculation of α1 [i, k] = (1 + Δ1) × α1 [i−1, k] is performed according to equations (28) to (32). When all the conditions are met.

Expressions (28) and (29) specify that the increase is performed only when both the priority sound and the non-priority sound are audible. Equation (30) works so that the log intensity (power) of the mixed sound does not exceed the sum of the log intensity of the priority sound and the non-priority sound (the principle of the sum of log intensity). Equation (31) works to keep the gain for the priority sound below a certain value (T _1H ). Equation (32) works to suppress the increase in power below a certain limit ( _TG ratio in amplitude ratio) even in the local area of the time-frequency plane as compared with the mixing in the case of simple addition (processing (3) ) Condition (a)).

It is desirable to correct the expressions (30) to (32) when determining the low SNR. This correction is performed by considering that the priority sound level has been raised by replacing P1 with (b _n / b _d ) P1.
(D) Condition for reducing α1 The reduction of α1, that is, the calculation of α1 [i, k] = (1 + Δ1) ⁻¹ × α1 [i−1, k] is performed by equations (33) to (37 ) Holds and equation (38) holds.

Expressions (33) and (34) indicate that the gain of the priority sound is returned when at least one of the priority sound and the non-priority sound does not satisfy the audible level at the point (i, k) on the time-frequency plane. Intended. Expression (35) works so as to return the gain of the priority sound when the log intensity of the mixed sound exceeds the sum of the log intensity of the priority sound and the non-priority sound. Equation (36) works in a direction to eliminate the excess when the gain α1 for the priority sound exceeds the preset upper limit T _1H (condition (b) of process (3)). Expression (37) works to return the gain of the priority sound when it exceeds a level (see FIG. 6) obtained by multiplying the mixed sound by simple addition by a predetermined magnification (ratio) _TG . Equation (38) indicates that the priority sound is decreased only when the gain value is larger than one.

  Expressions (33) to (36) are negations of Expressions (28) to (31). On the other hand, Expression (37) is not the negative of Expression (32). Equation (37) is a conditional expression before correction, and equation (32) is a conditional expression after correction. This difference prevents the gain from vibrating.

  By such a decrease operation, α1 returns to 1 when there is no need to increase it. When α1 [i, k] <1 is obtained by the reduction operation, α1 [i, k] = 1 is recovered by forcibly substituting 1. If there is this recovery operation, the condition of equation (38) is not necessarily required. However, in the case of software implementation, in order to prevent an unnecessary increase in multiplication time, in the case of FPGA implementation, to reduce power consumption. It is better to have the determination of the equation (38).

When neither the increase nor decrease condition of α1 is satisfied, the value is retained, that is, α1 [i, k] = α1 [i−1, k] is performed.
(E) Condition for decreasing α2 Reduction of α2, that is, the calculation of α2 [i, k] = α2 [i−1, k] −Δ2 is performed in both formulas (39) and (40). Is satisfied.

Equation (39) indicates that the power of the non-priority sound may be decreased as long as the amount does not exceed the power increase of the priority sound. Equation (40) works to keep the gain for non-priority sounds above a certain value (T _2L ).
(F) Condition for Increasing α2 Increase of α2, that is, the calculation of α2 [i, k] = α2 [i−1, k] + Δ2 is performed by both equation (41) and equation (42). This is the case.

In equation (41), when the gains α1 [i, k] and α2 [i−1, k] determined up to this point are used, the power decrease of the non-priority sound is larger than the power increase of the priority sound. It shows that it will become. The expression (41) is close to the negative of the expression (39), but there is a difference that the expression (41) is a conditional expression before the correction, whereas the expression (39) is a conditional expression after the correction. This difference prevents the gain from vibrating.

  With this operation, α2 returns to 1 when it is not necessary to decrease it. When α2 [i, k]> 1 due to an increase in α2, α2 [i, k] = 1 is recovered by forcibly substituting 1.

If neither the increase nor decrease condition of α2 is satisfied, the value is retained, that is, α2 [i, k] = α2 [i−1, k] is performed.
<Operation example>
FIG. 10 is a diagram illustrating an operation example of the mixing apparatus 1 according to the embodiment. Two sound source sets (set 1 and set 2) were prepared. In each sound source set, the sound was a priority sound and the instrument sound was a non-priority sound. FIG. 10A shows an example when the boost is not effective, and FIG. 10B shows an example when the boost is effective, both of which are directed to the sound source set 1. As described above, the boost process is performed when the priority sound is sound and has a low SNR. Both FIG. 10 (A) and FIG. 10 (B) plot various variables before correction.

In the figure, MUL is b _n P1 · P2 / b _d (bn / bd is a boost ratio), and PLUS is T _G ² (b _n P1 + b _d P2) / b _d . L in the figure is the audibility correction power L of the mixing output defined by the equation (23). The condition of equation (30) is to make L as large as possible within a range not exceeding MUL, and the condition of equation (32) is to make L as large as possible within a range not exceeding PLUS.

In both FIG. 10A and FIG. 10B, the magnitude relationship between MUL and PLUS depends on the frequency, and one of them is not always high. From this, it can be seen that both the conditions of the equations (30) and (32) are effective and should be used together.
<Development example 1>
As a first development example, an improvement example by smoothing the gain is shown. With the method described above, good results could be obtained for both of the two sound source sets, but it was found that the mixed sound was somewhat difficult to hear in the portion where the input SNR was low.

  As a result of searching for the cause, it was found that the increase in the gain α1 of the priority sound was too gentle to secure a necessary value. In order to cope with this, the gain increase step size Δ1 may be increased. However, if Δ1 is increased, there is a possibility that a large discontinuity occurs in the gain transition and the gain difference transition. In this case, spectrum dissipation (noise generation) occurs.

  Accordingly, in the first development example, α1 and α2 are smoothed as follows, and smoothed gains β1 and β2 are used. As a result, the problem of spectral dissipation can be avoided even if the gain adjustment step sizes Δ1 and Δ2 are increased by 10 times or more.

Here, η is a coefficient of the exponential smoothing method, and is derived from the smoothing time constant τ _α by the equation (44).

Β1 and β2 generated in this way provide a good mixing result for both the sound source set 1 and the set 2 described above. If there is no problem with the calculation load and the circuit scale, it is desirable to smooth the gain in the first development example.

  FIG. 11 shows an operation state of the mixing apparatus 1 when the sound source set 2 is used and mixing is performed with smoothed gains β1 and β2. In FIG. 11, the horizontal axis represents time, and the vertical axis represents frequency. 11A shows the sound as the priority signal X1, FIG. 11B shows the music as the non-priority signal X2, FIG. 11C shows the mixing result (X1 + X2) by the conventional simple addition, and FIG. It is a mixing result of an embodiment. FIG. 11E shows power E1 after smoothing the priority signal, and FIG. 11F shows power E2 after smoothing the non-priority signal (see FIG. 3). FIG. 11 (G) is a gradation display of power P1 after auditory correction, and FIG. 11 (H) is a gradation display of power P2 (see FIG. 4) after auditory correction.

  In FIGS. 11G and 11H, a light gray area is 0 dB or more and less than 20 dB, a black area is 20 dB or more and less than 40 dB, and a dark gray area is an area of 40 dB or more. That is, the region other than white is treated as audible by auditory correction. In FIG. 11G, the area indicated by the line segment e is the area where the sound is determined (e [i]), the area indicated by the line segment l is the area where the low SNR is determined (l [i]), and the line The area indicated by the minute b is an area for which a boost determination (b [i]) has been made. The used sound source set 2 is a sound source set having a low priority sound SNR, and in the time interval after 5 seconds, all the sound intervals are the targets of boost processing.

  FIG. 11 (I) is a gain mask for the priority sound created based on the gain β1 smoothed in the first development example, and is a diagram in which the logarithm of β1 is displayed in shades. White corresponds to 0 dB and black corresponds to 35 dB. FIG. 11J is a non-priority sound gain mask created based on the gain β2 smoothed in the first development example, and shows the value of β2 in grayscale. White corresponds to 1.0 and black corresponds to 0.0.

By performing gain adjustment using the gain masks of FIGS. 11 (I) and 11 (J) and then adding, fine mixing in the time-frequency plane becomes possible. In FIG. 11C according to the conventional method, only the non-priority sound (guitar) component can be seen in the low frequency region, whereas in FIG. 11D, the priority sound (sound) component is mixed.
<Development example 2>
In the above-described embodiment, in order to reduce the amount of calculation, the equation is not solved but sequential updating is performed by determining whether the inequality is true or false (processing (5)). In particular, when implementing an FPGA, we want to simplify the processing as much as possible. Therefore, a loudness curve of 70 phon and 20 phon is set in the signal processing unit 15, and C _Lp [k], A [k], and B [k] are sequentially derived from the equations (10) to (12). Instead, B [k] is given from the beginning. For example, the correction coefficient B [k] (reciprocal of the minimum audible power A [i]) is given as a constant table at the time of shipment. If you want to give a particularly strong priority even if you ignore the rationality temporarily during operation, for example, you can forcibly assign an arbitrary value to B [k] to give you the desired characteristics. It is also possible.

FIG. 12 shows a specific example of the correction coefficient B [i] when the equal loudness curve of FIG. 9B is set and the development example 2 is applied. In the case of FIG. 12, B [i] may be stored in advance as a function instead of being stored as a table.
<Processing flow>
13A and 13B are flowcharts showing an example of gain determination processing executed by the gain determination unit 151 of the signal processing unit 15 of the mixing apparatus 1. This processing flow corresponds to the first development example in which the gain α is smoothed to generate the gain β.

First, α1 [k], α2 [k], β1 [k], β2 [k] are initialized to “1” for all frequency bins k, and auditory correction total power Q1 = 0, Q2 = 0, i = 0. And the coefficient B [k] is read (S11). The process is started from k = 0 (S12), the smoothing powers E1 [i, k] and E2 [i, k] are read (S13), and the auditory correction powers P1 [i, k] and P2 [i, k ] Is calculated (S14), and audible correction total powers Q1 [i] and Q2 [i] are calculated (S15). The value of k is incremented (S16), and S13 to S16 are repeated until k reaches the frequency bin number _NH (NO in S17). This is the first pass of the loop for frequency bin k.

When k exceeds _NH (YES in S17), the sound determination result e [i], low SNR determination result l [i], boost determination result b [i], boost ratio numerator b _n [i], boost The ratio denominator b _d [i] is obtained (S18), and the process of the second pass of the loop for k is started (S19). For the frequency bin k, it is determined whether or not the priority sound k is within the range of the lowest bin k _1L and the highest bin k _1H for gain adjustment (S20). If within the range, the smoothing powers E1 [i, k] and E2 [i, k] are read (S21), P1 (priority audibility correction power), P2 (non-priority audibility correction power), L1 (Hearing correction power of priority sound with gain α1 before update), L _1p (Hearing correction power when gain of priority sound is increased), L2 (Hearing correction power of non-priority sound with gain α2 before update) (Power), L _2m (audience correction power when the gain of the non-priority sound is decreased by Δ2), L (formula (23)), and L _p (formula (25)) are obtained (S22).

  Using the obtained value, it is determined whether or not all of the equations (28) to (32) are satisfied, that is, whether or not α1 is increased (S23). If true (YES in S23), α1 is increased (S24). If not true (NO in S23), α1 is maintained.

  Next, it is determined whether or not any of Expression (33) to Expression (37) is satisfied and Expression (38) is satisfied, that is, whether or not α1 is decreased (S25). If the condition of S25 is not satisfied, α1 is maintained, and if satisfied, α1 is decreased (S26). Further, it is determined whether α1 after the decrease is less than 1 (S27). When α1 becomes less than 1, α1 is returned to 1 (S28), and when α1 is 1 or more, the updated α1 is maintained.

Subsequently, it is determined whether or not k of the non-priority sound is within the range of the lowest bin k _2L and the highest bin k _2H for gain adjustment (S29). If it is within the range, L _1a (audibility correction power related to the priority sound when the adjusted gain α1 is used) is obtained (S30), and whether or not the expressions (39) and (40) are satisfied, that is, It is determined whether or not α2 is decreased (S31). If true (YES in S31), α2 is decreased (S32). If not true (NO in S31), α2 is maintained.

  Next, it is determined whether or not Expression (41) and Expression (42) are satisfied, that is, whether or not α2 is increased (S33). When the condition of S33 is not satisfied, α2 is maintained, and when satisfied, α2 is increased (S34). Further, it is determined whether or not the increased α2 exceeds 1 (S35). If it is 1 or less, the increased α2 is maintained, and if it exceeds 1, α2 is returned to 1 (S36).

Next, based on Expression (43), α2 and α1 are respectively smoothed to generate β2 and β1 (S37 and S38), and β1 and α1 are output (S39). Thereafter, k is incremented (S40), and S20 to S40 are repeated until k reaches _NH (NO in S41).

When k exceeds _NH (YES in S41), the time block i is incremented (S42), and S12 to S42 are repeated until the last time block i is reached (NO in S43). When the process is finished, end the process.

In the process of FIGS. 13A and 13B, in order to determine the boost ratio numerator b _n [i] and denominator b _d [i] (S18), X1 [i, k] and X2 [i] for all frequency bins k. , K]. On the other hand, updating the gains α1 [k] and α2 [k] requires not only X1 [i, k] and X2 [i, k] but also b _n [i] and b _d [i]. Therefore, the gain cannot be updated once unless the loop processing for the frequency bin k is performed twice.

  Therefore, in FIG. 14, only the boost ratio is calculated with the previous sample, and this is diverted to reduce the processing from two passes to one pass. Thereby, simplification and high speed of the circuit are realized.

In FIG. 14, only the differences from FIGS. 13A and 13B will be described. After initialization, reading and calculation of necessary parameters in S11 to S15, the process jumps to S20 to adjust α1 by S21 to S28 and α2 by S29 to S36. Thereafter, α1 and α2 after adjustment are smoothed to obtain β1 and β2, and gains are obtained for all frequencies k in the processing range for the time block i of interest (repetition of S13 to S41). Thereafter, increments the time block i (S42), e [i ], l [i], b [i], b n [i], b d seek [i] (S51), these parameters: Used for processing of time block i = i + 1.

  As for this simplification, when the sound source set 2 with a low priority sound SNR was examined, the output values were not completely the same, but the difference was a minute level that could not be heard.

  By the method described above, an optimum gain value can be determined for each of the input signal 1 and the input signal 2 based on a reasonable determination criterion. In addition, the amount of calculation can be greatly reduced by sequential updating by only the addition / subtraction and multiplication.

  The processing of the signal processing unit 15 described above can be realized by hardware or software. By the processing of the signal processing unit 15, mixing processing is performed using the input signal as it is, the priority sound is made to stand out with a natural audibility (the principle of the sum of logarithmic intensity of the process (1)), and the non-priority sound is suppressed without a sense of incongruity. (Principle of filling hole in process (2)). Since the norm for determining the gain of the priority sound and the non-priority sound is reasonably determined (processing (1) to processing (3)), it is not necessary for the user to adjust the parameters while listening to the sound.

  Even when the power of the priority sound is extremely small with respect to the non-priority sound, the priority sound can be conspicuous (process (4)). For example, when narrating music, the whispers can be heard without being buried in the music.

  Further, the method of updating the gain only by multiplication and addition / subtraction facilitates hardware mounting on the FPGA (processing (5)). Also, a mixing device that is mounted as a plug-in on the DAW and operates in real time is realized. This is due to the fact that the calculation load is reduced by sequential updating and that the FFT score can be reduced to about 256 points because high-performance mixing is possible by adjusting only the gain.

  Note that smoothing of the power of the priority sound and the non-priority sound is not essential, and the gains α1 and α2 may be obtained directly from the power of the input signal developed on the time-frequency plane.

  The above-mentioned mixing device can automatically synthesize the input audio signal at high speed using a reasonably determined gain, so that it can be used not only for recording but also for breaking news, car navigation, disc jockey, conference, karaoke device, etc. Can be widely applied. For example, you can report emergency bulletins without disturbing the program, you can easily hear the car navigation sound even if you are playing music on a car stereo, and the disc jockey can talk without lowering the volume of the music. To make the voice stand out, automatically adjust the vocal sound to the accompaniment.

  In addition, by installing a mixing program in a user terminal device such as a personal computer or a smartphone, the user can mix desired music or superimpose desired music on a desired image and transmit it to a communication partner. On the receiving side, it is possible to further superimpose the voice on the received data and save or return it.

DESCRIPTION OF SYMBOLS 1 Mixing device 11 Signal input part 12 Frequency analysis part 15 Signal processing part 16 Frequency time conversion part 17 Signal output part 151 Gain determination part

Claims (9)

A frequency analysis unit that develops the first input signal and the second input signal in the time domain into a first signal and a second signal on a time frequency plane, respectively;
A signal processing unit for generating a mixed signal obtained by mixing the first signal and the second signal;
A frequency time conversion unit for converting the mixed signal into a time domain signal;
A signal output unit for outputting the converted signal;
Have
The signal processing unit
For each point on the time-frequency plane, the gain is determined on condition that the log intensity of the output signal does not exceed the sum of the log intensity of the first signal and the log intensity of the second signal, a first gain for adjusting the power of the signal in a first direction, said a gain that is determined by the condition that the first does not exceed the adjustment amount of the signal power, the power of the second signal said first A gain determining unit that determines a second gain to be changed in a second direction opposite to the direction ;
An adder for adding the first signal adjusted by the first gain and the second signal adjusted by the second gain;
A mixing apparatus comprising:   The gain determination unit sets (a) a first upper limit for the adjustment of the first gain so as not to exceed a certain multiple of power obtained by simply adding the first signal and the second signal, and (b) the first At least one condition of providing a fixed second upper limit for one gain or (c) providing a fixed lower limit for the second gain is added, and the first gain is within a range satisfying the added condition. The mixing apparatus according to claim 1, wherein the second gain is determined.   3. The gain determination unit according to claim 2, wherein the gain determination unit relaxes the at least one condition when a ratio of the power of the first signal to a power of the second signal is equal to or less than a predetermined ratio. Mixing device.   The gain determination unit corrects the power of the first signal and the power of the second signal to a first audible correction power and a second audible correction power based on the hearing limit level, respectively, and the logarithmic intensity of the output signal 2. The first gain and the second gain are determined within a range that does not exceed the sum of the logarithmic intensity of the first auditory correction power and the logarithmic intensity of the second auditory correction power. Mixing equipment. The gain determination unit generates a third gain obtained by smoothing the first gain and a fourth gain obtained by smoothing the second gain;
2. The mixing apparatus according to claim 1, wherein the adding unit adds the first signal adjusted with the third gain and the second signal adjusted with the fourth gain. 3.   The mixing apparatus according to claim 1, wherein the gain determination unit sequentially updates the first gain and the second gain for each point on the time-frequency plane.   7. The first input signal is a priority sound that is preferentially clarified by mixing processing, and the second input signal is a non-priority sound other than the priority sound. The mixing apparatus according to item 1. Receiving a first input signal and a second input signal in a time domain;
Expanding the first input signal and the second input signal into a first signal and a second signal on a time-frequency plane, respectively;
For each point on the time-frequency plane, the gain is determined on condition that the log intensity of the output signal does not exceed the sum of the log intensity of the first signal and the log intensity of the second signal, a first gain for adjusting the power of the signal in a first direction, said a gain that is determined by the condition that the first does not exceed the adjustment amount of the signal power, the power of the second signal said first Determining a second gain to be changed in a second direction opposite to the direction ;
Adding a first multiplication result obtained by multiplying the first signal by the first gain and a second multiplication result obtained by multiplying the second signal by the second gain to generate a mixed signal;
The mixed signal is converted into a time domain signal and output.
A signal mixing method characterized by the above. A mixing program for causing a computer to execute signal mixing processing, wherein the computer
Receiving a first input signal and a second input signal in a time domain;
Developing the first input signal and the second input signal into a first signal and a second signal on a time-frequency plane, respectively;
For each point on the time-frequency plane, the gain is determined on condition that the log intensity of the output signal does not exceed the sum of the log intensity of the first signal and the log intensity of the second signal, a first gain for adjusting the power of the signal in a first direction, said a gain that is determined by the condition that the first does not exceed the adjustment amount of the signal power, the power of the second signal said first A procedure for determining a second gain to be changed in a second direction opposite to the direction ;
Adding a first multiplication result obtained by multiplying the first signal by the first gain and a second multiplication result obtained by multiplying the second signal by the second gain to generate a mixed signal;
Converting the mixed signal into a time domain signal and outputting the signal;
A mixing program characterized in that it is executed.