JP6314393B2 - Acoustic signal analyzing apparatus, acoustic signal analyzing method, and computer program - Google Patents

Description

  The present invention relates to an acoustic signal analyzing apparatus that estimates sounds generated from each sound source based on a mixed sound obtained by mixing a plurality of sounds respectively generated from a plurality of sound sources.

  Conventionally, for example, as shown in Non-Patent Documents 1 and 2 below, acoustic signal analyzers that extract sounds generated from respective sound sources from mixed sounds are known. In the acoustic signal analyzer described in Non-Patent Document 1, a model of a first sound (speaker's voice) and a model of a second sound (background music) are learned, and the learning result is used. The first sound and the second sound are extracted from the mixed sound.

  Moreover, in the acoustic signal analyzer described in Non-Patent Document 2, the mixed sound is separated into disturbance and other components by using an independent component analysis method.

Haka Erdogan, Emad M. et al. Grais, “Semi-Blind Speech-Music Separation Using Sparsity and Continuity Priors”, Pattern Recognition (ICPR), 2010, 20th International Conference on, Aug. 2010, p. 4573-p. 4576 Satoshi Sawada, Jani Even, Hiroshi Saruwatari, Kiyohiro Shikano, Tomoya Takatani, “Examination of microphone array placement in an internal noise suppression type robotic speech dialogue system”, AI Challenge Study Group, November 2009, p. 26-p. 31

  According to the acoustic signal analysis apparatus of Non-Patent Document 1, only a mixed sound composed of sounds related to a learned model can be handled. That is, the acoustic signal analyzer of Non-Patent Document 1 lacks versatility.

  Further, according to the acoustic signal analysis device of Non-Patent Document 2, a plurality of sound collection devices (microphones) corresponding to the number of sound sources are required. Therefore, the configuration of the apparatus becomes complicated.

  Moreover, if the acoustic signal analyzers of Non-Patent Documents 1 and 2 are used, it is possible to estimate the maximum likelihood of the sound field characteristics of the room in which the acoustic signal analyzer is installed. The reliability of the estimated sound field characteristics depends on the sound generated from each sound source. That is, the reliability of the characteristics of the frequency band not included in the sound generated from each sound source is low. Since the sound generated from all sound sources is not known (the sound generated from at least one sound source is unknown), the reliability of which frequency band is high among the estimated sound field characteristics, It is impossible to identify which frequency band has low reliability. Therefore, it is difficult to accurately estimate the sound generated from each sound source.

The present invention has been made to address the above-described problems, and has as its purpose a highly versatile acoustic signal analyzer and acoustic signal analyzer that can accurately estimate the sound generated from each sound source based on mixed sound. It is to provide a method and a computer program . In addition, in the description of each constituent element of the present invention below, in order to facilitate understanding of the present invention, reference numerals of corresponding portions of the embodiment are described in parentheses, but each constituent element of the present invention is The present invention should not be construed as being limited to the configurations of the corresponding portions indicated by the reference numerals of the embodiments.

In order to achieve the above object, the present invention is characterized in that the sound emission means (161) for emitting a predetermined first sound, the emitted first sound, and the sound emission means are different. Sound collecting means (162) for collecting a mixed sound including the second sound emitted from the sound source, the second sound based on the collected mixed sound and the first sound, , And an estimation means (S12 to S15, S23 to S31) for simultaneously Bayesian estimation of the sound field characteristic in which the sound emission means and the sound collection means are installed, and the mixed sound includes the first sound and the sound The sound field is composed of a convolution sound and the second sound, and the sound field characteristic is expressed as a set of coefficients multiplied by the intensity of each frequency component of the first sound, The estimation means is configured such that a spectrum of the mixed sound, a time series of the spectrum of the second sound, and the set of coefficients are complex. A lower limit function that is a lower limit of the posterior distribution of a generation model representing generation according to a normal distribution and a generalized inverse Gaussian distribution, and is expressed using a plurality of auxiliary variables, and the second sound and the sound field An acoustic wave characterized in that the posterior distribution is approximately estimated by setting a lower limit function including a parameter relating to a characteristic and determining the lower limit function by repeatedly updating the auxiliary variable and the parameter. The signal analyzer (10, 20) is used.

  Further, the present invention is characterized in that a sound emitting means for emitting a predetermined first sound, the emitted first sound, and a second sound emitted from a sound source different from the sound emitting means. Sound collecting means for collecting mixed sound including sound, and the second sound, sound emitting means, and sound collecting means are installed based on the collected mixed sound and the first sound. An estimation means for performing Bayesian estimation simultaneously with the characteristic of the generated sound field, and the mixed sound includes a sound obtained by convolving the first sound and the characteristic of the sound field, and the second sound, The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound, and the estimation means includes a time series of the spectrum of the mixed sound and the spectrum of the second sound. And the posterior of the generation model that represents that the set of coefficients is generated according to the Poisson distribution and the gamma distribution, respectively A lower limit function which is a lower limit of the cloth, which is expressed using a plurality of auxiliary variables, sets a lower limit function including parameters relating to the characteristics of the second sound and the sound field, and repeats the auxiliary variables and the parameters The acoustic signal analysis apparatus is characterized in that the posterior distribution is approximately estimated by updating the lower limit function and determining the lower limit function.

  Further, the present invention is characterized in that a sound emitting means for emitting a predetermined first sound, the emitted first sound, and a second sound emitted from a sound source different from the sound emitting means. Sound collecting means for collecting mixed sound including sound, and the second sound, sound emitting means, and sound collecting means are installed based on the collected mixed sound and the first sound. Estimation means for simultaneously performing Bayesian estimation of the characteristics of the generated sound field, and the mixed sound includes a sound obtained by convolving the first sound and the characteristic of the sound field, and the second sound and the sound. The sound field characteristic is expressed as a set of coefficients multiplied by the intensity of each frequency component of the first sound, and the estimation means includes the mixed sound. Spectrum, the time series of the spectrum of the second sound, and the set of coefficients according to a complex normal distribution and a generalized inverse Gaussian distribution. A lower limit function that is a lower limit of the posterior distribution of a generation model representing generation of each, and is expressed using a plurality of auxiliary variables and includes parameters relating to characteristics of the second sound and the sound field And the auxiliary variable and the parameter are iteratively updated to determine the lower limit function to approximately estimate the posterior distribution. .

  Further, the present invention is characterized in that a sound emitting means for emitting a predetermined first sound, the emitted first sound, and a second sound emitted from a sound source different from the sound emitting means. Sound collecting means for collecting mixed sound including sound, and the second sound, sound emitting means, and sound collecting means are installed based on the collected mixed sound and the first sound. Estimation means for simultaneously performing Bayesian estimation of the characteristics of the generated sound field, and the mixed sound includes a sound obtained by convolving the first sound and the characteristic of the sound field, and the second sound and the sound. The sound field characteristic is expressed as a set of coefficients multiplied by the intensity of each frequency component of the first sound, and the estimation means includes the mixed sound. Spectrum, the second sound spectrum time series and the coefficient set are generated according to Poisson distribution and gamma distribution, respectively. A lower limit function that is a lower limit of the posterior distribution of the generation model representing that the generation model is expressed, and is expressed using a plurality of auxiliary variables, and includes a lower limit function including parameters relating to the characteristics of the second sound and the sound field In addition, the acoustic signal analyzing apparatus is characterized in that the posterior distribution is approximately estimated by repetitively updating the auxiliary variable and the parameter to determine the lower limit function.

  According to the acoustic signal analyzing apparatus configured as described above, it is not necessary to learn in advance a model of a sound that constitutes a mixed sound (or a model of each sound source that generates a sound that constitutes a mixed sound). The singing voice and sound field characteristics can be estimated for any mixed sound. That is, the acoustic signal analyzer 10 is more versatile than the acoustic signal analyzer of Non-Patent Document 1.

  In addition, the estimated sound field characteristic largely depends on the frequency characteristic of the musical sound (direct sound). However, according to the present invention, since the posterior distribution of the sound field characteristic is estimated, the uncertainty of the estimated sound field characteristic is uncertain. Can be certified. That is, it can be recognized that the reliability of the frequency band in which the variance of the estimated posterior distribution of the sound field characteristic exceeds the predetermined threshold is low and the reliability of the frequency band in which the variance is equal to or less than the predetermined threshold is high. If the sound field characteristic in the frequency band with low reliability is corrected using an equalizer or the like, the second sound can be extracted more accurately from the mixed sound.

According to another feature of the present invention, the sound collection means samples the second sound and the mixed sound in real time, and the estimation means updates the expected value of the lower limit function in real time. By optimizing, the posterior distribution is approximately estimated. Another feature of the present invention is a parameter related to characteristics of the sound field, parameter relating to the coefficient multiplied to the intensity of the predetermined frequency component, the up to now from the start of sounding of the first sound Only the total sum of the intensities of the predetermined frequency components of the spectrum of the first sound and the total sum of the intensities of the predetermined frequency components of the spectrum of the mixed sound from the start of sound generation to the present are updated (n). The update is based on the update formula set so as to be multiplied by the corresponding weighting coefficient (η _n ).

  According to this, since weighting is applied only to a specific variable, it is possible to suppress an increase in the number of iterative calculations when determining the lower limit function, compared to a case where a so-called “deterministic annealing” is employed. it can.

It is a block diagram which shows the production | generation process of the mixed sound which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the sound field characteristic of FIG. It is a block diagram which shows the structure of the acoustic signal analyzer which concerns on 1st and 2nd embodiment of this invention. It is a flowchart which shows the procedure in the case of batch-processing an acoustic signal. It is a flowchart which shows the procedure in the case of processing an acoustic signal in real time. It is a block diagram which shows the production | generation process of the mixed sound which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the sound field characteristic of FIG.

(First embodiment)
The acoustic signal analyzer 10 according to the first embodiment of the present invention will be described. First, the outline of the acoustic signal analyzer 10 will be described. The acoustic signal analyzer 10 emits a predetermined sound (predetermined music in this embodiment) from a sound emitting device (speaker 161: see FIG. 3) and a sound collecting device (microphone 162: see FIG. 3). Is used to pick up sounds around the acoustic signal analyzer 10 (in this embodiment, the singer's voice). In the present embodiment, the sound emitting device and the sound collecting device are installed at positions far away from each other. Therefore, the sound collecting device also collects the sound in which the acoustic characteristics (hereinafter referred to as sound field characteristics) of the room in which the sound emitting device and the sound collecting device are installed and the musical sound are convoluted. That is, not only the sound emitted from the sound emitting means (direct sound) but also the reflected sound (reverberation) reflected by the wall or floor of the room is collected. The sound collection device is installed at a position close to the singer. Therefore, the singing voice collected by the sound collecting device is not affected by the sound field. In the present embodiment, a sound in which a musical sound including reverberation and a singing voice (direct sound only) are mixed is called a mixed sound.

The relationship among the power spectrum Y of the mixed sound, the power spectrum X of the musical sound (direct sound), the sound field characteristic H, and the power spectrum S of the singing voice can be expressed as a block diagram as shown in FIGS. This model can be formulated as the following formula (1). Since the acoustic characteristic H and the power spectrum S of the singing voice are not directly observed, they are latent variables in this model. Note that the musical sound corresponds to the first sound of the present invention, and the singing voice corresponds to the second sound of the present invention. The acoustic signal analyzer 10 records (samples) the mixed sound, and uses the recorded mixed sound as observation data to simultaneously perform Bayesian estimation of the singing voice and the characteristics of the sound field.

“F” represents the frequency bin index (f = 1, 2,..., F), and “t” represents the index (t = 1, 2,...) Of the time frame (hereinafter simply referred to as a frame). .., T). Therefore, “X _{f, t} ” represents the strength (amplitude) of the f th frequency bin in the t th frame of the music. “Y _{f, t} ” represents the intensity (amplitude) of the f-th frequency bin in the t-th frame of the mixed sound. “S _{f, t} ” represents the intensity (amplitude) of the f-th frequency bin in the t-th frame of the singing voice. The sound field characteristic H is expressed as a set of coefficients H _{f, i} . “I” represents a coefficient index (i = 1, 2,..., I). That is, “H _{f, i} ” represents a coefficient to be multiplied by the intensity (amplitude) of the f-th frequency bin and delayed i times (time of i frames).

  Next, the configuration of the acoustic signal analyzer 10 will be described. As shown in FIG. 3, the acoustic signal analyzer 10 includes an input operator 11, a computer unit 12, a display 13, a storage device 14, an external interface circuit 15, and a sound system 16, which are connected to the bus BS. Connected through.

  The input operator 11 includes a switch corresponding to an on / off operation (for example, a numeric keypad for inputting a numerical value), a volume or rotary encoder corresponding to a rotation operation, a volume or linear encoder corresponding to a slide operation, a mouse, a touch panel, etc. Composed. These operators are operated by a player's hand, and are used to start or stop the process, to set various parameters relating to the analysis of the acoustic signal, and the like. When the input operator 11 is operated, operation information indicating the operation content is supplied to the computer unit 12 described later via the bus BS.

  The computer unit 12 includes a CPU 12a, a ROM 12b, and a RAM 12c connected to the bus BS. The CPU 12a reads out from the ROM 12b and executes a program representing a procedure for estimating the characteristics of the singing voice and the sound field based on the mixed sound. In addition to the program, the ROM 12b stores various data such as initial setting parameters, graphic data for generating display data representing an image displayed on the display 13, and character data. The RAM 12c temporarily stores data necessary for executing the program. For example, mixed sound data composed of a plurality of sample values obtained by sampling a mixed sound collected by a microphone 162 described later at a predetermined sampling period (for example, 1/444100 sec) is stored in the RAM 12c.

  The display 13 is configured by a liquid crystal display (LCD). The computer unit 12 generates display data representing contents to be displayed using graphic data, character data, and the like, and supplies the display data to the display unit 13. The display device 13 displays an image based on the display data supplied from the computer unit 12.

  The storage device 14 includes a large-capacity nonvolatile recording medium such as an HDD, FDD, CD, or DVD, and a drive unit corresponding to each recording medium. The storage device 14 stores music data representing the predetermined music. The music data is composed of a plurality of sample values obtained by sampling the performance of the predetermined music at a predetermined sampling period (for example, 1/444100 sec), and each sample value is sequentially recorded at consecutive addresses in the storage device 14. Has been. The music data includes title information representing the title of the music, data size information representing the capacity, and the like. The music data may be stored in the storage device 14 in advance, or may be taken in from the outside via the external interface circuit 15 described later.

  The external interface circuit 15 includes a connection terminal that enables the acoustic signal analyzer 10 to be connected to an external device such as an electronic music device or a personal computer. The acoustic signal analyzer 10 can be connected to a communication network such as a LAN (Local Area Network) or the Internet via the external interface circuit 15.

  The sound system 16 includes a D / A converter that converts music data into an analog sound signal, an amplifier that amplifies the converted analog sound signal, and a speaker 161 that converts the amplified analog sound signal into an acoustic signal and emits the sound. I have. The sound system 16 also includes a microphone 162 for collecting the mixed sound and an A / D converter for converting the mixed sound as the collected analog sound signal into a digital sound signal. Although the acoustic signal analyzer of Non-Patent Document 2 includes a plurality of microphones, the present embodiment includes only one microphone 162.

  Next, the operation (singing voice and sound field characteristic estimation procedure) of the acoustic signal analyzer 10 configured as described above will be described. As shown in FIG. 4, the singing voice and sound field characteristic estimation process is started in step S10. Next, in step S11, various variables (auxiliary variables described later, parameters of posterior distribution, etc.) are initialized. Next, in step S 12, the music data is supplied to the sound system 16 and sound emission of the music is started from the speaker 161, and sampling of the mixed sound collected by the microphone 162 is started. The sampled mixed sound data is stored in the RAM 12c. When the music is released, as described below, the singing voice and sound field characteristics are simultaneously (integrated) Bayesian estimated using the mixed sound data stored in the RAM 12c as observation data.

If the singing voice spectrum, musical tone spectrum, and mixed tone spectrum (short-time Fourier transform) are generated from a complex normal distribution, a generation model represented by the following equations (2) to (4) can be constructed. . Note that “GIG (a, b, c)” in the equations (2) to (4) represents a generalized inverse Gaussian distribution defined by the parameters a, b, c. In the following description, a variable name representing an element is described in parentheses in the upper right to indicate which element (S, H, etc.) the parameter relates to as necessary. For example, a parameter described as “a ^(H) ” is a parameter related to sound field characteristics.

The posterior distribution of the above model is calculated using the variational Bayes method. Here, the logarithmic simultaneous distribution is expressed as the following equation (5). In Equation (5), the constant term is ignored.

However, since the variational Bayes method cannot be applied to Equation (5), the lower limit is determined using an auxiliary function. Specifically, a lower limit function such as the following equation (6) is set, and the newly introduced auxiliary variable M and auxiliary variable Φ are updated.

Specifically, in step S13, the auxiliary variable M is optimized under the conditions defined by the equations (7) to (10), and the auxiliary variable Φ is optimized under the conditions defined by the equation (11). Is done.

Next, in step S14, the parameters of the posterior distribution are updated using the following equations (12) and (13). Each parameter in the equations (12) and (13) is defined as the following equations (14) to (19).

Next, in step S15, it is determined whether or not the lower limit function has converged. That is, it is determined whether the auxiliary variable M, the auxiliary variable Φ, and the parameters of the posterior distribution have converged. If the lower limit function has not converged, it is determined as “No”, and the auxiliary variable M, the auxiliary variable Φ, and the parameters of the posterior distribution are respectively updated in steps S13 and S14. On the other hand, when the lower limit function has converged, it is determined as “Yes”, and the singing voice and sound field characteristic estimation processing ends in step S16. As described above, the auxiliary variable M, the auxiliary variable Φ, and the parameters of the posterior distribution are updated repeatedly to determine the lower limit function, whereby the posterior distribution is approximately calculated. Thereby, Bayesian estimation of the singing voice and the sound field characteristic is performed simultaneously (integrally). Note that the singing voice is extracted from the mixed sound by applying the mask represented by the following equation (20) to the spectrum (short-time Fourier transform) of the mixed sound in the t-th frame and calculating the inverse Fourier transform. be able to.

  According to the acoustic signal analyzing apparatus 10 configured as described above, it is not necessary to learn in advance a model of a sound that constitutes a mixed sound (or a model of each sound source that generates a sound that constitutes a mixed sound). Therefore, singing voice and sound field characteristics can be estimated for any mixed sound. That is, the acoustic signal analyzer 10 is more versatile than the acoustic signal analyzer of Non-Patent Document 1.

  Further, unlike the acoustic signal analyzer of Non-Patent Document 2, it is sufficient to have one microphone, so the configuration of the apparatus is simple.

  Further, the estimated sound field characteristic largely depends on the frequency characteristic of the musical sound (direct sound). However, in this embodiment, the posterior distribution of the sound field characteristic is estimated, and thus the uncertainty of the estimated sound field characteristic is estimated. Can be certified. That is, the reliability of the frequency band in which the variance of the estimated posterior distribution of the sound field characteristic exceeds a predetermined threshold (for example, a predetermined value or a value proportional to the power of a specific frequency bin in the power spectrum X of the musical sound). The degree of reliability is low, and it can be recognized that the reliability of the frequency band whose variance is equal to or less than the predetermined threshold is high. If the sound field characteristics in the frequency band with low reliability are corrected using an equalizer or the like, the singing voice can be extracted more accurately from the mixed sound.

Instead of the generation model represented by the expressions (2) to (4) in the above embodiment, the generation model represented by the following expressions (21) to (23) may be adopted.

In this case, if the reproducibility of the Poisson distribution is used and the auxiliary variables M ^(S) and M ^(H) are constrained to satisfy the following expression (24), they are expressed by the above expressions (21) to (23). The generated model is equivalent to the generated model represented by the following equations (25) to (28).

The logarithmic simultaneous distribution is represented by the following equation (29).

The posterior distribution is updated using the following equations (30) to (33).

Note that Z _{f, t} in the above equation (31) is a normalization coefficient as shown in the following equation (34). Other parameters are defined as shown in the following equations (35) to (41).

In this case, a singing voice can be extracted from the mixed sound by adding the short-time Fourier transform phase of the mixed sound to the average value of “S _{f, t} ”.

In the above embodiment, “I” needs to be fixed. However, since “I” depends on the reverberation time of the sound field, in order to be robust against various sound fields, it is desirable that the dependence on “I” is weakened. Therefore, a variable g _j defined as in the following equation (42) may be introduced into the above equation (21), and the model may be formulated as in the following equation (43). That is, the variable g _j represents the gain of the singing voice of the current frame and the gain of the musical sound of the past frame.

  According to this, a behavior is obtained in which the value of “I” dynamically changes according to the sound field. Thereby, a singing voice can be extracted from a mixed sound more correctly.

  Further, in the first embodiment and the modification thereof, since the mathematical expression including the term representing the sum of the frames (sum related to “t”) is used, the singing voice and the sound field characteristics cannot be estimated in real time. . Therefore, a procedure for estimating the singing voice and sound field characteristics in real time by modifying the singing voice and sound field characteristics estimation procedure described using the equations (21) to (43) will be described.

First, consider the posterior distribution to be independent for each frequency bin and focus on one frequency bin. Then, the posterior distribution is expressed as the following formula (44) and formula (45).

The above posterior distribution is calculated using the variational Bayesian method. Specifically, the posterior distribution is approximately calculated by optimizing an objective function as shown in the following formula (46). “H _{q (θ)} ” in the equation (46) is defined as shown in the following equation (47).

Here, as shown in Expression (48), among the variables in the posterior distribution, variables that are independent for each frame are first optimized. Optimizing the objective function J ′ shown in Expression (48) is equivalent to optimizing the objective function J shown in Expression (46).

Furthermore, random variables _τ to Uniform (0, T) that are uniformly distributed between “0” and “T” are introduced, and the objective function J _τ is defined as the following equation (49). The objective function J _τ is an objective function of the objective function J ′ when it is assumed that all the observation data are of the frame τ.

Since the observation data is uniformly distributed, the following equation (50) is established. If the frame index “t” is incremented by “1”, the objective function J _τ (θ) is evaluated in each frame, and the optimum value is accumulated, the average value of J _I (θ) is calculated in real time. Can be updated.

The procedure of the real time estimation process of the above singing voice and sound field characteristics will be described with reference to FIG. In step S20, real time estimation processing of singing voice and sound field characteristics is started. In step S21, various variables are initialized. Next, in step S22, music playback is started. Next, in step S23, the spectrum of the mixed sound corresponding to the t-th frame is calculated. Specifically, sample values are sequentially stored in the buffer at a predetermined sampling period, and a plurality of sample values corresponding to the t-th frame among the sample values stored in the buffer are used. The spectrum of the mixed sound corresponding to the second frame is calculated. Initially, “t” is initialized to “0”, so the spectrum corresponding to the first frame is calculated. Next, in step S24, the auxiliary variable Φ is updated. Next, in step S25, the latent variable _St is updated. Next, in step S26, it is determined whether or not the auxiliary variable Φ has converged. When the auxiliary variable Φ has not converged, it is determined as “No”, Steps S24 and Step 25 are executed, and the auxiliary variable Φ and the latent variable _St are updated again. On the other hand, when the auxiliary variable Φ is converged, it is determined as "Yes" at step S27, voice of the audio signal is restored based on the average value of the latent variable S _t. In step S28, the parameters of the posterior distribution are updated. Next, in step S29, the posterior distribution is updated. However, “ρ _t ” is a coefficient that satisfies 0 ≦ ρ _t ≦ 1. Further, when the following equation (51) is satisfied, the convergence of the variable Φ is guaranteed.

  Next, in step S30, the index of the frame is updated (incremented). Next, in step S31, it is determined whether or not the last frame has been processed. That is, if the index (t) of the frame exceeds the index (T) of the final frame, it is determined that the final frame has already been processed, and the real-time processing ends in step S32. On the other hand, if the index (t) of the frame is equal to or less than the index (T) of the final frame, the process consisting of steps S23 to S31 is executed again.

In the real-time processing described above, “H” may be hierarchized to optimize the prior parameter superparameter. According to this, since “H” is independent between the frames, “H” can be updated only from the data of the current frame. However, if this is the case, the number of samples will be too small, and the estimation accuracy of the singing voice may be reduced. Therefore, the appropriate prior distribution of “H” is updated from the posterior distribution of “H” estimated in each frame. That is, the following formula (52) is used instead of the formula (22).

According to this, the update formula of “H _{f, t, i} ” does not include a term representing the sum of frames (sum related to “t”). Therefore, “H _{f, t, i} ” is updated independently in each frame.

Next, the hyper parameter is optimized by maximizing the objective function shown in the following equation (53). For any “f” and “i”, equations (54) to (57) hold.

Here, if the result of partial differentiation of G (a, b) with respect to “b” is equal to “0”, a relational expression “b = a / c” is obtained. Also, assuming that the result of partial differentiation of G (a, b) with respect to “a” is equal to “0”, the relational expression “d + logb−ψ (a) = 0” is obtained. If “a” is obtained, “b” is also obtained, but it is difficult to obtain “a”. Therefore, the following equation (58) combining the above two relational expressions is solved.

Here, if exp (a ′) = a, an objective function shown in the following formula (59) is derived.

When the Newton Raphson method is applied to this equation (59), an update equation shown in the following equation (60) is derived.

That is, “a” is expressed as in the following formula (61).

If an approximate expression of the digamma function is used, the following expressions (62) and (63) are established.

Therefore, an update formula as shown in the following formula (64) is derived. The parameters may be updated using this update formula.

In view of the equations (41) and (42), it is considered that the history should be simply accumulated. That is, for example, it is considered that an update equation such as the following equation (65) may be used.

However, the variational Bayes method uses iterative calculation, and the initial value of “S” is sparse (a value close to “0” is preferentially treated). On the other hand, as the observation data increases, the certainty of “a ^(H) ” and “b ^(H) ” increases. That is, the reliability is low because the number of observation data is small in the initial stage. Also, using equation (65), the new observation has a weak effect on the old observation. Therefore, as shown in the following formulas (66) and (67), a variable η _n is introduced that gives weight to the history accumulation. That is, the variable η _n is the sum of the intensities of predetermined frequency components of the musical sound spectrum from the start of sound generation (first sound) to the present, and the predetermined spectrum spectrum of the mixed sound from the start of sound generation to the present. Only the sum of the frequency component intensities corresponds to a weighting coefficient that is multiplied according to the number of updates. “N” represents the number of iterations. In the initial stage (that is, when “n” is small), the variable η _n is set to a value close to “0”, and the variable η _n is gradually brought close to “1” as the number of iterations increases.

  According to this, since weighting is applied only to a specific variable, it is possible to suppress an increase in the number of iterative calculations when determining the lower limit function, compared to a case where a so-called “deterministic annealing” is employed. it can.

  In addition, although the estimation procedure of the singing voice and the sound field characteristic described using the expressions (21) to (43) is modified and the procedure for estimating the singing voice and the sound field characteristic in real time has been described, the expressions (2) to ( The singing voice and sound field characteristics estimation procedure described using 20) can be similarly modified, and the singing voice and sound field characteristics can be estimated in real time.

(Second Embodiment)
Next, the acoustic signal analyzer 20 according to the second embodiment of the present invention will be described. First, an outline of the acoustic signal analyzer 20 will be described. The acoustic signal analysis device 20, like the acoustic signal analysis device 10, emits predetermined music from the sound emitting device (speaker 161) and collects the singer's singing voice using the sound collecting device (microphone 162). . However, in the present embodiment, unlike the first embodiment, the sound emitting device and the sound collecting device are installed at close positions. The sound collecting device is installed at a position far away from the singer and the sound emitting device. Therefore, the sound collecting device collects the sound in which the acoustic characteristics (sound field characteristics) of the room where the sound emitting device and the sound collecting device are installed, the musical sound and the singing voice are convoluted. That is, not only the sound emitted from the sound emitting device (direct sound) and the singer's singing voice (direct sound) but also the reflected sound (reflection) reflected from the wall or floor of the room is collected. In the present embodiment, a sound obtained by mixing a sound of music including reverberation and a singing voice including reverberation is referred to as a mixed sound.

The relationship among the power spectrum Y of the mixed sound, the power spectrum X of the musical sound (direct sound), the sound field characteristic H, and the power spectrum S of the singing voice can be expressed as a block diagram as shown in FIGS. The model can be formulated as the following formula (68). Since the acoustic characteristic H and the power spectrum S of the singing voice are not directly observed, they are latent variables in this model.

  The acoustic signal analyzer 20 records (samples) the mixed sound, and uses the recorded mixed sound as observation data to perform Bayesian estimation of the singing voice and the characteristics of the sound field simultaneously (integrally). Since the other structure of the acoustic signal analyzer 20 is the same as that of the acoustic signal analyzer 10, the description thereof is omitted.

  Next, the operation (singing voice and sound field characteristic estimation procedure) of the acoustic signal analyzer 20 configured as described above will be described. As in the first embodiment, as shown in FIG. 4, the singing voice and sound field characteristic estimation processing is started in step S10. Next, in step S11, various variables (auxiliary variables described later, parameters of posterior distribution, etc.) are initialized. Next, in step S 12, the music data is supplied to the sound system 16 and sound emission of the music is started from the speaker 161, and sampling of the mixed sound collected by the microphone 162 is started. The sampled mixed sound data is stored in the RAM 12c. When the music is released, as described below, the singing voice and sound field characteristics are simultaneously (integrated) Bayesian estimated using the mixed sound data stored in the RAM 12c as observation data.

Assuming that the spectrum of singing voice, the spectrum of musical sounds, and the spectrum of mixed sounds (short-time Fourier transform) are generated from a complex normal distribution, a generation model represented by the following equations (69) to (71) can be constructed. .

The posterior distribution of the above model is calculated using the variational Bayes method. Here, the logarithmic simultaneous distribution is represented by the following equation (72). In the equation (72), the constant term is ignored.

However, since the variational Bayes method cannot be applied to the equation (72), the lower limit is determined using an auxiliary function. Specifically, a lower limit function such as the following formula (73) is set, and the newly introduced auxiliary variable M and auxiliary variable Φ are updated.

Specifically, in step S13, the auxiliary variable M is optimized under the conditions defined by the equations (74) to (77), and the auxiliary variable Φ is optimized under the conditions defined by the equation (78). Is done.

Next, in step S14, the parameters of the posterior distribution are updated using the following equations (79) and (80). Each parameter in the formulas (79) and (80) is defined as the following formulas (81) to (86).

Next, in step S15, it is determined whether or not the lower limit function has converged. That is, it is determined whether the auxiliary variable M, the auxiliary variable Φ, and the parameters of the posterior distribution have converged. If the lower limit function has not converged, it is determined as “No”, and the auxiliary variable M, the auxiliary variable Φ, and the parameters of the posterior distribution are respectively updated in steps S13 and S14. On the other hand, when the lower limit function has converged, it is determined as “Yes”, and the singing voice and sound field characteristic estimation processing ends in step S16. As described above, the posterior distribution is approximately calculated. Thereby, Bayesian estimation of the singing voice and the sound field characteristic is performed simultaneously (integrally). Note that the singing voice is extracted from the mixed sound by applying the mask represented by the following formula (87) to the spectrum of the mixed sound (short-time Fourier transform) in the t-th frame and calculating the inverse Fourier transform thereof. be able to.

  The acoustic signal analyzer 20 configured as described above can also provide the same effects as those of the first embodiment.

Instead of the generation model represented by the equations (69) to (71) in the above embodiment, the generation model represented by the following equations (88) to (90) may be adopted.

In this case, if the reproducibility of the Poisson distribution is used and the auxiliary variables M ^(S) and M ^(H) are constrained to satisfy the following equation (91), they are expressed by the above equations (88) to (90). The generated model is equivalent to the generated model represented by the following equations (92) to (95).

In this case, the logarithmic simultaneous distribution is represented by the following equation (96).

The posterior distribution is updated using the following equations (97) to (100).

Note that Z _{f, t} in the above equation (98) is a normalization coefficient as shown in the following equation (101). Other parameters are defined as shown in the following equations (102) to (108).

In this case, a singing voice can be extracted from the mixed sound by adding the short-time Fourier transform phase of the mixed sound to the average value of “S _{f, t} ”.

  Also in the second embodiment, the sound field characteristics and the singing voice can be estimated in real time by modifying the mathematical formulas as in the modification of the first embodiment.

10, 20 ... Acoustic signal analyzer, 161 ... Speaker, 162 ... Microphone, Y ... Power spectrum of mixed sound, X ... Power spectrum of musical sound, H ... Sound field characteristics, S ... Singing voice power spectrum

Claims (14)

  Sound emission means for emitting a predetermined first sound;
  Sound collecting means for collecting a mixed sound including the first sound emitted and a second sound emitted from a sound source different from the sound emitting means;
  Estimation means for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emission means and the sound collection means are installed based on the collected mixed sound and the first sound; Prepared,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation means includes
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. An acoustic signal analyzer characterized by approximately estimating the posterior distribution by determining a function.   Sound emission means for emitting a predetermined first sound;
  Sound collecting means for collecting a mixed sound including the first sound emitted and a second sound emitted from a sound source different from the sound emitting means;
  Estimation means for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emission means and the sound collection means are installed based on the collected mixed sound and the first sound; Prepared,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation means includes
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the acoustic signal analyzer characterized by approximately estimating the posterior distribution.   Sound emission means for emitting a predetermined first sound;
  Sound collecting means for collecting a mixed sound including the first sound emitted and a second sound emitted from a sound source different from the sound emitting means;
  Estimation means for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emission means and the sound collection means are installed based on the collected mixed sound and the first sound; Prepared,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation means includes
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. An acoustic signal analyzer characterized by approximately estimating the posterior distribution by determining a function.   Sound emission means for emitting a predetermined first sound;
  Sound collecting means for collecting a mixed sound including the first sound emitted and a second sound emitted from a sound source different from the sound emitting means;
  Estimation means for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emission means and the sound collection means are installed based on the collected mixed sound and the first sound; Prepared,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation means includes
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the acoustic signal analyzer characterized by approximately estimating the posterior distribution.   The acoustic signal analyzer according to any one of claims 1 to 4,
  The sound collection means samples the second sound and the mixed sound in real time,
  The acoustic signal analyzer according to claim 1, wherein the estimation means approximately estimates the posterior distribution by updating and optimizing an expected value of the lower limit function in real time. The acoustic signal analyzer according to claim 5,
The parameter relating to the characteristic of the sound field, the parameter relating to the coefficient multiplied by the intensity of a predetermined frequency component, is the predetermined spectrum of the spectrum of the first sound from the start of sound generation to the present. Only the sum of frequency component intensities and the sum of intensities of the predetermined frequency components of the spectrum of the mixed sound from the start of sound generation to the present are set to be multiplied by a weighting coefficient corresponding to the number of updates. An acoustic signal analyzer that is updated based on an update formula.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. Acoustic signal analysis for simultaneously Bayesian estimation of the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound A method,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. An acoustic signal analysis method characterized in that the posterior distribution is approximately estimated by determining a function.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. Acoustic signal analysis for simultaneously Bayesian estimation of the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound A method,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the acoustic signal analysis method characterized by approximately estimating the posterior distribution.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. Acoustic signal analysis for simultaneously Bayesian estimation of the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound A method,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. An acoustic signal analysis method characterized in that the posterior distribution is approximately estimated by determining a function.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. Acoustic signal analysis for simultaneously Bayesian estimation of the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound A method,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the acoustic signal analysis method characterized by approximately estimating the posterior distribution.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. A sound collection step;
  An estimation step for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound. A computer program characterized by causing a computer to execute,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation step includes
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. A computer program comprising the step of approximately estimating the posterior distribution by determining a function.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. A sound collection step;
  An estimation step for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound. A computer program characterized by causing a computer to execute,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and the second sound,
  The estimation step includes
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the computer program is a step of approximately estimating the posterior distribution.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. A sound collection step;
An estimation step for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound. A computer program characterized by causing a computer to execute,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation step includes
  The lower limit function that is the lower limit of the posterior distribution of the generation model representing that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a complex normal distribution and a generalized inverse Gaussian distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including a parameter relating to the characteristics of the second sound and the sound field, and the auxiliary variable and the parameter are repeatedly updated to set the lower limit function. A computer program comprising the step of approximately estimating the posterior distribution by determining a function.   A mixed sound including a predetermined first sound emitted from the sound emitting means and a second sound emitted from a sound source different from the sound emitting means is collected using the sound collecting means. A sound collection step;
  An estimation step for performing Bayesian estimation simultaneously on the second sound and the characteristics of the sound field in which the sound emitting means and the sound collecting means are installed based on the collected mixed sound and the first sound. A computer program characterized by causing a computer to execute,
  The mixed sound includes a sound in which the first sound and the characteristics of the sound field are convoluted, and a sound in which the second sound and the characteristics of the sound field are convoluted,
  The characteristics of the sound field are expressed as a set of coefficients that are multiplied by the intensity of each frequency component of the first sound,
  The estimation step includes
  A lower limit function that is a lower limit of a posterior distribution of a generation model indicating that the spectrum of the mixed sound, the time series of the spectrum of the second sound, and the set of coefficients are generated according to a Poisson distribution and a gamma distribution, respectively. A lower limit function represented by a plurality of auxiliary variables and including parameters relating to the characteristics of the second sound and the sound field is set, and the lower limit function is determined by repeatedly updating the auxiliary variables and the parameters. Thus, the computer program is a step of approximately estimating the posterior distribution.

Images