JP2016001235A - Information processor, terminal device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of suppressing a load in a terminal device for processing of separation into a background sound and a speech sound.SOLUTION: An information processor includes an aural signal acquisition part for acquiring an aural signal in which a speech sound uttered from a person and a background sound are mixed, a phoneme component information calculation part for calculating phoneme component information showing phoneme components included in the aural signal with reference to phoneme information for showing phonemes constituting the speech sound, and a notification part for notifying a device that adjusts a ratio between the speech sound and the background sound in the aural signal, of the phoneme component information.

Description

  The present invention relates to an information processing device, a terminal device, and a program.

  Conventionally, a signal processing technique called sound source separation has been used to emphasize / suppress each sound from a sound signal in which background sound and sound uttered by a person (hereinafter referred to as speech sound) are mixed. . In this technique, a signal obtained by mixing audio signals derived from a plurality of sound sources is separated into signals derived from individual sound sources. However, depending on the mixing process, separation may be difficult. For example, when music is mixed with speech sound on television or radio, the number of musical instruments varies depending on the type of music. It must be separated from the mixed signal of only the right two channels.

  Such a sound source separation problem is often a defect setting problem in which the number of sound sources is larger than the number of mixed signals. For this reason, an exact solution cannot be obtained, and an approximate solution can be obtained by using some prior knowledge about the sound source. For example, in stereo broadcasting, the speech sound is localized to the center, and the background sound component and the speech sound are utilized by utilizing the property that the left / right channels are often mixed at the same level. There is a method of readjusting the mixing ratio after separating the components (see Patent Document 1). In addition, there is a method in which the mixing ratio is readjusted after the background sound and the speech sound are separated using a technique called non-negative matrix factorization (see Non-Patent Document 1).

JP 2009-025500 A

Makoto Hirohata and two others, “Non-delayed sound source separation technology for various video contents”, Acoustical Society of Japan, 2013 Spring Acoustical Society of Japan Proceedings, Presentation No. 3-10-18, p. 799-802, March 2013

  However, the conventional method has a problem that when it is used for a terminal device that receives a broadcast or the like, a processing load for separating the background sound and the speech sound may be large.

  The present invention has been made in view of such circumstances, and provides an information processing device, a terminal device, and a program capable of suppressing a load on a terminal device for processing to separate background sound and speech sound.

  The present invention has been made to solve the above-described problems, and one aspect of the present invention is an audio signal acquisition unit that acquires an audio signal in which a speech sound uttered by a person and a background sound are mixed, and a speech sound. A phoneme component information calculation unit that calculates phoneme component information that represents a component of the phoneme included in the speech signal, and the phoneme component information is converted into a speech sound in the speech signal. And a notification unit that notifies the device that adjusts the ratio between the background sound and the background sound.

  Another aspect of the present invention is the above-described information processing device, wherein the phoneme component information calculation unit refers to the phoneme information, and is a first non-negative value that is an audio signal subjected to a short-time Fourier transform. The matrix includes a second non-negative value matrix that is the phoneme information, a third non-negative value matrix that is the phoneme component information, and a fourth non-negative value related to a component other than the phoneme among the sounds represented by the speech signal. It is characterized by decomposing into a matrix and a fifth non-negative matrix.

  In another aspect of the present invention, a phoneme information storage unit that stores phoneme information representing a phoneme constituting a speech sound uttered by a person, and an audio signal that acquires an audio signal in which the speech sound and the background sound are mixed Referring to the acquisition unit, the phoneme component information acquisition unit that acquires phoneme component information representing the phoneme component included in the audio signal, the phoneme information and the phoneme component information, and included in the audio signal A terminal unit comprising: a separation unit that separates a speech sound and a background sound; and a mixing unit that adjusts and mixes a ratio of a voice uttered by a person separated by the separation unit and a background sound. It is.

  Another aspect of the present invention is the above-described terminal device, wherein the separation unit refers to the phoneme information and the phoneme component information, and represents information representing a speech sound included in the audio signal. A speech sound generation unit that generates a sound, and a background sound separation unit that generates information indicating the background sound by subtracting the sound represented by the information generated by the speech sound generation unit from the audio signal. To do.

  In another aspect of the present invention, a computer acquires a phoneme information storage unit that stores phoneme information representing a phoneme constituting a speech sound uttered by a person, and an audio signal in which the speech sound and the background sound are mixed. An audio signal acquisition unit; a phoneme component information acquisition unit that acquires phoneme component information representing a component of the phoneme included in the audio signal; and the phoneme information and the phoneme component information. A program for functioning as a separation unit that separates speech sound and background sound, and a mixing unit that adjusts and mixes the ratio of the sound produced by the person separated by the separation unit and the background sound.

  According to the present invention, it is possible to suppress the load on the terminal device for the process of separating the background sound and the speech sound.

It is a schematic block diagram which shows the structure of the audio | voice delivery system by one Embodiment of this invention. It is a schematic block diagram which shows the structure of the phoneme information delivery apparatus 11 by the embodiment. It is a schematic block diagram which shows the structure of the phoneme component information delivery apparatus 12 by the embodiment. It is a schematic block diagram which shows the structure of the terminal device 31 by the embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of an audio distribution system according to an embodiment of the present invention. The audio distribution system includes a phoneme information distribution device 11, a phoneme component information distribution device 12, an audio signal distribution device 13, a network 21, and a plurality of terminal devices 31. The phoneme information distribution device 11 extracts phoneme information Us representing each phoneme that is a sound constituting the speech sound from a large amount of the speech signal S including only the speech sound without including the background sound. Here, the phoneme information Us is a spectrum distribution of each phoneme. The phoneme information distribution device 11 distributes the extracted phoneme information Us to each terminal device 31 via the network 21. In addition, the phoneme information distribution device 11 notifies the extracted phoneme information Us to the phoneme component information distribution device 12. Note that the extraction and distribution of the phoneme information Us by the phoneme information distribution device 11 is performed prior to the distribution of the sound to the terminal device 31.

  The phoneme component information distribution device 12 (information processing device) refers to the phoneme information Us notified from the phoneme information distribution device 11 and indicates each component of the phoneme represented by the phoneme information Us in the audio signal P to be distributed. Phoneme component information Hp is calculated. The phoneme component information distribution device 12 distributes the calculated phoneme component information Hp to each terminal device 31 via the network 21. The calculation and distribution of the phoneme component information Hp by the phoneme component information distribution device 12 is performed simultaneously with the distribution of the audio signal P.

  The audio signal distribution device 13 distributes an audio signal P composed of a speech sound and a background sound to each terminal device 31 via the network 21. The audio signal distribution device 13 may distribute the audio signal P as it is, or may distribute it after compression encoding.

  The network 21 may be, for example, a packet switching network such as the Internet, or a network using broadcast waves. Note that the networks 21 when the phoneme information distribution device 11, the phoneme component information distribution device 12, and the audio signal distribution device 13 each distribute to the terminal device 31 may be different. For example, the phoneme information distribution device 11 and the phoneme component information distribution device 12 may distribute to the terminal device 31 via the Internet, and the audio signal distribution device 13 may distribute to the terminal device 31 via a broadcast wave. Good.

  The terminal device 31 receives and stores phoneme information Us distributed in advance via the network 21. When the terminal device 31 receives the speech signal P and the phoneme component information Hp, the terminal device 31 refers to the phoneme information Us stored in advance and the received phoneme component information Hp, and is mixed with the speech signal P. The ratio between the sound and the speech sound is adjusted, and the sound with the adjusted ratio is output.

  FIG. 2 is a schematic block diagram showing the configuration of the phoneme information distribution apparatus 11. The phoneme information distribution device 11 includes an audio signal acquisition unit 111, a short-time Fourier transform unit 112, and a phoneme information generation unit 113. The audio signal acquisition unit 111 acquires an audio signal S composed only of speech sounds. The short-time Fourier transform unit 112 performs a short-time Fourier transform on the audio signal S.

  The short-time Fourier transform is the following transformation. 1) A section with a certain time width is cut out from the time waveform of the sound while being shifted in order from the head at a certain time interval. 2) Discrete Fourier transform is performed on each extracted section. 3) Take the square of the absolute value of the result of the discrete Fourier transform. Thereby, the spectrogram showing the magnitude | size of the amplitude for every frequency component is obtained about each area.

  The short-time Fourier transform of the signal g (t) is expressed by equation (2) using G (f, n) expressed by equation (1). In Expression (1), Δt is a shift time interval. n = 1, 2,..., N are shift indices. T is the time width of the section to be cut out. DFT () is a discrete Fourier transform. f = 1, 2,..., F is an index of each frequency bin obtained by the discrete Fourier transform.

  Hereinafter, the spectrogram matrix X of Equation (2) when g (t) in Equation (1) is the speech signal S, that is, the conversion result of the speech signal S by the short-time Fourier transform unit 112 is referred to as spectrogram matrix Xs.

  The phoneme information generation unit 113 performs non-negative matrix factorization on the conversion result by the short-time Fourier transform unit 112, that is, the spectrogram matrix Xs. Thereby, the phoneme information generation unit 113 generates phoneme information Us. Non-negative matrix factorization is a decomposition of a matrix in which all elements, such as observation data, are non-negative (referred to as a non-negative matrix) into a product of two non-negative matrices. There are some known methods for calculating non-negative matrix factorization, and any of them may be used. Here, a method using KL-Divergence (Calbach libra information amount) will be described.

The KL-Diverence between the product of the matrices Us and Hs and the spectrogram matrix Xs is expressed by Expression (3). It is known that the matrices Us and Hs that minimize the KL-Diverence of Expression (3) can be obtained by repeating the operations of Expressions (4) and (5). Note that x _Sij , u _Sij , and h _Sij are elements of i rows and j columns of the matrices Xs, Us, and Hs, respectively.

  Therefore, the phoneme information generation unit 113 alternately performs the calculation of KL-Divergence according to Equation (3) and the calculations according to Equations (4) and (5). Then, the phoneme information generation unit 113 ends the repetition when the amount of decrease in KL-Divergence according to the expression (3) becomes equal to or less than a predetermined value before and after the calculations of the expressions (4) and (5). The matrix Us at that time is assumed to be phoneme information Us. It is known that the spectral pattern characteristically appearing in the speech sound is learned in the matrix Us by performing the non-negative matrix factorization of the spectrogram matrix Xs of the speech signal S consisting only of the speech sound in this way. (For example, refer nonpatent literature 1). The phoneme information generation unit 113 distributes the phoneme information Us to the terminal device 31 via the network 21. The phoneme information generation unit 113 notifies the phoneme component information distribution device 12 of the phoneme information Us.

FIG. 3 is a schematic block diagram showing the configuration of the phoneme component information distribution device 12. The phoneme component information distribution device 12 includes a phoneme information storage unit 121, an audio signal acquisition unit 122, a short-time Fourier transform unit 123, a phoneme component information calculation unit 124, and a phoneme component information distribution unit 125. The phoneme information storage unit 121 receives and stores the phoneme information Us notified by the phoneme information distribution device 11. The audio signal acquisition unit 122 acquires the audio signal P distributed to the terminal device 31. The short-time Fourier transform unit 123 performs short-time Fourier transform on the audio signal P acquired by the audio signal acquisition unit 122. The short-time Fourier transform unit 123 divides the audio signal P for each time width L, and performs the same calculation as the short-time Fourier transform unit 112 in FIG. 2 for each divided section. Hereinafter, the spectrogram matrix obtained by performing the short-time Fourier transform on the section from time t to time t + L is referred to as spectrogram matrix Xp ^(t) .

The phoneme component information calculation unit 124 refers to the phoneme information Us (second non-negative matrix) stored in the phoneme information storage unit 121, and the sound represented by the spectrogram matrix Xp ^(t) (first non-negative matrix) A matrix Hp ^(t) (third non-negative matrix) representing the ratio of speech sounds included at each time, and a matrix Hn ^(t) (fourth non-negative value ⁾ representing the ratio of sounds other than speech sounds, that is, background sounds. Matrix). The calculation of these matrices Hp ^(t) and Hn ^(t) corresponds to the problem of obtaining Hp ^(t) , Un ^(t) and Hn ^(t) that minimizes KL-Divergence expressed by Equation (6). To do. It is known that this problem can be obtained by repeatedly calculating equations (7), (8), and (9).

Therefore, the phoneme component information calculation unit 124 alternately repeats the calculation of KL-Divergence by Expression (6) and the calculations of Expressions (7), (8), and (9). Then, the phoneme component information calculation unit 124, when the amount of decrease in KL-Diverence according to the equation (6) becomes equal to or less than a predetermined value before and after the calculations of the equations (7), (8), and (9), The repetition is terminated, and the matrix Hp ^(t) at that time is included in the phoneme component information Hp. Similarly, the phoneme component information calculation unit 124 calculates matrices Hp ^{(t + L)} , Hp ^{(t + 2 × L)} ,... And includes them in the phoneme component information Hp.
The phoneme component information distribution unit 125 (notification unit) distributes the phoneme component information Hp calculated by the phoneme component information calculation unit 124 to the terminal device 31 via the network 21.

  FIG. 4 is a schematic block diagram illustrating the configuration of the terminal device 31. The terminal device 31 includes a phoneme information acquisition unit 301, a phoneme information storage unit 302, a phoneme component information acquisition unit 303, a speech sound generation unit 304, an audio signal acquisition unit 305, a short-time Fourier transform unit 306, a background sound separation unit 307, a speech A sound / background sound mixing unit 308, an inverse short-time Fourier transform unit 309, and an audio output unit 310 are included. The speech sound generation unit 304, the short-time Fourier transform unit 306, and the background sound separation unit 307 function as the separation unit 320.

The phoneme information acquisition unit 301 receives the phoneme information Us distributed via the network 21 by the phoneme information distribution device 11. The phoneme information storage unit 302 stores the phoneme information Us received by the phoneme information acquisition unit 301. The phoneme component information acquisition unit 303 receives the phoneme component information Hp distributed by the phoneme component information distribution device 12 via the network 21. The speech sound generation unit 304 multiplies the matrix Hp ^(t) included in the phoneme component information Hp received by the phoneme component information acquisition unit 303 by the phoneme information Us (matrix Us). Thereby, the spectrogram matrix of the speech sound is calculated.

The audio signal acquisition unit 305 receives the audio signal P distributed by the audio signal distribution device 13 via the network 21. The short-time Fourier transform unit 306 performs short-time Fourier transform on the audio signal P in the same manner as the short-time Fourier transform unit 123 of FIG. Thereby, the spectrogram matrix Xp ^(t) is calculated. The background sound separating unit 307 calculates a spectrogram matrix of the background sound by subtracting the spectrogram matrix calculated by the speech sound generating unit 304 from the spectrogram matrix Xp ^(t) . At this time, the spectrogram matrix calculated by the speech sound generation unit 304 and the spectrogram matrix Xp ^{(t) of} the speech signal P must be synchronized in time. For example, the phoneme component information distribution device 12 adds a time stamp to the phoneme component information Hp and distributes it, the audio signal distribution device 13 distributes the audio signal P with a time stamp, and the background sound separation unit 307 Synchronize using the time stamp.

The speech sound / background sound mixing unit 308 multiplies the spectrogram matrix of the speech sound calculated by the speech sound generation unit 304 by a preset coefficient α. Similarly, the speech sound / background sound mixing unit 308 multiplies the spectrogram matrix of the background sound calculated by the background sound separation unit 307 by a preset coefficient β. The speech sound / background sound mixing unit 308 calculates the sum of the spectrogram matrix of the speech sound multiplied by the coefficient α and the spectrogram matrix of the background sound multiplied by the coefficient β. Thereby, the spectrogram matrix Y ^(t) in which the ratio between the speech sound and the background sound is adjusted is calculated. The coefficients α and β may be set by the user of the terminal device 31, for example. The processing by the speech sound generation unit 304, the background sound separation unit 307, and the speech sound / background sound mixing unit 308 is expressed by Expression (10).

The inverse short-time Fourier transform unit 309 performs inverse short-time Fourier transform on the spectrogram matrix Y ^(t) calculated by the speech sound / background sound mixing unit 308, and the speech in which the ratio of the speech sound and the background sound is adjusted. A signal y is generated. In this inverse short-time Fourier transform, the inverse short-time Fourier transform unit 309 performs the square root of each element y _{ij in} the i-th row and j-th column of the spectrogram matrix Y ^(t) as shown in equations (11) and (12). And the inverse discrete Fourier transform is performed for each column of the matrix Y ′ to which the phase θ (i, j ⁾ of i rows and j columns of the spectrogram matrix Xp ^(t) is given, thereby generating the audio signal y.

In Expression (12), IDFT () is an inverse discrete Fourier transform.
The audio output unit 310 is a speaker that outputs audio in accordance with the audio signal y.
In addition, although the audio | voice delivery system in this embodiment has delivered the audio | voice signal P to the terminal device 31, you may distribute not only the audio | voice signal P but a video signal.
Moreover, the audio | voice delivery system does not have the phoneme information delivery apparatus 11, and the phoneme component information delivery apparatus 12 and the terminal device 31 may memorize | store the same phoneme information Us previously.

  Also, the audio signal distribution device 13 may encode and distribute the audio signal P by an irreversible compression method such as AAC (Advanced Audio Coding) or a reversible compression method. The phoneme component information distribution device 12 may encode and distribute the phoneme component information Hp by irreversible compression.

  In this way, the phoneme component information distribution device 12 refers to the phoneme information Us that represents the phonemes constituting the speech sound, and calculates the phoneme component information Hp that represents the phoneme components included in the speech signal P. And a phoneme component information distribution unit 125 that notifies the terminal device 31 of the phoneme component information Hp.

Thereby, the load of the process which isolate | separates into the background sound and speech sound in the terminal device 31 can be suppressed.
Also, it is possible to reduce the amount of information to be distributed rather than distributing the speech sound and the background sound separately. An example in which a monaural audio signal is distributed with a sampling frequency of 48 kHz will be described. When the time shift of the short-time Fourier transform is 5120 samples (about 100 ms) and the number of columns of the phoneme information Us is 100, the matrix Hp having 100 elements every 100 ms as the phoneme component information Hp. (T) is transmitted. If each element is a single precision floating point number (4 bytes), the bit rate is 100 × 4 bytes × 8 bits / 0.1 sec = 32 kbps. The loss can be reduced to about 16 kbps by half.

  On the other hand, in digital terrestrial broadcasting using AAC as an encoding method, the audio bit rate is 72 kbps. Therefore, when the speech sound and the background sound are distributed separately, 72 kbps × 2 = 144 kbps, but when one voice and phoneme component information are distributed, 72 kbps + 16 kbps = 88 kbps, which is a total bit more than the case where they are distributed separately. The rate can be lowered.

Furthermore, the phoneme component information calculation unit 124 refers to the phoneme information Us, converts the first non-negative value matrix Xp that is a short-time Fourier-transformed speech signal, and the second non-negative value matrix Us that is phoneme information, The third non-negative matrix Hp that is phoneme component information and the fourth non-negative matrix Hn and the fifth non-negative matrix Un related to components other than phonemes among the sounds represented by the speech signal P are decomposed.
Thereby, the phoneme component information distribution device 12 can calculate the phoneme component information Hp for separating the speech sound and the background sound included in the audio signal P.

In this way, the terminal device 31 receives the phoneme information storage unit 302 that stores the phoneme information Us that represents the phonemes that make up the speech sound that a person utters, and the phoneme component information Hp that represents the components of the phonemes included in the audio signal P. With reference to the acquired phoneme component information acquisition unit 303, the phoneme information Us and the phoneme component information Hp, the separation unit 320 that separates the speech sound and the background sound included in the audio signal P, and the separation unit 320 A speech sound / background sound mixing unit 308 that adjusts and mixes the ratio of the sound uttered by the person who separated the sound and the background sound.
Thereby, the load of the process which isolate | separates into the background sound and speech sound in the terminal device 31 can be suppressed.

  Further, the program for realizing the functions of the phoneme information distribution device 11, the phoneme component information distribution device 12, the audio signal distribution device 13, and the terminal device 31 in FIG. Each apparatus may be realized by causing the computer system to read and execute the program recorded on the computer. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 11 ... Phoneme information delivery apparatus 12 ... Phoneme component information delivery apparatus 13 ... Audio | voice signal delivery apparatus 21 ... Network 31 ... Terminal device 111 ... Audio | voice signal acquisition part 112 ... Short-time Fourier transform 113 ... Phoneme information generation part 121 ... Phoneme information storage part DESCRIPTION OF SYMBOLS 122 ... Phoneme component information acquisition part 123 ... Short-time Fourier transform part 124 ... Phoneme component information calculation part 125 ... Phoneme component information distribution part 301 ... Phoneme information acquisition part 302 ... Phoneme information storage part 303 ... Phoneme component information acquisition part 304 ... Speech Sound generation unit 305 ... Audio signal acquisition unit 306 ... Short time Fourier transform unit 307 ... Background sound separation unit 308 ... Speech sound / background sound mixing unit 309 ... Inverse short time Fourier transform unit 310 ... Audio output unit

Claims (5)

An audio signal acquisition unit for acquiring an audio signal in which speech sound and background sound uttered by a person are mixed;
A phoneme component information calculation unit that calculates phoneme component information representing a component of the phoneme included in the speech signal with reference to phoneme information representing a phoneme constituting the speech sound;
An information processing apparatus comprising: a notification unit that notifies the device of adjusting a ratio of a speech sound and a background sound in the audio signal.   The phoneme component information calculation unit refers to the phoneme information, converts a first non-negative matrix that is an audio signal subjected to short-time Fourier transform, a second non-negative matrix that is the phoneme information, and the phoneme component The third non-negative matrix that is information and the fourth non-negative matrix and the fifth non-negative matrix related to components other than the phonemes in the sound represented by the speech signal are decomposed. The information processing apparatus according to 1. A phoneme information storage unit that stores phoneme information representing a phoneme constituting a speech sound uttered by a person;
An audio signal acquisition unit for acquiring an audio signal in which a speech sound and a background sound are mixed;
A phoneme component information acquisition unit that acquires phoneme component information representing a component of the phoneme included in the audio signal;
A separation unit that separates a speech sound and a background sound included in the audio signal with reference to the phoneme information and the phoneme component information;
A terminal device comprising: a mixing unit that adjusts and mixes a ratio of a voice uttered by a person separated by the separation unit and a background sound. The separation unit is
A speech sound generator that generates information representing a speech sound included in the audio signal with reference to the phoneme information and the phoneme component information;
The terminal device according to claim 3, further comprising: a background sound separation unit that subtracts a sound represented by the information generated by the speech sound generation unit from the audio signal to generate information indicating the background sound. . Computer
A phoneme information storage unit that stores phoneme information representing a phoneme constituting a speech sound uttered by a person;
An audio signal acquisition unit for acquiring an audio signal in which a speech sound and a background sound are mixed;
A phoneme component information acquisition unit that acquires phoneme component information representing a component of the phoneme included in the audio signal;
A separation unit that separates a speech sound and a background sound included in the audio signal with reference to the phoneme information and the phoneme component information,
The program for functioning as a mixing part which adjusts and mixes the ratio of the sound uttered by the person separated by the separation part and the background sound.

Images