KR102035448B1 - Voice instrument - Google Patents

Abstract

An embodiment of the present invention relates to voice instrument. The voice instrument comprises: an input unit receiving source music information including lyric information, pitch information, and volume information from a user and including a skin vibration sensor; a storage unit storing data related to music including target music information including lyric information and pitch information; a control unit obtaining the target music information having lyric information and pitch information corresponding to the lyric information and pitch information of the source music information from the storage unit and amplifying the obtained target music information according to the volume information of the source music information to obtain the amplified target music information; and an output unit outputting the amplified target music information. The target music information includes tone information different from the source music information.

Description

Voice instrument

The present invention relates to a voice musical instrument.

Ordinary instruments produce different sounds with different tones, depending on the instrument's structure and how it sounds. As far as the sound of a person's song is concerned, the human oral structure itself is an instrument.

Regarding such a voice instrument, US 4,236,434 (Dec. 2, 1980) has attempted to make a sound by forming a formant filter that mimics a human voice using hardware. But hardware lexicons are not as diverse and sophisticated as human voices.

Meanwhile, in US 2009/0089063 A1 (Apr. 2, 2009), a method of converting a source voice frequency and replacing a portion of a frequency of a target voice by using a method of converting a voice is also presented. However, this method mixes the source and target voices, making it difficult to sing with the target voice alone.

In addition, US10008193 B1 (Jun. 26, 2018) presents a method of obtaining the average pitch of two songs and synthesizing the rhythm of the source voice and the pitch of the target voice in order to sing the source voice as the target voice. However, this method also has the disadvantage that the source and target voices are mixed and not very helpful to people who cannot sing.

Furthermore, conventionally, the sound of the song input to the microphone is too loud when the surrounding environment is too loud to achieve the desired purpose.

(Patent Document 1) US4236434 B1

(Patent Document 2) US20090089063 A1

(Patent Document 3) US10008193 B1

Some people sing well, but many do not. Often people who can't sing see the person who sings well and envy them that they want to sing so well.

The present invention is to meet the needs of these people, and relates to a voice musical instrument that can make a song sound, for example, in the voice of a desired singer even if it is not able to sing well.

In addition, the present invention can play the voice instrument of the present invention in an environment with other song sounds or noise, as can be played conventional instruments, such as guitars, violins, piano, drums, even in the presence of other song sounds or noise. To make it possible.

According to an exemplary embodiment of the present disclosure, a voice musical instrument may include: an input unit including a skin vibration sensor and receiving source song sound information including lyrics information, pitch information, and volume information from a user; A storage unit for storing song sound related data including target song sound information including lyrics information and pitch information; Obtaining the target song sound information having the lyrics information of the source song sound information, the lyrics information corresponding to the pitch information, and the pitch information from the storage unit, and obtaining the obtained target song sound information from the source song sound information; A control unit for amplifying the target song sound information by amplifying according to the volume information; And an output unit configured to output the amplified target song sound information, wherein the target song sound information may include tone information different from the source song sound information.

The input unit may further include a first input device configured to receive the pitch information of the source song sound information.

The input unit may further include a second input device configured to receive the volume information of the source song sound information.

The song sound related data stored in the storage unit may include a target song sound information table having different pitch information for each of a plurality of pieces of lyrics information that may be spoken for a unit time.

The unit time of the target song sound information stored in the storage unit may be longer than the unit time of the source song sound information.

The storage unit may store a plurality of feature vectors or AI variable vectors for classifying a plurality of pieces of lyrics information that may be spoken for a unit time.

The storage unit may include two or more groups of the target song sound information.

And an input device and an output device for inputting, outputting, selecting, storing, changing, and / or deleting the source song sound information and / or the song sound related data.

And a communication device for downloading and / or uploading the source song sound information and / or the song sound related data.

The control unit uses the artificial intelligence variable vector learned from the source song sound information input from the input unit as a convolutional neural network (CNN) or a generative adversarial network (GAN). Can be classified.

The target song sound information may be generated using a generator network AI variable vector trained with a generative hostile network.

The target song sound information may include rap, pansori, peking opera, accent and / or intonation data.

The voice processing method according to the present embodiment includes receiving source song sound information including lyrics information, pitch information, and volume information from an input unit including a skin vibration sensor from a user; Obtaining target song sound information having lyrics information and pitch information corresponding to the lyrics information of the source song sound information and the pitch information from a storage unit; Amplifying the acquired target song sound information according to the volume information of the source song sound information to obtain amplified target song sound information; And outputting the amplified target song sound information as song sounds, wherein the target song sound information may include tone information different from the source song sound information.

The computer program according to the present embodiment may be stored in a recording medium to implement the voice processing method according to the present embodiment.

The human oral structure is also a kind of musical instrument. Through the present invention, even a person who can not sing like a musical instrument can sing in a desired voice.

You can also sing not only solo but also double or triplet.

In addition, the source voice can be accurately distinguished regardless of the surrounding sounds such as the sound of another instrument, the song of another person, or the noise.

1 is a block diagram of a voice musical instrument according to a first embodiment of the present invention.
2 is a target song sound information table stored in a storage unit of a voice musical instrument according to a first embodiment of the present invention.
3 is a flowchart showing the function of the controller using the feature vector of the voice musical instrument according to the first embodiment of the present invention.
4 is a source song sound information table consisting of feature vectors of a voice musical instrument according to a first embodiment of the present invention.
5 is a flowchart showing the function of the controller to which the artificial intelligence algorithm of the voice musical instrument according to the modification of the first embodiment of the present invention is applied.
6A is a flowchart illustrating an artificial intelligence learning method using a convolutional neural network according to a modification of the first embodiment of the present invention, and FIG. 6A (b) illustrates a transformation process using a multi-layered layer of the convolutional neural network. The flow chart shown.
6B is a flowchart illustrating an artificial intelligence learning method using a generative antagonist network according to a modification of the first embodiment of the present invention.
7 is a block diagram of a voice musical instrument according to a second embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

However, the technical idea of the present invention is not limited to some embodiments described, but may be implemented in various forms, and within the technical idea of the present invention, one or more of the components between the embodiments may be selectively selected. Can be combined or substituted.

In addition, terms used in the embodiments of the present invention (including technical and scientific terms) are generally understood by those skilled in the art to which the present invention pertains, unless specifically defined and described. The terms commonly used, such as terms defined in advance, may be interpreted as meanings in consideration of the contextual meaning of the related art.

In addition, the terms used in the embodiments of the present invention are intended to describe the embodiments and are not intended to limit the present invention.

In this specification, the singular may also include the plural unless specifically stated in the text, and may be combined with A, B, and C when described as "at least one (or more than one) of A and B and C". It can include one or more of all possible combinations.

In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only to distinguish the components from other components, and the terms are not limited to the nature, order, order, etc. of the components.

And when a component is described as being 'connected', 'coupled', or 'connected' to another component, the component is 'connected', 'coupled', or 'connected' directly to the other component. In addition to the case, it may also include the case where the 'connected', 'coupled' or 'connected' due to another component between the component and the other component.

Hereinafter, the configuration of the voice musical instrument according to the first embodiment of the present invention will be described with reference to the drawings.

1 is a block diagram of a voice instrument according to a first embodiment of the present invention, and FIG. 2 is a target song sound information table stored in a storage unit of the voice instrument according to the first embodiment of the present invention.

The song sound information consists of lyrics information of the song, pitch information of the song sound, and volume information of the song sound. The input unit 100 of the present invention receives the source song sound information input by the user. At this time, the source song sound information input to the input unit 100 is not input using a microphone for detecting the vibration of the air, but using the skin vibration sensor 110 for detecting the vibration of the skin in contact with the ball or around the neck. Can be input. The microphone due to simple air vibration is vulnerable to other ambient noises, and even if the noise is removed by applying artificial intelligence technology, the noise reduction calculation takes a lot of load. What's more, if two people sing at the same distance from the microphone, it's ambiguous to see whose voice is the source voice and whose voice is the noise. Therefore, in the voice musical instrument 10 of the present invention, in order to cut off such load and ambiguity, it is possible to input song sound information using the skin vibration sensor 110. Other than the person to which the skin vibration sensor 110 is attached, the noise or the voice of another person has little effect on the skin vibration sensor 110. In order to more accurately acquire and analyze song sound information, the input unit 100 may include two or more skin vibration sensors 110. An example of the skin vibration sensor 110 includes a crack sensor that simulates a spider's sense organs. The skin vibration sensor 110 may be manufactured by depositing a material having a hard property on a polymer material having a viscoelastic property and then bending the sensor to generate an artificial crack. For example, the viscoelastic material may be PUA (Polyurethane acrylate), and the material having hard properties may be platinum metal.

The storage unit 200 stores song sound related data including target song sound information. The song sound related data may include source song sound information as well as characteristics of the source song sound, algorithm variables for analyzing the source song sound, artificial intelligence variables, song sounds played by the voice instrument 10, and the like. Furthermore, the target song sound information may have a different tone from the source song sound information. In this case, the tone of the target song sound information may be selected as the tone of the person desired by the user. The tone of the person desired by the user may be the tone of the singer. The unit time may be determined first to obtain source song sound information or target song sound information. Usually, the unit time is preferably about 20 ms to 50 ms, but is not necessarily limited thereto. The target song sound information of the present invention is composed of song sound information of different pitches for each of a plurality of pieces of lyrics information that can be uttered for a given unit time. Here, the unit time of the target song sound information for acquiring the song sound information and the unit time of the source song sound information do not necessarily need to be the same. Rather, the unit time for acquiring the target song sound information is preferably longer than the unit time for acquiring the source song sound information.

Therefore, the target song sound information stored in the storage 200 may have a table structure. FIG. 2 is an example of a target song sound information table representing target song sound information stored in the storage unit 200. For convenience, N song sound information according to each scale from 1 octave to 8 octaves is shown. For example, all song sound information in the line (1) represents the song sound information [a] uttered by the lyrics information / a / according to each scale of each octave. Here, [a] of the scale G and [a] of the scale A correspond to the same lyrics information / a / and have the same pitch, but are denoted by the same symbol [a] to avoid unnecessary symbol complexity. All song sound information in line (2) shows the song information [i] which uttered the lyrics information / i / according to each scale of each octave. Of course, it may be difficult for a person (singer) to utter / a / or / i / etc according to all pitches from 1 octave to 8 octaves, so it is not necessary to fill every cell of the table of FIG. none. If there is no song sound information, it can be replaced by a close one.

The lyrics information may vary depending on the length of the unit time, but may be distinctive features, phonemes, lexicons, and graphemes. The International Phonetic Alphabet (IPA) has 107 sound symbols, 52 accents, and 4 intonations, which can be used to create a target song sound information table.

The voice musical instrument 10 according to the first embodiment of the present invention may perform utterance such as rap, pansori, peking opera, etc. by adjusting the pitch information and the volume information. In addition, it is possible to perform a function of speaking in the voice of another person by various vocal methods having accents, intonations and the like. In this case, the target song sound information may include rap, pansori, peking opera, accent and / or intonation data.

The output unit 300 for outputting target song sound information may directly include a speaker, but is not limited thereto. For example, it may be connected to another instrument such as a synthesizer to output target song sound information.

The controller 400 may process the source song sound information input through the input unit 100 and transmit the processed source song sound information to the output unit 300.

Hereinafter, a voice processing method through a voice instrument according to the present invention will be described.

The voice processing method includes receiving source song sound information including lyrics information, pitch information, and volume information through an input unit 100 including a skin vibration sensor 110 from a user, and source song sound from a storage unit 200. Acquiring target song sound information having the lyrics information and the pitch information corresponding to the lyrics information and the pitch information of the information; and amplifying the target song sound information by amplifying the acquired target song sound information according to the volume information of the source song sound information. And acquiring the amplified target song sound information as a song sound. In this case, the target song sound information may include tone information different from the source song sound information.

Hereinafter, a voice processing method of a controller through a voice musical instrument according to a first embodiment of the present invention will be described with reference to the drawings.

3 is a flowchart showing the function of the controller using the feature vector of the voice instrument according to the first embodiment of the present invention, and FIG. 4 is the source song sound information consisting of the feature vector of the voice instrument according to the first embodiment of the present invention. Table.

Referring to FIG. 3, in the voice processing method 1000 using the feature vector, in step 1001 of receiving source song sound information from the input unit 100, the input source song sound information is analyzed to analyze lyrics of the song sound according to time. Step 1002 of classifying the information, the pitch information, and the volume information, and the target having the lyrics information and the pitch information corresponding to the lyrics information and the pitch information of the source song sound information from the target song sound information stored in the storage unit 200. Matching and acquiring the song sound information (1003), acquiring the amplified target song sound information by acquiring the amplified target song sound information according to the volume information of the source song sound information (1004), and obtaining the amplified target A step 1005 of transmitting the song sound information to the output unit 300 may be included.

In operation 1002, the lyrics information of the song sound, the pitch information, and the volume information of the song sound may be classified by analyzing the input source song sound information to analyze the analog source song sound information input through the skin vibration sensor 110. Pre-processing step 1002a for converting to digital, extracting feature data of the source song sound information passed through the pre-processing step 1002b, and classifying the extracted feature data using a source table 1002c. .

As mentioned above, the main song sound information of the present invention includes lyrics information, pitch information, and volume information. Since the sound of a song is a physical wave, the volume information of the song corresponds to the amplitude of the wave. On the other hand, lyrics information and pitch information of a song are closely related to the frequency of the wave.

The input source song sound information may be analyzed and classified into lyrics information and pitch information by using a fast Fourier transform (FFT) spectrum. To analyze and classify the sound of a song, extract the characteristic data from the spectrum using energy, root men square (RMS), spectral centroid (SC), mel frequency cepstral coefficient (MFCC), cepstrum, etc. Lyric information and pitch information can be obtained.

For example, the MFCC method is a method of dividing a song sound into a given unit time to extract a feature of the song sound by analyzing a spectrum of the interval. More specifically, the power spectrum is analyzed for a unit time to calculate the power of the so-called Mel scale, and the logarithm is taken to calculate the discrete cosine transform (DCT) and then the coefficients ( Example: coefficients 2 to 13 can be obtained to extract feature data of lyrics information of source song sound information.

The pitch information of the source song sound information is to find the fundamental tone of the song sound. For example, there may be a method such as a capstrum. The capstrum method is obtained by taking a log of the spectrum of a song sound and performing an inverse Fourier transform, which can be used to classify the pitch information of the song sound. However, it is not necessarily limited to this method.

As described above, a feature vector having each feature as a component can be obtained. For example, a vector consisting of coefficients 2 to 13 obtained by the MFCC method and the pitch value p of the song sound (c2, ..., c13, p) are examples of feature vectors.

Referring to FIG. 4, in FIG. 2, the song sound corresponding to the lyrics information / a / is represented by [a], while in FIG. 4, characteristic values of the song lyrics information / a / are a ₁ ,..., A _n . when represented by a feature vector _{<a> = <a 1, ···, a n} >. Like the [a] symbol, the <a> symbol may be displayed differently according to the scale, but the same symbol <a> is indicated to avoid unnecessary symbol complexity.

The feature vectors of FIG. 4 may be stored in advance in the storage unit 200 as feature vectors of known song sound information input during a unit time. When the feature vectors for the source song sound information are stored in the source song sound information table as shown in FIG. 4, the lyrics information of the song sound is compared with the stored value by comparing the result of the input feature song sound information with the stored values. And the pitch information of the song sound can be determined and classified (1002c). In this case, the k-Nearest Neighbors (k-NN) algorithm may be used to determine the lyrics information and the pitch information of the song sound from k neighbors having the closest feature as a method of comparing the stored feature vectors. Variables necessary to calculate such an algorithm may be stored in the storage unit 200.

Hereinafter, a voice processing method of a controller through a voice musical instrument according to a modification of the first embodiment of the present invention will be described with reference to the drawings.

5 is a flowchart showing the function of the controller to which the artificial intelligence algorithm of the voice musical instrument according to the modification of the first embodiment of the present invention is applied, and FIG. 6A (a) shows a convolution according to the modification of the first embodiment of the present invention. 6A is a flowchart illustrating an AI learning method using a neural network, and FIG. 6A (b) is a flowchart illustrating a transformation process using a multilayer layer of a convolutional neural network, and FIG. 6B is a generative example according to a modification of the first embodiment of the present invention. It is a flowchart illustrating an AI learning method using a hostile network.

Referring to FIG. 5, in the voice processing method 1000A using the artificial intelligence algorithm according to the modification of the first embodiment of the present invention, the step 1001 of receiving source song sound information from the input unit 100 may be performed. Classifying lyrics information, pitch information, and volume information of the song sound over time by analyzing the sound information (1002A), the lyrics information and the pitch of the source song sound information from the target song sound information stored in the storage unit 200; Acquiring the target song sound information having the lyrics information corresponding to the information and the target information (1003), and amplifying the obtained target song sound information according to the volume information of the source song sound information to obtain the amplified target song sound information. The method may include a step 1004 of obtaining and a step 1005 of transmitting the amplified target song sound information to the output unit 300. From the input unit 100 which is common in the speech processing method 1000 using the feature vector according to the first embodiment of the present invention and the speech processing method 1000A using the artificial intelligence algorithm according to the modification of the first embodiment of the present invention. Step 1001 of receiving the source song sound information, the target song sound information having the lyrics information and the pitch information corresponding to the lyrics information and the pitch information of the source song sound information from the target song sound information stored in the storage unit 200. Acquiring (1003) and obtaining the amplified target song sound information by amplifying the acquired target song sound information according to the volume information of the source song sound information (1004), and obtaining the amplified target song sound information. In step 1005 of transmitting to the output unit 300 to output the same function may be performed.

At this time, by analyzing the input source song sound information to classify the lyrics information, pitch information and volume information of the song sound over time may be applied to the artificial intelligence algorithm. The method of classifying by applying an artificial intelligence algorithm includes preprocessing step 1002a of converting analog source song sound information input through the skin vibration sensor 110 into digital for analysis, and artificial intelligence in the source song sound information that has been subjected to the preprocessing step. Applying an algorithm 1002d and classifying using artificial intelligence 1002e may be included.

쌍으로 이루어진 노래 소리 정보 분포

를 획득할 수 있다. 타겟 노래 소리 정보 y(1103)는 가사 정보, 음높이 정보 및 볼륨 정보 등이 포함될 수 있다. 다만, 후술할 제2실시예처럼 제2입력장치(130)가 있는 경우, 타겟 노래 소리 정보 y(1103)는 가사 정보와 음높이 정보만으로 이루어 질 수 있다. 또한, 제1 및 제2입력장치(120, 130)가 있는 경우에는, 타겟 노래 소리 정보 y(1103)는 가사 정보로만 이루어 질 수 있다. 나아가 타겟 노래 소리 정보 y(1103)에서 분류하고자 하는 정보의 개수는 K라고 할 수 있다. 예컨대, 가사 정보만을 분류한다면 도 2에서 타겟 노래 소리 정보의 개수 N이 타겟 노래 소리 정보 y(1103)에서 분류하고자 하는 정보의 개수 K가 된다.When the source song sound information x (1101) is referred to as source song sound information for AI learning, the target song sound information y (1103) corresponding thereto is included.

Paired Song Sound Information Distribution

Can be obtained. The target song sound information y 1103 may include lyrics information, pitch information, volume information, and the like. However, when the second input device 130 is present as in the second embodiment to be described later, the target song sound information y 1103 may be composed of only lyrics information and pitch information. In addition, when the first and second input devices 120 and 130 are present, the target song sound information y 1103 may be composed only of lyrics information. Further, the number of information to be classified in the target song sound information y 1103 may be referred to as K. For example, if only the lyrics information is classified, the number N of the target song sound information in FIG. 2 is the number K of information to be classified in the target song sound information y 1103.

In such a situation, the AI learning method can learn by constructing various AI neural networks.

(1102)은 도 6a의 (b)에서 볼 수 있듯이 conv2D(1102a), ReLU(1102b), Conv2D(1102c), ReLU(1102d), Max Pooling(1102e), Dropout(1102f), conv2D(1102g), ReLU(1102h), Conv2D(1102i), ReLU(1102j), Max Pooling(1102k), Dropout(1102l), FC(1102m), ReLU(1102n), Dropout(1102o), FC(1102p), Softmax(1102q)와 같은 복수의 레이어들을 순차적으로 포함할 수 있다. 여기서 레이어들의 각 연산에 입출입되는 데이터의 차원에 대한 변환과 기타 다양한 옵션 등이 있을 수 있으나 본 발명의 핵심은 아니므로 불필요한 복잡성을 피하기 위하여 생략하였다. 또한, 상기 레이어 구성은 다양하게 변형될 수 있음은 당업자에게 자명하다.Referring to FIG. 6A, a learning method using a convolutional neural network 1100 is a 2D convolution (Conv2D) 1102a, 1102c, 1102g, 1102i, which mixes source song sound information x (1101) using a filter, and converts it to nonlinearity. Rectified Linear Unit (ReLU) (1102b, 1102d, 1102h, 1102j, 1102n), which is an activation function that maximizes the size of the information by selecting the maximum value in a given area, Max Pooling 1102e, 1102k), Dropout (1102f, 1102l, 1102o), one layer that selects only a percentage of the network from one layer of neurons to the next layer of neurons to facilitate convergence of AI learning FC (Fully Connected) (1102m, 1102p) which converts all the neurons of the next layer and the neurons of the next layer, of the number of information K to be classified in the target song sound information y (1103) for the input data. The probability of belonging to any one You can be configured with a SoftMax (Softmax) a combination of a multi-layered layer constituting the mathematical transformation operation to each of the layers one by one, such as (1102q). For example, in the learning method 1100 using the convolutional neural network, the convolutional neural network

1102, conv2D 1102a, ReLU 1102b, Conv2D 1102c, ReLU 1102d, Max Pooling 1102e, Dropout 1102f, conv2D 1102g, as shown in FIG. ReLU (1102h), Conv2D (1102i), ReLU (1102j), Max Pooling (1102k), Dropout (1102l), FC (1102m), ReLU (1102n), Dropout (1102o), FC (1102p), Softmax (1102q) It may include a plurality of layers in sequence. Here, there may be a transformation and various other options for the dimension of data input and output in each operation of the layers, but are omitted in order to avoid unnecessary complexity since it is not the core of the present invention. In addition, it will be apparent to those skilled in the art that the layer configuration may be variously modified.

로 표현할 수 있다. The input source song sound information x 1101 passes through a multi-layered layer as shown in FIG. 6A (b).

Can be expressed as

에 위치하는 소스 노래 소리 정보 x(1101)에 대하여 상기와 같은 컨벌류션 신경망

(1102)을 통하여 변환된 값

가 타겟 노래 소리 정보 y(1103)의 y = 1, 2, …, K 중 어느 하나가 될 확률인

에 대한 [수학식 1]과 같은 크로스 엔트로피 로스(cross-entropy loss)를 포함하는 비용함수를 최소화되도록 학습시킬 수 있다. 이때, 컨벌류션 신경망

(1102)를 표현하는데 필요한 인공지능 변수 벡터

가 학습되어 최적화될 수 있다. 대응되는 타겟 노래 소리 정보 y(1103)를 알고 있는 소스 노래 소리 정보 x(1101)는 소스 노래 소리 정보

로 표현될 수 있다. Learning method using the convolutional neural network (1100) is the distribution of song sound information

The convolutional neural network as described above for the source song sound information x (1101) located at

Value converted via (1102)

Y = 1, 2,... Of the target song sound information y (1103); , Which is the probability that either K

It can be learned to minimize the cost function including the cross-entropy loss (Equation 1) for. At this time, the convolutional neural network

AI variable vector required to represent (1102)

Can be learned and optimized. Source song sound information x (1101) knowing the corresponding target song sound information y (1103) is source song sound information.

It can be expressed as.

가 저장될 수 있다. 상기 학습되어 저장된 인공지능 변수 벡터

를 이용하여 입력부(100)로부터 단위 시간 동안 입력되는 소스 노래 소리 정보 x(1101)를 변환하여 분류할 수 있다. In this case, the storage unit 200 has an AI parameter vector optimized in place of the feature vectors constituting the source song sound information table of FIG. 4.

Can be stored. AI variable vector learned and stored

The source song sound information x 1101 input through the input unit 100 for a unit time may be converted and classified by using.

 [Equation 1]

(1202)와 생성자 네트워크

(1205)로 이루어질 수 있다. 판별자 네트워크

(1202)는 예컨데 conv2D, Add b1, ReLU, Ave Pooling, conv2D, Add b2, ReLU, Ave Pooling, MatMul, Add b3, ReLU, MatMul, Add b4와 같은 복수의 레이어들을 순차적으로 포함할 수 있다(미도시). 여기서 Add b1, b2, b3, b4는 b1, …, b4와 같은 변수값을 추가로 더하는 연산이며, Ave Pooling은 도 6a의 (b)의 Max Pooling(1102e, 1102k)과 다르게 주어진 영역에서 최대값 대신 평균값을 선택하여 정보의 크기를 줄이는 연산이며, MatMul연산은 행렬을 곱하는 연산이다. 여기서 레이어의 각 연산에 입출입되는 데이터의 차원에 대한 고려 및 기타 다양한 옵션 등이 있을 수 있으나 본 발명의 핵심은 아니므로 불필요한 복잡성을 피하기 위하여 생략하였다. 또한, 상기 레이어 구성은 다양하게 변형될 수 있음은 당업자에게 자명하다.Referring to FIG. 6B, the learning method 1200 using the generative antagonist network has two neural networks, that is, the discriminator network, unlike the learning method 1100 using the convolutional neural network.

Constructor network with 1202

1205. Discriminator network

1202 may sequentially include a plurality of layers such as conv2D, Add b1, ReLU, Ave Pooling, conv2D, Add b2, ReLU, Ave Pooling, MatMul, Add b3, ReLU, MatMul, Add b4 (not shown). city). Where Add b1, b2, b3, b4 are b1,... , b4 is an operation of adding a variable value such as A4, and Ave pooling is an operation of reducing the size of information by selecting an average value instead of the maximum value in a given region, unlike Max Pooling (1102e, 1102k) of FIG. The MatMul operation is a multiplication operation of a matrix. Here, there may be a consideration about the dimension of the data input and output to each operation of the layer and various other options, but it is not the core of the present invention, so it is omitted to avoid unnecessary complexity. In addition, it will be apparent to those skilled in the art that the layer configuration may be variously modified.

(1205)는 예컨데 MatMul, Add c1, BN, ReLU, conv2D, Add c2, BN, ReLU, conv2D, Add c3, BN, ReLU, conv2D, Add c4, sigmoid와 같은 복수의 레이어들을 순차적으로 포함할 수 있다(미도시). 여기서 BN(Batch Normalization)은 도 6a의 (b)의 Dropout(1102f, 1102l, 1102o)처럼 인공지능 학습의 수렴이 용이하도록 하는 변환 중 하나이며, Sigmoid는 도 6a의 (b)의 ReLU(1102b, 1102d, 1102h, 1102j, 1102n) 와 같은 비선형 변환인 활성화 함수이다. 여기서 레이어의 각 연산에 입출입되는 데이터의 차원에 대한 고려 및 기타 다양한 옵션 등이 있을 수 있으나 본 발명의 핵심은 아니므로 불필요한 복잡성을 피하기 위하여 생략하였다. 또한, 상기 레이어 구성은 다양하게 변형될 수 있음은 당업자에게 자명하다.Constructor network

1205 may sequentially include a plurality of layers such as, for example, MatMul, Add c1, BN, ReLU, conv2D, Add c2, BN, ReLU, conv2D, Add c3, BN, ReLU, conv2D, Add c4, and sigmoid. (Not shown). Here, BN (Batch Normalization) is one of transformations to facilitate convergence of AI learning, such as Dropout (1102f, 1102l, 1102o) of FIG. 6A, and Sigmoid is ReLU 1102b, of FIG. 6B. 1102d, 1102h, 1102j, 1102n). Here, there may be a consideration about the dimension of the data input and output to each operation of the layer and various other options, but it is not the core of the present invention, so it is omitted to avoid unnecessary complexity. In addition, it will be apparent to those skilled in the art that the layer configuration may be variously modified.

쌍으로 이루어진 노래 소리 정보 분포

와, 소스 노래 소리 정보 x(1201)에 대응되는 타겟 노래 소리 정보 y(1203)는 모르고 소스 노래 소리 정보 x(1201)만으로 이루어진 노래 소리 정보 분포

로 이루어 질 수 있다. 나아가 분류할 타겟 노래 소리 정보 y(1203)에서 분류하고자 하는 정보의 개수는 컨벌류션 신경망을 이용한 학습 방법(1100)에서처럼 타겟 노래 소리 정보 y(1203)의 참(1203a)에 해당되는 y = 1, 2, …, K의 K개인 것 외에 거짓(1203b)에 해당되는 y = K+1인 것 1개를 더 포함하여 K+1개가 될 수 있다. In the learning method 1200 using the generative antagonist network, distribution of song sound information may be performed in two types. Specifically, when the source song sound information is referred to as the source song sound information x (1201), the target song sound information y (1203) corresponding thereto is included.

Paired Song Sound Information Distribution

And the song sound information distribution comprising only the source song sound information x 1201 without knowing the target song sound information y 1203 corresponding to the source song sound information x 1201.

Can be done with. Further, the number of information to be classified in the target song sound information y 1203 to be classified is y = 1, corresponding to true 1203a of the target song sound information y 1203, as in the learning method 1100 using the convolutional neural network. 2, … , K may be K + 1, in addition to K being K, and one more y = K + 1 corresponding to false 1203b.

에 위치하는 소스 노래 소리 정보 x(1201)는 판별자 네트워크

(1202)를 통하여 변환된 값

가 타겟 노래 소리 정보 y(1203)의 참(1203a)에 해당되는 y = 1,2, …, K 중 어느 하나가 될 확률

에 대하여 [수학식 1]과 같은 크로스 엔트로피 로스와, 노래 소리 정보 분포

에 위치하는 소스 노래 소리 정보 x(1201)가 판별자 네트워크

(1202) 통하여 변환된 값

가 타겟 노래 소리 정보 y(1203)의 참(1203a)에 해당되는 y = 1, 2, …, K에 속하는 확률, 즉 타겟 노래 소리 정보 y(1203)의 거짓(1203b)에 해당되는 y = K+1에 속하지 않도록 하는 확률

에 대한 [수학식 2]와 같은 크로스 엔트로피 로스와, 노이즈 데이터 z(1204)를 생성자 네트워크

(1205)를 통하여 변환한 후 생성된 소리 정보

(1206)를 판별자 네트워크

(1202)를 거쳐 노래 소리가 거짓(1203b)인 것처럼, 즉 타겟 노래 소리 정보 y(1203)의 거짓(1203b)에 해당되는 y = K+1에 속하도록 하는 확률에 대한 [수학식 3]과 같은 크로스 엔트로피 로스를 포함하는 비용함수의 조합이 최소화가 되도록 생성자 네트워크

(1205)를 고정시키고 판별자 네트워크

(1202)를 일정 횟수 학습시킬 수 있다.In this situation, song sound information distribution

Source song sound information x (1201) located in the discriminator network

Value converted through (1202)

Corresponds to true 1203a of target song sound information y 1203,. , Any probability of K

With respect to the cross entropy loss such as [Equation 1], song sound information distribution

Source song sound information x (1201) located in the discriminator network

Value converted through (1202)

Corresponds to true 1203a of target song sound information y 1203, y = 1, 2,... , The probability belonging to K, that is, the probability not to belong to y = K + 1 corresponding to the false 1203b of the target song sound information y 1203

Network generator with cross-entropy loss, noise data z (1204), such as

Sound information generated after the conversion through 1205

Discriminator Network (1206)

Equation 3 for the probability that through 1202 the song sound is false 1203b, i.e., y = K + 1 corresponding to false 1203b of the target song sound information y 1203. Constructor network so that the combination of cost functions containing the same cross entropy loss is minimized

Fixed 1205 and discriminator network

1202 can be learned a certain number of times.

 [Equation 2]

[Equation 3]

(1205)를 통하여 변환한 후 생성된 소리 정보

(1206)를 판별자 네트워크

(1202)를 거쳐 노래 소리가 실제(real)인 것처럼, 즉 타겟 노래 소리 정보 y(1203)의 참(1203a)에 해당되는 y = 1, 2, …, K에 속하도록 하는 확률에 대한 [수학식 4]와 같은 크로스 엔트로피 로스를 포함하는 비용함수가 최소화되도록 판별자 네트워크

(1202)를 고정시키고 생성자 네트워크

(1205)를 일정 횟수 학습시킬 수 있다.Further generation of noise data z (1204) network

Sound information generated after the conversion through 1205

Discriminator Network (1206)

Through 1202, y = 1, 2,... That corresponds to the true 1203a of the target song sound information y 1203, as if the song sound is real. , Discriminator network to minimize cost functions including cross-entropy losses, such as Eq.

Fixed 1202 constructor network

1205 can be learned a certain number of times.

[Equation 4]

(1202)와 생성자 네트워크

(1205)에 대한 인공지능 학습을 번갈아 가며 학습시키면 판별자 네트워크

(1202)와 생성자 네트워크

(1205)를 표현하는데 필요한 복수의 변수로 이루어진 판별자 네트워크 인공지능 변수 벡터

와 생성자 네트워크 인공지능 변수 벡터

가 학습되어 최적화될 수 있다. 이 경우, 저장부(200)에는 도 4의 소스 노래 소리 정보의 테이블을 구성하는 특징벡터들 대신에 최적화된 판별자 네트워크 인공지능 변수 벡터

가 저장될 수 있다. 학습되어 저장된 최적화된 판별자 네트워크 인공지능 변수 벡터

를 이용하여 입력부(100)로부터 단위 시간 동안 입력되는 소스 노래 소리 정보 x(1201)를 변환하여 분류할 수 있다. 물론, 저장부(200)에는 생성자 네트워크 인공지능 변수 벡터

도 저장할 수 있고, 이를 이용해서는 타겟 노래 소리 정보를 생성할 수 있다. 즉, 생성적 적대 네트워크를 이용한 학습 방법(1200)을 이용한 학습은 노이즈 데이터 z(1204)로부터 생성된 소리 정보

(1206)가 참(1203a)이 될 확률 0.5, 거짓(1203b)이 될 확률 0.5에서 종료되는데 이때 생성자 네트워크

(1205)를 통하여 생성된 소리 정보

(1206)는 참(1203a)과 거의 구분이 불가능하게 된다. 따라서 타겟 노래 소리 정보 y(1203)의 가사 정보 및 음높이 정보까지 분류하는 생성적 적대 네트워크를 이용한 학습 방법(1200)에서 생성된 소리 정보

(1206)은 타겟 노래 소리 정보로 활용될 수 있다. Discriminator network as above

Constructor network with 1202

Alternate AI Learning for (1205) for Discriminator Networks

Constructor network with 1202

Discriminator network AI variable vector consisting of a plurality of variables required to represent (1205)

And constructor network AI variable vector

Can be learned and optimized. In this case, the storage unit 200 has an optimized discriminator network AI parameter vector instead of feature vectors constituting the table of source song sound information of FIG. 4.

Can be stored. Learned and Stored Optimized Discriminator Network AI Variable Vectors

The source song sound information x 1201 input from the input unit 100 for a unit time may be converted and classified by using. Of course, the storage unit 200 has a generator network AI variable vector

It can also be stored, and can be used to generate target song sound information. That is, the learning using the learning method 1200 using the generative antagonist network includes sound information generated from the noise data z 1204.

(1206) ends with a probability 0.5 of true 1203a, and a probability 0.5 of false 1203b, where the producer network

Sound information generated through 1205

1206 becomes almost indistinguishable from true 1203a. Therefore, the sound information generated by the learning method 1200 using the generative hostile network classifying the lyrics information and the pitch information of the target song sound information y 1203.

1206 may be used as target song sound information.

(1202)와 생성자 네트워크

(1205)에 대한 인공지능 학습을 번갈아 가며 학습시키는 생성적 적대 네트워크를 이용한 학습 방법(1200)은 컨벌류션 신경망을 이용한 학습 방법(1100)보다 적은 수의 노래 소리 정보 분포

를 가지고도 소스 노래 소리 정보 x(1201)를 더 잘 분류할 수도 있다.Discriminator network as above

Constructor network with 1202

The learning method 1200 using a generative hostile network alternately learning AI learning about 1205 has less distribution of song sound information than the learning method 1100 using a convolutional neural network.

Even with the source song sound information x (1201) can be classified better.

At this time, each source song sound information x (1101, 1201) may be the waveform of the song sound itself, but the feature vector or the sound of the song sound such as <a> of FIG. 4 is converted to the brightness of the pixel in the time and frequency planes. It may be a spectrogram, which is a expressed two-dimensional image.

Here, as the volume information of the source song sound information x (1101, 1201) can be analyzed and classified separately, the source song sound information x (1101) is set to K, which is the number of information to be classified in the target song sound information y (1103, 1203). 1201) may also include pitch information, but only the lyrics information of the source song sound information x (1101, 1201) may be included and the pitch information may be separately analyzed and classified by the above-described capstrum method. That is, even in this case, as shown in FIG. 2, if the number of lyrics information is N, K = N, and only the lyrics information of the song sound can be artificially learned.

(1102) 또는 판별자 네트워크

(1202)의 계산 결과에 의하여 분류되어 타겟 노래 소리 정보 테이블의 각 요소로 매칭될 수 있다. 물론 인공지능 변수 벡터 외에 인공지능 상수값 등 인공지능 알고리즘 관련 데이터들도 저장부(200)에 저장될 수 있다. 이때, 인공지능 변수 벡터는 컨벌류션 신경망을 이용한 학습 방법(1100)에 의한 변수 벡터

, 생성적 적대 네트워크를 이용한 학습 방법(1200)에 의한 판별자 네트워크 인공지능 변수 벡터

및 생성자 네트워크 인공지능 변수 벡터

를 포함할 수 있다. After analyzing the input source song sound information and classifying lyrics information, pitch information, and volume information of the source song sound information over time, the target song sound information stored in the storage unit 200, such as source song sound information Song sound information having pitch information such as lyrics information may be matched and acquired (1003). In this case, when the voice processing method of FIG. 3 includes the feature vectors constituting the source song sound information table and is stored in the storage unit 200, each of the target song sound information tables corresponding to the corresponding position of the source song sound information table is stored. Can be matched to an element. On the other hand, when the artificial intelligence variable vector is stored by the speech processing method of FIG. 5, the convolutional neural network

(1102) or discriminator network

The result of the calculation of 1202 may be classified to match each element of the target song sound information table. Of course, in addition to the AI variable vector, AI algorithm related data such as AI constant values may also be stored in the storage unit 200. At this time, the AI variable vector is a variable vector by a learning method 1100 using a convolutional neural network.

Discriminator Network AI Variable Vector by Learning Method Using Genetic Hostile Network (1200)

And constructor network AI variables vector

It may include.

The unit time of the target song sound information stored in the storage 200 may be longer than the unit time of the input source song sound information so that the waveform of the output song sound is continuously connected. In this case, the target song sound information matching the source song sound information input for the unit time is clipped from the corresponding target song sound information in the target song sound information table so that it is continuously connected with the end of the waveform of the immediately matched target song sound information. It can be obtained by clipping. In addition, the input source song sound information may be superimposed and analyzed and classified.

The obtained target song sound information is amplified according to the classified volume information (1004) and transmitted to the output unit 300 to output the amplified target song sound information (1005). Amplifying the electrical signal and transmitting the output signal to the output unit 300 are well known methods used in the art, and thus detailed description thereof will be omitted.

Hereinafter, the configuration of a voice musical instrument according to a second embodiment of the present invention will be described with reference to the drawings.

7 is a block diagram of a voice musical instrument according to a second embodiment of the present invention.

The voice instrument 10 according to the first embodiment of the present invention classifies source song sound information input through the skin vibration sensor 110 of the input unit 100, and stores the target song previously stored in the storage unit 200. Compared to the sound information, the matching is obtained, and the obtained target song sound information is amplified and output. In this case, all of the lyrics information, the pitch information and the volume information can be obtained from the source song sound information input through the skin vibration sensor 110.

However, only the lyrics information is obtained from the source song sound information input from the skin vibration sensor 110, and a first input device 120 for inputting pitch information and a second input device 130 for inputting volume information are added. It is also possible to directly input the pitch information and volume information of the source song sound information.

Referring to FIG. 7, the input unit 100 may further include a first input device 120 for inputting pitch information of source song sound information. A keyboard such as a piano keyboard may be regarded as an example of the first input device 120 for inputting pitch information of source song sound information. Of course, the piano keyboard can be implemented in various ways such as touch buttons.

In addition, the input unit 100 may further include a second input device 130 for inputting volume information of source song sound information. The slide volume controller for adjusting the volume by changing the voltage by moving the electrode 131 on the linear resistor 132 may be an example of the second input device 130 for inputting volume information of the source song sound information. Of course, it can be implemented in a variety of ways, such as scaffolding volume control, rotary volume control rather than the slide volume control.

In addition, input and output devices 500 and 600 for inputting, outputting, selecting, storing, changing, and / or deleting song sound related data including source song sound information and / or target song sound information are stored in the storage unit 200. It may be further included. That is, the output device 500 for displaying, inputting, outputting, selecting, storing, changing, and / or deleting song sound related data such as source song sound information and / or target song sound information, and source song sound information and / or target song. The apparatus may further include an input device 600 for inputting, outputting, selecting, storing, changing, and / or deleting song sound related data such as sound information.

Furthermore, the target song sound information may be stored in two or more groups using the input / output devices 500 and 600. That is, m (m> 1) group can be saved. For example, a target song sound information table m group for the target song singers T1, ..., Tm (m> 1) is stored, and the selected singer voice is selected using the input / output devices 500 and 600, and then the selected group. Matches and amplifies the target song sound information can be output. Thus, double glazing, triple glazing and the like are also possible. At this time, it is obvious that information on the chord can also be input and output using the input and output devices (500, 600).

Furthermore, a plurality of people S1,... For example, one or more source song sound information tables for Sk (k> 1) may be stored. That is, when S1 plays, not only the feature vectors of S1 may be stored in the storage unit 200 in advance, but also the feature vectors of S2 may be stored in order to play S2. The error of recognition can be further reduced by storing the feature vectors of the player himself. When S2 wants to sing in the voice of T3, the source song sound information S2 or the target song sound information T3 can be selected using an output device 500 such as a display and an input device 600 such as a keyboard.

The apparatus 700 may also include a communication device 700 for downloading and / or uploading song sound related data including source song sound information and / or target song sound information. That is, a communication device 700 for downloading and / or uploading a target voice or the like from an electronic device such as the Internet or another smart phone may be included.

Hereinafter, a computer program for implementing a voice processing method using a voice instrument of any one of the first and second embodiments of the present invention will be described.

The computer program may be a computer program stored in a recording medium in order to implement the voice processing method through the voice instrument 10 of any one of the first and second embodiments of the present invention. In order to implement the voice processing method, a computer program stored in a recording medium may be executed by a general purpose computer or a dedicated computer.

The voice processing method via the voice instrument 10 of any one of the first and second embodiments according to the present invention may be stored in a computer readable recording medium having recorded thereon a program for executing the voice processing method on a computer. . Recording media include all types of recording media on which programs and / or data are stored for reading by a computer system. For example, magnetic disks such as magnetic tape, flexible disk or hard disk, optical disk such as CD or DVD, magneto-optical disk, semiconductor memory such as USB memory or memory card. The voice processing method can be executed by installing a program in a general-purpose computer using such a recording medium.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (15)

An input unit configured to receive source song sound information including lyrics information, pitch information, and volume information from a user;
A storage unit for storing song sound related data including target song sound information including lyrics information and pitch information;
Obtaining the target song sound information having the lyrics information of the source song sound information, the lyrics information corresponding to the pitch information, and the pitch information from the storage unit, and obtaining the obtained target song sound information from the source song sound information; A control unit for amplifying the target song sound information by amplifying according to the volume information; And
An output unit for outputting the amplified target song sound information,
The input unit includes a skin vibration sensor for receiving at least one of the lyrics information, the pitch information and the volume information of the source song sound information,
The target song sound information includes tone information different from the source song sound information,
And the controller classifies the lyrics information from the source song sound information input from the input unit by using an AI parameter vector trained using an AI network. The method of claim 1,
The input unit further includes a first input device for receiving the pitch information of the source song sound information,
And the skin vibration sensor receives the lyrics information and the volume information of the source song sound information when the first input device receives the pitch information of the source song sound information. The method of claim 1,
The input unit further includes a second input device for receiving the volume information of the source song sound information,
And the skin vibration sensor receives the lyrics information and the pitch information of the source song sound information when the second input device receives the volume information of the source song sound information. The method of claim 1,
The input unit may include a first input device configured to receive the pitch information of the source song sound information; And
The input unit includes a second input device for receiving the volume information of the source song sound information,
When the first input device receives the pitch information of the source song sound information and the second input device receives the volume information of the source song sound information, the skin vibration sensor is configured to perform the lyrics of the source song sound information. Voice instrument to receive information. The method of claim 1,
The song sound related data stored in the storage unit includes a target song sound information table having different pitch information for each of a plurality of pieces of lyrics information that can be spoken for a unit time. The method of claim 1,
And a unit time of the target song sound information stored in the storage unit is longer than a unit time of the source song sound information. The method of claim 1,
The storage unit stores a plurality of feature vectors or artificial intelligence variable vector for classifying a plurality of lyrics information that can be spoken for a unit time. The method of claim 1,
And the storage unit includes at least two groups of the target song sound information. The method of claim 1,
And an input device and an output device for inputting, outputting, selecting, storing, changing, and / or deleting the source song sound information and / or the song sound related data. The method of claim 1,
And a communication device for downloading and / or uploading the source song sound information and / or the song sound related data. The method of claim 1,
The artificial intelligence network is a convolutional neural network (Convolution Neural Network, CNN) or a generative adversarial network (GAN). The method of claim 1,
The target song sound information is generated using a generator network AI variable vector trained with a generative hostile network. The method of claim 1,
The target song sound information includes rap, pansori, peking opera, accent and / or intonation data. Receiving at least one of the lyrics information, the pitch information, and the volume information of the source song sound information including lyrics information, pitch information, and volume information through a skin vibration sensor of an input unit;
Obtaining target song sound information having lyrics information and pitch information corresponding to the lyrics information of the source song sound information and the pitch information from a storage unit;
Amplifying the acquired target song sound information according to the volume information of the source song sound information to obtain amplified target song sound information; And
And outputting the amplified target song sound information as a song sound,
The target song sound information includes tone information different from the source song sound information,
The acquiring of the target song sound information from the storage unit may include classifying the lyrics information by using an AI variable vector trained using an artificial intelligence network from the source song sound information input from the input unit. Voice processing method. A computer program stored in a recording medium for implementing the voice processing method of claim 14.

Images