KR102621021B1 - Method for training speech transformation model to generate emotion-neutral speech - Google Patents

Abstract

A method of training a speech conversion model performed by a computing device according to an embodiment of the present disclosure is disclosed. The method includes obtaining voice data for training; Converting the learning voice data into emotion-neutral voice data from which emotions have been removed, using a voice conversion model; and training the speech conversion model using a loss function associated with the converted emotion-neutral speech data, wherein the speech conversion model can be trained to generate an emotion-neutral speech.

Description

Method for training a speech transformation model that generates emotion-neutral speech {METHOD FOR TRAINING SPEECH TRANSFORMATION MODEL TO GENERATE EMOTION-NEUTRAL SPEECH}

The present invention relates to a method for learning a voice conversion model, and more specifically, to a technology for learning a voice conversion model that generates an emotion-neutral voice that does not contain emotions.

As voice recognition technology develops, the use of more convenient interfaces between humans and machines is rapidly increasing. Recently, as the practical use of voice recognition systems has increased, application products that can be useful in real life are being developed. Currently, voice recognition technology has advanced considerably, recognizing hundreds of thousands of words in vocabulary, and recognition performance is also improving to the point where it can be put to practical use.

However, the problem that voice recognition technology still has is that the performance of these systems is greatly dependent on environmental changes such as surrounding noise and channel characteristics and psychological changes such as emotional states.

In particular, in the case of speech processing models such as speech recognition (STT; Speech To Text) model, speaker verification model, speaker separation model, and Voice Activity Detection (VAD) model, inference ( Since the ability to inference is greatly affected, emotions are often taken into consideration when learning these speech processing models and new models are often learned or created according to emotions.

Republic of Korea Patent No. 10-2171559 (2020.10.23) discloses a method of generating training data for a voice synthesis model and a method of learning a voice synthesis model.

The purpose of the present disclosure is to generate emotion-neutral speech in order to improve the performance of various speech processing techniques that are heavily influenced by emotions. For example, the purpose of the present disclosure is to provide “a model for converting and outputting voice data containing emotions into emotion-neutral voice data that does not contain emotions when voice data containing emotions is obtained.”

Meanwhile, the technical problem to be achieved by the present disclosure is not limited to the technical problems mentioned above, and may include various technical problems within the scope of what is apparent to those skilled in the art from the contents described below.

According to an embodiment of the present disclosure for realizing the above-described problem, a method of learning a voice conversion model performed by a computing device is disclosed. The method includes obtaining voice data for training; Converting the learning voice data into emotion-neutral voice data from which emotions have been removed, using a voice conversion model; and training the speech conversion model using a loss function associated with the converted emotion-neutral speech data, wherein the speech conversion model can be trained to generate an emotion-neutral speech.

In one embodiment, the step of acquiring the voice data for learning includes acquiring a voice data set for learning, wherein the voice data set for learning includes emotional voice data for learning including emotions; And it may include emotion-neutral voice data for learning that does not contain emotions.

In one embodiment, the step of acquiring the voice data set for learning includes outputting example sentences and emotion guide information through a user interface; Obtaining emotional voice data for learning including the emotion based on the user interface; and acquiring emotion-neutral voice data for learning that does not contain the emotion based on the user interface.

In one embodiment, the step of converting the learning voice data into emotion-neutral voice data from which emotions are removed includes removing emotions from the learning emotion voice data containing the emotions using a voice conversion model, thereby removing the converted emotions. It may include generating neutral voice data.

In one embodiment, the step of training the speech conversion model using a loss function associated with the converted emotion-neutral speech data, based on a distance between the converted emotion-neutral speech data and the emotion-neutral speech data for training, It may include calculating a first loss function.

In one embodiment, the emotional voice data for training corresponds to a 1-1 voice vector, and the converted emotion-neutral voice data generated based on the emotional voice data for training corresponds to a 1-2 voice vector, The emotion-neutral voice data for learning corresponds to a second voice vector, and the distance between the converted emotion-neutral voice data and the emotion-neutral voice data for learning is based on the distance between the 1-2 voice vector and the second voice vector. Calculated, the 1-1st voice vector, the 1-2nd voice vector, and the second voice vector may correspond to vectors having the length of the voice as a dimension.

In one embodiment, the step of training the speech conversion model using a loss function associated with the converted emotion-neutral speech data includes analyzing a relationship between the converted emotion-neutral speech data and an emotion variable. 2 calculating a loss function; And it may further include calculating a final loss function based on the first loss function and the second loss function.

In one embodiment, calculating a second loss function based on analyzing a relationship between the converted emotion-neutral speech data and an emotion variable comprises: based on a speech vector corresponding to the converted emotion-neutral speech data; extracting a feature vector; and calculating the second loss function based on analysis of a correlation relationship between the extracted feature vector and the emotional variable.

According to an embodiment of the present disclosure for realizing the above-described problem, a method for preventing changes in voice processing performance due to changes in emotions performed by a computing device is disclosed. The method includes obtaining voice data; Converting the voice data into emotion-neutral voice data from which emotions are removed using a voice conversion model; And it may include inputting the converted emotion-neutral voice data into a voice processing model.

A neural network structure for processing voice data is disclosed according to an embodiment of the present disclosure for realizing the above-described problem. The neural network structure includes a voice conversion model for outputting emotion-neutral voice data with emotions removed from the original voice data; and a voice processing model for receiving the emotion-neutral voice data from the voice conversion model and processing the emotion-neutral voice data from which the emotion has been removed instead of the original voice data.

In one embodiment, the speech processing model may correspond to a model learned based on emotion-neutral speech data instead of speech data associated with a plurality of emotions.

In one embodiment, the speech processing model may include an architecture that is independent of a plurality of emotions, instead of an architecture that includes a plurality of emotion processing modules learned for a plurality of emotions.

In one embodiment, the speech processing model may include at least one of a speech recognition (STT; Speech To Text) model, a speaker verification model, a speaker separation model, or a Voice Activity Detection (VAD) model. .

According to an embodiment of the present disclosure for realizing the above-described object, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it causes the one or more processors to perform the following operations for training a speech conversion model, which operations include: acquiring speech data for training; Converting the learning voice data into emotion-neutral voice data from which emotions have been removed, using a voice conversion model; and an operation of training the speech conversion model using a loss function associated with the converted emotion-neutral speech data, wherein the speech conversion model can be trained to generate an emotion-neutral speech.

A computing device according to an embodiment of the present disclosure for realizing the above-described problem is disclosed. The device includes at least one processor; and a memory, wherein the at least one processor acquires voice data for training; Converting the learning voice data into emotion-neutral voice data from which emotions have been removed using a voice conversion model; And configured to learn the speech conversion model using a loss function associated with the converted emotion-neutral speech data, and the speech conversion model can be trained to generate an emotion-neutral speech.

The present disclosure can provide a technology for generating emotion-neutral speech in order to improve the performance of various speech processing techniques that are heavily influenced by emotion. For example, the present disclosure may provide “a model for converting and outputting voice data containing emotions into emotion-neutral voice data that does not contain emotions when voice data containing emotions is acquired.”

In addition, the present disclosure can significantly improve the performance of an existing speech processing model without changing the pipeline of the existing speech processing model, with respect to processing of speech containing emotions. For example, the present disclosure utilizes a “model for generating a voice with emotions removed from the input voice” as a kind of pre-processing model by attaching it to the front end of the existing voice processing model, thereby creating a pipeline of the existing voice processing model. The performance of existing speech processing models can be significantly improved without changes.

Additionally, the present disclosure can significantly save resources that would otherwise be required in connection with the processing of speech containing emotion. For example, the present disclosure can save resources required for emotion processing in the learning stage of various speech processing models, and typically solves the problem of having to learn separate models for each emotion for speech processing. . For example, the present disclosure generates a voice with the emotions of the input voice removed through a pre-processing process, so it is necessary to train a voice processing model only for emotion-neutral voices, and a separate learning process for each emotion is required or the emotion is removed. There is no need to have models with very different structures.

Additionally, the present disclosure can facilitate the acquisition of training data for speech processing models. For example, according to the present disclosure, it is sufficient to use only emotion-neutral voices (since emotions are removed through pre-processing) to train a speech processing model, since these emotion-neutral voices are the voices with the most learning data. , the process of acquiring training data for speech processing models can be made very easy.

Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a network function according to an embodiment of the present disclosure.
Figure 3 is a diagram schematically showing a plurality of modules for training a voice conversion model according to an embodiment of the present disclosure.
Figure 4 is a diagram schematically showing a neural network structure for processing voice data according to an embodiment of the present disclosure.
Figure 5 is a schematic flowchart of a method for learning a speech conversion model according to an embodiment of the present disclosure.
Figure 6 is a schematic flowchart of a method for preventing changes in voice processing performance due to changes in emotions, according to an embodiment of the present disclosure.
7 is a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the disclosure. However, it is clear that these embodiments may be practiced without these specific descriptions.

As used herein, the terms “component,” “module,” “system,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an implementation of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. Components can transmit signals, for example, with one or more data packets (e.g., data and/or signals from one component interacting with other components in a local system, a distributed system, to other systems and over a network such as the Internet). Depending on the data being transmitted, they may communicate through local and/or remote processes.

Additionally, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the related listed items.

Additionally, the terms “comprise” and/or “comprising” should be understood to mean that the corresponding feature and/or element is present. However, the terms “comprise” and/or “comprising” should be understood as not excluding the presence or addition of one or more other features, elements and/or groups thereof. Additionally, unless otherwise specified or the context is clear to indicate a singular form, the singular terms herein and in the claims should generally be construed to mean “one or more.”

And, the term “at least one of A or B” should be interpreted to mean “a case containing only A,” “a case containing only B,” and “a case of combining A and B.”

Those skilled in the art will additionally recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or a combination of both. It must be recognized that it can be implemented with To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. A skilled technician can implement the described functionality in a variety of ways for each specific application. However, such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

The description of the presented embodiments is provided to enable anyone skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Therefore, the present invention is not limited to the embodiments presented herein. The present invention is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

In this disclosure, network function, artificial neural network, and neural network may be used interchangeably.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include different configurations for performing the computing environment of the computing device 100, and only some of the disclosed configurations may configure the computing device 100.

The computing device 100 may include a processor 110, a memory 130, and a network unit 150.

The processor 110 may be composed of one or more cores, and may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit) may include a processor for data analysis and deep learning. The processor 110 may read a computer program stored in the memory 130 and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 is used for learning neural networks, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Calculations can be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, CPU and GPGPU can work together to process learning of network functions and data classification using network functions. Additionally, in one embodiment of the present disclosure, the processors of a plurality of computing devices can be used together to process learning of network functions and data classification using network functions. Additionally, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150.

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, hard disk type, multimedia card micro type, or card type memory (e.g. (e.g. SD or -Only Memory), and may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in connection with web storage that performs a storage function of the memory 130 on the Internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

The network unit 150 according to an embodiment of the present disclosure includes Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), and VDSL ( A variety of wired communication systems can be used, such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN).

In addition, the network unit 150 presented in this specification includes Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), and SC-FDMA ( A variety of wireless communication systems can be used, such as Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit 150 may be configured regardless of communication mode, such as wired or wireless, and may include a local area network (LAN), a personal area network (PAN), or a wide area network (WAN). It can be composed of various communication networks such as Wide Area Network. Additionally, the network may be the well-known World Wide Web (WWW), or may use wireless transmission technology used for short-distance communication, such as Infrared Data Association (IrDA) or Bluetooth.

The techniques described herein can be used in the networks mentioned above, as well as other networks.

Figure 2 is a schematic diagram showing a network function according to an embodiment of the present disclosure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node. Nodes (or neurons) that make up neural networks may be interconnected by one or more links.

Within a neural network, one or more nodes connected through a link may form a relative input node and output node relationship. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the data of the output node may be determined based on the data input to the input node. Here, the link connecting the input node and the output node may have a weight. Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network can be determined according to the number of nodes and links within the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if the same number of nodes and links exist and two neural networks with different weight values of the links exist, the two neural networks may be recognized as different from each other.

A neural network may consist of a set of one or more nodes. A subset of nodes that make up a neural network can form a layer. Some of the nodes constituting the neural network may form one layer based on the distances from the first input node. For example, a set of nodes with a distance n from the initial input node may constitute n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanation purposes, and the order of a layer within a neural network may be defined in a different way than described above. For example, a layer of nodes may be defined by distance from the final output node.

The initial input node may refer to one or more nodes in the neural network through which data is directly input without going through links in relationships with other nodes. Alternatively, in a neural network network, in the relationship between nodes based on links, it may mean nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes that do not have an output node in their relationship with other nodes among the nodes in the neural network. Additionally, hidden nodes may refer to nodes constituting a neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. You can. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. You can. A neural network according to another embodiment of the present disclosure may be a neural network that is a combination of the above-described neural networks.

A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to the input layer and output layer. Deep neural networks allow you to identify latent structures in data. In other words, it is possible to identify the potential structure of a photo, text, video, voice, or music (e.g., what object is in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.) . Deep neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), auto encoders, generative adversarial networks (GAN), and restricted Boltzmann machines (RBM). machine), deep belief network (DBN), Q network, U network, Siamese network, Generative Adversarial Network (GAN), etc. The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network to output output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be placed between input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically and reduced from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform nonlinear dimensionality reduction. The number of input layers and output layers can be corresponded to the dimension after preprocessing of the input data. In an auto-encoder structure, the number of nodes in the hidden layer included in the encoder may have a structure that decreases as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, not enough information may be conveyed, so if it is higher than a certain number (e.g., more than half of the input layers, etc.) ) may be maintained.

A neural network uses at least one of supervised learning, self-supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. It can be learned. Learning of a neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

Neural networks can be trained to minimize output errors. In neural network learning, learning data is repeatedly input into the neural network, the output of the neural network and the error of the target for the learning data are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. This is the process of updating the weight of each node in the neural network through backpropagation. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (i.e., labeled learning data), and in the case of non-teacher learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of teacher learning regarding data classification, the learning data may be data in which each learning data is labeled with a category. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of non-teachable learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weight of each node in each layer of the neural network can be updated according to backpropagation. The amount of change in the connection weight of each updated node may be determined according to the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, in the early stages of neural network training, a high learning rate can be used to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stages of training, a low learning rate can be used to increase accuracy.

In the learning of neural networks, the training data can generally be a subset of real data (i.e., the data to be processed using the learned neural network), and thus the error for the training data is reduced, but the error for the real data is reduced. There may be an incremental learning cycle. Overfitting is a phenomenon in which errors in actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that learned a cat by showing a yellow cat fails to recognize that it is a cat when it sees a non-yellow cat may be a type of overfitting. Overfitting can cause errors in machine learning algorithms to increase. To prevent such overfitting, various optimization methods can be used. To prevent overfitting, methods such as increasing the learning data, regularization, dropout to disable some of the network nodes during the learning process, and use of a batch normalization layer can be applied. You can.

According to an embodiment of the present disclosure, the computing device 100 includes an input module 10, a speech conversion model 20, a feature vector extraction module 21, a loss function calculation module 22, and a speech processing model 30. It can be included. Meanwhile, a plurality of modules and models that may be included in the computing device 100 may be controlled by the processor 110 or implemented by the operation of the processor 110. In addition, modules and models that may be included in the computing device 100 in relation to the operation of learning a voice conversion model and processing voice data are not limited to the plurality of modules and models examined above, and additional Modules and models may be included. Below, a plurality of modules and models for learning a voice conversion model will be described in more detail through FIG. 3, and a plurality of modules and models for processing emotion-neutral voice data from which emotions have been removed will be described through FIG. 4. I would like to explain this in more detail.

Figure 3 is a diagram schematically showing a plurality of modules for training a voice conversion model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the input module 10 can acquire voice data for learning. Additionally, the input module 10 can acquire a set of voice data for learning. Here, the voice data set for training may include emotional voice data for training (voice with sentiment) and neutral voice data for training (voice without sentiment) without sentiment. For example, emotional voice data for learning that includes emotions may include voice data uttered (pronounced) by a speaker (user) in an emotional state corresponding to joy, sadness, anger, depression, etc. In addition, the input module 10 can acquire “emotional voice data for learning containing emotions” and “emotion-neutral voice data for learning without emotions” uttered by the same speaker regarding the same text through the user interface. For reference, emotion-neutral voice data for learning that does not contain emotions can be used as the correct answer data related to learning of the voice conversion model.

According to one embodiment, the input module 10 may output example sentences and emotion guide information through a user interface. For example, the input module 10 may output example sentences corresponding to “Hello, nice to meet you” and emotion guide information (e.g., joy, sadness, angry, depressed, etc.) through the user interface. Additionally, the input module 10 may acquire emotional voice data for learning including emotions based on the user interface. The input module 10 may acquire voice data uttered by the user with a specific emotion (e.g., joy) in response to the example sentence “Hello, nice to meet you,” output through the interface. Additionally, the input module 10 may acquire emotion-neutral voice data for learning that does not contain emotions based on the user interface. In other words, the input module 10 can acquire voice data in which the example sentence “Hello, nice to meet you” was uttered with a neutral emotion (no emotion included) through the user interface.

According to an embodiment of the present disclosure, the voice conversion model 20 can convert voice data for training into emotion-neutral voice data with emotions removed. Additionally, the speech conversion model 20 can be trained to generate emotion-neutral speech. For reference, the voice conversion model 20 may be a model that converts voice data containing emotion (Voice with sentiment) into emotion-neutral voice data (Voice without sentiment) close to emotion. In other words, the voice conversion model 20 can receive voice data containing emotions as input and output converted emotion-neutral voice data.

For example, the voice data for learning includes the voice data set for learning, and the voice data set for learning includes “emotional voice data for learning containing emotions” and “emotion-neutral voice data for learning not containing emotions.” In this case, the voice conversion model 20 may generate converted emotion-neutral voice information by removing emotions from “emotional voice data for learning containing emotions.”

According to one embodiment, the voice conversion model 20 may learn using a loss function associated with the converted emotion-neutral voice data. Specifically, the speech conversion model 20 may perform learning using the final loss function calculated from the loss function calculation module 22. For example, referring to FIG. 3, when the voice data for training is “emotional voice data for learning containing emotions,” the voice conversion model 20 converts the voice data into emotion-neutral voice data with the emotions removed, and converts the converted emotion-neutral voice data into emotion-neutral voice data. Learning can be performed using a loss function associated with voice data.

According to an embodiment of the present disclosure, the loss function calculation module 22 calculates a first loss function (L ₁ ) for the distance between “converted emotion-neutral speech data” and “emotion-neutral speech data for learning” and “transformed emotion-neutral speech data”. This is a module that calculates the final loss function (L) based on the second loss function (L ₂ ) that analyzes the relationship between “neutral speech data” and “emotion variables.”

For a more specific example, the loss function calculation module 22 calculates a first loss based on the distance between the “converted emotion-neutral speech data” and the “emotion-neutral speech data for learning” (playing the role of correct answer data). The function (L ₁ ) can be calculated. For example, the first loss function is L ₁ = D( , ) can be expressed as follows. Here, D may be a function that calculates the distance between vectors. In addition, “emotional voice data for learning” is the 1-1 voice vector ( ), and the “converted emotion-neutral voice data” generated based on the emotional voice data for learning is the 1-2 voice vector ( ) can correspond to . In addition, the “emotionally neutral voice data for learning” is a second voice vector ( ) can correspond to. In addition, the distance between the “converted emotion-neutral voice data” and the “emotion-neutral voice data for learning” is the 1-2 voice vector ( ) and the second negative vector ( ) can be calculated based on the distance between them. Meanwhile, the 1-1 negative vector ( ), the 1-2 negative vector ( ), and the second negative vector ( ) may correspond to vectors that have the length of the voice as a dimension. In addition, the 1-2 negative vector ( ) is emotion-neutral voice data converted from the voice conversion model (T), =T( ) It can be expressed as follows. here, is the length of the voice.

Additionally, the loss function calculation module 22 may calculate a second loss function (L ₂ ) based on analyzing the relationship between the converted emotion-neutral voice data and emotion variables. In one embodiment, the feature vector extraction module 21 may extract feature vectors for the converted emotion-neutral voice data. The loss function calculation module 22 can analyze the relationship between the feature vector extracted from the feature vector extraction module 21 and the emotional variable. For example, the 1-2 voice vector corresponding to the converted emotion-neutral voice data in the feature vector extraction module 21 ( ), the feature vector (z) is extracted, and the loss function calculation module 22 can analyze the relationship between the extracted feature vector (z) and the emotional variable. For reference, the extracted feature vector is z = E( ) It can be expressed as follows. Here, E is the feature vector calculation module, and d is the dimension of the feature vector.

Additionally, the loss function calculation module 22 may calculate the second loss function (L ₂ ) based on analysis of the correlation between the extracted feature vector (z) and the emotional variable (s). . In this case, the second loss function can be expressed as L ₂ = R(z, s). where s is the emotion variable, and s It can be expressed as Additionally, R may be a function that analyzes correlation. Meanwhile, the emotion variable (s) may be composed of various types of variables that quantitatively represent emotions. For example, the emotion variable s may include variables that quantitatively represent emotions in a real number range between -1 and 1, or may include various types of variables in addition to these variables.

Additionally, the loss function calculation module 22 may calculate a final loss function based on the first loss function and the second loss function. For example, the final loss function is It can be expressed as here, , > 0. Additionally, the weight for each loss function is , is a hyperparameter that can be changed and tuned depending on the usage environment. In other words, the final loss function may be calculated based on the weighted sum of the first loss function and the second loss function.

For example, the distance calculation function (eg, function D) associated with the first loss function and the correlation analysis function (eg, function R) associated with the second loss function may include various types of functions depending on the purpose. In one embodiment, function D and function R can be expressed as follows. However, the following expression is only an example and is not limited thereto, and various embodiments may exist.

Meanwhile, the loss function calculation module 22 determines the final loss function ( ) can be trained to minimize the voice conversion model 20. These final loss functions ( ) Based on this, the voice conversion model 20 can be learned in such a way that emotions can be well removed from voice data and correlations with emotions can also be reduced, and through this, emotion-neutral voices can be well generated. direction can be learned.

Figure 4 is a diagram schematically showing a neural network structure for processing voice data according to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the input module 10 may acquire original voice data. The original voice data may include at least one of emotional voice data containing emotion or emotion-neutral voice data not containing emotion.

According to an embodiment of the present disclosure, the voice conversion model 20 may output emotion-neutral voice data in which emotions are removed from the original voice data. Additionally, when the original voice data is emotion-neutral voice data with emotions removed, the voice conversion model 20 may output result data that is substantially the same as the corresponding voice data. In other words, the voice conversion model 20 is a model learned to generate an emotion-neutral voice. According to one embodiment, the voice conversion model 20 may be a model learned through the learning process described above. That is, the voice conversion model may be a model learned through the learning process described in FIG. 3.

According to an embodiment of the present disclosure, the voice processing model 30 receives the emotion-neutral voice data from the voice conversion model 20, and processes the emotion-neutral voice data from which the emotion has been removed instead of the original voice data. This is a model for doing this. Additionally, the speech processing model 30 may correspond to a model learned based on emotion-neutral speech data instead of speech data associated with a plurality of emotions. Additionally, the speech processing model 30 may include an architecture independent of a plurality of emotions, instead of an architecture including a plurality of emotion processing modules learned for a plurality of emotions. In one embodiment, the speech processing model 30 may include at least one of a speech recognition (STT) model, a speaker verification model, a speaker separation model, or a voice activity detection (VAD) model. You can. However, the voice processing model 30 is not limited to this, and various types of models may be included.

According to one embodiment of the present disclosure, speech recognition (STT or ASR; Speech To Text, or Automatic Speech Recognition) is a dictation technology that converts speech into text. In other words, speech recognition (STT) is a technology that generates text (correct spelling) corresponding to speech. The input of this voice recognition (STT) may include at least one of a voice signal, a spectrogram converted from a voice signal, or a voice feature. Additionally, the output of speech recognition (STT) is text in the form of a string. Meanwhile, the speech recognition (STT) model can be implemented in various types of models, including a neural network model. Additionally, speech recognition (STT) models can be divided into modular and non-modular end-to-end (e2e) methods depending on how they are implemented. Here, the modular method is divided into an acoustic model (a model that indicates what form a voice signal can be expressed in), a language model (a model that assigns the probability of occurrence to words based on given sentences and words), and a pronunciation dictionary. It may include, but is not limited to, traditional models that perform recognition (e.g., some models of Kaldi toolkit-based ASR models, Hybrid-ASR models, etc.). On the other hand, non-modularized methods mainly refer to e2e models (e.g., transformer-based encoder decoder models, etc.), and models can be created by learning a lot of data without having sub-modules. Meanwhile, the Beam Search technique is a representative decoding technique. The Beam Search technique does not predict only one word that is closest to the correct answer depending on the situation, but opens up various possibilities and considers the entire sentence to find the most optimal one. This is a way to find the correct answer.

According to an embodiment of the present disclosure, speaker verification can be divided into an enrollment process and a verification process. The input of speaker verification is an audio stream, and the output can be classified as authentication success or failure through binary classification. In addition, speaker verification involves vectorizing the voice through speaker embedding extraction and comparing similarity. If it exceeds the threshold, it is judged to be the same person, and if it is below the threshold, it is judged to be a different person. You can. As an example, speaker verification may represent each speaker with a one-hot speaker code. In addition, similarity includes cosine similarity and Euclidean similarity, and it is assumed that speaker embeddings of the same speaker have similar similarities. Representative speaker embedding extraction methods may include I-vector, X-vector, etc. Additionally, speaker verification can be classified into Text independent SV and Text dependent SV. Text dependent SV is an authentication module specialized for specific sentences (e.g., ok google), and the original performance of the speaker authentication module is not used when uttering other sentences. On the other hand, Text independent SV is a module that can perform speaker authentication for general sentences. Errors in speaker authentication include false positives and false negatives, and in speaker authentication, the false positive rate tends to be more important than the false negative rate. Generally, a measure such as minimum detection cost can be used to reflect this.

Speaker Diarization according to an embodiment of the present disclosure is a technology for determining “who spoke and when” in an input audio stream. After detecting voice activity based on Voice Activity Detection (VAD), speaker features can be extracted based on a “Speaker embedding extractor” such as ResNet, Res2Net, or ECAPA-TDNN. Clustering by features. Clustering can be done based on UIS-RNN, unsupervised clustering (agglomerative hierarchical clustering, spectral clustering), etc., and evaluation metrics can include DER (Diarization error rate) and JER (Jaccard error rate).

Voice Activity Detection (VAD), according to an embodiment of the present disclosure, is a technology that determines whether voice activity is detected in an audio stream. Voice activation detection (VAD) may be subject to preprocessing techniques such as noise removal and speech enhancement before applying speech recognition (STT). In addition, voice activation detection (VAD) is based on a binary classification algorithm that distinguishes whether each short speech section (e.g., 0.01 s) was pronounced by a person (1) or not (0). You can determine whether it is a voice section in various ways. In addition, voice activation detection (VAD) is a probability distribution-based classification that determines which of the two is closer to voice or noise based on the probability distribution based on statistical grounds, using the distribution of spoken voices and the distribution of noise voices. based classification) algorithm may be mainly used, but is not limited to this.

According to an embodiment of the present disclosure, since the voice processing model 30 receives emotion-neutral voice data from the voice conversion model 20 and processes the emotion-neutral voice data from which emotions have been removed instead of the original voice data, various Resources can be significantly saved in the learning stage of the speech processing models 30. In other words, the speech processing model 30 only needs to learn each task (e.g., speech recognition, speaker verification, speaker separation, and voice activity detection) on emotion-neutral speech data. Additionally, since emotion-neutral voice data is the voice with the most learning data, it is much easier to obtain learning data for the voice processing models 30.

In addition, since emotion-neutral voice data is received from the voice conversion model 20 and emotion-neutral voice data is processed instead of the original voice data, the voice is used in the inference stage of the various voice processing models 30. Processing performance can be improved. In particular, since the speech conversion model 20 operates as a separate model, the performance of the models can be improved without changing the pipeline of the existing speech processing model.

Figure 5 is a schematic flowchart of a method for learning a speech conversion model according to an embodiment of the present disclosure.

The method of learning the voice conversion model shown in FIG. 5 may be performed by the computing device 100. Even if there is no detailed mention below, the contents described above regarding the computing device 100 can be equally applied to a description of a method of training a voice conversion model.

Referring to FIG. 5, a method of learning a voice conversion model according to an embodiment of the present disclosure includes acquiring voice data for learning (S110), using a voice conversion model to transform the voice data for learning into an emotion-removed form. It may include converting to emotion-neutral voice data (S120) and learning the voice conversion model using a loss function associated with the converted emotion-neutral voice data (S130). Additionally, the method of learning a voice conversion model according to an embodiment of the present disclosure may be performed by the computing device 100.

The step S110 is a step of acquiring voice data for learning. This step S110 includes acquiring a voice data set for learning, and the voice data set for learning may include emotional voice data for learning containing emotions and emotion-neutral voice data for learning not containing emotions. there is.

The step S120 is a step of converting the learning voice data into emotion-neutral voice data from which emotions have been removed, using a voice conversion model. At this time, the voice conversion model can be learned to generate an emotion-neutral voice. Step S120 may include generating the converted emotion-neutral speech data by removing emotions from the emotion-containing learning emotion speech data using a speech conversion model.

The step S130 is a step of learning the voice conversion model using a loss function associated with the converted emotion-neutral voice data. Step S130 may include calculating a first loss function based on the distance between the converted emotion-neutral speech data and the emotion-neutral speech data for learning. Additionally, step S130 includes calculating a second loss function based on analyzing the relationship between the converted emotion-neutral voice data and emotion variables; And it may further include calculating a final loss function based on the first loss function and the second loss function.

The steps mentioned in the above description may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

Figure 6 is a schematic flowchart of a method for preventing changes in voice processing performance due to changes in emotions, according to an embodiment of the present disclosure.

The method for preventing changes in voice processing performance due to emotional changes shown in FIG. 6 may be performed by the computing device 100. Even if no detailed mention is made below, the contents described above regarding the computing device 100 may be equally applied to a description of a method for preventing changes in voice processing performance due to changes in emotions.

Referring to FIG. 6, a method for preventing changes in voice processing performance due to emotional changes according to an embodiment of the present disclosure includes acquiring voice data (S210), utilizing a voice conversion model to It may include converting to emotion-neutral voice data from which emotions are removed (S220) and inputting the converted emotion-neutral voice data into a voice processing model (S230). A method for preventing changes in voice processing performance due to emotional changes according to an embodiment of the present disclosure may be performed by the computing device 100.

The step S210 is a step of acquiring voice data.

The step S220 is a step of converting the voice data into emotion-neutral voice data from which emotions have been removed, using a voice conversion model. Here, the voice conversion model is a model learned through FIG. 5.

The step S230 is a step of inputting the converted emotion-neutral voice data into a voice processing model. Here, the speech processing model may correspond to a model learned based on emotion-neutral speech data instead of speech data associated with a plurality of emotions. For reference, the speech processing model may include at least one of a speech recognition (STT; Speech To Text) model, a speaker verification model, a speaker separation model, or a voice activity detection (VAD) model.

The steps mentioned in the above description may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

According to an embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed.

Data structure can refer to the organization, management, and storage of data to enable efficient access and modification of data. Data structure can refer to the organization of data to solve a specific problem (e.g., retrieving data, storing data, or modifying data in the shortest possible time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. Logical relationships between data elements may include connection relationships between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements that are physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure may specifically include a set of data, relationships between data, and functions or instructions applicable to the data. Effectively designed data structures allow computing devices to perform computations while minimizing the use of the computing device's resources. Specifically, computing devices can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and search through effectively designed data structures.

Data structures can be divided into linear data structures and non-linear data structures depending on the type of data structure. A linear data structure may be a structure in which only one piece of data is connected to another piece of data. Linear data structures may include List, Stack, Queue, and Deque. A list can refer to a set of data that has an internal order. The list may include a linked list. A linked list may be a data structure in which data is connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer may contain connection information to the next or previous data. A linked list can be expressed as a singly linked list, doubly linked list, or circular linked list depending on its form. A stack may be a data listing structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (for example, inserted or deleted) at only one end of the data structure. Data stored in the stack may have a data structure (LIFO-Last in First Out) where the later it enters, the sooner it comes out. A queue is a data listing structure that allows limited access to data. Unlike the stack, it can be a data structure (FIFO-First in First Out) where data stored later is released later. A deck can be a data structure that can process data at both ends of the data structure.

A non-linear data structure may be a structure in which multiple pieces of data are connected behind one piece of data. Nonlinear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structure may include a tree data structure. A tree data structure may be a data structure in which there is only one path connecting two different vertices among a plurality of vertices included in the tree. In other words, it may be a data structure that does not form a loop in the graph data structure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Below, it is described in a unified manner as a neural network. Data structures may include neural networks. And the data structure including the neural network may be stored in a computer-readable medium. Data structures including neural networks also include data preprocessed for processing by a neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, neural network It may include a loss function for learning. A data structure containing a neural network may include any of the components disclosed above. In other words, the data structure including the neural network includes preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, neural network It may be configured to include all or any combination of the loss function for learning. In addition to the configurations described above, a data structure containing a neural network may include any other information that determines the characteristics of the neural network. Additionally, the data structure may include all types of data used or generated in the computational process of a neural network and is not limited to the above. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node.

The data structure may include data input to the neural network. A data structure containing data input to a neural network may be stored in a computer-readable medium. Data input to the neural network may include learning data input during the neural network learning process and/or input data input to the neural network on which training has been completed. Data input to the neural network may include data that has undergone pre-processing and/or data subject to pre-processing. Preprocessing may include a data processing process to input data into a neural network. Therefore, the data structure may include data subject to preprocessing and data generated by preprocessing. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. (In this specification, weights and parameters may be used with the same meaning.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. A neural network may include multiple weights. Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. Based on the weight, the data value output from the output node can be determined. The above-described data structure is only an example and the present disclosure is not limited thereto.

As an example and not a limitation, the weights may include weights that are changed during the neural network learning process and/or weights for which neural network learning has been completed. Weights that change during the neural network learning process may include weights that change at the start of the learning cycle and/or weights that change during the learning cycle. Weights for which neural network training has been completed may include weights for which a learning cycle has been completed. Therefore, the data structure including the weights of the neural network may include weights that are changed during the neural network learning process and/or the data structure including the weights for which neural network learning has been completed. Therefore, the above-mentioned weights and/or combinations of each weight are included in the data structure including the weights of the neural network. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or a different computing device and later reorganized and used. Computing devices can transmit and receive data over a network by serializing data structures. Data structures containing the weights of a serialized neural network can be reconstructed on the same computing device or on a different computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure to increase computational efficiency while minimizing the use of computing device resources (e.g., in non-linear data structures, B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree) may be included. The foregoing is merely an example and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of a neural network. And the data structure including the hyperparameters of the neural network can be stored in a computer-readable medium. A hyperparameter may be a variable that can be changed by the user. Hyperparameters include, for example, learning rate, cost function, number of learning cycle repetitions, weight initialization (e.g., setting the range of weight values subject to weight initialization), Hidden Unit. It may include a number (e.g., number of hidden layers, number of nodes in hidden layers). The above-described data structure is only an example and the present disclosure is not limited thereto.

7 is a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has generally been described above as being capable of being implemented by a computing device, those skilled in the art will understand that the present disclosure can be implemented in combination with computer-executable instructions and/or other program modules that can be executed on one or more computers and/or in hardware and software. It will be well known that it can be implemented as a combination.

Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Additionally, those skilled in the art will understand that the methods of the present disclosure are applicable to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. It will be appreciated that each of these may be implemented in other computer system configurations, including those capable of operating in conjunction with one or more associated devices.

The described embodiments of the disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Computer-readable media can be any medium that can be accessed by a computer, and such computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media refers to volatile and non-volatile media, transient and non-transitory media, removable and non-removable, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Includes media. Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage. This includes, but is not limited to, a device, or any other medium that can be accessed by a computer and used to store desired information.

A computer-readable transmission medium typically implements computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transport mechanism. Includes all information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal have been set or changed to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106, to processing unit 1104. Processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

System bus 1108 may be any of several types of bus structures that may further be interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, and EEPROM, and is a basic input/output system that helps transfer information between components within the computer 1102, such as during startup. Contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

Computer 1102 may also include an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)—the internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes - a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM for reading the disk 1122 or reading from or writing to other high-capacity optical media such as DVDs). Hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to system bus 1108 by hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For computer 1102, drive and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to removable optical media such as HDDs, removable magnetic disks, and CDs or DVDs, those skilled in the art will also recognize removable optical media such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other types of computer-readable media, such as the like, may also be used in the example operating environment and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as mouse 1140. Other input devices (not shown) may include microphones, IR remote controls, joysticks, game pads, stylus pens, touch screens, etc. These and other input devices are connected to the processing unit 1104 through an input device interface 1142, which is often connected to the system bus 1108, but may also include a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, It can be connected by other interfaces, etc.

A monitor 1144 or other type of display device is also connected to system bus 1108 through an interface, such as a video adapter 1146. In addition to monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and is generally connected to computer 1102. For simplicity, only memory storage device 1150 is shown, although it includes many or all of the components described. The logical connections depicted include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154. These LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, computer 1102 is connected to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed thereon for communicating with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158 or be connected to a communicating computing device on the WAN 1154 or to establish communications over the WAN 1154, such as over the Internet. have other means. Modem 1158, which may be internal or external and a wired or wireless device, is coupled to system bus 1108 via serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and that other means of establishing a communications link between computers may be used.

Computer 1102 may be associated with any wireless device or object deployed and operating in wireless communications, such as a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communications satellite, wirelessly detectable tag. Performs actions to communicate with any device or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows connection to the Internet, etc. without wires. Wi-Fi is a wireless technology, like cell phones, that allows these devices, such as computers, to send and receive data indoors and outdoors, anywhere within the coverage area of a base station. Wi-Fi networks use wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, the Internet, and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). .

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description include voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. It can be expressed by particles or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be used in electronic hardware, (for convenience) It will be understood that it may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally with respect to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. A person skilled in the art of this disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash. Includes, but is not limited to, memory devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of illustrative approaches. It is to be understood that the specific order or hierarchy of steps in processes may be rearranged within the scope of the present disclosure, based on design priorities. The appended method claims present elements of the various steps in a sample order but are not meant to be limited to the particular order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not limited to the embodiments presented herein but is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (15)

1. A method of training a speech conversion model, performed by a computing device, comprising:
Obtaining a learning speech data set including emotional speech data for learning containing emotions and emotion-neutral speech data for learning not containing emotions;
Converting the emotional voice data for learning including the emotion into emotion-neutral voice data from which the emotion is removed, using a voice conversion model;
calculating a first loss function based on the distance between the converted emotion-neutral speech data and the emotion-neutral speech data for learning;
A feature vector is extracted based on the voice vector corresponding to the converted emotion-neutral voice data, and a second loss function is based on a correlation relationship analysis between the extracted feature vector and an emotion variable that quantitatively represents emotion. Calculating; and
Calculating a final loss function based on the first loss function and the second loss function, and using the final loss function to train the speech conversion model.
Including,
The speech conversion model is learned to generate emotion-neutral speech,
method.
delete According to claim 1,
The step of acquiring the learning voice data set is,
Outputting example sentences and emotion guide information through a user interface;
Obtaining emotional voice data for learning including the emotion based on the user interface; and
Obtaining emotion-neutral voice data for learning that does not contain the emotion based on the user interface.
Including,
method.
delete delete According to claim 1,
The emotional voice data for learning corresponds to the 1-1 voice vector,
The converted emotion-neutral voice data generated based on the emotional voice data for learning corresponds to a 1-2 voice vector,
The emotion-neutral voice data for learning corresponds to a second voice vector,
The distance between the converted emotion-neutral voice data and the emotion-neutral voice data for learning is calculated based on the distance between the first-second voice vector and the second voice vector,
The 1-1 voice vector, the 1-2 voice vector, and the second voice vector correspond to vectors having the length of voice as a dimension,
method.
delete delete As a method for preventing changes in speech processing performance due to emotional changes, performed by a computing device,
Obtaining voice data containing emotions;
Converting the emotion-containing voice data into emotion-neutral voice data from which the emotion is removed, using a voice conversion model; and
Inputting the converted emotion-neutral voice data into a voice processing model
Including,
The voice conversion model is learned based on a learning voice data set including emotional voice data for learning and emotion-neutral voice data for learning that does not contain emotions,
The voice conversion model is,
a first loss function calculated based on a distance between the converted emotion-neutral speech data and the emotion-neutral speech data for learning;
A feature vector is extracted based on a voice vector corresponding to the converted emotion-neutral voice data, and a second loss is calculated based on correlation analysis between the extracted feature vector and an emotional variable that quantitatively represents emotion. function; and
Learned using a final loss function calculated based on the first loss function and the second loss function,
method.
delete According to clause 9,
The voice conversion model is,
Corresponding to a model learned based on emotion-neutral speech data instead of speech data associated with a plurality of emotions,
method.
According to claim 11,
The voice conversion model is,
Instead of an architecture comprising a plurality of emotion processing modules learned for a plurality of emotions, an architecture comprising an architecture independent of a plurality of emotions,
method.
According to claim 12,
The voice conversion model is,
Comprising at least one of a speech recognition (STT) model, a speaker verification model, a speaker separation model, or a voice activity detection (VAD) model,
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, causes the one or more processors to perform the following operations for training a speech conversion model, the operations comprising:
An operation of acquiring a learning speech data set including emotional speech data for learning containing emotion and emotion-neutral speech data for learning not containing emotion;
Converting the emotional voice data for learning including the emotion into emotion-neutral voice data from which the emotion is removed, using a voice conversion model;
calculating a first loss function based on the distance between the converted emotion-neutral speech data and the emotion-neutral speech data for learning;
A feature vector is extracted based on the voice vector corresponding to the converted emotion-neutral voice data, and a second loss function is based on a correlation relationship analysis between the extracted feature vector and an emotion variable that quantitatively represents emotion. An operation that calculates; and
calculating a final loss function based on the first loss function and the second loss function and training the speech conversion model using the final loss function;
Including,
The speech conversion model is learned to generate emotion-neutral speech,
A computer program stored on a computer-readable storage medium.
As a computing device,
at least one processor; and
Memory
Including,
The at least one processor,
Obtaining a learning speech data set including emotional speech data for learning containing emotions and emotion-neutral speech data for learning not containing emotions;
Using a speech conversion model, convert the emotional speech data for learning containing the emotion into emotion-neutral speech data with the emotion removed;
calculating a first loss function based on the distance between the converted emotion-neutral speech data and the emotion-neutral speech data for training;
A feature vector is extracted based on the voice vector corresponding to the converted emotion-neutral voice data, and a second loss function is based on a correlation relationship analysis between the extracted feature vector and an emotion variable that quantitatively represents emotion. Calculate; and
Configured to calculate a final loss function based on the first loss function and the second loss function, and to train the speech conversion model using the final loss function,
The speech conversion model is learned to generate emotion-neutral speech,
Device.

Images