KR20200127917A - Method to train model - Google Patents

Abstract

Disclosed is a computer program stored in a computer-readable storage medium. The computer program includes the operations of: generating a learning dataset including one or more pieces of learning data having learning voice data as an input and having a facial animation based on the learning voice data as a label; and learning a facial animation generation model including a first network function and a second network function based on the learning dataset. The operation of learning the facial animation generation model including the first network function and the second network function based on the learning dataset includes an operation of learning the facial animation generation model by differently setting an update degree of a weight for the first network function and an update degree of a weight for the second network function. According to the present invention, an animation can be generated based on voice data.

Description

Model training method {METHOD TO TRAIN MODEL}

The present invention relates to a model learning method, and more particularly, to a method of learning a model for generating animation based on voice data.

With the recent rapid development of animation-related technologies, the movement to apply animation to the entire industry is increasing. Animation is used not only in manga-related industries but also in various industries, and it takes considerable time and cost to create animation.

In the case of the game industry, voice actors record their voices based on the game story, and based on the voices, an animation suitable for the voice is generated and provided to game players. The amount of game stories recorded by voice actors is vast, and the time and cost of creating animations based on them is enormous.

Accordingly, there is a need in the art to save time and cost in creating animations based on voice data.

Korean Patent Publication No. 10-2019-0008137 discloses a deep learning-based speech synthesis apparatus and method using multiple speaker data.

The present disclosure has been devised in response to the above-described background technology, and an object of the present disclosure is to provide a model learning method.

A computer program stored in a computer-readable storage medium according to an embodiment of the present disclosure for realizing the above-described problem, the computer program inputting learning speech data, and labeling a face animation based on the learning speech data. Generating a training data set including one or more training data to be used; Including an operation of training a face animation generation model including a first network function and a second network function based on the training data set, and comprising a first network function and a second network function based on the training data set The operation of training the face animation generation model may include an operation of training the face animation generation model by differently setting an update degree of the weight for the first network function and an update degree of the weight for the second network function. I can.

In an alternative embodiment of computer program operations that perform the following operations to provide a model learning method, the update degree of the weight for the first network function and the update degree of the weight for the second network function are set differently. The operation of training the face animation generation model by setting the update degree of the weight for the first network function to be greater than the update degree of the weight for the second network function to train the face animation generation model can do.

In an alternative embodiment of computer program operations that perform the following operations for providing a model learning method, the update degree of the weight for the first network function is set to be greater than the update degree of the weight for the second network function. The training of the face animation generation model includes backpropagating based on an output obtained by calculating the training speech data as an input of the face animation generation model and an error of the face animation, which is a label included in the training data. In this case, it may include an operation of updating the weights only for the first network function except for the second network function.

In an alternative embodiment of computer program operations that perform the following operations to provide a model learning method, the update degree of the weight for the first network function and the update degree of the weight for the second network function are set differently. In the operation of training the face animation generation model, the update degree of the weight for the first network function and the update degree of the weight for the second network function are differently set during a predetermined epoch, so that the face animation generation model is constructed. It may include an operation to learn.

In an alternative embodiment of computer program operations that perform the following operations for providing a model learning method, the second network function includes a first layer and a second layer, and the first layer and the second layer The size of can be different.

In an alternative embodiment of computer program operations that perform the following operations for providing a model learning method, the second layer may be a layer closer to an output layer than the first layer.

In an alternative embodiment of computer program operations that perform the following operations to provide a model learning method, the degree of update of the weights for the first network function and the weights for the second network function after a predetermined learning epoch An operation of learning the face animation generation model by setting the update degree to the same may be further included.

In an alternative embodiment of computer program operations for performing the following operations for providing a model learning method, the first network function is a network function for outputting a feature vector for a face pose based on the training speech data, In addition, the second network function may be a network function for generating the face animation based on a feature vector related to the face pose.

In an alternative embodiment of computer program operations that perform the following operations for providing a model learning method, the first network function includes two or more convolutional layers, and the second network function includes two or more pulley connections. It may include a tie layer.

A model learning method according to an embodiment of the present disclosure for realizing the above-described task, comprising: at least one learning data including learning speech data as input and a face animation based on the learning speech data as a label Creating a data set; And training a face animation generation model including a first network function and a second network function based on the training data set, and comprising a first network function and a second network function based on the training data set. The training of the face animation generation model may include training the face animation generation model by differently setting an update degree of the weight for the first network function and an update degree of the weight for the second network function. I can.

A server for providing a model learning method according to an embodiment of the present disclosure for realizing the above-described problem, comprising: a processor including one or more cores; And a memory, wherein the processor generates a training data set including at least one training data having a face animation based on the training voice data as an input, and the training data set based on the training data set. Thus, the face animation generation model including the first network function and the second network function is trained, and the learning of the face animation generation model including the first network function and the second network function based on the training data set is It may include training the face animation generation model by differently setting an update degree of the weight for the first network function and an update degree of the weight for the second network function.

According to an embodiment of the present disclosure, a method for model training may be provided.

1 is a diagram illustrating a block diagram of a computing device performing an operation for providing an animation generating method according to an embodiment of the present disclosure.
2 is a diagram illustrating a face animation generation model according to an embodiment of the present disclosure.
3 is a diagram illustrating a face animation generation model according to an embodiment of the present disclosure.
4 is a diagram illustrating a face animation generation model according to an embodiment of the present disclosure.
5 is a diagram illustrating a method of training a face animation generation model according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of generating a face animation based on a face animation generation model according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of learning a face animation generation model according to an embodiment of the present disclosure.
8 is a block diagram of a computing device according to an embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is clear that these embodiments may be implemented without this specific description.

The terms "component", "module", "system" and the like as used herein refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component may be, but is not limited to, a process executed on a processor, a processor, an object, an execution thread, a program, and/or a computer. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized on a single computer. A component can be distributed between two or more computers. In addition, these components can execute from a variety of computer readable media having various data structures stored therein. Components can be, for example, via a signal with one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or a signal through another system and a network such as the Internet. Depending on the data being transmitted), it may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or is not clear from the context, "X employs A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, when X uses both A and B, “X uses A or B” can be applied to either of these cases. In addition, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

In addition, the terms "comprising" and/or "comprising" are to be understood as meaning that the corresponding features and/or components are present. However, it is to be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. In addition, unless otherwise specified or when the context is not clear as indicating a singular form, the singular in the specification and claims should be interpreted as meaning "one or more" in general.

Those of skill in the art would further describe the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein, including electronic hardware, computer software, or a combination of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the specific application and design restrictions imposed on the overall system. Skilled technicians can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

A description of the presented embodiments is provided so that a person of ordinary skill in the art of the present disclosure can use or implement the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features presented herein.

In an embodiment of the present disclosure, the server may include any type of computing device. The server is a digital device, and may be a digital device equipped with a processor, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone, and equipped with a computing power having a memory. The server may be a web server that processes services. The types of servers described above are only examples, and the present disclosure is not limited thereto.

1 is a diagram illustrating a block diagram of a computing device performing an operation for providing an animation generating method according to an embodiment of the present disclosure.

The computing device 100 for providing a method for generating a face animation or a method for learning a model for generating a face animation according to an embodiment of the present disclosure may include a network unit 110, a processor 120, and a memory 130. .

The network unit 110 may transmit/receive voice data, an emotional state, and the like according to an embodiment of the present disclosure, with other computing devices or servers.

The processor 120 may be composed of one or more cores, and a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit) may include a processor for providing a face animation generation model. The processor 120 may read a computer program stored in the memory 130 to provide a face animation generation model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 120 may perform calculation for generating a face animation based on voice data. According to an embodiment of the present disclosure, the processor 120 may perform calculation for training a face animation generation model.

2 is a diagram illustrating a face animation generation model according to an embodiment of the present disclosure.

The face animation generation model 800 according to an embodiment of the present disclosure may generate a face animation 600 corresponding to the voice data 200 based on the voice data 200.

The processor 120 may receive the voice data 200. The voice data 200 may include data on a person's voice. The voice data 200 may be data including sound information output to a game player during a game. For example, the voice data may include a digital file including information for reproducing sound information. Voice data may be stored in a format such as WAV, AIFF, AU, FLAC, TTA, MPE, AAC, ATRAC, etc., but is not limited thereto. In addition to the above-described format, the voice data 200 may be stored in an arbitrary format including at least a portion of sound information. The acoustic information may be information about a character of a game player, a sound uttered by a non-player character (NPC), or a sound uttered by a playable character played by a game player other than one game player. For example, when a game provides a quest to game players, the game quest may be delivered using animation. For example, the voice data 200 may be data about the voice of a character or NPC in an animation effect that delivers quest-related content. As another example, the voice data 200 may be data related to voice spoken by a character or NPC during game play in a game. The audio data 200 may be audio recordings of content related to the game quest. The voice data 200 may be data previously stored in the game. The voice data 200 may be data previously stored in the memory of the game server. Alternatively, the voice data 200 may be data related to voice recorded in advance or in real time by a game player or another game player. The description of the above-described voice data is only an example, and the present disclosure is not limited thereto.

The processor 120 may receive voice data through the network unit 110. The reception of voice data according to an embodiment of the present disclosure may be receiving or loading voice data stored in the memory 130. Receiving the voice data may be receiving or loading the voice data from another storage medium, another computing device, or a separate processing module within the same computing device based on a wired/wireless communication means.

The processor 120 may undergo a preprocessing process before inputting the voice data 200 to the face animation generating model 800. The preprocessing of the voice data 200 may refer to a process of processing in a format for processing by one or more network functions included in the face animation generation model 800. For example, the preprocessing of the voice data 200 may include an operation of deleting, converting, or converting at least some data of the voice data 200 into a format for input to the animation generating model 800. have. Pre-processing of the voice data 200 may mean partially correcting the waveform of the voice data 200. For example, the pre-processing of the voice data 200 may mean separating only voice data of a portion where voice is present using Voice Activity Detection (VAD), which is voice detection. Alternatively, the preprocessing of the voice data is to remove noise from the signal, amplify the frequency band where the signal may exist, cut the frequency band where the signal may exist, or increase the amplitude of the signal to the maximum value and assign it to the digital signal. The maximum number of bits can be used, the highest frequency or the lowest frequency is normalized based on a predetermined criterion, or whitening is performed to change the average of the signal to 0 and the standard deviation to 1. The specific description of the above-described pretreatment is only an example, and the present disclosure is not limited thereto.

The processor 120 may input the voice data 200 to the face animation generating model 800. The face animation generation model 800 may include a first network function 300 and a second network function 500. The first network function 300 may perform dimension reduction of input data. The second network function 500 may perform dimensional expansion of input data. The processor 120 may perform a dimension reduction operation by inputting the voice data 200 to the first network function 300. The processor 120 may use the first network function 300 to calculate and output a result as an input of the second network function 500. The processor 120 may perform a dimensional expansion operation of data input to the second network function 500. The processor 120 may generate the face animation 600 based on the voice data 200 based on the dimensional expansion operation of the second network function 500.

The first network function 300 included in the face animation generation model 800 may include two or more dimensional reduction layers. The second network function 500 included in the face animation generation model 800 may include two or more dimensional extension layers.

The processor 120 may calculate the voice data 200 by using the first network function 300 including two or more dimensional reduction layers. The processor 120 may calculate the voice data 200 by using a first network function 300 including two or more convolutional layers. The first network function 300 may output a feature vector 400 related to a face pose for generating the face animation 600. The feature vector 400 related to the face pose may be a feature vector for describing a face pose (or a face shape) based on the voice data 200. The feature vector 400 related to the face pose may be a feature vector for generating a face animation based on the voice data 200. For example, the feature vector 400 for the face pose may be determined based on the number of feature maps based on the voice data 200, the dimension of the time axis, and the dimension of the phoneme that is the basis of the voice data 200. A detailed description of the feature vector for the above-described face pose is only an example, and a method of calculating it will be described in detail later.

The processor 120 may calculate the feature vector 400 for the face pose using the second network function 500 including two or more dimensional enhancement layers. The processor 120 may generate the face animation 600 by calculating the feature vector 400 related to the face pose using a second network function 500 including two or more deconvolutional layers. . The second network function 500 may include two or more deconvolutional layers. The processor 120 may perform linear transformation on the feature vector 400 related to the face pose based on the second network function 500. The processor 120 may calculate a feature vector 400 related to the face pose based on the second network function 500. The processor 120 may generate coordinates for generating a face animation based on an operation of the second network function 500. The coordinates may be a reference point that is the basis of face animation. For example, the coordinates may be vertices. A vertex position on the face animation for generating the face animation 600 may be output. The processor 120 may generate the face animation 600 based on the vertex position calculated using the second network function 500. The processor 120 may determine a vertex of the face animation based on the calculated vertex position and generate the face animation 600 accordingly. The facial animation 600 may include data on the face of a person, player, or character. The face animation 600 may be an animation based on sound information output to a game player during a game. The facial animation 600 may be an animation depicting an expression matched with sound information output to a game player during a game. The face animation 600 may be an animation expressed through a player's character or NPC. For example, the face animation 600 may be an animation expressed through a character or NPC in an animation effect delivering quest-related content. As another example, the face animation 600 may be an animation expressed through a character or NPC during game play. For example, the face animation 600 may be a 3D-type animation. The detailed description of the above-described face animation is only an example, and the present disclosure is not limited thereto.

A convolutional neural network is a kind of deep neural network and includes a neural network including a convolutional layer. A convolutional neural network is a type of multilayer perceptron designed to use minimal preprocessing. The CNN may consist of one or several convolutional layers and artificial neural network layers combined therewith. CNN can additionally utilize weights and pooling layers. Thanks to this structure, CNN can fully utilize the input data of the two-dimensional structure. The convolutional neural network can be used to recognize objects in an image. The convolutional neural network can process image data by representing it as a matrix having dimensions. For example, in the case of red-green-blue (RGB) encoded image data, each R, G, and B color may be represented as a two-dimensional (for example, two-dimensional image) matrix. That is, in the case of image data encoded in RGB, it may be configured as a two-dimensional matrix of three channels. That is, the color value of each pixel of the image data may be a component of the matrix, and the size of the matrix may be the same as the size of the image. Accordingly, image data can be represented by three two-dimensional matrices (three-dimensional data arrays).

In a convolutional neural network, a convolutional process (input and output of a convolutional layer) can be performed by multiplying the convolutional filter and matrix components at each position of the image while moving the convolutional filter. The convolutional filter may be composed of an n*n matrix. The convolutional filter may consist of a fixed type filter that is generally smaller than the total number of pixels in the image. That is, when inputting an m*m image into a convolutional layer (for example, a convolutional layer having a convolutional filter size of n*n), a matrix representing n*n pixels including each pixel of the image This convolutional filter can be multiplied by components (ie, the product of each component of the matrix). A component matching the convolutional filter may be extracted from an image by multiplying it with the convolutional filter. For example, a 3*3 convolutional filter for extracting the upper and lower linear components from an image is composed of [[0,1,0], [0,1,0], [0,1,0]]. I can. When a 3*3 convolutional filter for extracting upper and lower linear components from an image is applied to an input image, upper and lower linear components matching the convolutional filter may be extracted and output from the image. The convolutional layer may apply a convolutional filter to each matrix for each channel representing an image (ie, R, G, and B colors in the case of R, G, and B coded images). The convolutional layer may extract features matching the convolutional filter from the input image by applying a convolutional filter to the input image. The filter value of the convolutional filter (that is, the value of each component of the matrix) may be updated by backpropagation in the learning process of the convolutional neural network.

A subsampling layer is connected to the output of the convolutional layer to simplify the output of the convolutional layer, thereby reducing the amount of memory and computation. For example, if the output of the convolutional layer is input to a pooling layer with a 2*2 max pooling filter, the image is compressed by outputting the maximum value included in each patch for each 2*2 patch from each pixel of the image. I can. The aforementioned pooling may be a method of outputting a minimum value from a patch or an average value of a patch, and any pooling method may be included in the present disclosure.

The convolutional neural network may include one or more convolutional layers and sub-sampling layers. The convolutional neural network may extract a feature from an image by repeatedly performing a convolutional process and a subsampling process (eg, the aforementioned max pooling). Through repetitive convolutional processes and subsampling processes, the neural network can extract global features of the image.

The output of the convolutional layer or the subsampling layer may be input to a fully connected layer. A fully connected layer is a layer where all neurons in one layer and all neurons in neighboring layers are connected. The fully connected layer may mean a structure in which all nodes of each layer are connected to all nodes of another layer in a neural network.

In an embodiment of the present disclosure, in order to perform an operation for generating a face animation based on voice data, the neural network may include a deconvolutional neural network (DCNN). The deconvolutional neural network performs an operation similar to that of calculating the convolutional neural network in the reverse direction. The deconvolutional neural network may output features extracted from the convolutional neural network as a feature map related to original data. A description of the specific configuration of the convolutional neural network and the deconvolutional neural network is discussed in more detail in US Patent No. US9870768B2, which is incorporated by reference in its entirety in this application. The description of the facial animation generation model described in an embodiment of the present disclosure is the papers TERO KARRAS, TIMO AILA, SAMULI LAINE, ANTTI HERVA, JAAKKO LEHTINEN “Audio-Driven Facial Animation by Joint End,” which are all incorporated by reference in this application. -to-End Learning of Pose and Emotion” is discussed in more detail in July 2017.

Hereinafter, the first network function 300 included in the face animation generation model 800 will be described in detail. The processor 120 may calculate and output a feature vector 400 related to a face pose based on the voice data 200 using the first network function 300.

3 is a diagram illustrating a face animation generation model according to an embodiment of the present disclosure.

The processor 120 calculates the voice data 200 using a first network function 300 including two or more convolutional layers to generate a face animation 600 with a feature vector 400 related to a face pose. ) Can be printed. Two or more layers included in the first network function according to an embodiment of the present disclosure may output an output based on an input. For example, the output can be expressed as (number of feature maps) * (dimension of time axis) * (dimension of formant axis). Specific description of the above-described output is only an example, and the present disclosure is not limited thereto.

The processor 120 may convert the voice data 200 into image data 310 based on the voice data. The processor 120 may convert the preprocessed voice data 200 into image data 310 based on the voice data. The processor 120 may convert the preprocessed voice data 200 into image data 310 based on the voice data in order to calculate the voice data 200 based on the convolutional layer. The image data 310 may be data representing sound information based on the voice data 200 as an image. The image data 310 may be an image different from other image data based on other sounds that are the basis of the image data 310. The image data 310 may be processed by a first network function 300 including a convolutional layer. A function for converting the voice data 200 into image data 310 based on the voice data may be a predetermined image data conversion function. The predetermined image data conversion function may be updated based on an error when the face animation generating model 800 is trained, or may perform transformation into the image data 310 fixedly without updating when the face animation generating model is trained. have. The predetermined image data conversion function may be converted into image data 310 based on the frequency of the voice data 200. Detailed description of the above-described image data conversion function is only an example, and the present disclosure is not limited thereto.

The processor 120 may calculate the image data 310 using the first network function 300 to output a feature vector 400 related to the face pose.

The first network function 300 may include a first sub network function 320 and a second sub network function 340. Each of the first sub-network function 320 and the second sub-network function 340 may include a convolutional layer. More specifically, the first sub-network function 320 may include two or more convolutional layers. The second sub-network function 340 may include two or more convolutional layers and correction layers. The correction layer will be described later in detail.

The processor 120 may calculate the image data 310 based on the first sub-network function 320 and output the voice feature 330. The voice feature 330 may mean a feature in which the input voice data 200 can be distinguished from other voice data. The voice feature 330 may mean a feature in which sound information output to a game player during a game can be distinguished from other sound information. The voice feature 330 may be a feature of a voice spoken by a character or NPC during game play in a game. The voice feature 330 may be a feature based on voice data related to facial animation. For example, the voice feature 330 may be a feature related to a facial animation related to an intonation of a voice, an emphasis of a voice, a specific phoneme of a voice, and the like. The voice feature 330 may be information calculated based on the image data 310 converted from the voice data. The specific description of the voice characteristics is only an example, and the present disclosure is not limited thereto.

The processor 120 may calculate the voice feature 330 based on the second sub-network function 340 and output a feature vector 400 related to the face pose. The processor 120 may calculate a feature based on voice data related to animation based on the second sub-network function 340. The processor 120 calculates a voice feature 330 including features related to, for example, intonation of voice, emphasis of voice, and specific phoneme of voice, using the second sub-network function 340 to calculate a face pose feature. Vector 400 can be output. A method of outputting a feature vector for a face pose based on a voice feature will be described later in detail.

4 is a diagram illustrating a face animation generation model 800 according to an embodiment of the present disclosure.

The processor 120 may calculate the voice feature 330 based on the second sub-network function 340 and output a feature vector 400 related to the face pose.

Even in the case of having the same voice characteristics, the facial animation may be expressed differently according to the emotional state of the speaker. Even in the case of voice data generated by pronouncing the same word with the same sound and emphasis, face animation may be expressed differently in case of different emotions. Accordingly, in order to increase the accuracy of facial animation expression, facial animation may be generated in consideration of emotional state data related to voice data.

The processor 120 may input emotional state data 342 related to the voice data 200 to the second sub-network function 340.

The emotional state data 342 may be data determined based on the voice data 200. The emotional state data 342 may be data related to an expression expressed in the facial animation 600. The emotional state data 342 may be data representing a classification regarding an emotional state expressed in the face animation 600. The emotional state data 342 may be an N-dimensional vector for indicating a classification of an emotional state. The emotional state data 342 may be a vector in a form that can be calculated in each of two or more combining layers included in the second sub-network function 340. For example, the emotional state data 342 may be a vector related to sadness, joy, and joy. The emotional state data 342 may be stored in the memory 130 based on classification related to the emotional state. Alternatively, the emotional state data 342 may be received from another computing device. The detailed description of the above-described emotional state data is only an example, and the present disclosure is not limited thereto. For example, when the emotional state data is an N-dimensional vector, the outputs of the first layer 343 and the second layer 344 may be (256+N)*32*1, and the third and fourth layers The output of the layer may be (256+N)*16*1. Detailed description of the above-described emotional state data and layers is only an example, and the present disclosure is not limited thereto.

The emotional state data 342 according to an embodiment of the present disclosure may be data manually labeled with voice data. The processor 120 may use the manually labeled emotional state data 342 based on the voice data as an input of the second sub network function 340 included in the face animation generation model 800. For example, a person may manually label emotion state data 342 on each of two or more voice data to input the face animation generation model 800. For example, an N-dimensional vector that is joy emotion state data stored in the memory 130 based on manual labeling may be input as an input of the face animation generation model 800. The detailed description of the above-described emotional state data is only an example, and the present disclosure is not limited thereto.

The emotional state data 342 according to an embodiment of the present disclosure may be learned according to the learning of the face animation generation model 800. The initial vector for the emotion state data 342 may be determined as a random value, and the value of the emotion state data 342 may be updated based on the face animation output value and the face animation label. For example, an initial vector for the emotional state data 342 may be randomly determined based on a Gaussian distribution. Details of the learning of emotional state data will be described later in detail.

The processor 120 may calculate the voice feature 330 based on the second sub network function 340 to output a sub feature vector related to a face pose. The processor 120 may output a feature vector 400 relating to the face pose by correcting the sub feature vector relating to the face pose based on the emotion state data 342.

The processor 120 may calculate the voice feature 330 and the emotion state data 342 using the second sub-network function 340 including at least two convolutional layers and a combination layer. The convolutional layer and the combination layer exist as a pair, and may be located in the second sub-network function 340 in the order of a convolutional layer, a combination layer, a convolutional layer, and a combination layer. Each of the two or more combined layers may output a sub-feature vector related to a face pose based on the output of the immediately preceding convolutional layer and the emotion state data 342. The processor 120 may use the sub-feature vector for the face pose output from each of the two or more combining layers as an input of a convolutional layer located next to the combining layer or as an input of a second network function.

The processor 120 uses the voice feature 330 as an input of the first layer 343 included in the second sub-network function 340 to generate a sub feature vector for a first face pose based on the voice feature 330. Can be printed. The first layer 343 may be a convolutional layer.

The processor 120 receives the subfeature vector related to the first face pose and the emotion state data 342 as inputs to the second layer 344 included in the second subnetwork function 340, and Feature vectors can be output. The second layer 344 may be a concatenation layer. The second layer 344 may perform an operation based on the sub-feature vector related to the first face pose and the emotion state data 342. The second layer 344 may correct the sub-feature vector related to the first face pose based on the emotion state data 342. The second layer 344 may output a sub feature vector for a first face pose and a sub feature vector for a second face pose based on the emotion state data 342.

The processor 120 may output the sub feature vector for the third face pose by using the sub feature vector for the second face pose as an input of the third layer. The third layer may be a layer closer to the output layer than the first layer 343. The third layer may be a convolutional layer. The third layer may output a sub-feature vector having a size smaller than that of the first layer.

The processor 120 receives the sub-feature vector related to the third face pose and the emotion state data 342 as inputs of the fourth layer included in the second sub-network function 340 to obtain a sub-feature vector related to the fourth face pose. Can be printed. The fourth layer may be a combination layer. The fourth layer may output a sub-feature vector having a size smaller than that of the second layer.

For example, when the outputs of the first layer 343 and the second layer 344 included in the second sub-network function 340 are (256+N)*32*1, the third layer and the fourth layer The output of may be (256+N)*16*1. Detailed description of the size of the above-described layer output is only an example, and the present disclosure is not limited thereto.

The processor 120 may generate the face animation 600 by calculating the feature vector 400 for the face pose using the second network function 500 including two or more deconvolutional layers. The processor 120 may calculate a feature vector 400 for a face pose using a second network function 500 including two or more deconvolutional layers. The sizes of two or more deconvolutional layers included in the second network function 500 may gradually increase. The size of the first deconvolutional layer included in the second network function 500 and the size of the second deconvolutional layer adjacent to the first deconvolutional layer may be a layer that does not differ by more than a threshold value. Since the sizes of two or more deconvolutional layers included in the second network function 500 gradually increase toward the output layer, a face animation may be gradually predicted based on the size. A detailed description of the second network function including the two or more deconvolutional layers described above is only an example, and the present disclosure is not limited thereto.

The processor 120 may determine the positions of two or more vertices included in the face by using the feature vector 400 related to the face pose. Vertices may be points that are the basis of facial animation. The processor 120 may determine a shadow, an outline, and the like of the face based on the position of the vertex. The processor 120 may generate a face animation by connecting two or more vertices. The processor 120 may generate the face animation based on the positions of two or more vertices included in the face. For example, the vertex may be a coordinate of a point determined to generate a 3D animation. Detailed description of the above-described vertex is only an example, and the present disclosure is not limited thereto.

Hereinafter, a method of training a face animation generation model according to an embodiment of the present disclosure will be described.

5 is a diagram illustrating a method of training a face animation generation model according to an embodiment of the present disclosure.

The face animation generation model 800 including the first network function 300 and the second network function 500 for generating the face animation based on the voice data 200 is a training voice data 200 As an input, a face animation based on the learning voice data 200 may be learned based on a training data set including two or more training data having a label 604.

Hereinafter, a method of generating a training data set to train a face animation generation model according to an embodiment of the present disclosure will be described.

The processor 120 may generate a training data set including one or more training data having a face animation based on the training voice data as an input and a label of the training voice data. The training voice data may be voice data that is a basis for learning the face animation generation model 800. The training voice data may be data including sound information output to a game player that is a basis for learning the face animation generation model 800. For example, the learning voice data may be data on sound uttered by a character of a game player or an NPC. The specific description of the learning voice data is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training voice data that is the basis of the training data set for training the face animation generation model 800 may be voice data based on a pangram. Fangram can mean a sentence created using all the letters included in the language. For example, in the case of English, voice data may be generated based on sentences made using all alphabets included in English. The English fangram may be, for example, the quick brown fox jumps over the lazy dog, and the French fangram may be, for example, Buvez de ce whiskey que le patron juge fameux. The voice that the user reads the fangram aloud may be the learning voice data. The detailed description of the above-described learning voice data is only an example, and the present disclosure is not limited thereto.

The processor 120 may generate training data by determining a face animation to be output as a label corresponding to the training voice data for learning of the face animation generating model 800. The label-in-face animation may be an animation based on learning voice data. For example, a face animation, which is a label, may be an animation expressed by a character or NPC generated based on the learning voice data. The specific description of the above-described label is only an example, and the present disclosure is not limited thereto.

The processor 120 may augment the training data to generate the training data set.

The training data set may include first training data and second training data.

The second learning data includes, as input, second learning speech data generated by moving at least one section of the first learning speech data, which is an input of the first learning data, by a predetermined time unit, and is a label of the first learning data. A second face animation generated by converting the first face animation to correspond to the first learning voice data may be used as a label.

The processor 120 may time shift a section of the first voice data by a predetermined time unit to generate the second voice data. For example, when there is voice data corresponding to 0 to 10 seconds, the processor 120 advances the voice data of the 3 to 4 second interval by 1 second, and moves the voice data for the 2 to 3 second interval. And the existing voice data overlap, and the voice data is converted to 0 for a period of 3 to 4 seconds, thereby generating second voice data. The detailed description of the second voice data described above is merely an example, and the present disclosure is not limited thereto.

The processor 120 may generate the second face animation by converting the first face animation, which is a label of the first voice data, to correspond to generating the second voice data based on the first voice data. Based on moving the first voice data by a predetermined time unit, the processor 120 may generate a second face animation by moving a vertex of a section of the first face animation by a predetermined time. In the above-described example, when the second voice data is generated by advancing the voice data of the first voice data in a period from 3 to 4 seconds by 1 second, the vertex corresponding to the face animation in the period from 3 to 4 seconds is advanced by 1 second, A second face animation may be generated by combining with a vertex corresponding to a face animation in a period of 2 to 3 seconds. The specific description of the second facial animation described above is only an example, and the present disclosure is not limited thereto.

The processor 120 may train a face animation generation model 800 including a first network function 300 and a second network function 500 based on the training data set. The first network function 300 is a network function for outputting a feature vector 400 related to a face pose based on the learning voice data, and the second network function 500 is a feature vector related to the face pose. It may be a network function for generating the face animation 600 based on 400. The first network function 300 may include two or more convolutional layers, and the second network function 500 may include two or more fully connected layers.

The processor 120 calculates and calculates the learning voice data 200 as an input of the face animation generation model 800 and calculates the error between the output 602 and the label 604 included in the training data. It can be learned on the basis of.

The error is based on at least one of a difference between the position of two or more vertices of each of the face animation, which is the label 604 included in the output 602 and the training data, and whether the speed of motion included in the output face animation is appropriate. Can be determined. Facial animation may be generated based on two or more vertices. The error may be determined based on the difference between the positions of two or more vertices included in the face animation of each of the output 602 and the label 604. The error may be determined based on the number of portions in which the vertex positions differ, a distance based on the vertex positions, and the like. An error may be determined based on a time change of the output face animation and a position of a vertex according to time change. The error may be determined based on a difference between the position of the first vertex included in the face animation at the first time and the position of the second vertex included in the face animation at the second time. A threshold value of a distance between the first vertex and the second vertex may be determined based on the speed at which the human facial muscles move. When the difference between the distance between the position of the first vertex at the first time and the position of the second vertex at the second time is greater than the threshold value, the output 602 and the label 604 of the face animation generation model 800 It can be determined that there is an error of. A detailed description of the error based on the learning of the above-described face animation generation model is only an example, and the present disclosure is not limited thereto.

An error calculated based on the output 602 and the label 604 of the face animation generation model 800 may be backpropagated from the output layer in the reverse direction of the face animation generation model 800 to the input layer direction.

The processor 120 sets the update degree of the weight for the first network function 300 and the update degree of the weight for the second network function 500 differently to train the face animation generation model 800. I can. The processor 120 differently sets the update degree of the weight for the first network function 300 and the update degree of the weight for the second network function 500 during a predetermined epoch, so that the face animation generation model 800 ) Can be learned. For example, a predetermined epoch could mean the first 10 epochs after the start of learning. The specific description of the above-described predetermined epoch is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the second network function 500 for training by setting differently the update degree of the weight of the first network function 300 and the update degree of the weight of the second network function 500 is It may be a network function based on a fully connected layer. The second network function 500 may include two or more fully connected layers. The second network function included in the face animation generation model may include a first layer and a second layer. Each of the first layer and the second layer may be a fully connected layer. The size of the first layer and the second layer may be different. The size of the data output calculated based on the first layer and the second layer may be different. For example, the size of the data output calculated based on the first layer and outputted may be 150, and the size of the data output calculated based on the second layer and outputted may be 15000. The second layer may be a layer closer to the output layer than the first layer. The processor 120 may calculate the output of the first layer as an input of the second layer. The first layer and the second layer may be adjacent layers. The detailed description of the above-described first layer and second layer is only an example, and the present disclosure is not limited thereto.

If the size of an adjacent layer among two or more layers included in the second network function differs by more than a threshold value, there may be attenuation in the update degree of the weight of the first network function located in front of the second network function when the face animation generation model is trained. I can. Therefore, in order to prevent attenuation of the weight of the first network function, the face animation generation model can be trained by setting different degrees of update of the weights for the first network function and the second network function during a predetermined epoch during training. .

The processor 120 may train the face animation generation model 800 by setting the update degree of the weight for the first network function 300 to be greater than the update degree of the weight for the second network function 500. have. The processor 120 may not update the weights for the second network function 300 and may update the weights for the first network function 300. When the processor 120 performs backpropagation based on an error of the face animation, which is a label included in the training data and an output obtained by calculating the training voice data as an input of the face animation generation model, the second network Excluding the function, the weight can be updated only for the first network function. When backpropagating based on an error, the processor 120 may not update the weight of the second network function 500 and may update only the weight of the first network function 300. The processor 120 may update the weight based on the error of only the first network function 300. The processor 120 updates the link between two or more layers by propagating an error from the output layer included in the first network function 300 to the input layer through one or more hidden layers excluding the second network function 500. You can perform the operation

The processor 120 converts the training voice data 200 included in the training data into the training voice data 200 based on the first network function 300 and the second network function 500 included in the face animation generation model 800. ) May be performed to generate the output 602 face animation. During a predetermined epoch after the start of learning, the processor 120 performs an operation on the training speech data 200 based on the first network function 300 and the second network function 500, while a weight based on error The update of is based only on the first network function 300 and may not be based on the second network function 500. For example, the processor 120 may perform an update of the weight based on the error only for the first network function 300 during the first 10 epochs after learning starts. The specific description of the above-described predetermined epoch is merely an example, and the present disclosure is not limited thereto.

The processor 120 may perform training of the face animation generation model 800 by updating weights for each of the first network function 300 and the second network function 500 after a predetermined learning epoch. have. The processor 120 sets the update degree of the weight for the first network function 300 and the update degree of the weight for the second network function 500 to be the same after a predetermined learning epoch to generate the face animation model. You can learn (800). The processor 120 may perform an update of the weight based on the error during the predetermined learning epoch only for the first network function 300, and for learning after the predetermined learning epoch, the first network function 300 and the second Weights may be updated for the network function 500. The processor 120 propagates the error from the output layer included in the second network function 500 to the input layer after a predetermined learning epoch, and at least one hidden layer from the output layer included in the first network function 300 By propagating the error to the input layer through the second network function 500 and the first network function 300, each of the weights of two or more links included in each may be updated. For example, when the predetermined learning epoch is 10 epochs, the weights may be updated for each of the first network function 300 and the second network function 500 for learning after the weight of 10 epochs is updated. . The detailed description of the above-described learning is only an example, and the present disclosure is not limited thereto.

In learning according to an embodiment of the present disclosure, connection weights of nodes of each layer may be updated according to backpropagation backpropagated from an output layer to an input layer. A change amount may be determined according to a learning rate in the connection weight of each node to be updated. Calculation of network functions on input data and backpropagation of errors can constitute a learning cycle. The learning rate may be applied differently according to the number of repetitions of the learning cycle of the network function. For example, in the initial learning of a network function, a high learning rate is used to allow the network function to quickly secure a certain level of performance, thereby increasing efficiency. For example, in the later stages of learning of a network function, accuracy can be improved by using a low learning rate.

In generating the face animation generation model 800, the processor 120 may set a dropout so that a part of the output of the hidden node is not transmitted to the next hidden node in order to prevent overfitting.

The training epoch inputs training speech data 200 for all training data included in the training data set into each of one or more input nodes included in the input layer of one or more network functions of the face animation generation model 800. And, by comparing the facial animation (i.e., correct answer) labeled in the learning voice data 200 and the facial animation (i.e., output) of the face animation generation model, an error is derived, and the derived error is one of the facial animation generation models. It may be an operation of updating weights set for each link by propagating from the output layer of the network function to the input layer through one or more hidden layers. That is, when the calculation using the face animation generation model 800 and the weight update process of the face animation generation model 800 are performed on all training data included in the training data set, it may be 1 epoch.

In generating the face animation generation model 800, the processor 120 pre-determines the learning rate of the face animation generation model 800 when the learning epoch for training the face animation generation model 800 is less than a predetermined epoch. It can be set above the determined value. In generating the face animation generation model 800, the processor 120 advances the learning rate of the face animation generation model 800 when the learning epoch for training the face animation generation model 800 is equal to or greater than a predetermined epoch. It can be set below the determined value. The learning rate may mean an update degree of a weight. For example, by setting a high learning rate at the beginning of learning (that is, increasing the degree of updating of the weights), the output of the training data can quickly access the label of the training data. For example, in the second half of training, the learning rate is set low (i.e., the update degree of the weight is small), so that the error between the output of the training data and the label of the training data is reduced (i.e., to increase accuracy). can do. The disclosure of the above-described learning rate is merely an example, and the present disclosure is not limited thereto.

The processor 120 may perform learning of the face animation generating model 800 by more than a predetermined epoch, and then determine whether to stop learning by using the verification data set. The predetermined epoch may be part of the overall learning target epoch. The processor 120 may use a part of the training data set as a verification data set. The verification data is data corresponding to the training data, and may be data for determining whether to stop learning. The processor 120 may determine whether the learning effect of the face animation generating model 800 is equal to or higher than a predetermined level by using the verification data after learning of the face animation generating model 800 is repeated by a predetermined epoch or more. For example, if the processor 120 performs learning with a target repetition learning number of 100,000 times using 1 million learning data, it performs repetitive learning of 10,000 times, which is a predetermined epoch, and then uses 1000 verification data. Thus, by performing 10 iterative learning (that is, 10 epochs), if the change in the output of the neural network is less than a predetermined level during 10 iterative learning, it is determined that further learning is meaningless and the learning can be terminated. That is, the verification data may be used to determine completion of learning based on whether the effect of learning for each epoch in the iterative learning of the neural network is greater than or equal to a certain level. The above-described number of training data and verification data and the number of repetitions are only examples, and the present disclosure is not limited thereto.

The processor 120 may use at least one of the training data sets as a test data set. The test data is data corresponding to the training data, and may be data used to verify performance after learning of the face animation generating model 800 is completed. The processor 120 may use one or more voice data and a face animation labeled with the voice data as a test data set. The processor 120 inputs voice data included in the test data set to the face animation generation model 800, compares the output output from the face animation generation model 800 with the labeled face animation, and compares the labeled face animation with the test data set. It is possible to determine a correct answer rate of the face animation generation model 800 for. The processor 120 inputs voice data included in the test data into the face animation generation model 800, and the face animation (ie, output) output from the face animation generation model 800 and the test data When face animations (ie, correct answers) are compared and an error is less than a predetermined value, activation of the face animation generation model 800 may be determined. The processor 120 compares the face animation (i.e., output) output from the face animation generation model 800 with the face animation included in the test data (i.e., the correct answer), and when the error is greater than or equal to a predetermined value, the face Learning of the animation generation model 800 may be further performed by a predetermined epoch or more, or the face animation generation model 800 may be deactivated. When the processor 120 deactivates the face animation generation model 800, the face animation generation model 800 may be discarded. According to an embodiment of the present disclosure, the processor 120 may independently learn one or more network functions included in each face animation generation model 800 to generate a plurality of face animation generation models 800, and performance By evaluating, only neural networks with a certain performance or higher can be used for facial animation generation.

According to an embodiment of the present disclosure, the face animation generation model 800 is trained by adding noise of a predetermined size to an operation for each of two or more convolutional layers included in the first network function 300 Can be. The noise of a predetermined size may be noise applied when an operation of each of two or more convolutional layers is performed. The above-described operation may be any operation including a multiplication operation. The processor 120 may perform learning by adding noise of a predetermined size to a multiplication operation performed in the convolutional layer included in the first network function 300. The processor 120 may apply noise of a predetermined size to each of two or more convolutional layers included in the first network function 300. The processor 120 may apply noise of a predetermined size to each of two or more feature maps learned based on the first network function 300. The detailed description of the above-described learning is only an example, and the present disclosure is not limited thereto.

Hereinafter, a method of learning emotional state data will be described.

The processor 120 may determine random emotional state data and perform calculation of the face animation generation model 800. The processor 120 outputs 602 through the calculation of the face animation generation model 800 based on the voice data 200 and the emotional state data determined as an initial value, and the label 604 of the voice data 200 Emotional state data determined as an initial value may be changed based on an error of the in-face animation. Processor 120 may update the emotional state data by backpropagating based on the error between the output 602 and the label 604. For example, based on the emotion state data related to sadness, the error between the facial animation output 602 from the face animation generation model 800 and the facial animation as the label 604 is greater than or equal to the threshold value, and the emotion state data related to joy If the error between the face animation output 602 from the face animation generation model 800 and the face animation that is the label 604 is less than a threshold value, emotional state data related to the voice data 200 may be determined as joy. . The detailed description of the above-described learning of the emotional state data is only an example, and the present disclosure is not limited thereto.

6 is a flowchart illustrating a method of generating a face animation based on a face animation generation model according to an embodiment of the present disclosure.

The computing device 100 may receive 610 voice data. The voice data 200 may include data on a person's voice. For example, when a game provides a quest to game players, the game quest may be delivered using animation. In this case, the voice data 200 may be recorded by voice of the contents of the game quest. The description of the above-described voice data is only an example, and the present disclosure is not limited thereto.

The computing device 100 may calculate the voice data using a first network function including two or more convolutional layers to output 620 a feature vector for a face pose for generating a face animation. The first network function may include a first sub network function and a second sub network function.

The computing device 100 may undergo a preprocessing process before inputting the voice data into the face animation generation model. The preprocessing of the voice data may mean partially correcting the waveform of the voice data. The computing device 100 may normalize the highest frequency or lowest frequency of speech to correspond to a predetermined value.

The computing device 100 may convert the voice data into image data based on the voice data. The computing device 100 may convert the preprocessed voice data into image data based on the voice data in order to calculate voice data based on the convolutional layer. The function for converting voice data into image data based on the voice data may be a predetermined image data conversion function. The predetermined image data conversion function may be updated based on an error when the face animation generation model is trained, or may be converted into image data on a fixed basis without updating when the face animation generation model is trained. The predetermined image data conversion function may convert into image data based on the frequency of the voice data.

The computing device 100 may calculate the image data using the first network function and output a feature vector for the face pose.

The computing device 100 may calculate the image data based on the first sub-network function and output a voice characteristic. The first sub-network function may include two or more convolutional layers. The voice feature may include a feature in which the input voice data can be distinguished from other voice data. The voice characteristic may be information about intonation of voice, emphasis of voice, specific phoneme of voice, and the like. The voice characteristic may be information calculated based on image data converted from the voice data.

The computing device 100 may calculate the voice feature based on the second sub-network function and output a feature vector for the face pose. The second sub network function may include two or more convolutional layers and correction layers.

The computing device 100 may input emotional state data related to the voice data to the second sub-network. The computing device 100 may calculate the voice feature based on the second sub network function to output a sub feature vector related to a face pose. The computing device 100 may output a feature vector for the face pose by correcting the sub feature vector for the face pose based on the emotion state data. The emotional state data may be data indicating classification of an emotional state. The emotional state data may be an N-dimensional vector for indicating a classification of an emotional state. The emotional state data may be a vector in a form that can be calculated in each of two or more combining layers included in the second sub-network function. Emotional state data according to an embodiment of the present disclosure may be data manually labeled with voice data. Emotional state data according to an embodiment of the present disclosure may be learned by learning a face animation generation model.

The computing device 100 may generate 630 a face animation by calculating a feature vector for the face pose using a second network function including two or more deconvolutional layers. The sizes of two or more deconvolutional layers included in the second network function may gradually increase. The size of the first deconvolutional layer included in the second network function and the size of the second deconvolutional layer adjacent to the first deconvolutional layer may be a layer that does not differ by more than a threshold value.

The computing device 100 may determine positions of two or more vertices included in the face by using the feature vector for the face pose. The computing device 100 may generate the face animation based on the positions of two or more vertices included in the face.

The face animation generation model including the first network function and the second network function for generating the face animation based on the voice data receives training voice data as an input, and labels a face animation based on the training voice data. The facial animation, which is a label included in the training data, and an output obtained by calculating the training voice data as an input of the facial animation generation model and being trained based on a training data set including two or more training data It may be a model learned based on the error of.

The training data set includes first training data and second training data, and the second training data is generated by moving at least one section of the first training voice data, which is an input of the first training data, by a predetermined time unit. One second learning voice data may be input, and a second face animation generated by converting a first face animation, which is a label of the first training data, to correspond to the first training voice data, may be used as a label. The error may be determined based on at least one of a difference between the output and a position of two or more vertices of each of the face animation, which is a label included in the training data, and whether a motion speed included in the output face animation is appropriate.

The face animation generation model may be a model learned by adding noise of a predetermined size to an operation for each of two or more convolutional layers included in the first network function.

The learning voice data may be voice data based on a pangram.

7 is a flowchart illustrating a method of learning a face animation generation model according to an embodiment of the present disclosure.

The computing device 100 may generate (710) a training data set including one or more training data having a face animation based on the training voice data as an input and a label of the training voice data.

The computing device 100 may train 720 a face animation generation model including a first network function and a second network function based on the training data set.

The computing device 100 may train 730 the face animation generation model by differently setting an update degree of a weight for the first network function and an update degree of a weight for the second network function. The first network function is a network function for outputting a feature vector related to a face pose based on the training speech data, and the second network function is configured to generate the face animation based on a feature vector related to the face pose. May be a network function for The first network function may include two or more convolutional layers, and the second network function may include two or more fully connected layers.

The computing device 100 may train the face animation generation model by setting an update degree of the weight for the first network function to be greater than an update degree of the weight for the second network function. When the computing device 100 performs backpropagation based on an error of the facial animation, which is a label included in the training data and an output obtained by calculating the training speech data as an input of the face animation generation model, the second Excluding the network function, the weight can be updated only for the first network function. The computing device 100 may train the face animation generation model by differently setting an update degree of a weight for the first network function and an update degree of a weight for the second network function during a predetermined epoch. The second network function includes a first layer and a second layer, and sizes of the first layer and the second layer may be different. The second layer may be a layer closer to an output layer than the first layer.

The computing device 100 may train the face animation generation model by setting the update degree of the weight for the first network function and the update degree of the weight for the second network function equally after a predetermined learning epoch. .

8 is a block diagram of a computing device according to an embodiment of the present disclosure.

8 shows a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

While the present disclosure has generally been described above with respect to computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and/or as a combination of hardware and software. will be.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Further, to those skilled in the art, the method of the present disclosure is not limited to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable household appliances, etc. It will be appreciated that it may be implemented with other computer system configurations, including one or more associated devices).

The described embodiments of the present disclosure may also be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium accessible by the computer can be a computer-readable medium. Computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Includes the medium. Computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device, or other magnetic storage device, Or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as other transport mechanism, and includes all information delivery media. do. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal is set or changed to encode information in the signal. By way of example, and not limitation, computer-readable transmission media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above-described media are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment 1100 is shown that implements various aspects of the present disclosure including a computer 1102, which includes a processing device 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to, system memory 1106 to processing device 1104. The processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

The system bus 1108 may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read-only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110 such as ROM, EPROM, EEPROM, etc. This BIOS is a basic input/output system that helps transfer information between components in the computer 1102, such as during startup. Includes routines. RAM 1112 may also include high speed RAM such as static RAM for caching data.

The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)-this internal hard disk drive 1114 can also be configured for external use within a suitable chassis (not shown). Yes-, magnetic floppy disk drive (FDD) 1116 (for example, to read from or write to removable diskette 1118), and optical disk drive 1120 (e.g., CD-ROM To read the disk 1122 or read from or write to other high-capacity optical media such as DVD). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are each connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128. ) Can be connected. The interface 1124 for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of the computer-readable medium above refers to a removable optical medium such as a HDD, a removable magnetic disk, and a CD or DVD, those skilled in the art may use a zip drive, a magnetic cassette, a flash memory card, a cartridge, etc. It will be appreciated that other types of media readable by a computer, such as the like, may also be used in the exemplary operating environment and that any such media may contain computer executable instructions for performing the methods of the present disclosure.

A number of program modules, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136, may be stored in the drive and RAM 1112. All or part of the operating system, applications, modules, and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on a number of commercially available operating systems or combinations of operating systems.

A user may input commands and information to the computer 1102 through one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through the input device interface 1142, which is connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, It can be connected by other interfaces such as etc.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communication. The remote computer(s) 1148 may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other common network node, and is generally connected to the computer 1102. Although it includes many or all of the components described for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or to a larger network, eg, a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies, and facilitate an enterprise-wide computer network such as an intranet, all of which can be connected to a worldwide computer network, for example the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and/or wireless communication network interface or adapter 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connected to a communications computing device on the WAN 1154, or through the Internet, to establish communications over the WAN 1154. Have other means. The modem 1158, which may be an internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for the computer 1102 or portions thereof may be stored in the remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing communication links between computers may be used.

Computer 1102 is associated with any wireless device or entity deployed and operated in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communication satellite, wireless detectable tag. It operates to communicate with any device or place and phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet, etc. without wires. Wi-Fi is a wireless technology such as a cell phone that allows such devices, for example computers, to transmit and receive data indoors and outdoors, ie anywhere within the coverage area of a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a,b,g, etc.) to provide a secure, reliable and high-speed wireless connection. Wi-Fi can be used to connect computers to each other, to the Internet and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz radio bands, for example at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). have.

Those of ordinary skill in the art of the present disclosure will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. Or particles, or any combination thereof.

A person of ordinary skill in the art of the present disclosure includes various exemplary logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein, electronic hardware, (convenience). For the sake of clarity, it will be appreciated that it may be implemented by various forms of program or design code or a combination of both (referred to herein as "software"). To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. A person of ordinary skill in the art of the present disclosure may implement the described functions in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CD, DVD, etc.), smart cards, and flash memory. Devices (eg, EEPROM, card, stick, key drive, etc.), but is not limited to these. In addition, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on the design priorities, it is to be understood that within the scope of the present disclosure a specific order or hierarchy of steps in processes may be rearranged. The appended method claims provide elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (1)

A computer program stored on a computer-readable storage medium.

Images