KR20220034396A - Device, method and computer program for generating face video - Google Patents

Abstract

A device that generates a face image comprises: an input part that receives the speech data and an original face image for a speaker; a speech characteristic information extraction part that analyzes the speech data and extracts the speech characteristic information; a face image extraction part that extracts a face image through a face image storage part based on the extracted speech characteristic information; a learning part that learns a face generation model based on the speech data and the extracted face image; and a generation part that generates the face image for the unidentified speech data through the learned face generation model. Therefore, the present invention is capable of generating a real face image with high accuracy.

Description

DEVICE, METHOD AND COMPUTER PROGRAM FOR GENERATING FACE VIDEO

It relates to an apparatus, method, and computer program for generating a face image based on voice data.

In general, a method of generating a face image based on voice data is configured to generate a face image only with frequency image information converted by converting voice data into a frequency image, and an arbitrary face composed of animations rather than the face of the actual speaker create

Accordingly, in the prior art, there is a difficulty in generating an accurate actual face image of a speaker or inferring information about the speaker only with voice data. In this regard, Korean Patent Application Laid-Open No. 10-2019-0046371, which is a prior art, discloses an apparatus and method for generating facial expressions.

The conventional facial expression generating apparatus may generate a virtual character facial expression by estimating a vowel included in the input speaker's voice, and reflecting and combining weights with a plurality of standard expressions predefined as the estimated vowel.

However, as described above, the conventional facial expression generating apparatus only generates a virtual character facial expression, not an actual facial image of the speaker, by combining predefined standard facial expressions only with vowel information included in the speaker's voice. There is a difficulty in generating the actual face image of the speaker and information about the speaker.

Korean Patent Publication No. 10-2019-0046371 (published on May 7. 2019) Korean Patent Publication No. 10-2096598 (Registered on March 27, 2020)

The present invention is to solve the problems of the prior art, and to provide a face image generating apparatus, method, and computer program capable of generating an actual face image of a speaker based on a result of analyzing voice data.

In addition, the present invention not only generates the actual face image of the speaker by analyzing the contents of speech included in the voice data, but also derives the psychological state of the speaker to generate speaker information, and based on this, the non-face-to-face speaker (suspect) is identified. An object of the present invention is to provide an apparatus, method, and computer program for generating a face image that can be tracked (arrested).

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides an apparatus for generating a face image based on voice data, comprising: an input unit for receiving voice data and an original face image of a speaker; a voice characteristic information extraction unit that analyzes the voice data and extracts voice characteristic information; a face image extracting unit for extracting a face image through a face image storage unit based on the extracted voice feature information; a learning unit for learning a face generation model based on the voice data and the extracted face image; and a generator for generating a face image with respect to the de-identified voice data through the learned face generating model.

Another embodiment of the present invention provides a method for generating a face image based on voice data, the method comprising: receiving voice data and an original face image of a speaker; extracting voice characteristic information by analyzing the voice data; extracting a face image through a face image storage unit based on the extracted voice feature information; training a face generation model based on the voice data and the extracted face image; and generating a face image for de-identified voice data through the learned face generating model.

Another embodiment of the present invention provides a computer program stored in a computer-readable recording medium including a sequence of instructions for generating a face image based on voice data, wherein the computer program is executed by a computing device to be Receive voice data and original face image for a person, extract voice feature information by analyzing the voice data, extract a face image through a face image storage unit based on the extracted voice feature information, and extract the voice data and the extract A computer program stored in a computer-readable recording medium comprising a sequence of instructions for training a face generation model based on the acquired face image and generating a face image for unidentified voice data through the learned face generation model. can provide

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to analyze the collected voice data and generate body information and emotional information about the talker, so that the actual face image of the speaker reflecting the speaker's expression can be generated with high accuracy. An apparatus, method, and computer program for generating a face image may be provided.

In addition, the face image generated through the face image model is compared and analyzed with the original face image of the speaker, and the result of comparative analysis is compared and analyzed again with the speaker's voice feature information. By doing so, it is possible to provide a face image generating apparatus, method, and computer program capable of generating a more accurate face image of the speaker through the face generating model.

In addition, by analyzing the speech information included in the collected voice data, the psychological state of the speaker is identified and the speaker information is generated, thereby providing information to track (arrest) the non-face-to-face speaker (suspect). Face image Generating devices, methods, and computer programs may be provided.

1 is a block diagram of an apparatus for generating a face image according to an embodiment of the present invention.
2 is an exemplary view for explaining a voice feature information extracting unit according to an embodiment of the present invention.
3 is an exemplary view for explaining a face image extractor according to an embodiment of the present invention.
4 is an exemplary view for explaining a learning unit according to an embodiment of the present invention.
5 is an exemplary view for explaining a generator according to an embodiment of the present invention.
6 is a flowchart of a method for generating a face image according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating a face image according to an embodiment of the present invention. Referring to FIG. 1 , the face image generating apparatus 100 may include an input unit 110 , a voice feature information extracting unit 120 , a face image extracting unit 130 , a learning unit 140 , and a generating unit 150 . can Components that can be controlled by the face image generating apparatus 100 are illustrated by way of example.

Each component of the face image generating apparatus 100 of FIG. 1 is generally connected through a network. A network refers to a connection structure in which information can be exchanged between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The input unit 110 according to an embodiment of the present invention may receive voice data and an original face image of the speaker. For example, the input unit 110 may transmit voice data of the received speaker to the voice characteristic information extracting unit 120 .

2 is an exemplary view for explaining a voice feature information extracting unit according to an embodiment of the present invention. Referring to FIG. 2 , the voice characteristic information extractor 120 may analyze the voice data 20 to extract the voice characteristic information 230 .

The voice characteristic information extractor 120 may convert the voice data 20 into the frequency image 210 , and analyze the converted frequency image 210 to extract the voice characteristic information 230 . For example, the voice feature information extraction unit 120 extracts the voice data 20 from Mel (Mel-scaled power spectrogram), MFCC (Mel-frequency Cepstral Coefficient), Chroma (power spectrogram Chroma), Contrast (spectral contrast), It can be converted into Tonnetz (tonal centroid feature) and the like and used as an input of the voice analysis model 220 . Here, Mel, MFCC, Chroma, Contrast, and Tonnetz are examples, and different types of features may be used according to the voice analysis model 220 .

Referring to FIG. 2 , the voice feature information extraction unit 120 uses various types of voice analysis models 220 to extract information such as frequency bands, voice strengths, and utterance times of voices extracted from voice data 20 . The voice characteristic information 230 may be output by analyzing the speaker's age, gender, race, utterance content, emotion, and the like. Here, the voice analysis model 220 shown in FIG. 2 is an example, and in some cases, the voice analysis model 220 that analyzes other information may be added or deleted. The voice characteristic information extraction unit 120 according to an embodiment of the present invention analyzes the content of speech and the strength, height, frequency domain, etc. of the voice included in the voice data 20 , and adds the extracted voice characteristic information 230 to the extracted voice characteristic information 230 . Based on this, it is possible to more accurately generate the speaker's face image reflecting the speaker's expression.

The voice characteristic information 230 according to an embodiment of the present invention may include body information 231 of the speaker including information on at least one of gender, age, and race of the speaker. For example, the voice feature information extraction unit 120 uses the voice analysis model 220 such as an age analysis model, a gender analysis model, and a race analysis model based on the converted frequency image 210 to determine the height and intensity of the voice. By analyzing the speaker, information on the speaker's gender (eg, male), age (eg, 20's), and race (eg, Asian) can be extracted.

Also, the voice characteristic information 230 may include emotional information 232 about the speaker. For example, the voice feature information extraction unit 120 uses the voice analysis model 220 such as an emotion analysis model and a speech content analysis model based on the converted frequency image 210 to obtain the height and intensity of the voice and voice data. By analyzing the content of the utterance included in (20), the speaker's emotions (eg, happiness) can be analyzed. Specifically, the voice feature information extraction unit 120 may, for example, when the voice data 20 includes utterance contents such as “I was really happy today”, not only the utterance contents, but also the height, intensity, and utterance time of the voice. It is possible to determine whether the content of the utterance contains an objectionable expression by analyzing the utterance, and based on this, it is also possible to analyze the speaker's emotions at the time of the utterance.

3 is an exemplary view for explaining a face image extractor according to an embodiment of the present invention. Referring to FIG. 3 , the face image extractor 130 may extract a face image 330 through the face image storage 320 based on the extracted voice feature information 310 .

The face image extractor 130 may generate the face image 330 by synthesizing at least one piece of facial feature information 321 matching the voice feature information 310 in the face image storage 320 .

The face image storage unit 320 according to an embodiment of the present invention includes skin color, face shape and eye color suitable for race, wrinkles suitable for age, face type suitable for gender, and mouth shape suitable for emotions and speech contents. and face image information corresponding to facial expressions and the like.

For example, the face image extraction unit 130 lists a plurality of pieces of facial feature information 321 that are matched in the face image storage unit 320 using the extracted speaker body information 311 and emotion information 312 . Then, it is possible to generate the face image 330 by synthesizing it after finding and searching. Specifically, referring to FIG. 3 , the face image extracting unit 130 may include, for example, information on faces in their twenties in the face image storage unit 320 based on age information (eg, in their twenties) in the body information 311 . can be searched for facial feature information 321 such as facial skin and wrinkles, Facial feature information 321 such as nose, mouth, face size, and eyes may be selected.

As another example, the face image extraction unit 130 extracts wrinkle information, skin color, eye color, Facial feature information 321 matching the hair color and face shape may be selected.

In addition, referring to FIG. 3 , the face image extractor 130 , for example, when the extracted speaker's emotional information 312 is a sad emotion, the facial image storage unit 320 shows an expression that may be displayed when the emotion is sad. Facial feature information 321 for eye, nose, mouth, and facial posture coordinate information for .

In this way, the face image extraction unit 130 searches and synthesizes facial feature information 321 matching the extracted speaker body information 311 and emotion information 312 in the face image storage unit 320 to obtain a face image. 330 may be generated.

According to an embodiment of the present invention, the face image extractor 130 searches for the facial feature information 321 from the face image storage 320 in proportion to the number of voice feature information 310 extracted from the voice data. can Therefore, as the number of voice feature information 310 extracted from voice data increases, the face image storage unit 320 can search for more facial feature information 321 for the corresponding voice data. Based on this, it is possible to more accurately synthesize the speaker's face image.

4 is an exemplary view for explaining a learning unit according to an embodiment of the present invention. Referring to FIG. 4 , the learning unit 140 may train the face generation model 420 based on the voice data and the extracted face image 412 . The learning unit 140 examines whether the face image 430 generated through the face generation model 420 has been properly generated, and reflects the review result in the learning of the face generation model 420 , the face generation model 420 . ) to learn to generate a more accurate speaker's face.

For example, the learning unit 140 combines the frequency image 411 obtained by converting the speaker's voice data and the face image 412 extracted by the face image extracting unit 130 based on the speaker's voice data to obtain a face. An input image 410 to be input to the generation model 420 may be generated. The learner 140 may input the input image 410 into the face generation model 420 to learn to generate the face image 430 . In this case, the learning unit 140 may use a generative adversarial network (GAN) model, and the generative adversarial network may be used by changing various types of models. A generative adversarial neural network is one of the machine learning methods that automatically creates images, videos, and voices close to the real thing while a generative model and a discriminant model compete with each other.

The learner 140 may reflect the result of comparison analysis between the face image 430 generated by the face generation model 420 and the speaker's original face image 440 . For example, the learner 140 may perform the image analysis 450 on the face image 430 generated by the face generation model 420 and the original face image 440 of the speaker.

Specifically, the learning unit 140 matches the deep learning feature values extracted from the face image 430 generated by the face generation model 420 and the original face image 440 of the speaker to obtain a difference between the feature values as a loss function. can be reflected in learning. The learning unit 140 may analyze the face image 430 using various types of image deep learning models such as age analysis models and gender analysis models for the face image 430 generated by the face generation model 420 . there is. 4 , the model used for the image analysis 450 is illustrated as an example of an age analysis model, a gender analysis model, a race analysis model, an expression analysis model, and the like, but variously added according to the voice characteristic information 460 . and may be deleted.

The learning unit 140 uses the result of the image analysis 450 analyzed on the face image 430 generated through the learned face generation model 420 and the original face image 440 of the face generation model 420 . can be reflected in learning.

The learner 140 may further reflect the result of comparison analysis between the face image 430 generated through the face generation model 420 and the speaker's voice characteristic information 460 . For example, the learning unit 140 compares and analyzes the face image 430 generated through the face generation model 420 and the original face image 440 to display the result of the image analysis 450 on the speaker's voice characteristic information. (460) can be compared and analyzed. Specifically, the learning unit 140 compares and analyzes the image analysis 450 result and the speaker's voice characteristic information 460, and if the analysis results do not match, it may not be reflected in the learning of the face generation model 420. there is.

This is because an analysis with high accuracy can be obtained only when two analysis results match. Therefore, learning to compare with an erroneous analysis result may cause an error in learning of the face generation model 420 .

Through this, the learning unit 140 compares and analyzes the face image 430 generated through the face generation model 420 and the original face image 440, the result of the image analysis 450 and the speaker's voice characteristic information ( By reflecting all of 460 in the training of the face generating model 420 , the face generating model 420 may be trained to generate a more accurate face image 430 of the speaker.

5 is an exemplary view for explaining a generator according to an embodiment of the present invention. Referring to FIG. 5 , the generator 150 may generate a face image 520 for de-identified voice data through a pre-learned face generation model 510 .

For example, the generator 150 may generate a face image 520 by analyzing de-identified voice data, and voice feature information 540 obtained by extracting the generated face image 520 from the de-identified voice data. A more accurate speaker's face image 520 may be generated based on the matching face information 530 by comparing and analyzing with .

The generator 150 uses the result of comparative analysis on the face information 530 obtained by analyzing the voice feature information 540 for the unidentified voice data and the face image 520 generated through the face generation model 510 . Thus, the speaker information 550 for the face image 520 may be generated.

For example, the generation unit 150 analyzes the face image 520 generated through the face generation model 510 using an age analysis model, a gender analysis model, a race analysis model, and an expression analysis model to analyze facial information ( The speaker information 550 including the speaker's facial expression and emotional state may be generated based on the matching information by comparing and analyzing the 530 and the voice characteristic information 540 .

Specifically, as for the speaker information 550 generated according to an embodiment, referring to FIG. 5 , the result of analyzing the face image 520 generated through the face generation model 510 using the age analysis model (eg, : 20's) and age information (eg, 20's) included in the voice feature information 540 are compared and analyzed and when the age information of "twenties" matches, "20's" for the face image 520 It may include the speaker information 550 of ".

As another example, the generating unit 150 generates a face image 520 through the face generation model 510, and compares and analyzes the generated face image 520 and the voice feature information 540 to determine the speaker's face, In addition to facial expression, age, gender, and race, in particular, since it is possible to generate the speaker information 550 deducing the authenticity of the contents of the utterance and the psychological state at the time, it is possible to estimate and arrest the face of criminals such as voice phishing criminals or kidnapping threats. You can provide helpful information.

Specifically, as for the speaker information 550 generated according to another embodiment, referring to FIG. 5 , the result of analyzing the face image 520 generated through the face generation model 510 using the expression analysis model (eg, : Sadness) and speech content information (eg, "I'm happy") included in the voice feature information 540 are compared and analyzed to determine the authenticity of the speech content with respect to the corresponding face image 520 is "a lie". information 550 .

As described above, the face image generating apparatus 100 according to an embodiment of the present invention tracks (arrests) the speaker (suspect) by generating information about the speaker as well as the actual face image of the speaker for the de-identified voice data. You can provide information that can help you do this.

For example, the face image generating apparatus 100 may analyze the age, gender, race, content and emotion of the speaker only with the unidentified voice data, and the speaker's face matched with the corresponding voice data based on the analysis result An image may be generated, and by analyzing the authenticity of the speaker's utterance with respect to the generated face image and the psychological state at the time of the utterance, it is possible to provide speaker information that can help track the speaker.

For another example, the face image generating apparatus 100 may analyze only the voice data input through a phone call or a recorded voice in order to arrest a suspect for a voice phishing crime and a kidnapping crime, and the corresponding voice data may be analyzed. It is possible to create a face image of a suspect based on the result of analyzing the voice data, and provides suspect information that can help in arresting the suspect by analyzing the authenticity of the suspect's utterances and the psychological state at the time of the utterance. can do.

6 is a flowchart of a method for generating a face image according to an embodiment of the present invention. The method for generating a face image shown in FIG. 6 includes steps of time-series processing according to the embodiment shown in FIGS. 1 to 5 . Therefore, even if omitted below, the method for generating a face image based on voice data in the face image generating apparatus 100 according to the embodiment shown in FIGS. 1 to 5 is also applied.

In operation S610, the apparatus for generating a face image may receive voice data and an original face image of the speaker.

In step S620, the face image generating apparatus may extract voice feature information by analyzing the voice data.

In operation S630, the face image generating apparatus may extract a face image through the face image storage unit based on the extracted voice feature information.

In operation S640, the face image generating apparatus may train a face generating model based on the voice data and the extracted face image.

In operation S650 , the face image generating apparatus may generate a face image for unidentified voice data through the learned face generating model.

In the above description, steps S610 to S650 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

The method for generating a face image based on voice data in the face image generating apparatus 100 described with reference to FIGS. 1 to 6 is a computer program stored in a computer-readable recording medium executed by a computer or instructions executable by the computer. It may also be implemented in the form of a recording medium including In addition, the method for generating a face image based on voice data in the face image generating apparatus 100 described with reference to FIGS. 1 to 5 is also implemented in the form of a computer program stored in a computer-readable recording medium executed by a computer. can be

A computer-readable recording medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable recording medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. do.

100: face image generating device
110: input unit
120: voice feature information extraction unit
130: face image extraction unit
140: study unit
150: generator

Claims (17)

An apparatus for generating a face image based on audio data, the apparatus comprising:
an input unit for receiving voice data and an original face image of the speaker;
a voice characteristic information extraction unit that analyzes the voice data and extracts voice characteristic information;
a face image extractor configured to extract a face image through a face image storage unit based on the extracted voice feature information;
a learning unit for learning a face generation model based on the voice data and the extracted face image; and
A generation unit that generates a face image for unidentified voice data through the learned face generation model
Containing, a face image generating device.
The method of claim 1,
The voice feature information includes body information of the speaker including information on at least one of gender, age, and race of the speaker.
The method of claim 1,
The voice characteristic information includes emotional information about the speaker.
The method of claim 1,
The face image extraction unit,
and generating the face image by synthesizing at least one piece of facial feature information matching the extracted voice feature information from the face image storage unit.
The method of claim 1,
The learning unit,
The face image generating apparatus of claim 1, wherein a result of comparative analysis between the generated face image and the original face image is reflected in the face generating model.
6. The method of claim 5,
The learning unit,
The face image generating apparatus of claim 1, wherein the result of comparison analysis between the generated face image and the voice feature information is further reflected in the face generating model.
The method of claim 1,
The generating unit,
and generating speaker information for the face image based on a result of comparative analysis of voice feature information on the de-identified voice data and the generated face information.
The method of claim 1,
The voice characteristic information extracting unit converts the voice data into a frequency image,
The apparatus for generating a face image, wherein the voice characteristic information extractor analyzes the converted frequency image and extracts the voice characteristic information.
A method for generating a face image based on audio data by a face image generating apparatus, the method comprising:
receiving voice data and an original face image of the speaker;
extracting voice characteristic information by analyzing the voice data;
extracting a face image through a face image storage unit based on the extracted voice feature information;
training a face generation model based on the voice data and the extracted face image; and
generating a face image for unidentified voice data through the learned face generation model;
Including, a face image generation method.
10. The method of claim 9,
The voice feature information includes body information of the speaker including information on at least one of gender, age, and race of the speaker.
10. The method of claim 9,
The voice feature information includes emotional information about the speaker.
10. The method of claim 9,
The step of extracting the face image,
and generating the face image by synthesizing at least one piece of facial feature information matching the extracted voice feature information from the face image storage unit.
10. The method of claim 9,
reflecting a result of comparative analysis between the generated face image and the original face image in the face generation model
The method of generating a face image further comprising a.
14. The method of claim 13,
Reflecting a comparative analysis result between the face image and the voice feature information in the face generation model
The method of generating a face image further comprising a.
10. The method of claim 9,
generating speaker information for the face image based on a result of comparative analysis of voice feature information on the de-identified voice data and the generated face information
The method of generating a face image further comprising a.
10. The method of claim 9,
converting the audio data into a frequency image; and
extracting the voice feature information by analyzing the converted frequency image
The method of generating a face image further comprising a.
A computer program stored in a computer-readable recording medium comprising a sequence of instructions for generating a face image based on voice data, the computer program comprising:
When the computer program is executed by a computing device,
Receive voice data and original face image of the speaker,
extracting voice characteristic information by analyzing the voice data;
extracting a face image through a face image storage unit based on the extracted voice feature information,
learning a face generation model based on the voice data and the extracted face image,
A computer program stored in a computer-readable recording medium comprising a sequence of instructions for generating a face image for de-identified voice data through the learned face generation model.

Images