KR20230151156A - Device for providing an avatar speech service synchronized with each speaker's voice and face image - Google Patents

Abstract

An avatar speech service providing device according to an embodiment of the present invention includes a collection unit that collects video information from one or more video platform servers; an extraction unit that extracts voice and face images for each speaker from the image information and provides time synchronization information; Based on the voice extraction and face image extraction results for each speaker and the time synchronization information, voice-text matching information in which voice and text are matched is constructed using the TTS learning model and STF learning model of the model processing unit, and the speaker's face A matching unit that configures face-voice matching information in which the voice is matched; A model processing unit that constructs a TTS learning model for an avatar speech service based on artificial intelligence neural network learning based on the voice-text matching information and an STF learning model for an avatar speech service based on artificial intelligence neural network learning based on the face-voice matching information. ; and a service provider that configures avatar speech service information using the TTS learning model and the STF learning model and provides it to the user terminal.

Description

Device for providing an avatar speech service synchronized with each speaker's voice and face image}

The present invention relates to a service provision device. More specifically, the present invention relates to an avatar speech service providing device that is synchronized with the voice and face image of each speaker.

The technology for converting sentences and text into speech (SPEECH) is called TTS (TEXT TO SPEECH), and with the development of various artificial intelligence-based learning technologies, quite natural speech conversion is now possible.

Furthermore, Speech To Face (STF), a voice signal face conversion technology that applies an effect to change the mouth shape of a person or character's face on the screen, such as an avatar, according to the speech voice, was recently developed.

Accordingly, in order to create a face that matches the voice, a face creation process is added, which reflects the speaker's speaking habits, intonation, speed, etc., as well as the original facial appearance and mouth shape, so that the speaker actually speaks. A video with the same effect applied is provided.

In addition, these TTS and STF technologies are currently being developed significantly in each field by utilizing methods using artificial intelligence neural networks.

Recently, services that utilize these artificial intelligence neural network technologies to produce speech videos using avatars have been proposed, but with current technology, only test videos that speak only simple text can be produced over a long period of time, and are mechanically structured. Due to the existence of various problems in terms of commercialization, such as the heterogeneity of the image, poor image quality, unnatural mouth shape, and difficulties in real-time processing, it is still in the research and experimental stage.

The present invention was created to solve the problems described above, and effectively integrates and applies TTS and STF technologies based on artificial intelligence neural network learning to process high-quality natural emotional expression and accurate mouth shape synchronization in avatar speech videos. The purpose is to provide an avatar speech service providing device and an operating method that can provide various avatar speech application services based on this.

A method according to an embodiment of the present invention for solving the problems described above is a method of operating an avatar speech service providing device, comprising: collecting video information from one or more video platform servers; providing time synchronization information by extracting voice and face image for each speaker from the image information; Based on the voice extraction and face image extraction results for each speaker and the time synchronization information, configure voice-text matching information in which voice and text are matched, and configure face-voice matching information in which the speaker's face and voice are matched. step; Constructing a TTS learning model for an avatar speech service based on artificial intelligence neural network learning based on the voice-text matching information and an STF learning model for an avatar speech service based on artificial intelligence neural network learning based on the face-voice matching information, respectively; and configuring avatar speech service information using the TTS learning model and the STF learning model and providing it to the user terminal.

In addition, an apparatus according to an embodiment of the present invention for solving the problems described above is an avatar speech service providing apparatus, comprising: a collection unit that collects video information from one or more video platform servers; an extraction unit that extracts voice and face images for each speaker from the image information and provides time synchronization information;

Based on the voice extraction and face image extraction results for each speaker and the time synchronization information, voice-text matching information in which voice and text are matched is constructed using the TTS learning model and STF learning model of the model processing unit, and the speaker's face A matching unit that configures face-voice matching information in which the voice is matched;

A model processing unit that constructs a TTS learning model for an avatar speech service based on artificial intelligence neural network learning based on the voice-text matching information and an STF learning model for an avatar speech service based on artificial intelligence neural network learning based on the face-voice matching information. ; and a service provider that configures avatar speech service information using the TTS learning model and the STF learning model and provides it to the user terminal.

According to an embodiment of the present invention, based on the results of voice extraction and face image extraction for each speaker and the time synchronization information, voice-text matching information in which voice and text are matched is configured, and the speaker's face and voice are matched. It constitutes face-voice matching information, and includes a TTS learning model for an avatar speech service based on artificial intelligence neural network learning based on the voice-text matching information, and an avatar speech service based on artificial intelligence neural network learning based on the face-voice matching information. Each STF learning model can be constructed, and avatar speech service information using the TTS learning model and the STF learning model can be configured and provided to the user terminal.

Accordingly, according to an embodiment of the present invention, by effectively converging and applying TTS and STF technologies based on artificial intelligence neural network learning, high-quality natural emotional expression and accurate mouth shape synchronization in avatar speech images are processed, and various An avatar speech application service can be provided.

1 is a conceptual diagram schematically showing the entire system according to an embodiment of the present invention.
Figure 2 is a block diagram illustrating in more detail a service providing device according to an embodiment of the present invention.
3 to 13 are diagrams to explain in more detail the operation of processing units according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art will be able to invent various devices that embody the principles of the present invention and are included in the spirit and scope of the present invention, although not explicitly described or shown herein. In addition, it is understood that all conditional terms and embodiments listed herein are, in principle, expressly intended only for the purpose of ensuring that the concept of the invention is understood, and are not limited to the embodiments and conditions specifically listed as such. It has to be.

Additionally, it is to be understood that any detailed description reciting principles, aspects, and embodiments of the invention, as well as specific embodiments, is intended to encompass structural and functional equivalents thereof. In addition, these equivalents should be understood to include not only currently known equivalents but also equivalents developed in the future, that is, all elements invented to perform the same function regardless of structure.

Accordingly, for example, the block diagrams herein should be understood as representing a conceptual view of an example circuit embodying the principles of the invention. Similarly, all flow diagrams, state transition diagrams, pseudo-code, etc. are understood to represent various processes that can be substantially represented on a computer-readable medium and are performed by a computer or processor, whether or not the computer or processor is explicitly shown. It has to be.

The functions of the various elements shown in the figures, which include functional blocks represented by processors or similar concepts, may be provided by the use of dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which may be shared.

Additionally, the clear use of terms such as processor, control, or similar concepts should not be construed as exclusively referring to hardware capable of executing software, and should not be construed as referring exclusively to hardware capable of executing software, including without limitation digital signal processor (DSP) hardware and ROM for storing software. It should be understood as implicitly including ROM, RAM, and non-volatile memory. Other hardware for public use may also be included.

In the claims of this specification, components expressed as means for performing the functions described in the detailed description include, for example, a combination of circuit elements that perform the functions or any form of software including firmware/microcode, etc. It is intended to include any method of performing a function, coupled with suitable circuitry for executing the software to perform the function. Since the present invention defined by these claims combines the functions provided by various listed means and is combined with the method required by the claims, any means capable of providing the above functions are equivalent to those identified from the present specification. It should be understood as

The above-described purpose, features and advantages will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art will be able to easily implement the technical idea of the present invention. There will be. Additionally, in describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

1 is a conceptual diagram schematically showing the entire system according to an embodiment of the present invention.

Referring to Figure 1, first, the system according to an embodiment of the present invention includes an avatar speech service providing device 100, a user terminal 200, a crawling target video platform server 300, a TTS processing device 400, and an STF processing. Includes device 500.

And, the avatar speech service providing device 100 according to an embodiment of the present invention includes a crawling target video platform server 300, a TTS processing device 400, and an STF processing device to provide a service according to an embodiment of the present invention. It can be connected to each of 500 and the user terminal 200 through a wired/wireless network, and can communicate with each other.

Here, each of the above networks is a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), and a mobile communication network ( It can be implemented as any type of wired/wireless network, such as a mobile radiocommunication network or satellite communication network.

Additionally, the user terminal 200 may be a terminal device that receives an avatar speech video produced through the avatar speech service providing device 100 or a platform server to which the avatar speech video is uploaded.

When the user terminal 200 is a terminal device, it may be an individual device such as a mobile phone, a smart phone, a smart pad, or a personal digital assistant (PDA), according to an embodiment of the present invention. It may be a display device that receives and outputs the avatar speed image provided by the service providing device 100.

Meanwhile, the crawling target video platform server 300 may include various separately operated video platform servers, and various speech images required for learning an artificial intelligence neural network to provide avatar speech images in the avatar speech service providing device 100. It may be equipped with .

In addition, the avatar speech service providing device 100 according to an embodiment of the present invention can collect video data by performing web crawling or web scraping from the crawling target video platform server 300.

Additionally, the avatar speech service providing device 100 can extract high-quality and high-quality sound images by applying an image evaluation algorithm to the collected image data. As an image evaluation algorithm for this, quality evaluation-based extraction algorithms such as No-Reference Video Quality Assessments and No-Reference Audio Quality Assessments can be used.

In addition, the avatar speech service providing device 100 can perform voice extraction and face image extraction for each speaker from the extracted original image, and provides time synchronization information to the TTS processing device 400 and the STF processing device 500. ) can be distributed to each.

And, the avatar speech service providing device 100 configures the voice-text matching information processed by the TTS processing device 400, and the face-voice matching the speaker's face and voice processed by the STF processing device 500. a matching unit that configures matching information; It consists of a TTS learning model for avatar speech service based on artificial intelligence neural network learning based on voice-text matching information, and a model processing unit that builds an STF learning model for avatar speech service based on artificial intelligence neural network learning based on face-voice matching information. You can.

In order to efficiently process such learning model construction by automating it, the avatar speech service providing device 100 may be provided with various engine processing units.

For example, the avatar speech service providing device 100 may be equipped with a Speaker Diarization engine that can secure voice data for each speaker using a deep learning model that separates one audio for each speaker.

Additionally, the avatar speech service providing device 100 may be equipped with a Face Detection engine that extracts a portion containing a face using a deep learning model that determines whether a face exists.

Additionally, the avatar speech service providing device 100 may be equipped with an Active Speaker Detection engine that secures the image portion of the speaker that matches the audio using a deep learning model that determines the speaker speaking at one point in time.

In addition, the avatar speech service providing device 100 reversely performs STT processing to label the voice-text matching data so that the voice-text matching data processed through the TTS processing device 400 can be applied to a deep learning learning model. An STT (SPEECH TO TEXT) engine that provides data may be further provided.

Additionally, the avatar speech service providing device 100 may further include a verification unit to verify the accuracy of each data processing engine.

Meanwhile, the TTS processing device 400 can configure voice-text matching data based on speech or text information received from the avatar speech service providing device 100, and can perform either voice-to-text conversion or text-to-voice conversion for this purpose. can be processed.

In addition, the STF processing device 500 can configure facial image-voice matching data based on the facial image or voice information received from the avatar speech service providing device 100, and can perform voice-face image conversion or facial image for this. -Can handle any one of voice conversion.

According to this system configuration, the avatar speech service providing device 100 provides various services based on avatar speech generation and includes various processing units capable of improving processing speed and lip sync accuracy, applying head motion, improving image quality, etc. Additional elements may be provided, and this will be explained in more detail with reference to FIG. 2.

Figure 2 is a block diagram illustrating in more detail a service providing device according to an embodiment of the present invention, and Figures 3 to 13 are drawings for explaining in more detail the operation of processing units according to an embodiment of the present invention.

Referring to FIG. 2, the avatar speech service providing device 100 according to an embodiment of the present invention includes a multilingual global video communication service unit, a real-time AI avatar creation unit based on text input, a lip sync image generation unit based on high-definition data processing, and a non-learning unit. Person response AI avatar creation unit, facial image composition unit based on emotion analysis in voice, lip sync image processing unit based on mouth shape synchronization of voice, voice-based lip sync 3D face data processing unit, emotion analysis based facial expression and head movement control unit, short-term voice data learning processing unit, It includes a learning-based emotion production voice processing unit, an audiobook production service unit, a speaker face-voice matching data collection automation unit, and an emotion-based text-voice matching data collection automation unit.

First, the multilingual global video communication service unit generates avatar speech video information based on multilingual voices and speech images in real time in response to characters input from the user terminal 200, and provides a data service to the user terminal 200. Process it.

Accordingly, the multilingual global video communication service department will provide an application that enables global video communication using only text by generating a multilingual global video communication avatar speech video in real time according to an embodiment of the present invention when the user simply inputs text. As the demand for global video communication increases, real-time communication is possible, but it can solve the problem of limitations in language barriers.

For example, in the case of conventional technology, non-face-to-face communication technology is currently provided using microphones and cameras for real-time video communication, but language barriers always exist and there are systematic limitations in that microphones and camera devices are required. do.

Accordingly, the multilingual global video communication service unit according to an embodiment of the present invention provides a service in which, when a language to be translated is selected on the user terminal 200 and the native language is input as text, an avatar image uttering a translation voice is output. You can.

In addition, the multilingual global video communication service unit according to an embodiment of the present invention provides an avatar speech service including multilingual voices and a lip-sync video that can be generated in real time to the user terminal 200 through the artificial intelligence learning model and engine processing described above. can be provided.

Accordingly, it is possible to provide a service that enables global video communication even without a camera and microphone, and the multilingual global video communication service unit according to an embodiment of the present invention combines a translation function, a voice output function, and an avatar creation function, It is possible to provide a service that allows global communication without camera equipment.

According to the processing of the multilingual global video communication service department according to this embodiment of the present invention, even in the case of disabled people who have difficulty communicating by voice, they can express their opinions audio-visually by simply entering text, enabling smooth communication. , In a global environment where language barriers exist, it is possible to communicate in multiple languages just by inputting text in situations where speaking in multiple languages is difficult, thereby eliminating communication barriers between ethnic groups and cultures.

Meanwhile, the text input-based real-time AI avatar generator generates an AI human avatar that is output in real time in response to text input from the user terminal 200, and sends speech image data of the AI human avatar to the user terminal 200. The process provided can be processed.

More specifically, the real-time AI avatar creation unit based on text input according to an embodiment of the present invention is intended to overcome limitations due to the time required to output an image when producing an AI announcer, and only requires text input from the user terminal 200. , It provides a function to output an avatar speech video in which voice and speech video are matched in real time from the user terminal 200, and this implements a service that enables video communication and avatar speech video production using only text.

For example, conventional AI announcer technologies are being experimentally proposed, but they learn about 8 hours of voice and face data from a single speaker with an artificial intelligence neural network, and when text is input, the learned announcer's voice and face are output as an image. As this technology uses a high-definition-low-efficiency model, there is a problem in that a long rendering time of more than 1 minute occurs even for short video output of about 10 seconds.

Accordingly, the real-time AI avatar generator based on text input according to an embodiment of the present invention may include a lip-sync image output module that guarantees a rendering time of at least 1 second after text input, and real-time image output through optimization of the structure of the generation algorithm. Video output can be provided.

The real-time AI avatar generator based on text input according to an embodiment of the present invention can build a lip-sync image generation model capable of real-time generation by utilizing the model lightweighting technique as shown in FIG. 3. For example, model optimization using parameter pruning and quantization, knowledge distillation, etc. can be implemented through the deep learning model optimization process as shown in Figure 3.

Accordingly, the real-time AI avatar creation unit based on text input enables smooth communication by allowing disabled people who have difficulty communicating by voice to express their opinions audio-visually by simply entering text, and in a global environment where language barriers exist. In this regard, there is a difference in that it allows multilingual communication just by inputting text in situations where speaking in multiple languages is difficult, thereby resolving communication barriers between people who speak different languages.

Meanwhile, the high-definition data processing-based lip sync image generator according to an embodiment of the present invention generates a high-definition lip sync image based on high-definition data and processes service data provided to the user terminal 200.

Here, the lip-sync image generator based on high-definition data processing according to an embodiment of the present invention was designed to expand the use of AI humans using large concierges due to the development of display technology, and is a domestic AI announcer according to the prior art. This is to solve the image quality problem that occurs because the output images collect, process, and utilize data through face images of HD (720p) or lower. The lip sync video generation unit based on high-definition data processing overcomes the technical limitations of using low-quality data and solves the problem of the quality problem of AI avatar speech videos becoming an entry barrier to commercialization.

To this end, the lip sync image generator based on high-definition data processing according to an embodiment of the present invention builds a high-definition dataset of speech images, secures a high-definition dataset with an automated data collection system, and specifically improves the clarity of the mouth shape and teeth. An advanced creation process can be performed.

More specifically, referring to FIG. 4, the high-definition data processing-based lip sync image generator can generate high-definition data processing-based lip sync images by utilizing the latest GAN models that generate high-definition faces, such as Face super resolution and StyleGAN3.

Accordingly, the lip sync video generation unit based on high-definition data processing can provide high-definition avatar speech videos through services that require high-definition, such as dramas and movies, which have improved video quality. This leads to increased utilization of video production and video production due to improved video quality. It dramatically saves time and cost.

Meanwhile, the AI avatar creation unit corresponding to the non-learning person generates AI human avatar speech in which the voice and image of the non-learning person are output and outputs it to the user terminal 200.

Typically, when producing a video, wasteful elements occur in terms of time, cost, and energy required for filming, and the AI avatar creation unit corresponding to a non-learning person according to an embodiment of the present invention generates a non-learning person from the user terminal 200. By simply entering information and text, it creates and provides an AI avatar for non-learning people that combines voice and video, allowing users to easily create videos even without a separate filming space.

In other words, taking conventional technology as an example, the current AI announcer technology is a technology that learns several hours (about 8 hours) of voice and face data of a single speaker with an artificial intelligence neural network, and outputs the learned voice and face of the announcer as an image when text is input. This has been proposed, but since it requires more than 8 hours of video data from a single speaker, the cost and time from learning data collection to processing and output are considerable. As a result, it does not solve the difficulties of producing human video and creates a barrier to entry into the production of AI announcers. Additionally, because it is a model learned from video data from a single speaker, it has limitations in that it cannot be synchronized with the voices of other speakers.

Accordingly, the AI avatar generator corresponding to an untrained person according to an embodiment of the present invention is capable of generating mouth shapes from untrained video data through learning data so that the voice of an unspecified speaker and the video of an unspecified person can be freely synchronized. A generalized generation engine can be built through artificial intelligence neural network learning, an AI avatar corresponding to an untrained person can be created based on this, and an avatar speech image based on the avatar can be provided to the user terminal 200.

To this end, referring to FIG. 5, the non-learning person corresponding AI avatar generator according to an embodiment of the present invention embeds voice information and creates speech features (various information about voice, which implicitly expresses pitch, tone, volume, etc.). a speech encoder that extracts information), an image encoder that extracts face features (information that implicitly expresses high-dimensional information of the face image) by embedding face information, predicts the mouth shape through the speech features, and extracts face features. It may include a decoder that generates a base face, and a lip sync discriminator that determines the accuracy of the lip sync key by measuring similarity between the speech feature and the face feature.

In addition, the AI avatar creation unit for non-learning people can process the creation of mouth shapes that match the voice without language restrictions, and for this purpose, it can be equipped with a language-specific customized lip sync engine that can learn the mouth shape characteristics of each language. .

Accordingly, the AI avatar creation unit for non-learning people will eliminate waste of time and resources required for video production along with innovation in the video communication environment, lowering the entry barrier to video production and allowing production personnel's time to be used more effectively. , Through free synchronization of voice and face images, an avatar speech image without language restrictions can be created and provided to the user terminal 200.

Meanwhile, the facial image component based on emotion analysis in the voice analyzes the emotion in the voice, generates an avatar speech image with changing facial expressions, and outputs it to the user terminal 200.

The facial image component based on emotion analysis in the voice was created in response to the need for more friendly digital humans in a situation where digital humans are appearing in various activity areas due to the increased demand for digital humans, and the voice of the digital human included in the avatar speech video is We can provide a service that allows you to express your facial expressions naturally.

For example, the digital human of the prior art is a technology that generates an image of a person's appearance from already learned image data, and outputs faces and expressions based on the learning data, so only the gestures included in the image data with an expressionless face are used. This is the output situation. As a result, there are limitations in terms of lack of friendliness and usability.

Accordingly, as shown in FIG. 6, the facial image component based on emotion analysis in voice according to an embodiment of the present invention analyzes the emotion in the voice signal, and is capable of controlling the emotion of the avatar speech image according to the emotion analysis result. A learning model can be built. To this end, the facial image composition unit based on emotion analysis in speech can build a learning model by learning the mouth shape through speech features output from the audio encoder and learning facial expressions with emotions through expression features output from the expression encoder. there is.

By driving the facial image component based on emotional analysis within the voice, it becomes possible to create images with more natural facial expressions than conventional models that only consider the pronunciation of voice signals, which makes it possible to introduce digital humans into art and cultural content that requires emotional expression.

In addition, the facial image component based on emotion analysis in the voice provides a digital human capable of expressing various emotions to the user terminal 200, which establishes a human-friendly digital human industry that can form intimacy with the user and even experience emotional exchange. make it possible

In addition, the lip sync image processing unit based on mouth shape synchronization of the voice can build a lip sync image generation model in which the mouth shape matches the voice.

More specifically, the lip-sync image processing unit based on voice mouth synchronization was developed in a situation where expectations for related technology are increasing along with the increase in demand for AI announcers and a more natural lip-sync technology is required. When text is input, an avatar speech generation model that is output by accurately matching the voice and speech video can be built based on learning.

For example, the AI announcer technology of the prior art is a technology that learns about 8 hours of voice and face data of a single speaker with an artificial intelligence neural network, and outputs the learned announcer's voice and face as an image when text is input. When you input a language, lip sync mismatch problems may occur.

Accordingly, as shown in FIG. 7, the lip synchronization image processor based on mouth shape synchronization of voice according to an embodiment of the present invention can build a learning model for generating multilingual lip sync images according to multilingual language recognition. To this end, the lip sync image processing unit based on speech lip synchronization can process lip sync discriminator learning taking language into account after collecting the data set.

In addition, the lip sync image processing unit based on speech mouth shape synchronization can perform processing to determine sync through similarity calculation of lower half visual features and audio features in order to build a lip sync image generation model with high accuracy for each language.

Accordingly, the lip sync image processing unit based on synchronization of the mouth shape of the voice can help with global avatar video communication by resolving unnatural mouth shapes for each language, and provides accurate mouth shape images according to the language, providing language education videos. An avatar speech video service that can also be used can be provided to the user terminal 200.

Additionally, the voice-based lip sync 3D face data processing unit performs a process of generating a 3D face that lip syncs according to the voice.

This was developed as various metahumans based on 3D technology emerged due to the development of the metaverse industry. The voice-based lip sync 3D face data processing unit can implement a face generation model that enables smooth voice lip sync even in a 3D environment.

In other words, conventional 3D face production and voice lip sync production environments utilize moshen capture-based technology, but this requires high development costs and environments, so the barrier to entry is high, the production period is considerable, and it is difficult to produce natural mouth shapes. Because the situation is causing a sense of heterogeneity, the voice-based lip sync 3D face data processing unit can process the 3D face creation process to solve this problem.

More specifically, referring to FIG. 8, the voice-based lip sync 3D face data processing unit according to an embodiment of the present invention can perform voice and 3D data synchronization learning using 3D mesh data as a learning dataset. To this end, the voice-based lip sync 3D face data processing unit can input a 3D template mesh and output a moving mesh.

Additionally, the voice-based lip sync 3D face data processing unit can build a lip sync generation model based on arbitrary neutral face mesh data and audio information.

Additionally, the voice-based lip sync 3D face data processing unit can perform audio-based 3D face data generation processing by embedding a speech signal with an audio encoder.

Accordingly, the voice-based lip sync 3D face data processing unit can provide the user terminal 200 with an avatar speech service that can be utilized in a variety of ways not only in a 2D video environment but also in a 3D-based metaverse environment, which requires expensive motion capture equipment. Even without , it is possible to provide an environment where you can control the facial expressions of 3D avatars.

Meanwhile, the emotion analysis-based facial expression and head movement control unit analyzes emotions in the voice to generate a facial image capable of facial expression changes and head movement, and configures avatar speech service information based on this to provide it to the user terminal 200. .

The emotion analysis-based facial expression and head movement control unit was created in response to the need for more natural digital humans in a situation where digital humans are appearing in various activity areas due to the increasing demand for digital humans, and has the function to naturally produce the voice and head of the digital human. can be provided.

Digital humans in the prior art are a technology that generates images of human appearances from previously learned image data, and outputs faces and expressions based on the learning data. Only the stiff posture and recorded gestures included in the learned image data are used. This is an output situation, and as a result, there are limitations such as unnaturalness and poor usability.

Accordingly, as shown in FIG. 9, the emotion analysis-based facial expression and head movement control unit according to an embodiment of the present invention uses audio-to-animation generator technology to extract mouth parameters, eyebrow parameters, and head from audio and face. By extracting parameters and providing a module to predict head pose, head motion control can be performed.

Accordingly, the emotion analysis-based facial expression and head movement control unit processes head motion control to predict and generate a natural head pose according to the voice.

In particular, the emotion analysis-based facial expression and head movement control unit can process images or images with static head motion to produce avatar speech that generates natural head motion according to voice signals, and can provide various services accordingly. Expansion becomes possible.

Meanwhile, the short-term voice data learning processing unit performs voice generation learning processing that can be learned only with short-term voice data (for example, within 10 minutes).

Typically, as the demand for AI voice actors increases, the demand to turn one's own voice into an AI voice actor is also increasing. However, with conventional TTS technology alone, it is necessary to learn about 8 hours of voice data from a single speaker to complete a natural AI voice actor model. As it is a technology, large amounts of high-quality voice data are needed to develop AI voice actors, creating an entry barrier to technology development.

Accordingly, the short-term voice data learning processing unit according to an embodiment of the present invention is a short-term voice data processing unit that enables learning of learning data for the Text to Speech (TTS) function, in which voice is output by inputting only letters, with less than 10 minutes of data. A data learning process can be provided.

More specifically, referring to FIG. 10, the short-term voice data learning processor according to an embodiment of the present invention collects data from audio through a speaker encoder in order to build a voice synthesis model that can be learned with only short-term voice data of about 10 minutes. End-to-end deep learning can be performed by extracting speaker information.

In addition, the short-term voice data learning processing unit uses an existing basemodel learned from a large amount of data to build a TTS learning efficiency system, and can build a model that matches model adaptation with only a small amount of personal speaker data.

Accordingly, the short-term voice data learning processing unit can develop AI voice actors with a small amount of voice data, enabling TTS personalized service, which can lead to rapid development of AI voice actor technology and rapid expansion of the AI voice actor industry.

Meanwhile, the learning-based emotion production voice processing unit can build a voice generation engine that can freely express emotions based on artificial intelligence neural network learning.

The learning-based emotion production voice processing unit was developed as the demand for AI voice actors increases, and the demand for using AI voice actors expressing various emotions also increases. When a text is input from the user terminal 200, a voice is output. In the text to speech (TTS) function, emotion rendering processing is performed to enable various emotions to be expressed.

In the case of the prior art, as a technology for learning voice data synchronized with text, there is a problem in that there are limitations in various voice expressions because only consistent voice conditions are learned and output.

Accordingly, referring to FIG. 11, the learning-based emotion production voice processing unit according to an embodiment of the present invention performs data learning processing that classifies the emotion of the voice through Speech Emotion Recognition, performs emotion classification of the voice data, and By performing a learning process in which artificial intelligence learns the characteristics of classified voices, a learning model for the learning-based emotion production voice processor can be built.

Here, the learning-based emotion production voice processing unit can build a model that can generate various emotions (happy, sad, angry, etc.), which consists of a model that can learn and generate voice characteristics of classified emotional data. It can be.

Accordingly, the learning-based emotion production voice processing unit can configure avatar speech information that can be used for various cultural contents such as dramas, movies, novel audiobooks, etc. through a TTS model capable of emotion production and provide it to the user terminal 200. In addition, by driving the learning-based emotion production voice processing unit, the unnatural voice of an AI voice actor can be produced by mixing various emotions, which can lead to the expansion of the AI voice actor industry.

The audiobook production service department can shorten the audiobook production period by automating the provision of audiobook production services.

The audiobook production service department was created as the demand for creating large quantities of audiobooks in a short period of time increased as the technology of AI voice actors improved. This is a service function that quickly produces large quantities of audiobooks by improving the existing TTS technology. provides.

For example, in the case of audiobooks in the prior art, they are recorded and produced by having human voice actors read the script, and since reading the script takes a lot of time, the production period is long and production capabilities are limited.

To overcome this, the audiobook production service department according to an embodiment of the present invention configures an audiobook production system using AI voice actors, and performs a process in which the AI voice actor's voice is immediately output as a file when script information is input. . In addition, the audiobook production service unit may configure a user interface capable of inputting various emotions on the user terminal 200 and provide the user interface to the user terminal 200.

In addition, the audiobook production service department configures an AI voice actor learning model and system that can appropriately produce emotions and speakers, and provides an automated avatar speech-based audiobook production service based on this to the user terminal 200. .

Accordingly, mass production of audiobooks becomes possible, so expansion of the existing audiobook market can be expected, and the introduction of AI voice actor technology can dramatically improve the audiobook production environment.

Meanwhile, the speaker face-voice matching data collection automation unit efficiently and quickly builds an STF learning database by an automated process based on an artificial intelligence algorithm when collecting and processing the speaker's face-voice matching data.

In the case of the prior art, there is a problem that excessive time and cost are consumed by manually collecting and processing face-voice matching data.

Accordingly, referring to FIG. 12, the speaker face-voice matching data collection automation unit according to an embodiment of the present invention includes an image crawling engine, which is a web crawler that collects images from a website, and the quality of video and voice of the collected images. A video quality evaluation analysis unit that evaluates (No-Reference Video Quality Assessments, No-Reference Audio Quality Assessments) and stores only those that exceed a certain standard, and a deep learning-based face function that determines whether a face is present in each frame of the video. A recognition unit, a deep learning-based speaker separation unit that separates audio by speaker from the input audio, a voice reinforcement unit that secures the voice of each speaker using a deep learning model that separates one audio by speaker, and an audio and It includes a viewpoint matching unit that matches the time section where the face is present, and a deep learning-based speaker identification unit that finds the face of the person speaking in the video.

Accordingly, the speaker face-voice matching data collection automation unit can quickly collect a large amount of data matching the speaker's face and voice by automating the data collection process using an artificial intelligence algorithm.

Meanwhile, the emotion-based text-voice matching data collection automation unit configures an automated system based on artificial intelligence algorithms to collect and process emotion-based text-voice matching data.

In the case of the prior art, there is a problem that excessive time and cost are consumed by manually collecting and processing text-voice matching data.

Accordingly, referring to FIG. 13, the emotion-based text-voice matching data collection automation unit according to an embodiment of the present invention includes an audio crawling engine that secures data by automatically crawling audio with a web crawler, and a No-Reference Audio Quality An audio evaluation algorithm processing unit that secures only high-quality audio through assessments, a Speaker Diarization engine processing unit that secures voices for each speaker using a deep learning model that separates one audio by speaker, and noise is removed from voice signals containing noise. A Speech enhancement processing unit that preserves the voice, a Speech To Text processing unit that secures data labeling text that matches the voice using a deep learning model that converts the voice into text, and a Speech Emotion Recognition processing unit that classifies the emotions of the audio. It recognizes the speaker, but includes a speaker identification processing unit that adds speaker information if the speaker is not in the existing data and stores it as the speaker ID if present.

In accordance with the operation of the emotion-based text-voice matching data collection automation unit according to an embodiment of the present invention, the data collection process is automated using an artificial intelligence algorithm to quickly collect a large amount of data matching the speaker's face and voice. There is an advantage.

The method according to the present invention described above can be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of computer-readable recording media include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, optical data storage devices, etc.

The computer-readable recording medium is distributed in a computer system connected to a network, so that computer-readable code can be stored and executed in a distributed manner. And, functional programs, codes, and code segments for implementing the method can be easily deduced by programmers in the technical field to which the present invention pertains.

In addition, although preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the invention pertains without departing from the gist of the present invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

Claims (5)

In the avatar speech service providing device,
A collection unit that collects video information from one or more video platform servers;
an extraction unit that extracts voice and face images for each speaker from the image information and provides time synchronization information;
Based on the voice extraction and face image extraction results for each speaker and the time synchronization information, voice-text matching information in which voice and text are matched is constructed using the TTS learning model and STF learning model of the model processing unit, and the speaker's face A matching unit that configures face-voice matching information in which the and voice are matched;
A model processing unit that constructs a TTS learning model for an avatar speech service based on artificial intelligence neural network learning based on the voice-text matching information and an STF learning model for an avatar speech service based on artificial intelligence neural network learning based on the face-voice matching information. ; and
Comprising a service provider that configures avatar speech service information using the TTS learning model and the STF learning model and provides it to the user terminal.
Avatar speech service provision device. According to paragraph 1,
The extraction unit,
From the collected video information, obtaining voice data for each speaker using a deep learning model that separates one audio for each speaker and providing time synchronization information.
Avatar speech service provision device. According to paragraph 1,
The extraction unit,
From the collected image information, extracting the part where the face is using a deep learning model to determine whether there is a face and providing time synchronization information.
Avatar speech service provision device. According to paragraph 1,
The extraction unit,
From the collected video information, using a deep learning model to determine the speaker speaking at one point in time, obtain the video portion of the speaker that matches the audio and provide time synchronization information.
Avatar speech service provision device. According to paragraph 1,
The service provider department,
Multilingual global video communication service department, real-time AI avatar creation unit based on text input, lip sync video creation unit based on high-definition data processing, AI avatar creation unit for non-learning people, facial image composition unit based on emotion analysis in voice, mouth shape synchronization based on voice Lip sync video processing unit, voice-based lip sync 3D facial data processing unit, emotion analysis-based facial expression and head movement control unit, short-term speech data learning processing unit, learning-based emotion production voice processing unit, audiobook production service unit, speaker face-voice matching data collection automation unit, Constructing avatar speech service information according to the operation of at least one of the emotion-based text-voice matching data collection automation unit and providing it to the user terminal
Avatar speech service provision device.