KR20230160105A - Real-time video chat rendering technology using character IP - Google Patents

Abstract

Real-time video chat rendering technology using character IP is launched. Through an embodiment of the present invention, an image containing a human face is analyzed, measurement values for defined features are extracted, a basic model is created, and the character shape corresponding to the human face is compared with the measurement values given to the character. Avatars can be created automatically.

Description

Real-time video chat rendering technology using character IP}

The present invention relates to a real-time video chat rendering technology using character IP using a human face. More specifically, it analyzes an image containing a human face and automatically generates an avatar in the shape of a character corresponding to the human face. Real-time video chat rendering technology using the character IP, a computer device that performs the real-time video chat rendering technology using the character IP, and combined with a computer to execute the real-time video chat rendering technology using the character IP on the computer. It relates to a computer program stored on a computer-readable recording medium and the recording medium.

An avatar is a visual image used as a user's alter ego in cyberspace and refers to a virtual graphic character that represents the user in community sites, Internet chatting, shopping malls, online games, etc. For example, Korean Patent Publication No. 10-2007-0036857 relates to real-time video chat rendering technology using user-based character IP, which generates pattern data by scanning the user in three dimensions through a 3D scanner. A technology for generating a 3D avatar image of a user using pattern data is being disclosed.

Real-time video chat rendering technology using character IP that can automatically create an avatar with a character shape corresponding to the human face by analyzing images containing a human face, and real-time video chat rendering technology using the character IP. A computer program stored on a computer-readable recording medium and the recording medium are provided to execute the real-time video chat rendering technology using the character IP in combination with a computer device and a computer.

For each of a plurality of character images including the character's face, generating a first measurement value set that quantifies the character's face; storing a first set of measurement values corresponding to each character classification obtained from the plurality of character images in a database; generating a basic model for each character classification; determining a character classification corresponding to the person's face by receiving a second set of numerical measurements of the person's face and comparing it with the first set of measurement values stored in the database; Confirming a basic model created corresponding to the determined character classification; and processing the confirmed basic model to correspond to the person's face to provide an avatar in the shape of a character corresponding to the person's face.

According to one side, the step of generating the first set of measurement values includes extracting measurement values for each predefined measurement item for each of the features predefined as facial components for the face of the character and performing the first measurement. A set of values can be created, and the second set of measurement values can be created by extracting measurement values for each measurement item for each of the features as a facial component for the person's face.

According to another aspect, the step of generating the first set of measurement values includes analyzing the plurality of character images using a deep learning model to classify the characters included in each of the plurality of character images to determine the character classification. It may include steps.

According to another aspect, the step of generating the basic model includes calculating average values of elements corresponding to each other in first sets of measurement values stored in association with the same character classification in the database, and based on the set of calculated average values. Thus, the basic model can be created.

According to another aspect, the step of determining the character classification may include sequentially comparing the second set of measurement values with the first set of measurement values stored in the database to select n corresponding to the m measurement value sets in the order of high similarity. It may include the step of determining character classifications (where n is a natural number), and the step of checking the basic model may include checking n basic models generated for each of the n character classifications.

According to another aspect, a final model may be generated through synthesis between the confirmed n basic models.

According to another aspect, the step of processing the confirmed basic model in response to the human face includes modifying a value corresponding to the confirmed basic model based on the second measurement value set, The model can be customized according to the person's face.

Receiving an image containing a human face from a user terminal through a network; generating a first set of measurement values by quantifying the human face included in the image; transmitting the first set of measurement values to an analysis server; A database in which a second set of measurement values generated by analyzing the character image from the analysis server and digitizing the character's face is stored for each character category and a basic model for the character classification selected based on the first set of measurement values transmitted are analyzed. Receiving from a server; And a step of sharing the basic model or a final model generated through the basic model with the user terminal to provide an avatar with a character shape corresponding to the person's face. Real-time video chat rendering using character IP. Provides technology.

Creating or selecting an image containing a human face; transmitting the selected image to a server through a network; Receiving a character model selected based on a comparison between a first set of measurement values generated by quantifying the face of a person included in the image from the server and a second set of measurement values generated by quantifying the face of a character included in the character image. step; and providing a real-time video chat rendering technology using character IP, including the step of displaying a character avatar generated by rendering the received character model on the screen.

A computer program stored on a computer-readable recording medium is provided to be combined with a computer and execute the control method on the computer.

Provided is a computer-readable recording medium on which a program for executing the control method on a computer is recorded.

At least one processor implemented to execute instructions readable by a computer device, wherein, for each of a plurality of character images including a face of the character, the face of the character is quantified by the at least one processor. 1 Generate a set of measurement values, store the first set of measurement values corresponding to each character classification obtained from the plurality of character images in a database, generate a basic model for each character classification, and quantify the human face. By receiving a set of measurements and comparing them with a first set of measurements stored in the database, determine a character classification corresponding to the face of the person, identify a base model generated corresponding to the determined character classification, and determine the face of the person Provided is a computer device that processes the identified basic model to correspond to the person's face to provide an avatar with a corresponding character shape.

Comprising at least one processor implemented to execute instructions readable by a computer device, receiving an image including a human face from a user terminal through a network by the at least one processor, and identifying a person included in the image A first set of measurement values is generated by quantifying the face of the character, the first set of measurement values is transmitted to an analysis server, and a second set of measurement values generated by analyzing the character image is stored for each character classification. Receive from the analysis server a basic model for a character classification selected based on a database and the transmitted first set of measurement values, and provide the basic model or the basic model to provide an avatar of a character shape corresponding to the face of the person A computer device is provided that shares the final model generated through the user terminal.

Comprising at least one processor implemented to execute readable instructions in a computer device, generating or selecting an image including a human face by the at least one processor; transmitting the selected image to a server through a network; Receiving a character model selected based on a comparison between a first set of measurement values generated by quantifying the face of a person included in the image from the server and a second set of measurement values generated by quantifying the face of a character included in the character image. step; and a computer device that displays a character avatar generated by rendering the received character model on a screen.

According to the present invention, an avatar in the shape of a character corresponding to a person's face can be automatically created by analyzing an image containing a person's face.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
Figure 3 is a flowchart showing an example of an avatar creation process according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The control method according to embodiments of the present invention may be implemented through a computer device, such as an electronic device, which will be described later. At this time, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the control method according to the embodiments of the present invention according to the control of the driven computer program. The above-described computer program can be combined with a computer device and stored in a computer-readable recording medium in order to cause the computer to execute the control method.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment in FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, and 140, a plurality of servers 150 and 160, and a network 170. Figure 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in Figure 1.

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals implemented as computer devices or mobile terminals. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, Personal Digital Assistants (PDAs), and Portable Multimedia Players (PMPs). ), tablet PC, etc. For example, in Figure 1, the shape of a smartphone is shown as an example of electronic device 1 (110). However, in embodiments of the present invention, electronic device 1 (110) actually connects the network 170 using a wireless or wired communication method. It may refer to one of various physical computer devices capable of communicating with other electronic devices 120, 130, 140 and/or servers 150, 160.

The communication method is not limited, and may include not only a communication method utilizing a communication network that the network 170 may include (for example, a mobile communication network, wired Internet, wireless Internet, and a broadcast network), but also short-range wireless communication between devices. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 170 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computers that communicate with a plurality of electronic devices 110, 120, 130, 140 and a network 170 to provide commands, codes, files, content, services, etc. It can be implemented with devices. For example, the server 150 provides services (e.g., content provision service, social network service, messaging service, search service) to a plurality of electronic devices 110, 120, 130, and 140 connected through the network 170. , mail service, etc.).

Previously, FIG. 1 illustrates an example of a communication environment between a server and a client, but this is just one embodiment, and the control method according to embodiments of the present invention is performed in each of a plurality of electronic devices 110, 120, 130, and 140. It can also be done individually. For example, the control method according to the embodiment of FIG. 1 may be a description of a case where the electronic device 110 receives content from the server 150 and plays it through a media player. On the other hand, in another embodiment, the electronic device 110 may play content stored in the local storage of the electronic device 110 through a media player without separate communication with the server 150.

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices 110, 120, 130, and 140 described above or each of the servers 150 and 160 may be implemented by the computer device 200 shown in FIG. 2. For example, a computer program according to an embodiment may be installed and driven in the computer device 200, and the computer device 200 may play and share sound sources according to embodiments of the present invention under the control of the driven computer program. method can be performed.

As shown in FIG. 2, this computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 200 as a separate permanent storage device that is distinct from the memory 210. Additionally, an operating system and at least one program code may be stored in the memory 210. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on computer programs installed by files received over network 170.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170. For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as memory 210 is transmitted to the network ( 170) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170. Signals, commands, data, etc. received through the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc. may be stored in a storage medium (as described above) that the computer device 200 may further include. It can be stored as a permanent storage device).

The input/output interface 240 may be a means for interfacing with the input/output device 250. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. The input/output device 250 may be configured as a single device with the computer device 200.

Additionally, in other embodiments, computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most prior art components. For example, the computer device 200 may be implemented to include at least some of the input/output devices 250 described above, or may further include other components such as a transceiver, a database, etc.

Figure 3 is a flowchart showing an example of real-time video chat rendering technology using character IP according to an embodiment of the present invention. The real-time video chat rendering technology using character IP according to the embodiment of FIG. 3 shows an example of the entire process performed by the user terminal 310, front server 320, and analysis server 330. Here, the user terminal 310, like any one of the plurality of electronic devices 110, 120, 130, and 140 described above, is a physical device in which a computer program for real-time video chat rendering technology using character IP is installed and driven. It may correspond to a device, and the front server 320 and analysis server 330 may correspond to the servers 150 and 160 described above. As already described, each of the plurality of electronic devices 110, 120, 130, and 140 and each of the servers 150 and 160 may be implemented by the computer device 200 described with reference to FIG. 2. Depending on the embodiment, the front server 320 and the analysis server 330 are implemented by one physical device, or the front server 320 and/or the analysis server 330 are implemented by two or more physical devices linked to each other. It can also be implemented by:

In step S311, the user terminal 310 may create or select an image. For example, the user terminal 310 can capture an image through a camera included in the user terminal 310 or confirm an image selected by the user among images stored in a local storage included in the user terminal 310. there is.

In step S312, the user terminal 310 may upload an image. For example, the user terminal 310 may transmit the image created or selected in step S311 to the front server 320 through the network 170.

As a more specific example, the user terminal 310 may provide a function for capturing images by driving a camera under the control of an application installed and run on the user terminal 310 or at least one of the images stored in the local storage of the user terminal 310. A function for selecting one can be provided to the user, and the image created or selected through this function can be uploaded to the front server 320.

In step S321, the front server 320 can digitize the image. For example, the front server 320 may analyze the image received from the user terminal 310 and quantify the facial components of the human face included in the image. For example, quantification may be implemented in the form of extracting measurement values for each predefined measurement item for each feature predefined as a facial component. As a more specific example, predefined features as facial components may be selected from eyes, nose, mouth, face shape, etc., and predefined measurement items may be selected from area, ratio, angle, length, etc. Depending on the embodiment, one feature may consist of one or more detailed features. For example, the feature 'eye' consists of the subfeatures 'left eye' and 'right eye', or the subfeatures 'left eye', 'left corner of the eye', 'right eye', and 'right corner of the eye'. It can be configured. At this time, the measurement items required according to the feature (or detailed feature) may include all the same measurement items such as area, ratio, and angle, but the required measurement items may vary depending on the embodiment. For example, width and height may be required for the feature 'eyes', and angle may be required as features for the detailed feature 'left eye corner'. Measured values calculated as measurement items can be measured as relative values based on a standard rather than actual physical unit values. For example, the width of the eyes or the distance between the eyes can be calculated as a relative value to the width of the face, and the height of the eyes can be calculated as a value relative to the height of the face. The measurement values may form a set of multiple values extracted for each feature/measurement item (hereinafter referred to as 'measurement value set').

In step S322, the front server 320 may transmit the measured value to the analysis server 330. For example, the front server 320 may transmit a set of measurement values extracted as the image is digitized in step S321 to the analysis server 330.

In step S331, the analysis server 330 may quantify the character image. For example, the analysis server 330 may quantify the face of the character included in the image in the same manner as the front server 320 quantifies the face of the person included in the image in step S321. For example, the analysis server 330 digitizes the character image by extracting measurement values for each pre-defined measurement item for each of the character facial features that are pre-defined as facial components and generating a set of measurement values. can be implemented. Additionally, deep learning can be used to classify the types of characters included in character images. For example, the analysis server 330 analyzes input character images and analyzes each of a plurality of character images using a deep learning model such as CNN (Convolutional Neural Network), which has been learned to classify the type of character included in the character image. You can create a character classification for the characters derived from .

In step S332, the analysis server 330 may build a database. For example, the analysis server 330 may classify characters included in a character image through the deep learning described above and build a database to store sets of measurement values extracted for each character classification.

In step S333, the analysis server 330 may generate a basic model for each classified character. In one example, a base model may be created based on a set of measurements stored in a database. If two or more sets of measurement values are stored in the database in connection with one character classification, the average value of the corresponding elements in the two or more measurement value sets can be calculated, and a basic model can be created through the set of calculated average values. there is.

These steps (S331) to (S333) may be performed in parallel with the steps (S311, S312, S321, and S322) described above, and before the start of the steps (S311, S312, S321, and S322) described above. A database can be built in advance and a basic model can be created.

In step S334, the analysis server 330 may select a basic model. For example, the analysis server 330 may sequentially compare the set of measurement values received from the front server 320 with the measurement value sets stored in the database to determine the most similar m (m is a natural number) set of measurement values. , n (n is a natural number less than or equal to m) character classifications corresponding to these m sets of measurement values can be determined. In other words, the analysis server 330 can determine a character classification corresponding to the human face by receiving a set of numerical measurement values of the human face and comparing it with a set of measurement values stored in a database. If there are two or more measurement value sets corresponding to the same character classification among the m measurement value sets, one character classification is determined for the two or more measurement value sets, so the number of character classifications determined is the determined measurement value. It can be equal to or less than the number of sets. Additionally, the analysis server 330 may select n basic models corresponding to the determined n character classifications. Depending on the embodiment, the analysis server 330 may select and provide only one basic model through one measurement value set with the most similar measurement value set, or select and provide n basic models to the front server 320 or The user terminal 310 may select one of n basic models, and as in an embodiment described later, a new character model may be created by combining n basic models.

In step S335, the analysis server 330 may customize the basic model. In other words, the analysis server 330 may process the confirmed basic model corresponding to the human face to provide an avatar in the shape of a character corresponding to the human face. For example, the analysis server 330 adjusts the item values of the facial components of the basic model according to the set of measurement values received from the front server 320, thereby customizing the basic model to fit the target person's face. You can. In other words, the facial features of the target person can be applied to the base model. At this time, each of the n basic models selected in step S334 may be customized, and the customized n basic models may be transmitted to the front server 320.

Additionally, depending on the embodiment, customization may be performed in the front server 320. For example, in step S334, the analysis server 330 may transmit the selected n basic models to the front server 320, and the front server 320 may transmit each of the n basic models calculated in step S331. It can be customized according to the number. In this case, the front server 320 can customize the basic model according to the person's face by modifying the values corresponding to the basic model based on a set of numerical measurement values of the person's face.

In step S323, the front server 320 may create a final model. In one embodiment, the front server 320 may synthesize n customized basic models to create a completely new character model as a final model. In another embodiment, the front server 320 may simply generate each of the n basic models as a final model. Meanwhile, the customized n basic models are not only used to create the final model, but are also stored in the database along with the set of measurements and can be used to create another avatar later.

In step S324, the front server 320 may share the final model. For example, the front server 320 may transmit the final model to the user terminal 310 through the network 170 and share the generated final model. In other words, the front server 320 may share the basic model or the final model generated through the basic model with the user terminal 310 in order to provide an avatar in the shape of a character corresponding to a human face. For example, when customization or synthesis is not applied, the basic model can be provided to the user terminal 310, and when customization or synthesis is applied, the final model to which customization or synthesis is applied is provided to the user terminal 310. can do.

In step S313, the user terminal 320 may render the final model. For example, the user terminal 320 may receive a character model from the front server 320 and process a task to actually display the modeled character model on the screen. In other words, the user terminal 320 displays the graphic elements finally created through modeling on the screen to create and display a character avatar corresponding to the user's face included in the image created or selected in step S311. do. Meanwhile, depending on the embodiment, the user terminal 320 may provide the user with the ability to customize the final model again according to the control of the application described above before rendering. In this case, the user can modify the facial features of the final model as desired through the provided intelligence, and the final modified character model can be rendered by the user terminal 320 and displayed on the screen.

In this way, through the real-time video chat rendering technology using the character IP according to this embodiment, it is possible to automatically generate an avatar in the shape of a character corresponding to the person's face by analyzing an image containing a person's face. At this time, the character-shaped avatar that is created does not simply apply the user's facial expressions to an already created character avatar template, but finds a character similar to the user's face and automatically creates an avatar with the character's shape. can be provided.

Additionally, the generated character avatar can be used to create a sticker package containing a plurality of sticker images through combination with various poses, facial expressions, and other effects. Meanwhile, as an example, when continuous frames input through a camera are transmitted to the front server 320 instead of a single image from the user terminal 310, the front server 320 extracts data through each of the consecutive frames. By sequentially applying sets of measurement values to the final model, a service can be provided so that the user's facial expressions or actions are applied to the character avatar in real time. In this way, automatically generated character avatars can be used to provide various services.

As such, according to embodiments of the present invention, an avatar in the shape of a character corresponding to a person's face can be automatically created by analyzing an image containing a person's face.

The system or device described above may be implemented with hardware components, software components, or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. These recording media may be a variety of recording or storage means in the form of a single or several pieces of hardware combined, and are not limited to media directly connected to any computer system and may be distributed over a network. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

Claims (4)

For each of a plurality of character images including the character's face, generating a first measurement value set that quantifies the character's face;
storing a first set of measurement values corresponding to each character classification obtained from the plurality of character images in a database;
generating a basic model for each character classification;
determining a character classification corresponding to the person's face by receiving a second set of numerical measurements of the person's face and comparing it with the first set of measurement values stored in the database;
Confirming a basic model created corresponding to the determined character classification; and
Customizing the identified basic model according to the person's face to provide an avatar with a character shape corresponding to the person's face.
Real-time video chat rendering technology using character IP, including: According to paragraph 1,
Generating the first set of measurement values includes:
Generating the first set of measurement values by extracting measurement values for each predefined measurement item for each of the predefined features as facial components for the character's face,
The second measurement value set is a facial component of the person's face and is generated by extracting measurement values for each measurement item for each of the features. Real-time video chat rendering technology using character IP. According to paragraph 1,
Generating the first set of measurement values includes:
Analyzing the plurality of character images using a deep learning model to classify the characters included in each of the plurality of character images to determine the character classification.
Real-time video chat rendering technology using character IP, including. According to paragraph 1,
The step of creating the basic model is,
Real-time video using Character IP, calculating the average value of elements corresponding to each other in first measurement value sets stored in association with the same character classification in the database, and generating the basic model based on the set of calculated average values. Chat rendering technology.