KR102382521B1 - User device, call relay server and method for enabling video call with avatar - Google Patents

Abstract

A user terminal performing a video call using an avatar includes a video call requesting unit for requesting a video call between the user terminal and at least one other user terminal, and when a video call between the user terminal and the other user terminal is connected, the user A photographing unit that shoots a user's actual image using a camera interlocking with the terminal, transmits the photographed user's actual image to a call relay server, and receives the other user's actual image taken in the other user terminal from the call relay server A communication unit, a display unit for displaying the actual image of the user on the first area of the screen of the user terminal through the general call mode for the video call, and the other user's actual image on the second area of the screen, and the video call and a mode change unit configured to change the normal call mode to the avatar call mode when receiving a request for an avatar call mode for The live-action image is replaced with a preset user avatar and displayed, and further comprising an avatar generator generating a user face model based on facial features of the user of the user terminal and a pre-learned average face model, wherein the avatar generator includes a A user avatar is generated by synthesizing a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the modeled beauty model and the user face model.

Description

A user terminal, call relay server and method for performing a video call using an avatar {USER DEVICE, CALL RELAY SERVER AND METHOD FOR ENABLING VIDEO CALL WITH AVATAR}

The present invention relates to a user terminal, a call relay server, and a method for performing a video call using an avatar.

A smart phone is a mobile device developed so that some of the tasks that can be done with a computer can also be performed on a mobile phone. People carry their smartphones with them all the time to browse the Internet, send and receive mail, and shoot and edit videos or photos. In addition, smartphone users may exchange content such as videos, music, and photos with each other.

Recently, video calls have been made possible using a camera mounted on a smartphone, and with respect to a technology for performing a video call in such a smart phone, Korea Patent Application Laid-Open No. 2017-0110469, a prior art, discloses a method for providing a video call and An electronic device for this purpose is disclosed.

However, some users are reluctant to answer an unexpected video call depending on the user's current status. In addition, users who use video calls want to show their individuality to the other party or to be seen as a more beautiful appearance to the other party.

An object of the present invention is to provide a user terminal, a call relay server, and a method for generating a user avatar based on a user's facial features, converting a user's image taken during a video call into a user avatar, and performing a video call with a counterpart.

An object of the present invention is to provide a user terminal, a call relay server, and a method that allow the user's avatar's expression to change according to the user's facial expression captured during a video call, so that the user's current facial expression and emotions are transmitted to the other party through the user avatar.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides a video call request unit for requesting a video call between a user terminal and at least one other user terminal, and a video call between the user terminal and the other user terminal is connected, a photographing unit that shoots a user's actual image using a camera interlocked with the user terminal, transmits the photographed user's actual image to a call relay server, and the other photographed in the other user terminal from the call relay server A communication unit for receiving the user's live-action image, displays the user's live-action image in the first area of the screen of the user terminal through the general call mode for the video call, and displays the other user's live-action image in the second area of the screen and a mode change unit for changing the normal call mode to the avatar call mode when a request for an avatar call mode for the video call is received, wherein the display unit displays the second screen of the screen when the avatar call mode is requested. The user's live-action image displayed in area 1 is replaced with a preset user avatar and displayed, and further comprising an avatar generator for generating a user face model based on facial features of the user of the user terminal and a pre-learned average face model, , the avatar generator is a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model, and the user's face model through synthesis It is possible to provide a user terminal that generates a user avatar.

Another embodiment of the present invention provides a requesting unit for receiving a request for a video call from a user terminal to at least one other user terminal, and when the video call is accepted by the other user terminal, by receiving user video data from the user terminal and a communication unit for transmitting to the other user terminal, receiving the other user's image data from the other user's terminal and transmitting it to the user terminal, wherein when the video call is in a normal call mode, the user image data is transmitted from the user terminal's When the video call is changed from the normal call mode to the avatar call mode in the user terminal, the user image data includes a live-action user image, and the other user image data includes a live-action image of another user of the other user terminal. includes an image of a user avatar replaced from the user's actual image, wherein a user face model is generated by the user terminal based on facial features of the user of the user terminal and a pre-learned average face model, the user The avatar is generated by synthesizing a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model and the user face model. A call relay server may be provided.

Another embodiment of the present invention includes the steps of receiving a request for a video call from a user terminal to at least one other user terminal, and when the video call is accepted by the other user terminal, receiving user video data from the user terminal transmitting to the other user terminal; receiving the other user's image data from the other user's terminal and transmitting the other user's image data to the user terminal; of a user's live-action image, wherein the other user's image data includes another user's live-action image of the other user terminal, and when the video call is changed from the general call mode to the avatar call mode in the user terminal, the user image The data includes an image for a user avatar displayed by being replaced from the user's actual image, wherein a user face model is generated by the user terminal based on facial features of the user of the user terminal and a pre-learned average face model, The user avatar is generated by synthesizing the user face model and a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model. It is possible to provide a call relay method that is.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, a user terminal that generates a user avatar based on the user's facial features, converts the image of the user photographed during a video call into a user avatar, and performs a video call with the other party; A call relay server and method may be provided.

It is possible to provide a user terminal, a call relay server, and a method that allow the user's avatar's expression to change according to the user's facial expression captured during a video call, so that the user's current facial expression and emotions are transmitted to the other party through the user avatar. .

1 is a block diagram of a video call system according to an embodiment of the present invention.
2 is a block diagram of a user terminal according to an embodiment of the present invention.
3A to 3F are exemplary views for explaining a process of generating a user avatar in a user terminal according to an embodiment of the present invention.
4A to 4C are exemplary views for explaining a process of changing a replacement requested part among parts included in a user's avatar's face by a user into a replacement element according to an embodiment of the present invention.
5A to 5E are exemplary diagrams for explaining a process of modifying a part of an avatar created in a user terminal according to an embodiment of the present invention.
6A to 6C are exemplary diagrams for explaining a process of selecting one of a plurality of user avatars as a representative user avatar in a user terminal according to an embodiment of the present invention.
7A and 7B are exemplary diagrams for explaining a process of requesting a video call from a user terminal to another user terminal according to an embodiment of the present invention.
8A to 8D are exemplary diagrams for explaining a process of performing a video call with another user terminal in a user terminal according to an embodiment of the present invention.
9 is a flowchart of a method of performing a video call using an avatar in a user terminal according to an embodiment of the present invention.
10 is a block diagram of a call relay server according to an embodiment of the present invention.
11 is a flowchart of a method of performing a video call using an avatar through a call relay server according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a video call system according to an embodiment of the present invention. Referring to FIG. 1 , a video call system 1 may include a user terminal 110 , another user terminal 120 , and a call relay server 130 . The user terminal 110 , the other user terminal 120 , and the call relay server 130 exemplarily illustrate components that can be controlled by the video call system 1 .

Each component of the video telephony system 1 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1 , the user terminal 110 may be connected to the call relay server 130 at the same time or at a time interval.

A network refers to a connection structure in which information can be exchanged between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The user terminal 110 may generate a user avatar based on facial features of the user of the user terminal 110 . For example, the user terminal 110 receives a photographed image of the user of the user terminal 110 , adjusts at least one of a size or a resolution of the input photographed image, and detects a feature point of the user from the adjusted photographed image and a user face model may be generated based on the pre-learned average face model including the detected feature points and a plurality of vertices. In this case, the user terminal 110 may generate a user avatar using the user face model and the beauty offset. For example, the beauty offset may be a beauty weight applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in a pre-modeled beauty model.

The user terminal 110 may be requested to replace at least one part included in the face of the generated user avatar, and may receive a replacement element for at least one part to be replaced. For example, the user terminal 110 may change vertex coordinates corresponding to at least one part requested to be replaced into vertex coordinates corresponding to the selected replacement element.

When there are a plurality of generated user avatars, the user terminal 110 may receive any one of the plurality of user avatars set as the representative user avatar.

The user terminal 110 may request a video call between the user terminal 110 and at least one other user terminal 120 . For example, the user terminal 110 receives a selection of at least one friend from the list of friends registered in the user terminal 110 , and conducts a video call with another user terminal 120 corresponding to the selected friend to the call relay server 130 . ) can be requested.

When a video call is connected between the user terminal 110 and the other user terminal 120 , the user terminal 110 may use a camera interworking with the user terminal 110 to take a user's actual image.

The user terminal 110 may transmit the captured user's actual image to the call relay server 130 , and may receive the other user's actual image captured by the other user terminal 120 from the call relay server 130 . For example, when the general call mode is performed, the user terminal 110 may encode the captured user's live-action image and transmit it to the call relay server 130 . As another example, when the avatar call mode is performed, the user terminal 110 may encode an image of the user avatar and transmit it to the call relay server 130 .

The user terminal 110 may display the user's live-action image in the first area of the screen of the user terminal 110 through the general call mode for the video call, and may display the other user's live-action image in the second area of the screen.

According to an embodiment, the user terminal 110 may perform a video call by changing from the normal call mode based on the actual user image to the avatar call mode based on the user avatar. In this case, when the user terminal 110 receives a request for the avatar call mode for the video call, the user terminal 110 may change the normal call mode to the avatar call mode. For example, the user terminal 110 may display the user's live-action image displayed on the first area of the screen by replacing it with a preset user avatar. In this case, the user terminal 110 may track the facial expression of the user of the user terminal 110 from the captured user's actual image, and change the facial expression of a preset user avatar according to the tracked facial expression. Here, when the avatar call mode is performed, the user terminal 110 may display a user avatar set as a representative user avatar.

According to another embodiment, the user terminal 110 may perform a video call by applying the beauty effect mode in the normal call mode based on the actual user image. For example, when the user terminal 110 receives a request for a beauty effect mode during a video call, the user terminal 110 may apply and display the beauty effect to the user's actual image displayed in the first area of the screen.

The other user terminal 120 may receive a video call from the user terminal 110 through the call relay server 130 . Here, the other user terminal 120 may perform a video call through the general call mode, the avatar call mode, and the beauty effect mode in the same process as the user terminal 110 .

The call relay server 130 may receive a request for a video call from the user terminal 110 to at least one other user terminal 120 . For example, the call relay server 130 receives the selection of at least one friend from the list of friends registered in the application of the user terminal 110 from the user terminal 110, and the other user terminal 120 corresponding to the selected friend. You can request a video call from

When the video call is accepted by the other user terminal 120 , the call relay server 130 may receive user image data from the user terminal 110 and transmit it to the other user terminal 120 . Also, the call relay server 130 may receive the other user's image data from the other user terminal 120 and transmit it to the user terminal 110 .

The call relay server 130 may receive image data based on a call mode performed by the user terminal 110 or the other user terminal 120 . For example, when the general call mode is performed in the user terminal 110 , the call relay server 130 receives the user image data encoded with the actual user image from the user terminal 110 , and the other user terminal 120 . When the normal call mode is performed in , it is possible to receive the other user's image data encoded with the actual image of the other user from the other user terminal 120 . For another example, when the avatar call mode is performed in the user terminal 110 , the call relay server 130 receives user image data in which an image of the user avatar is encoded from the user terminal 110 , and another user terminal When the avatar call mode is performed in 120 , image data of another user in which an image of the other user's avatar is encoded may be received from the other user terminal 120 .

2 is a block diagram of a user terminal according to an embodiment of the present invention. Referring to FIG. 2 , the user terminal 110 includes an avatar generating unit 210 , a video call requesting unit 220 , a photographing unit 230 , a communication unit 240 , a display unit 250 , and a mode changing unit 260 . may include Here, the configuration included in the user terminal 110 is the same as the configuration included in the other user terminal 120 , and the other user terminal 120 may also perform a video call through the same operation as the user terminal 110 . .

The avatar generator 210 may generate a user avatar based on facial features of the user of the user terminal 110 . The process of generating the user avatar will be described in detail with reference to FIGS. 3A to 3F .

3A to 3F are exemplary views for explaining a process of generating a user avatar in a user terminal according to an embodiment of the present invention.

Referring to FIG. 3A , a user may enter a process for generating a user avatar by receiving an avatar creation 301 button through the application 300 .

Referring to FIG. 3B , the avatar generator 210 may receive a captured image 302 of the user of the user terminal 110 . For example, the avatar generator 210 may photograph a user's face to generate a user avatar, and may receive a photographed photographed image 302 . For another example, the avatar generator 210 selects one of a plurality of pre-photographed images stored in an album of the user terminal 110 as the photographed image 302 to generate a user avatar from the user, and selects the photographed image ( 302) can be entered.

The avatar generator 210 may adjust at least one of a size and a resolution of the input captured image. For example, the avatar generator 210 receives a reduction input of the captured image 302 through a pinch-in input from the user, and receives the captured image 302 through a pinch-out. By receiving the magnification of , the size of the user's face may correspond to the size of a predetermined area (eg, a circle or a rectangle). As another example, the avatar generator 210 receives, through a drag input, from the user so that the user's face included in the photographed image 302 is positioned at the center of the predetermined area 303 , for example, 1024x1024 The size of the photographed image 302 may be adjusted by the size.

Referring to FIG. 3C , the avatar generator 210 may receive a user's gender 304 from the user, and generate a user avatar based on the input gender 304 .

Referring to FIG. 3D , the avatar generator 210 may detect the feature point 310 of the user from the adjusted captured image. Here, the feature point may be shape information such as the user's eyes, nose, mouth, and face outline.

The avatar generator 210 may generate the user face model 312 based on the pre-learned average face model 311 including the detected feature points 310 and a plurality of vertices. For example, the avatar generator 210 may generate the user face model 312 by projecting the previously learned average face model 311 on the detected feature points 310 of the user. Here, the average face model 311 may be formed of a unique vector value representing a degree of change of a face by learning a plurality of face meshes based on Principal Component Analysis (PCA).

Referring to FIG. 3E , the avatar generator 210 may generate a user avatar 314 by synthesizing a user face model 312 with a beauty offset. Here, the beauty offset may be derived by multiplying an offset obtained by subtracting a plurality of vertex coordinates included in the average face model 311 from a plurality of vertex coordinates included in the pre-modeled beauty model 313 by a beauty weight. The beauty weight may indicate the strength of beauty. For example, the avatar generator 210 may generate the user avatar 314 by combining the beauty offset with the user face model 312 and summing the vertex coordinates.

Referring to FIG. 3F , after checking the user avatar 315 that has been created, the user can store (316) the user avatar 315 and manage it.

Returning to FIG. 2 , the avatar generator 210 may receive a request to modify the generated user avatar from the user, and may modify the user avatar based on the modification request. The process of modifying the user avatar will be described in detail with reference to FIGS. 4A to 5E .

4A to 4C are exemplary views for explaining a process of changing a replacement requested part among parts included in a user's avatar's face by a user into a replacement element according to an embodiment of the present invention.

Referring to FIG. 4A , the avatar generator 210 receives a request for replacement of at least one part included in the face of the generated user avatar 400 , and selects a replacement element 410 for at least one part to be replaced. can be chosen

Referring to FIG. 4B , when a replacement element 410 is simply replaced with a replacement element 410 for a part of the face of the user avatar 400 , there is a disadvantage in that unnaturalness occurs due to the simple replacement.

Referring to FIG. 4C , the avatar generator 210 may change the vertex coordinates corresponding to at least one part requested to be replaced into vertex coordinates corresponding to the selected replacement element in order to prevent unnaturalness caused by simple replacement. there is. Through this, the shape of the part requested to be replaced by the user avatar 400 is changed to approximate the shape of the replacement element 410 , thereby preventing unnaturalness.

5A to 5E are exemplary diagrams for explaining a process of modifying a part of an avatar created in a user terminal according to an embodiment of the present invention.

Referring to FIG. 5A , the avatar generator 210 may receive an avatar edit button 501 of the application 500 from the user.

Referring to FIG. 5B , the avatar generator 210 may display the generated user avatar. In this case, the avatar generator 210 may display various buttons 510 for modifying the user avatar, such as an avatar face button 511 , an avatar body button, an avatar accessory button, and an avatar background button.

The avatar face button 511 may call a menu for changing the shape of the user's avatar's face element, such as a hairstyle and eyebrow shape, and the avatar generator 210 receives an avatar face button 511 from the user, You can display the face menu.

The avatar body button may call a menu for changing an appearance element currently worn by the user's avatar, and the avatar generator 210 may display an avatar body menu upon receiving an avatar body button input from the user.

The avatar accessory button may call a menu for selecting accessories such as glasses to be worn on the user's avatar, and the avatar generator 210 may display an accessory menu when receiving an accessory button input from the user.

The avatar background button may call a menu for changing the background button of the user's avatar, and the avatar generating unit 210 may display a background menu when receiving an avatar background button from the user.

Referring to FIG. 5C , when the avatar face button 511 is input by the user, the avatar generator 210 includes a face such as a hair category, a face type category, an eyebrow category, an eye category, a nose category, a mouth category, and a beard category. Various categories 520 related to can be displayed.

Referring to FIG. 5D , when an eyebrow category is selected by the user, the avatar generator 210 may display various eyebrow shapes and receive a selection of any one eyebrow shape 521 from among various eyebrow shapes.

Alternatively, the avatar generator 210 may receive an adjustment point corresponding to a kind of node for adjusting the overall shape of the face from the user, and change the shape of a part corresponding to each category based on the adjustment point. The control points can be adjusted up/down or left/right, and if the control points are arranged symmetrically and one control point is adjusted, the other control points can also be adjusted.

Referring to FIG. 5E , the avatar generator 210 may display the user avatar 530 to which the selected eyebrow shape 521 is applied.

Returning to FIG. 2 , when there are a plurality of generated user avatars, the avatar generator 210 may receive any one of the plurality of user avatars as the representative user avatar. The process of setting the representative user avatar will be described in detail with reference to FIGS. 6A to 6C .

6A to 6C are exemplary diagrams for explaining a process of selecting one of a plurality of user avatars as a representative user avatar in a user terminal according to an embodiment of the present invention.

*65 Referring to FIG. 6A , the avatar generator 210 may receive a representative avatar change button 600 of the application from the user and enter a process for setting the representative user avatar.

Referring to FIG. 6B , when there are a plurality of generated user avatars, the avatar generator 210 selects any one user avatar 611 among the plurality of user avatars 610 from the user and then clicks the Done button 612 . can be input.

Referring to FIG. 6C , the avatar generator 210 may set the selected user avatar 611 as the representative user avatar 620 . Here, the representative user avatar 620 may be used when the user terminal 110 performs a video call with the other user terminal 120 using the avatar, and is later changed to another user avatar by the user's selection. can be

*68 Returning to FIG. 2 again, the video call request unit 220 may request a video call between the user terminal 110 and at least one other user terminal 120 . The process of requesting a video call will be described in detail with reference to FIGS. 7A and 7B .

7A and 7B are exemplary diagrams for explaining a process of requesting a video call from a user terminal to another user terminal according to an embodiment of the present invention.

Referring to FIG. 7A , the video call request unit 220 may receive a call button 701 displayed on the screen 700 of the application from the user.

Referring to FIG. 7B , the video call request unit 220 receives a selection of at least one friend from the friend list 710 registered in the user terminal 110 , and performs an image with the other user terminal 120 corresponding to the selected friend. A call may be requested to the call relay server 130 . For example, the video call request unit 220 may display a list of friends capable of video calls (710), select a friend from the user (711), and then request (712) a video call to the call relay server 130. there is. Here, the video call request unit 220 may request a video call with one friend or a video call with a plurality of friends.

Returning to FIG. 2 again, when the video call between the user terminal 110 and the other user terminal 120 is connected, the photographing unit 230 may take a user's actual image by using a camera interworking with the user terminal 110 . . For example, the photographing unit 230 may drive the front camera of the user terminal 110 to photograph the user's live-action image.

The communication unit 240 may transmit the captured user's live-action image to the call relay server 130 , and may receive the other user's live-action image captured by the other user terminal 120 from the call relay server 130 . For example, when the normal call mode is performed, the communication unit 240 may encode the captured user's live-action image and transmit it to the call relay server 130 . For another example, when the avatar call mode is performed, the communication unit 240 may encode an image of the user avatar and transmit it to the call relay server 130 .

The display unit 250 may display the user's live-action image in the first area of the screen of the user terminal 110 through the general call mode for the video call, and may display the other user's live-action image in the second area of the screen. For example, the display unit 250 may display the user's live-action image in the first area located at the upper right of the screen, and display the other user's live-action image in the second area corresponding to the remaining area except for the first area.

According to an embodiment, the user terminal 110 may perform a video call by changing from the normal call mode based on the actual user image to the avatar call mode based on the user avatar.

When receiving a request for an avatar call mode for a video call, the mode change unit 260 may change the normal call mode to the avatar call mode.

*77 When the avatar call mode is requested, the display unit 250 may replace or change the user's live-action image displayed on the first area of the screen with a preset user avatar and display it. Here, when the avatar call mode is performed, the display unit 250 may display a user avatar set as a representative user avatar.

When the avatar call mode is performed, the display unit 250 may track the facial expression of the user of the user terminal 110 from the captured user's live-action image, and may change the preset facial expression of the user avatar according to the tracked facial expression.

According to another embodiment, the user terminal 110 may perform a video call by applying the beauty effect mode in the normal call mode based on the actual user image.

The mode change unit 260 may receive a request for a beauty effect mode during a video call.

The display unit 250 may display a beauty effect applied to the user's actual image displayed in the first area of the screen.

Although not shown in FIG. 2 , the user terminal 110 may perform a video call using the general call mode, the avatar call mode, and the beauty effect mode with the other user terminal 120 and chat on the same screen at the same time.

8A to 8D are exemplary diagrams for explaining a process of performing a video call with another user terminal in a user terminal according to an embodiment of the present invention.

8A is an exemplary diagram illustrating a process of performing a video call with another user terminal 120 through a normal call mode in the user terminal 110 according to an embodiment of the present invention. Referring to FIG. 8A , the user terminal 110 may perform a video call between the user 800 and another user through a normal call mode. In this case, the user terminal 110 may perform a 1:1 video call with the user 800 and one other user, and may perform a multi-party video call with the user 800 and a plurality of other users.

8B to 8D are exemplary diagrams illustrating a process of performing a video call with another user terminal through an avatar call mode in a user terminal according to an embodiment of the present invention.

Referring to FIG. 8B , the user terminal 110 may conduct a video call with one other user terminal 120 . For example, when the user terminal 110 performs a video call through the normal call mode and the other user terminal 120 changes from the normal call mode to the avatar call mode to perform the video call, the user terminal 110 may display the user's live-action image in the second area 820 of the screen of the user terminal 110, and display the image of the other user's avatar in the first area 810 of the screen.

Referring to FIG. 8C , the user terminal 110 may conduct a video call with a plurality of other user terminals 120 . For example, when the user terminal 110 conducts a video call with a plurality of other user terminals 120 , the user terminal 110 may divide the screen based on the total number of people participating in the video call. Here, the sizes of the divided screens may all be the same, but are not limited thereto.

For example, the user terminal 110 may receive a change in the location of each split screen by the user, and display each split screen based on the changed location. For another example, when an enlargement input is received from the user on one of the plurality of split screens, the user terminal 110 may enlarge and display the corresponding screen and display the remaining screens by reducing the screen at the same ratio.

The user and a plurality of other users may display respective images based on the general call mode or the avatar call mode performed in each terminal. For example, when the user terminal 110 performs a video call with a plurality of other user terminals 120 through the avatar call mode, the user terminal 110 moves to the first area 830 of the plurality of divided areas. The user's appearance may be replaced with a user avatar and displayed.

Referring to FIG. 8D , when a video call is performed through the avatar call mode, the user terminal 110 tracks the facial expression of the user of the user terminal 110 from the captured user's live-action image, and follows the tracked facial expression. It is possible to change the facial expression of the user avatar. For example, when the user makes a winking expression, the user avatar 840 may also make a wink expression according to the user's expression.

9 is a flowchart of a method of performing a video call using an avatar in a user terminal according to an embodiment of the present invention. The method of performing a video call using an avatar in the user terminal 110 shown in FIG. 9 includes steps processed in time series by the video call system 1 according to the embodiment shown in FIGS. 1 to 8D . do. Therefore, even if omitted below, it is also applied to the method of performing a video call using the avatar in the user terminal 110 according to the embodiment shown in FIGS. 1 to 8D .

In step S910 , the user terminal 110 may request a video call between the user terminal and at least one other user terminal 120 .

In step S920 , when a video call is connected between the user terminal 110 and the other user terminal 120 , the user terminal 110 may use a camera interworking with the user terminal 110 to take a user's actual image.

In step S930, the user terminal 110 transmits the captured user live-action image to the call relay server 130, and may receive the other user's live-action image captured in the other user terminal 120 from the call relay server 130. .

In step S940, the user terminal 110 displays the user's live-action image on the first area of the screen of the user terminal 110 through the general call mode for the video call, and displays the other user's live-action image on the second area of the screen. can

In step S950 , when the user terminal 110 receives a request for the avatar call mode for the video call, the user terminal 110 may change the normal call mode to the avatar call mode.

When the user terminal 110 receives a request for the avatar call mode in step S960, the user's live-action image displayed on the first area of the screen may be replaced with a preset user avatar and displayed.

In the above description, steps S910 to S960 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

10 is a block diagram of a call relay server according to an embodiment of the present invention. Referring to FIG. 10 , the call relay server 130 may include a request unit 1010 and a communication unit 1020 .

The requester 1010 may receive a request for a video call from the user terminal 110 to at least one other user terminal 120 . For example, the requesting unit 1010 receives a selection of at least one friend from the list of friends registered in the application of the user terminal 110 from the user terminal 110, and communicates with the other user terminal 120 corresponding to the selected friend. You may be asked to make a video call.

When the video call is accepted by the other user terminal 120 , the communication unit 1020 may receive user image data from the user terminal 110 and transmit it to the other user terminal 120 . Also, the communication unit 1020 may receive other user image data from the other user terminal 120 and transmit it to the user terminal 110 .

The communication unit 1020 may receive image data based on a call mode performed by the user terminal 110 or the other user terminal 120 . For example, when the general call mode is performed in the user terminal 110 , the communication unit 1020 receives the user image data encoded with the actual user image from the user terminal 110 , and in the other user terminal 120 , the general call mode is performed. When the call mode is performed, the other user's image data in which the actual image of the other user is encoded may be received from the other user terminal 120 . For another example, when the avatar call mode is performed in the user terminal 110 , the communication unit 1020 receives user image data in which an image of the user avatar is encoded from the user terminal 110 , and the other user terminal 120 . ), when the avatar call mode is performed, the other user's image data in which an image of the other user's avatar is encoded may be received from the other user terminal 120 .

That is, the call relay server 130 based on the video call mode selected in each of the user terminal 110 and the other user terminal 120 from the user terminal 110 and the other user terminal 120, in the case of a normal call mode, the actual image The encoded image data may be received, and in the case of the avatar call mode, the encoded image data for the avatar may be received.

11 is a flowchart of a method of performing a video call using an avatar in a call relay server according to an embodiment of the present invention. The method of performing a video call using an avatar in the call relay server 130 shown in FIG. 11 includes steps processed in time series by the video call system 1 according to the embodiments shown in FIGS. 1 to 10 . include Therefore, even if omitted below, it is also applied to the method of performing a video call using the avatar in the call relay server 130 according to the embodiment shown in FIGS. 1 to 10 .

In step S1110 , the call relay server 130 may receive a request for a video call from the user terminal 110 to at least one other user terminal 120 .

In step S1120 , when the video call is accepted by the other user terminal 120 , the call relay server 130 may receive user image data from the user terminal 110 and transmit it to the other user terminal 120 .

In step S1130 , the call relay server 130 may receive the other user's image data from the other user terminal 120 and transmit it to the user terminal 110 .

In the above description, steps S1110 to S1130 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

The method of performing a video call using an avatar in the user terminal and the call relay server described with reference to FIGS. 1 to 11 includes a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. It can also be implemented in a form. Also, the method of performing a video call using an avatar in the user terminal and the call relay server described with reference to FIGS. 1 to 11 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

110: user terminal
120: other user terminal
130: call relay server
210: avatar generator
220: video call request unit
230: shooting department
240: communication department
250: display
260: mode change unit
1010: request
1020: communication department

Claims (16)

In the user terminal performing a video call using an avatar,
a video call request unit for requesting a video call between the user terminal and at least one other user terminal;
When the video call is connected between the user terminal and the other user terminal, a photographing unit for photographing a user's live-action image using a camera interworking with the user terminal;
a communication unit for transmitting the captured user's actual image to a call relay server and receiving the other user's actual image taken from the other user's terminal from the call relay server;
a display unit for displaying the user's actual image on the first area of the screen of the user terminal through the general call mode for the video call, and displaying the other user's actual image on the second area of the screen; and
When receiving a request for an avatar call mode for the video call, a mode change unit for changing the normal call mode to the avatar call mode
including,
When the display unit receives a request for the avatar call mode, the user's live-action image displayed on the first area of the screen is replaced and displayed with a preset user avatar,
Further comprising an avatar generator for generating a user face model based on the facial features of the user of the user terminal and the pre-learned average face model,
The avatar generator generates a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model, and the user through synthesis with the user face model. A user terminal that generates an avatar.
The method of claim 1,
The communication unit, when the normal call mode is performed, the user terminal by encoding the captured user live-action image and transmitting it to the call relay server.
The method of claim 1,
When the avatar call mode is performed, the regular communication unit encodes an image of the user avatar and transmits the encoded image to the call relay server.
The method of claim 1,
When the avatar call mode is performed, the display unit tracks the facial expression of the user of the user terminal from the captured user live-action image, and changes the facial expression of the preset user avatar according to the tracked facial expression. , user terminal.
The method of claim 1,
The video call request unit receives a selection of at least one friend from a list of friends registered in the user terminal, and requests a video call with another user terminal corresponding to the selected friend to the call relay server.
The method of claim 1,
The avatar generating unit receives, when there are a plurality of generated user avatars, one of the plurality of user avatars set as a representative user avatar;
When the avatar call mode is performed, the display unit displays a user avatar set as the representative user avatar.
The method of claim 1,
The avatar generating unit,
receiving an input of a photographed image for a user of the user terminal;
Adjusting at least one of the size or resolution of the input captured image,
Detecting the feature point of the user from the adjusted photographed image,
generating the user face model based on the pre-learned average face model including the detected feature points and a plurality of vertices.
The method of claim 1,
wherein the avatar generator receives a request for replacement of at least one part included in the face of the generated user avatar, and receives a selection of a replacement element for the at least one part to be replaced.
9. The method of claim 8,
and the avatar generator changes vertex coordinates corresponding to the at least one part requested to be replaced into vertex coordinates corresponding to the selected replacement element.
The method of claim 1,
The mode change unit receives a request for a beauty effect mode during the video call,
The display unit, the user terminal by applying a beauty effect to the user's live-action image displayed in the first area of the screen.
In the call relay server for performing a video call using an avatar,
a request unit for receiving a request for a video call from the user terminal to at least one other user terminal; and
When the video call is accepted by the other user terminal, the communication unit receives the user image data from the user terminal and transmits it to the other user terminal, and receives the other user image data from the other user terminal and transmits it to the user terminal including,
When the video call is a normal call mode, the user image data includes a user live-action image of the user terminal, and the other user image data includes an actual image of another user of the other user terminal,
When the video call is changed from the normal call mode to the avatar call mode in the user terminal, the user image data includes an image of the user avatar replaced from the actual user image,
A user face model is generated by the user terminal based on facial features of the user of the user terminal and a pre-learned average face model,
The user avatar is generated by synthesizing the user face model and a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model. that is, a call relay server.
12. The method of claim 11,
When the general call mode is performed in the user terminal, the communication unit receives the user image data in which the actual user image is encoded from the user terminal,
When the normal call mode is performed in the other user terminal, the call relay server to receive the other user's image data encoded with the actual image of the other user from the other user terminal.
12. The method of claim 11,
When the avatar call mode is performed in the user terminal, the communication unit receives the user image data encoded with the image for the user avatar from the user terminal, and when the avatar call mode is performed in the other user terminal , A call relay server that receives the other user's image data encoded with an image of the other user's avatar from the other user's terminal.
A method of performing a video call using an avatar through a call relay server, the method comprising:
receiving a video call request from the user terminal to at least one other user terminal;
receiving user image data from the user terminal and transmitting the user image data to the other user terminal when the video call is accepted by the other user terminal; and
Receiving other user image data from the other user terminal and transmitting the image data to the user terminal,
When the video call is a normal call mode, the user image data includes a user live-action image of the user terminal, and the other user image data includes an actual image of another user of the other user terminal,
When the video call is changed from the normal call mode to the avatar call mode in the user terminal, the user image data includes an image for the user avatar displayed instead of the actual user image,
A user face model is generated by the user terminal based on facial features of the user of the user terminal and a pre-learned average face model,
The user avatar is generated by synthesizing the user face model and a beauty offset in which a beauty weight is applied to an offset obtained by subtracting a plurality of vertex coordinates included in the average face model from a plurality of vertex coordinates included in the pre-modeled beauty model. What is a call relay method.
15. The method of claim 14,
When the general call mode is performed in the user terminal, receiving from the user terminal the user image data encoded with the actual user image; and
When the normal call mode is performed in the other user terminal, the method further comprising the step of receiving the other user's image data encoded with the actual image of the other user from the other user terminal, the call relay method.
15. The method of claim 14,
receiving, from the user terminal, the user image data in which an image of the user avatar is encoded, when the avatar call mode is performed in the user terminal; and
When the avatar call mode is performed in the other user terminal, the method further comprising: receiving, from the other user terminal, the other user's image data encoded with an image of the other user's avatar.

Images