KR101189053B1 - Method For Video Call Based on an Avatar And System, Apparatus thereof - Google Patents

Abstract

The present invention relates to an avatar-based video call method and system and a terminal supporting the same. In a video call, the terminal displays an avatar corresponding to the user of the opposite terminal, recognizes at least one of a gesture and a voice of the other terminal user. The avatar execution code generated corresponding thereto is received from the counterpart terminal, and at least one of the state and operation of the avatar is adjusted and output according to the received avatar execution code. According to the present invention, various emotional states of the other party can be transmitted to the user through an avatar output based on the other party's motion or voice during a video call.

Handset, video call, avatar, gesture, voice

Description

Avatar-based video call method and system, and a terminal supporting the same {Method For Video Call Based on an Avatar And System, Apparatus}

The present invention relates to a video call technology, and more particularly, to an avatar-based video call method for recognizing at least one of a gesture and a voice of a speaker during a video call, and controlling at least one of a specific avatar's state and motion based thereon; System, and a terminal supporting the same.

Recently, terminals have been able to implement various optional functions thanks to the rapid development of technology for miniaturizing the size while maintaining the electrical capacity of the battery. For example, while the camera is mounted, the terminal supports a function of collecting an image of a specific subject in association with a call function. Accordingly, the terminal user may use a video call function with another terminal user based on a camera mounted on the terminal.

Since the video call function makes a call while checking the face or background of the other party as compared to the time when the call is made by voice only, the advantage of communicating the user's emotional state to the other party is better than simply making a voice call. have.

However, the current video call is restricted by various environmental conditions, for example, bandwidth, transmission speed, data processing capacity of the terminal, and thus, video is often disconnected or not transmitted properly. Accordingly, there is a problem that it is difficult to accurately recognize the image related to the other party even when performing a video call.

Accordingly, an object of the present invention is to recognize at least one of a gesture or a voice of a speaker during a video call and to adjust at least one of the state and operation of a specific avatar, and then to output the adjusted avatar. The present invention provides a terminal.

In order to achieve the above object, the present invention provides an avatar-based video call system including a first terminal and a second terminal. When the video call channel is formed, the first terminal outputs an avatar corresponding to the second terminal, and adjusts at least one of the state and operation of the avatar based on the avatar execution code received from the second terminal. The second terminal outputs the status and operation of the avatar based on at least one of a specific gesture recognized from an operation state and an operation change of a subject collected by the camera and a specific voice recognized from an audio signal collected by a microphone. Generate the avatar execution code to adjust at least one, and transmit the avatar execution code to the first terminal.

The present invention also provides an avatar-based video call system including a communication network and an avatar providing server. The communication network controls a video call between the first terminal and the second terminal. When the avatar providing server receives an avatar request signal from the first or second terminal after a video call channel is formed through the communication network, the avatar providing server transmits an avatar corresponding to the other party to the terminal that sent the avatar request signal. Receive video call data from a counterpart terminal of the terminal receiving the avatar, generate an avatar execution code based on a specific gesture and voice recognition taken by the other party from the received video call data, and convert the avatar execution code into the avatar Send to the received terminal.

The present invention also provides an avatar providing server of an avatar-based video call system including a transceiver and a server controller. The transceiver unit communicates with the first terminal or the second terminal via a communication network. When the server control unit receives an avatar request signal from the first or second terminal after the video call channel is formed through the communication network, the server controller transmits the avatar corresponding to the other party to the terminal that sent the avatar request signal, and the avatar Receive video call data from a counterpart terminal of the receiving terminal, generate an avatar execution code based on a specific gesture and voice recognition taken by the other party from the received video call data, and receive the avatar execution code from the avatar execution code. Send to one terminal.

The present invention also provides a terminal of an avatar-based video call system including a camera, a display unit, a controller, and a communication unit and a storage unit. The camera collects an image to be transmitted to the opposite terminal for a video call, and the display unit outputs an image of the opposite party received from the opposite terminal. The controller outputs an avatar corresponding to the counterpart terminal to the display unit when a video call channel is established with the counterpart terminal, and recognizes at least one of a gesture and a voice of the counterpart terminal from the counterpart terminal. And control at least one of a state and an operation of the avatar according to the avatar execution code to output to the display unit. The communication unit may support to form a video call channel for a video call and to transmit the avatar execution code using the video call channel or a message service channel. When the avatar receives an avatar from the opposite terminal or designates a specific avatar stored in its own terminal as the avatar of the opposite terminal, the storage unit may store the avatar by connecting to the telephone number of the opposite terminal.

The present invention also provides a counterpart to a terminal to which the avatar providing server sends the avatar request signal when receiving an avatar request signal from the first or second terminal after a video call channel is formed between the first terminal and the second terminal. An avatar transmitting step of transmitting an avatar, a receiving step of the avatar providing server receiving video call data from a counterpart terminal of the terminal receiving the avatar, and the avatar providing server specifying the counterpart from the received video call data; Generating an avatar execution code based on a gesture and a voice recognition; and transmitting the avatar execution code by the avatar providing server to the terminal receiving the avatar. to provide.

The present invention also provides a channel forming step of forming a video call channel between a first terminal and a second terminal, an output step of outputting an avatar corresponding to the second terminal by the first terminal, and collecting by the camera of the second terminal. Avatar execution code for adjusting at least one of the avatar's state and operation based on at least one of a specific gesture recognized from the operation state and the change of motion of the image and the audio signal collected by the microphone An avatar-based image including a reception step received by the first terminal from the second terminal, and an output step of adjusting and outputting at least one of a state and an operation of the avatar according to the avatar execution code by the first terminal; Provide a call method.

According to the video call method and system of the present invention and a terminal supporting the same, during a video call, the terminal displays an avatar corresponding to the user of the opposite terminal, recognizes at least one of a gesture and a voice of the other terminal user, and Since the generated avatar execution code is received from the counterpart terminal or the avatar providing server and the at least one of the avatar's state and operation is output according to the received avatar execution code, the avatar execution code is effectively transmitted to the user through the avatar. Can be. That is, the video call system recognizes the avatar's current emotional state, such as joy, sadness, surprise, pleasure, anger, annoyance, and the like through an avatar through a specific gesture and voice taken by the other party. Through this, the video call can be smoothly performed while recognizing the emotional state of the other party. Accordingly, various emotional states of the other party may be transmitted to the user through an avatar output based on the other party's motion or voice during a video call.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, only parts necessary for understanding the operation according to the embodiment of the present invention will be described, and the description of other parts will be omitted so as not to disturb the gist of the present invention.

The terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary meanings and the inventor is not limited to the meaning of the terms in order to describe his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention, and not all of the technical ideas of the present invention are described. Therefore, It is to be understood that equivalents and modifications are possible.

Here, 'Avatar' is a graphic content that represents the emotional state of the terminal user, such as animated characters, video, still images, UCC (User Created Contents), emoticons, flash, haptic content combined with video and vibration, etc. It includes.

1 is a diagram schematically illustrating a configuration of a video call system for supporting video call operation according to an exemplary embodiment of the present invention.

In the following description, the terminal is described as a terminal using a mobile communication network to describe the video call service of the present invention. However, the present invention is not limited thereto. That is, the video call service of the present invention may be applied to various terminals such as a general wired terminal, a fixed terminal, an IP terminal, and the like, which are equipped with a camera, as well as a terminal using a mobile communication network. When the terminal is a wired terminal capable of supporting a video call, the mobile communication network may be a switch network capable of supporting a video call based on a wired line. In addition, when the terminal is an IP terminal capable of supporting a video call, the mobile communication network may be replaced with an IP network capable of supporting a video call. That is, a communication network supporting a video call service between terminals includes a mobile communication network, an exchange network, and an IP network.

Referring to FIG. 1, the video call system of the present invention may include a first terminal 100, a second terminal 200, and a mobile communication network 300.

In the video call system of the present invention having the configuration as described above, the first terminal 100 or the second terminal 200 may perform a video call request to the opposite terminal through the mobile communication network 300. When the call channel is formed, at least one of the first terminal 100 and the second terminal 200 may output a counterpart avatar corresponding to the counterpart terminal on the screen. In this case, the counterpart avatar may be previously stored in the storage unit of each terminal, or may be received from the counterpart terminal and output.

After the video call channel is formed, the first terminal 100 and the second terminal 200 acquire a still image or a moving image of a subject collected through a camera, and obtain a speaker's voice. The first terminal 100 and the second terminal 200 extract a specific operation state or a change in the operation from the image of the subject to check whether it corresponds to a specific gesture, and recognize whether the specific voice is recognized by recognizing the voice. The first terminal 100 and the second terminal device 200 generates an avatar execution code corresponding to the specific gesture when the operation state and the operation change correspond to a specific gesture or the voice recognition result is a specific voice, and the counterpart side generates the avatar execution code. Can be sent to the terminal. In addition, the first terminal 100 and the second terminal 200 may control at least one of an expression, a state, and an operation of the other avatar by applying the avatar execution code received from the other terminal to the other avatar being output on the current screen. have. Here, each of the first terminal 100 and the second terminal 200 may output its own avatar to its display unit, and at least one of a state and an operation of the avatar according to the avatar execution code transmitted to the counterpart terminal. One may adjust one and output the adjusted one's avatar to the display unit. Through this process, the first terminal 100 and the second terminal 200 can accurately and quickly recognize what the speaker wants to indicate through the specific operation, the specific state, and the specific voice of the counterpart terminal user through the change of the avatar. have.

In order to support the video call service according to the present embodiment, the first terminal 100 and the second terminal 200 may include a configuration as shown in FIG. 2. Here, since the first terminal 100 and the second terminal 200 may include the same configuration to substantially perform the transmission function and the reception function, in the following description, the first terminal 100 and the second terminal The terminal 200 will be described as a terminal and the drawing number will be described by giving the same as the drawing number of the first terminal 100.

2 is a block diagram schematically illustrating a configuration of a terminal according to an exemplary embodiment of the present invention.

2, the terminal 100 of the present invention includes a communication unit 110, an input unit 120, an audio processor 130, a display unit 140, a storage unit 150, a camera 170, and a controller 160. It may include the configuration of).

The communication unit 110 transmits and receives video call data under the control of the controller 160. The communication unit 110 may include a radio frequency transmitter for upconverting and amplifying a frequency of a transmitted signal, and a radio frequency receiver for low noise amplifying and downconverting a received signal. In particular, the communication unit 110 transmits a communication channel for transmitting its avatar to the counterpart terminal under control of the controller 160, and a communication channel for transmitting an avatar execution code for adjusting its avatar state or motion. It can be formed with the terminal. In this case, the communication unit 110 generates a separate data communication channel under the control of the control unit 160 to transmit data related to the avatar, namely, avatar data and avatar execution code for adjusting the state and operation of the avatar, or have already been formed. The avatar data and the avatar execution code may be transmitted through the video call channel. In addition, the communication unit 110 transmits the avatar data and the avatar execution code through the currently created video call channel, or a separate data communication channel or a message service channel, for example, a short message service (SMS) or a multimedia (MMS). Message Service) can be temporarily created to transmit the avatar data and the avatar execution code.

The input unit 120 includes a plurality of input keys and function keys for receiving numeric or character information and setting various functions. The function keys may include direction keys, side keys, shortcut keys, and the like, which are set to perform a specific function. In addition, the input unit 120 generates a key signal input in relation to user setting and function control of the terminals, and transmits it to the controller 160. That is, the input unit 120 inputs an input signal for inputting a phone number of a counterpart terminal, an input signal for forming a video call call based on the input phone number, a mode selection signal for setting an avatar use during a video call, and a video call. An input signal for termination and the like are generated according to a user request and transmitted to the controller 160.

The audio processor 130 includes a speaker SPK for playing audio data transmitted and received during the video call, and a microphone MIC for collecting a user's voice or other audio signal during the video call. Here, the audio processor 130 may further include a speaker for outputting audio data received during a video call separately from the speaker installed for the voice call. The audio processor 130 may transmit the audio signal collected by the microphone MIC to the controller 160 for speech recognition.

The display unit 140 may be a liquid crystal display (LCD), an OLED, or the like, and when the LCD is applied, the display unit 140 may include an LCD controller, a memory capable of storing data, and an LCD display device. When a display device such as an LCD and an OLED is implemented by using a touch screen method, the screen of the display unit 140 may be operated as an input unit. In particular, the display unit 140 of the present invention may display at least one of image data collected and image-processed through the camera 170 and image data transmitted from the counterpart terminal during the video call. For example, assuming that the first terminal 100 and the second terminal 200 perform a video call, the display unit 140 of the first terminal 100 receives the image data transmitted by the second terminal 200. And a second screen area for displaying image data collected and image-processed by the camera 170 mounted on the first terminal 100. In this case, the first screen area and the second screen area may be displayed by dividing the screen on the same plane, and the second screen area may be displayed overlapping on the first screen area. Meanwhile, the first screen area displayed on the display unit 140 may output not only image data transmitted by the second terminal 200, but also the other party's avatar corresponding to the second terminal 200. Here, the preset avatar may be an avatar selected by the user of the first terminal 100. In addition, the second screen area may output not only image data collected by the camera 170 of the first terminal 100, but also its own avatar corresponding to the first terminal 100. Similar to the first terminal 100, the second terminal 200 also outputs at least one of a first screen area and a second screen area for outputting its own image data, its own avatar, its opponent's image data, and its opponent's avatar. Can be. The display interface of the display unit 140 will be described in more detail with reference to FIG. 4.

The storage unit 150 includes an application program for operating a function according to an embodiment of the present invention, an application program required for a video call, an application program for operating the camera 170, avatar data, and an avatar execution code for controlling an avatar. Save them. The storage unit 150 may be responsible for buffering the video call data transmitted / received during a video call by a predetermined size. The storage unit 150 may largely include a program area and a data area.

The program area may include an operating system (OS) for booting the terminal 100, an application program for operating a camera 170 and collecting audio for a video call, and other option functions of the terminal 100, for example, a sound. Stores applications required for playback, image or video playback. When the terminal 100 activates the above functions in response to a user request, the terminal 100 provides each function by using the corresponding application programs under the control of the controller 160. In particular, the program area of the present invention may include an avatar service module, a gesture recognition module, and a voice recognition module to support an avatar. The avatar service module, the gesture recognition module, and the voice recognition module are set to the avatar-based video call service mode of the present invention, and when the video call is executed, the avatar service module, the gesture recognition module, and the voice recognition module may be an application program that can be loaded and activated in the controller 160. have. The gesture recognition module may include a routine for recognizing a specific gesture from an operation state or an operation change of a subject collected by the camera 170, and a routine for generating an avatar execution code corresponding to the recognized specific gesture. When the speech recognition module corresponds to a routine for recognizing an audio signal collected by a microphone (MIC) of the audio processor 130, a routine for determining whether a speech recognized word or sentence corresponds to a specific speech, It may include a routine for generating an avatar execution code mapped thereto. When the avatar service module receives the avatar execution code from the gesture recognition module and the voice recognition module, the avatar service module applies a routine to apply the received avatar execution code to its avatar and transmits the received avatar execution code to the counterpart terminal. It may include.

The data area is an area in which data generated according to the use of the terminal 100 is stored. In a video call, data recorded by the user, data to be stored by the user among images collected by the camera 170, and the terminal ( User data related to various optional functions provided by 100 may be stored, for example, a video, phone book data, audio data, information corresponding to corresponding content or user data, and the like. In particular, the data area of the present invention may store various avatar data for implementing an avatar capable of outputting a certain type of operation and state as an image. Using this, the user of the terminal 100 may set an avatar corresponding to a specific phone number when storing the phone book data. If there is a partner avatar linked to the specific phone number and the video call service of the present invention is performed with the terminal having the phone number, the partner avatar is loaded by the controller 160 and output to the display unit 140. Can be. At this time, the avatar of the user may be output to the display unit 140 according to the user's selection. The avatar may perform an animation including various states or various actions according to the avatar execution code. For example, the avatar may be changed to another shape or shape according to a specific avatar execution code, or may be a form in which another image is added to a specific image. The avatar execution code for the avatar's presentation form may be deleted or added through a setting operation of the designer or the user of the avatar. Here, the shape of the avatar may be changed according to the shape of each avatar, and may also be changed by the avatar designer or the user's setting manipulation. On the other hand, since the avatar execution code is transmitted to the counterpart terminal to adjust the state and operation of the avatar outputted by the counterpart terminal, it is preferable that the avatar execution code is mutually regulated in order to transmit the avatar execution code. The data area may temporarily store the first avatar execution code received from the opponent terminal and the second avatar execution code for controlling his / her avatar, and execute the previous first avatar until the new first avatar execution code is received from the opponent terminal. You can keep the code saved, or save it semi-permanently depending on your settings.

The data area may include a gesture recognition DB (Data Base) for gesture recognition and a voice recognition DB for voice recognition. The gesture recognition DB is a database that provides a criterion for recognizing a specific operation state or a change in motion of a subject as a specific gesture. For example, when the subject is a human, the gesture recognition DB may include reference information about various gestures that can be expressed as a stationary state or an operational state that a human can take. For example, it is assumed that the gesture recognition DB has information about a palm, a predetermined angle, a left-right direction, and a predetermined number of times as a gesture reference for a "shake". Then, when the subject 100 shows the palm of the object in the image collected by the camera 170, and moves the object more than a predetermined number of times in the left and right direction within a predetermined angle, the terminal 100 moves the above-described image based on the gesture recognition DB. Shake "gesture. On the other hand, the speech recognition DB is the first information on the criteria for determining which voice corresponds to the signal transmitted from the audio signal, and the second for the criteria for comparing whether the speech recognition information corresponds to a predetermined specific voice May contain information. The first information is information provided to recognize a phoneme, syllable, word, word, sentence, and the like from a signal collected by the microphone (MIC), and the second information is used to convert the recognized information into an avatar execution code. Information for comparing whether a particular syllable, word, word, or sentence is set.

The camera 170 collects images for video call and gesture recognition. The camera 170 photographs an image photographed through a lens, a camera sensor (not shown) for converting the photographed optical signal into an electrical signal, and a signal for converting an analog image signal photographed from the camera sensor into digital data. A processing unit (not shown). Here, the camera sensor may be a charge coupled device (CCD) sensor, and the signal processor may be implemented as a digital signal processor (DSP), but is not limited thereto. The camera 170 may be activated when an input signal for using a camera function is received, and may be activated when using a video call function. The camera 170 transmits the collected image to the counterpart terminal or the controller 160 for gesture recognition.

The controller 160 may initialize each component of the terminal 100 and perform necessary signal control to support the avatar-based video call service of the present invention. In particular, the controller 160 controls the transmission and reception of the avatar execution code and the avatar output based on a specific gesture and voice recognition taken by the counterpart during a video call.

The controller 160 may set an avatar corresponding to the other party and an avatar execution code for operating the set avatar in the avatar setting mode. The avatar and the avatar execution code may be variously classified and provided according to a user's interests or hobbies such as general, weather, sports (golf, basketball, baseball, etc.). In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the other party likes or is interested in.

For example, the avatar execution code mapped according to a specific gesture and voice of the opponent may operate the avatar as shown in Table 1. At this time, the emotional state is described as being limited to joy, sadness, surprise, but is not limited thereto. In addition, the motion of the avatar according to each emotional state is not limited thereto.

pleasure sadness surprised Normal Avatar laughs Avatar crying
Turm Avatar eyes
Large moxibustion weather Avatar is sunny
Strolling in the weather Avatar rained Avatar head up
Lightning strikes golf The ball put by the avatar
I like the hole cup Avatar tee shot in water
I burst out and cry The tee shot is hole-in-one
Avatar is surprised basketball Avatar is on three point shot
Succeeded Avatar hit the ball
Deprived Avatar shot
Blocked

If the avatar execution code is not set separately, the controller 160 may use the avatar execution code set as a default. For example, a general item may be set as a default.

When the avatar-based video call service request is made, the controller 160 performs negotiation and channel acquisition for the video call connection with the counterpart terminal through the communication unit 110. When there is a video call request or a video call channel is formed, the controller 160 activates the camera 170 to collect an image of a subject, and controls to output the collected image to the display unit 140, Can be processed to meet currency standards. In this case, the controller 160 provides a function of outputting an avatar based on an avatar execution code for changing at least one of a state and an operation of the avatar from at least one of a gesture and a voice of the subject.

To this end, the controller 160 may include a gesture recognition unit 165, a voice recognition unit 167, a video call module 163, and an avatar service module 161 as shown in FIG. 3.

The gesture recognition unit 165 recognizes a specific operation state or a specific operation change of the image collected by the camera 170 as a gesture based on the gesture recognition DB stored in the storage unit 150, and corresponds to the recognized gesture. The avatar generating code is generated and delivered to the avatar service module 161. To this end, the gesture recognition unit 165 extracts a feature point from the image collected by the camera 170, and determines what action the subject takes from the arrangement state or change of the arrangement state of the feature point, and the gesture recognition DB is determined. You can compare it with the stored information. Here, as described above, the gesture recognition DB may not only provide a criterion of whether a specific operation state and change of a subject correspond to a specific gesture, but also store information on which avatar execution code corresponds to a specific gesture. . In order to recognize the gesture by extracting the feature points of the collected image by the gesture recognition unit 165, for example, adaboost may be used through PCA (principal component analysis). Here, adaboost is a learning algorithm for extracting the shape and feature points of an object. "A decision-theoretic generalization of on-line learning and an application to boosting", In Computational Learning Theory: Eurocolt 'by Yoav Freund and Robert E. Schapire. 95, pp. 23-37, Springer-Verlag, 1995.

The gesture recognition unit 165 may use object change confirmation, not feature point confirmation, for gesture evaluation of the image collected by the camera 170. In more detail, the video call module 163 may use various image processing methods. For example, the video call module 163 may use MPEG4 image processing. The MPEG4 may classify an image into hierarchical object units. For example, an image collected during a video call may be classified into a background object and a target object. Here, when the target object is a human, the target object may be further divided into a head object, an arm object, a torso object, and the like, and the arm object may be further divided into a finger object, a palm object, a back object, a cuff object, and the like. have. In addition, the finger node may be divided into respective nodes. When the video call module 163 processes the image in a hierarchical manner of the image, the gesture recognition unit 165 does not perform a gesture evaluation on the entire image, and a hand object that can occupy the most part of the gesture. Only the track can be tracked, and a change in the hand can be used to determine if a particular gesture is taken. Through this, the gesture recognition unit 165 may improve the processing speed for the recognition of the gesture. When the gesture is recognized, the gesture recognition unit 165 may generate an avatar execution code corresponding to the gesture and transmit it to the avatar service module 161.

The speech recognition unit 167 performs speech recognition based on a predetermined speech recognition model of the audio signal transmitted from the audio processor 130. For example, the voice recognition unit 167 may sample the voice signal received by the microphone MIC at 16 KHz, quantize the data to 16 bits, and generate the voice recognition model. The quantized speech data may be analyzed by pre-emphasis by a transfer function of a predetermined value and multiply by a Hamming window of 25 ms and moving by 10 ms. Through this, voice characteristic parameters can be determined. For example, a total of 39 characteristic parameters can be determined including the first and second difference components in the 12th LPC-MEL spectrum coefficient and the normalized logarithmic energy. This feature parameter can be applied to the speech recognition model of the present invention. The speech recognition model generates a phoneme decision tree for each state position of the model, and uses SSS (Successive State Splitting: SSS) using learning speech data. We can apply the method of learning the state string of the context-dependent acoustic model. Since this method is fast in performing state division, it is possible to select and divide a state divided by the SSS, perform state division for the entire partitionable state, and select a state in which the likelihood is maximized. A hidden Markov network may be applied to the acoustic model applied by the speech recognizer 167. In addition, the speech recognizer 167 may apply a method of frequency-analyzing speech waves based on various algorithms to extract and separate a sound region or a feature equivalent to the vowel. The voice recognition unit 167 may apply not only the above-described voice recognition algorithm but also various voice recognition algorithms. When the voice is recognized based on these various algorithms, the voice recognition unit 167 may determine whether the recognized voice corresponds to a predetermined specific voice. In this case, the specific voice is a voice corresponding to a specific avatar execution code, and may be designated by a designer who designs an avatar execution code or by a designer who designs an operation or state change of an avatar. The speech recognition unit 167 may generate an avatar execution code corresponding to a specific speech using a speech recognition DB. When the voice recognition unit 167 generates an avatar execution code for the collected specific voice, the voice recognition unit 167 may transmit the generated avatar code to the avatar service module 161.

The avatar service module 161 controls to output the counterpart avatar corresponding to the counterpart terminal to the display unit 140 when the video call channel is formed, and displays the avatar corresponding to the terminal according to a user setting. Control to output to one side. The avatar execution code received from at least one of the gesture recognition unit 165 and the voice recognition unit 167 is transmitted to the counterpart terminal or controlled to be applied to the own avatar. That is, the avatar service module 161 may transmit the avatar execution code to the video call module 163 to transmit to the counterpart terminal or to the counterpart terminal using a message service under the control of the controller 160. Meanwhile, the avatar service module 161 may receive a first avatar execution code for controlling a partner's avatar from the partner terminal through the video call module 163 or a message service. The avatar service module 161 may control to change the state or operation of the partner avatar outputted to the display unit 140 based on the received first avatar execution code. The avatar service module 161 may control to change at least one of its avatar state and operation using a second avatar execution code received from at least one of the gesture recognition unit 165 and the voice recognition unit 167. Can be. In this case, when the avatar service module 161 does not output its own avatar or does not perform its own avatar adjustment function according to a user's selection, the avatar service module 161 does not perform its own avatar adjustment function and controls the other party's avatar. It can also perform a function only.

Meanwhile, when the video call channel is formed but there is no avatar corresponding to the counterpart terminal, the avatar service module 161 may request the user to select the counterpart avatar. That is, the avatar service module 161 may output a pop-up window indicating that the opponent avatar does not exist to the display unit 140 and activate a menu for selecting the opponent avatar. Thereafter, the avatar service module 161 outputs the counterpart avatar selected by the user to the display unit 140, and updates the phone book data by linking to the phone number of the counterpart terminal on which the current video call channel is formed. In addition, the avatar service module 161 may transmit a message requesting avatar data to the counterpart terminal when there is no counterpart avatar. When the avatar service module 161 receives the avatar data from the counterpart terminal, the avatar service module 161 stores the avatar data in the storage unit and outputs the counterpart avatar to one side of the display unit 140. When the avatar service module 161 receives the avatar data from the counterpart terminal, the avatar service module 161 may control to update the phone book data by linking and storing the avatar data in the counterpart phone number. In addition, the avatar service module 161 outputs a counterpart avatar implemented on the display unit 140 based on the avatar data received from the counterpart terminal when the counterpart avatar specified by the user exists, but the avatar data is received from the counterpart terminal. Can be controlled. The avatar service module 161 may control to update the counterpart avatar previously stored in the counterpart terminal phone number with newly received avatar data.

Even when the avatar service module 161 does not exist, the avatar service module 161 selectively outputs a popup window indicating that the avatar does not exist, and the terminal user can select the avatar of the user. The menu can be controlled to output. When the terminal user changes his avatar, the avatar service module 161 may generate data about the avatar change and transmit it to the counterpart terminal.

The video call module 163 converts the video data collected by the camera 170 and the audio data collected by the microphone to meet the video call standard, and then transmits the converted video data to the opposite terminal through the communication unit 110, After receiving the transmitted signal to extract the image data and the audio data, and to control the output through the display unit 140 and the speaker. The video call module 163 includes one of an image codec for processing an image collected by the camera 170, for example, H.263, JPEG, Wavelet, mpeg2, mpeg4, and H.264. The video call module 163 may include a video codec including H.324M including the above-described video codec for generating the video data. The video call module 163 may generate video call data using various other video codecs.

For example, a process of generating and transmitting video call data using H, 263 and H.324M will be described below.

The H.263 converts the image signal output from the camera 170 into frame data by converting the image signal into frame data, and transforms the frame data to fit the display characteristics and size of the display unit 140 and outputs the image data. In this case, the H.263 may compress the image data. That is, H.263 compresses the image data displayed on the display unit 140 in a preset manner or restores the compressed image data to the original image data. The H.263 may be replaced with JPEG, Wavelet, mpeg2, mpeg4, H.264 and the like.

The H.324M muxes the video data generated by the H.263 together with other data to generate video call data, and transmits the video call data to the communication unit 110. To this end, the H.324M may include an audio codec, for example, AMR, for encoding the audio data collected by the audio processor 130. The H.324M may include H.245 for generating a control signal for synchronizing and controlling the video data and the audio data during a video call. The H.324M may include H.223 which receives the image data from the H.263, the audio data from the AMR, and the control signal from the H.245 and transmits the received control signal to the communication unit 110. That is, the H.223 generates video call data by muxing the video data, the audio data and the control signal, and transfers the video call data to the communication unit 110.

In particular, the video call module 163 of the present invention may integrate the avatar data transmitted by the avatar service module 161 together with the video data generated by the image codec, and transmit the same to the counterpart terminal through the communication unit 110. After separating and extracting the image data and the avatar data from the received signal, the two pieces of data may be separated and output to the display unit 140. In addition, the video call module 163 extracts the first avatar execution code transmitted from the counterpart terminal in the process of demuxing the video data, the audio data and the control signal, and extracts the extracted first avatar execution code from the avatar service module. Can be passed to 161. In addition, the video call module 163 generates a signal muxing the second avatar execution code transmitted by the avatar service module 161 in the process of muxing the video data, the audio data, and the control signal. Can be controlled to send to.

Meanwhile, although the video call module 163 receives the avatar data and the avatar execution code from the avatar service module 161, muxes the signals for the video call, and transmits them, the present invention has been described as an example. It is not limited. That is, the controller 160 of the present invention can operate the avatar service module 161 independently of the video call module 163 to transmit the avatar data and the avatar execution code to the counterpart terminal through the communication unit 110 and also receive the received data. The avatar signal may be processed by the avatar service module 161 and output to the display unit. To this end, the controller 160 may generate a communication channel for transmitting avatar data and an avatar execution code, for example, a data communication channel or a message service channel, independently of the video call channel generated by the video call module 163. .

The video call module 163 may use MPEG4 to encode and decode data necessary for a video call. In this case, the video call module 163 may select an image object related to an image, an audio object related to audio, or the like. It can control transmission and reception to support data transmission required for a video call. In this case, the video call module 163 may provide some objects to the gesture recognition unit 165, and the gesture recognition unit 165 converts objects recognized by the video call module 163 as described above. Specific gesture recognition may be performed.

As described above, the terminal according to an exemplary embodiment of the present invention enables an avatar to change at least one of an avatar's state and operation through gesture recognition of a subject collected by the camera 170 and audio recognition collected by a microphone. Execution code can be generated, and based on this, the state and motion of the avatar can be changed during a video call to more accurately or more expressively express an expression or mood that the user wants to express.

Meanwhile, the terminal may selectively activate the gesture recognition unit 165 and the voice recognition unit 167 according to a user's selection, and support the avatar-based video call service based thereon. That is, the terminal may deactivate the voice recognition function when the user sets the avatar operation based on the gesture recognition, and conversely, when the user sets the avatar operation based on the voice recognition. In addition, the control may be controlled to deactivate the gesture recognition function. As described above, the terminal may activate both gesture recognition and voice recognition. In this process, the avatar execution codes may be simultaneously or almost formed at the same time, and may be transmitted to the opposite terminal. The receiving terminal may output two avatar execution codes simultaneously to the avatar, but may not be simultaneously applied. The avatar may be controlled to output an avatar to which each avatar execution code is applied at a predetermined time period.

4 is a diagram illustrating an example of a screen interface for operating a video call of a terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the display unit 140 of the terminal includes a first screen area 141 that outputs data received from the other terminal as large as the screen 401 and a second screen that outputs images collected by the camera of the user. Area 143 may be included.

The first screen area 141 includes a counterpart image data output area 141a for outputting an image collected by a camera of the counterpart terminal and a counterpart avatar output area 141b corresponding to the counterpart terminal. Here, the opponent avatar output area 141b may be output to be distinguished from the opponent image data output area 141a. That is, the first screen area 141 may divide the screen and then allocate the partner avatar output area 141b and the partner image data output area 141a to the divided area. In addition, the first screen area 141 may be output by overlaying the partner avatar on the partner image data output area 141a without dividing the screen.

The second screen area 143 includes its own image data output area 143a and its own avatar output area 143b collected by the camera. Here, the avatar output area 143b of the user may be removed according to the size limit setting of the display unit 140 or the user selection. The second screen area 143 may divide the screen similarly to the first screen area 141, and then arrange its image data output area 143a and its avatar output area 143b separately. . The second screen area 143 may output the user avatar to be overlaid on the first screen area 141.

On the other hand, when the terminal receives an avatar execution code that can change at least one of the state and operation of the other avatar from the other terminal, the terminal may change and output the state and the operation of the other avatar as shown on the screen 402. For example, when the avatar execution code corresponding to the "shaking operation" is received from the opposite terminal through a specific channel such as a video call channel, a message service channel, or a voice call channel, the terminal responds to the "shaking operation". Control to manipulate the state and operation of the avatar based on the avatar execution code. Here, the change of the image or animation corresponding to the state and the motion of the avatar to correspond to the "shake of the hand" may be adjusted according to the intention of the designer, the preference, the setting of the user, and the like. On the other hand, since the terminal continuously receives the opponent's image data based on the video call module, the opponent's image data is output to the opponent's image data output area 141a. In this case, when the opposite terminal user shakes his hand, an actual image corresponding thereto may be output to the opposite image data output area.

In addition, although not shown, the terminal user may generate a specific voice, and when the terminal recognizes the specific voice and generates a specific avatar execution code based on the specific voice, the terminal and the counterpart terminal recognize the specific voice. Avatars that are changed according to the avatar execution code generated based on the output may be output.

In the above, a system for operating a video call according to an embodiment of the present invention, a terminal and a screen interface constituting the system have been described. Hereinafter, an avatar based video call operating method of the present invention will be described in detail with reference to the accompanying drawings.

5 is a flowchart illustrating an operation of a transmitting terminal in a video call method according to an exemplary embodiment of the present invention. On the other hand, for convenience of description, the transmitting terminal is referred to as a second terminal 200 and the receiving terminal is described as a first terminal 100 as follows.

Referring to FIG. 5, in the video call operating method according to an exemplary embodiment of the present invention, the second terminal 200 first initializes each configuration of the second terminal 200 when power is supplied, and displays a preset standby screen. It can be output in step S101.

Thereafter, when the second terminal 200 receives an input signal for requesting a video call connection from the input unit 120, the second terminal 200 attempts to connect the video call in step S103. In more detail, the user of the second terminal 200 inputs a phone number of the first terminal 100 capable of making a video call for a video call connection, and inputs an input signal for instructing the video call connection. Can be generated using Then, the second terminal 200 determines an environment for a video call through negotiation with the first terminal 100 corresponding to the input telephone number.

Meanwhile, before performing step S103, the first and second terminals 100 and 200 may select an avatar corresponding to the other party in the avatar setting mode and an avatar execution code for operating the set avatar according to a user's selection signal through the input unit 120. Can be set. In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the other party likes or is interested in.

In this case, the second terminal 200 may determine whether the video call is performed based on the avatar mode in step S105. To this end, the second terminal 200 may provide a menu for selecting a general video call mode and an avatar-based video call mode. In addition, when the user requests a video call connection in step S103, the second terminal 200 may output a pop-up window asking whether to proceed to the avatar mode. When proceeding to the avatar mode, the second terminal 200 may transmit information to confirm that the message is a message for requesting an avatar-based video call connection to the first terminal 100 during the video call connection process.

On the other hand, when the second terminal 200 is not the avatar mode, the second terminal 200 may support to perform a general video call function by step S107. That is, the second terminal 200 transmits the image collected by the camera 170 and the audio signal collected by the microphone MIC to the first terminal 100 without a separate avatar output, and has its own display unit ( The collected image may be selectively output to 140. The second terminal 200 may output an image and an audio signal received from the first terminal 100.

When the avatar mode is set in operation S105, the avatar corresponding to the first terminal 100 is output to one side of the display unit 140, and the avatar corresponding to the user's terminal 200 is displayed according to a user setting. ) Can be output to one side. If there is no avatar corresponding to the first terminal 100, the second terminal 200 requests the first terminal 100 to receive avatar data corresponding to the avatar or to the storage unit 150. The stored specific avatar may be set as an avatar corresponding to the first terminal 100.

Thereafter, the second terminal 200 branches to step S109 to perform at least one of gesture recognition and voice recognition of the subject collected by the camera 170. As described above, the gesture recognition may include determining whether the state and the change correspond to the preset specific gesture when the specific part or the whole of the subject has the specific operation state or the specific operation change. The voice recognition recognizes the audio signal collected by the microphone as at least one of phoneme, syllable, word, word, and sentence based on a voice recognition DB, and whether the recognized information corresponds to a preset specific voice. It includes the process of judging.

In operation S111, the second terminal 200 generates a specific avatar execution code based on gesture recognition and voice recognition. To this end, the second terminal 200 may store a table in which various avatar execution codes are mapped to at least one of a specific gesture and a specific voice, and with reference to this, the specific avatar execution code according to a specific gesture recognition and a specific voice recognition generation. Will generate

Next, the second terminal 200 transmits the generated avatar execution code to the first terminal 100 in step S113. In this case, the second terminal 200 transmits the avatar execution code to the first terminal 100 through the video call channel, or newly generates a new data communication channel, a message service channel, etc. without being associated with the video call channel, It may transmit to the first terminal 100.

In addition, the second terminal 200 may control to change the state or operation of the avatar by applying the avatar execution code in step S115. This process corresponds to a case in which the user sets his avatar to be outputted to his display unit 140. If there is no user setting, the process may be omitted. In this case, the degree of change in the state and operation of the avatar according to the avatar execution code may vary according to the intention of the designer who designed the avatar, and may take various types of states and actions according to various avatars. Even so, it is preferable that the specific emotions, expressions or actions to be represented by the specific avatar executable code are designed identically.

Next, the second terminal 200 checks whether the video call ends in step S117, and if there is no termination, branches to the step S109 to support the avatar-based video call service, and then repeatedly performs the process. have.

In the above description, an avatar who takes a specific state or action by the avatar execution code may continuously take the state and action until a new avatar execution code is generated or received, and is set to apply the real time of the video call. After a certain time or set to take a specific state or action, it may return to the default state and action.

6 is a flowchart illustrating an operation of a receiving terminal in an avatar based video call operating method according to an exemplary embodiment of the present invention. On the other hand, for convenience of description, the transmitting terminal is referred to as a second terminal 200 and the receiving terminal is described as a first terminal 100 as follows.

Referring to FIG. 6, when power is supplied, the first terminal 100 performs a booting process using the supplied power, and when the booting is completed, the preset standby screen is controlled to be output in step S201. Can be.

Thereafter, the first terminal 100 checks whether a signal for requesting a video call is received in step S203, and if a separate call request signal is not received in this process, branches to step S205 to request a user. Accordingly, a specific function of the first terminal 100 may be performed, for example, a file play function, a file search function, a broadcast reception function, an image collection function, a game function, and the like. Thereafter, the first terminal 100 may continue to monitor step S203 by branching to step S203 while performing step S205. Substantially, the first terminal 100 waits for step S203 while performing step S205. When the video call request signal reception is transmitted from the outside, the first terminal 100 may be scheduled in an interrupt manner before receiving a separate video call request signal. In step S205 can support the user function.

On the other hand, when receiving a video call request signal in step S203, the first terminal 100 may determine whether the received video call request signal is performed based on the avatar mode. To this end, the video call request signal preferably includes a message requesting to perform the operation based on the avatar mode, and the first terminal 100 determines whether to perform the video call service in the avatar mode by checking the corresponding message. A pop-up window that asks can be displayed on the display. Alternatively, the first terminal 100 may basically support a video call service based on the avatar mode. In this case, step S207 and step S209 for performing a normal video call mode may be omitted.

When it is set to the avatar mode in step S207, the first terminal 100 may enter the step S211 and output an avatar corresponding to the opponent's avatar, that is, an avatar corresponding to the second terminal 200, to one side of the display unit 140. When the avatar corresponding to the second terminal 200 is stored in the phone book and receives an avatar-based video call request from the second terminal 200, the avatar corresponding to the second terminal 200 may be activated and output to the display unit 140. In this case, when there is no avatar corresponding to the second terminal 200, the first terminal 100 transmits a message requesting avatar data corresponding to the avatar to the second terminal 200, and the second terminal ( 200, corresponding avatar data corresponding thereto may be received. Also, the first terminal 100 may set a specific avatar stored in its storage unit 150 as an avatar corresponding to the user of the second terminal 200 without performing this process.

In operation S213, the first terminal 100 determines whether to receive an avatar execution code instructing to change the state and operation of the avatar corresponding to the second terminal 200, and receives a separate avatar execution code. If there is not, branching to step S211 can be performed repeatedly the following process. Here, when an input signal for terminating a video call is generated, the first terminal 100 may end the mode process regardless of whether an avatar execution code is received.

On the other hand, when receiving the avatar execution code in step S213, the first terminal 100 branches to step S215 and changes at least one of the state or operation of the avatar of the second terminal 200 based on the received avatar execution code. Can be controlled. In operation S217, the first terminal 100 determines whether the video call is terminated. If the video call is maintained, the first terminal 100 may branch to the previous step S211 to repeatedly perform the following process.

In addition, like the second terminal 200, the first terminal 100 may output its avatar to one side of the display unit 140, and perform at least one of gesture recognition and voice recognition performed by the second terminal 200. And it can be adjusted to change its avatar state and operation based on the avatar execution code obtained through this.

In summary, an avatar-based video call method and system and a terminal supporting the same may be based on at least one of an avatar corresponding to the first terminal 100 performing a video call and an avatar corresponding to the second terminal 200. It may be stored, or may receive and store data on the opponent's avatar from the opponent, and output the data with the image reproduced based on the image data. In addition, the first terminal 100 and the second terminal 200 of the present invention recognize a gesture or a specific voice of the user, generate an avatar execution code that can change at least one of the state and operation of the avatar, and then transmit and receive the same. Accordingly, at least one of the avatar state and the action of the avatar being output by the counterpart terminal may be changed, and the at least one of the avatar state and the action of the avatar is output to the own terminal based on the generated avatar execution code. .

Meanwhile, in the video call system according to an exemplary embodiment of the present invention, the first terminal 100 or the second terminal 200 calls and outputs the partner avatar stored in the storage unit 150, and the terminal recognizes the other party's gesture or recognizes the voice. An example of receiving an avatar execution code generated based on the received information from an opponent terminal and controlling a state or an operation of the opponent avatar is output, but is not limited thereto. That is, as shown in FIG. 7, the first terminal 100 or the second terminal 200 may receive and output the partner avatar and the avatar execution code corresponding to the partner through the avatar providing server 400.

In the video call system according to another embodiment of the present invention, as shown in FIG. 7, the first terminal 100, the second terminal 200, and the avatar providing server 400 connected through the mobile communication network 300. It is configured to include.

The first terminal 100 and the second terminal 200 perform a video call with each other via the mobile communication network 300. The first terminal 100 or the second terminal 200 outputs the partner's avatar corresponding to the partner's partner received from the avatar providing server 400 through the mobile communication network 300, and through a specific gesture and voice recognition taken by the partner's partner. The avatar execution code corresponding to the identified emotion state of the opponent is received, and at least one of the state and the operation of the opponent avatar is adjusted and output.

The mobile communication network 300 performs a series of data transmission / reception operations for data transmission and information exchange between the first terminal 100, the avatar providing server 400, and the destination terminal 200. In particular, upon receiving a video call connection request from the first terminal 100, the mobile communication network 300 forms a video call channel for a video call between the first terminal 100 and the second terminal 200. When a video call is initiated through the formed video call channel, the mobile communication network 300 receives the avatar of the other party and the avatar execution code from the avatar providing server 400 and transmits the received avatar to the first or second terminal 100 or 200. In this case, the mobile communication network 300 uses the currently formed video call channel for the other party's avatar and the avatar execution code, or forms a data communication channel or a message service channel separately from the formed video call channel to the first or second terminals 100 and 200. send.

When the avatar providing server 400 receives the avatar request signal from the first or second terminals 100 and 200 after the video call channel is formed through the mobile communication network 300, the avatar providing server 400 transmits the avatar request signal to the counterpart. Send the corresponding partner avatar. The avatar providing server 400 receives video call data from the counterpart terminal of the terminal receiving the counterpart avatar, and generates the avatar execution code based on a specific gesture and voice recognition taken by the counterpart from the received video call data. The avatar providing server 400 transmits the generated avatar execution code to the terminal receiving the counterpart avatar. At this time, the terminal receiving the opponent avatar adjusts and outputs at least one of the status and operation of the opponent avatar according to the received avatar execution code.

In particular, the avatar providing server 400 according to another exemplary embodiment of the present invention includes a transceiver 410, a database 420, and a server controller 430.

The transceiver 410 communicates with the first terminal 100 and the second terminal 200 via the mobile communication network 300. The transceiver 410 receives an avatar request from the mobile communication network 300 under the control of the server controller 430, and transmits the requested avatar to the mobile communication network 300. The transceiver 410 transmits the avatar execution code to the mobile communication network 300 under the control of the server controller 430.

The database unit 420 stores the partner avatar connected to the phone number of the partner terminal, the gesture recognition database for gesture recognition, the voice recognition database for voice recognition, the recognized gesture and the avatar execution code mapped to the specific voice. . That is, the database unit 420 stores the partner avatar set by the user of the first and second terminals 100 and 200 and the avatar execution code for operating the partner avatar. The user may set the opponent avatar to correspond to the opponent party. The partner avatar includes an avatar set by mapping to a phone number of the counterpart according to a user's selection signal and a general avatar set by default. The general avatar set as a default may be used when performing a video call with a person other than the other party set by the user.

The server controller 430 performs an overall control operation of the avatar providing server 400. In particular, the server controller 430 controls the output of the avatar of the user terminal based on a specific gesture and voice recognition that the other party takes during the video call.

In the avatar setting mode, the server controller 430 may set an opponent avatar corresponding to the other party according to the user's selection signals through the first and second terminals 100 and 200, and set an avatar execution code for operating the avatar. have. In this case, since the setting of the partner avatar and the avatar execution code by the server controller 430 may be performed in the same manner as the setting method by the controller 160 of FIG. 2, a detailed description thereof will be omitted.

After the video call channel is formed between the first and second terminals 100 and 200 via the mobile communication network 300, the server controller 430 may display an avatar from the first or second terminals 100 and 200 through the mobile communication network 300. When receiving the request signal, the other party's avatar is transmitted to the terminal that sent the avatar request signal. The server controller 430 receives video call data from a counterpart terminal of the terminal receiving the counterpart avatar. The server controller 430 generates an avatar execution code based on a specific gesture and voice recognition that the other party takes from the received video call data. The server controller 430 transmits the generated avatar execution code to the terminal receiving the counterpart avatar. At this time, the terminal receiving the opponent avatar adjusts and outputs at least one of the status and operation of the opponent avatar according to the received avatar execution code.

At this time, the server controller 430 calls the partner avatar from the database unit 420 or receives it from the counterpart terminal and transmits it to the user terminal. That is, when the video call channel is formed, the server controller 430 checks whether the partner avatar corresponding to the partner is set in the database unit 420. If the check result is set, the server controller 430 calls the set counterpart avatar. If the check result is not set, the server control unit 430 may call the general avatar stored in the database unit 420 or request and receive the other party's avatar from the other party's terminal. The server controller 430 transmits the called or received counterpart avatar to the user terminal. In this case, when the server controller 430 requests and receives the partner's avatar from the counterpart terminal, the server controller 430 may use the currently formed video call channel, or may form a data communication channel or a message service channel separately from the video call channel. The server controller 430 may store the partner avatar received from the counterpart terminal in the database unit 420 by connecting the counterpart avatar to the phone number of the counterpart.

The server controller 430 recognizes the other party's gesture from the received video call data and recognizes the other party's voice from the received video call data. That is, the server controller 430 extracts a counterpart image from the received image, tracks a region of interest in the extracted counterpart image, and recognizes an object gesture. In this case, the ROI includes at least one of a face, both hands, and both arms. The server controller 430 recognizes the voice of the other party from the received voice whether the word or phrase expressing the emotion of the other party is included and the height of the voice tone.

The server controller 430 may generate an avatar execution code as follows through gesture recognition. That is, the specific state and motion change are extracted from the video of the received video call data. The server controller 430 compares whether the extracted specific state and operation change correspond to a predetermined specific gesture. As a result of the comparison, in response to the specific gesture, the server controller 430 extracts and generates the avatar execution code mapped to the specific gesture.

The server controller 430 may generate an avatar execution code as follows through speech recognition. That is, the server controller 430 recognizes the voice of the received video call data as described above. The server controller 430 determines whether there is a predetermined voice in the recognized voice. If there is a specific voice as a result of the determination, the server controller 430 extracts and generates an avatar execution code that is previously mapped to the specific voice.

Meanwhile, the server controller 430 may extract and generate an avatar execution code mapped to a specific gesture and a specific voice.

The server controller 430 transmits the generated avatar execution code to the user terminal through a currently set video call channel, a separate data communication channel, or a message service channel.

As such, when the video call channel is formed, the avatar providing server 430 of the video call system according to another embodiment of the present invention transmits a counterpart avatar corresponding to the counterpart to the first and second terminals 100 and 200, and gestures of the counterpart. And transmitting the avatar execution code generated based on the voice recognition to the terminal receiving the counterpart avatar. Since the terminal receiving the opponent avatar adjusts and outputs at least one of the state and operation of the opponent avatar according to the avatar execution code, the terminal may effectively transmit the emotional state of the opponent to the user through the opponent avatar.

A video call method for transmitting an opponent's emotion in a video call system according to another embodiment of the present invention will be described with reference to FIGS. 7 and 8 as follows. 8 is a flowchart illustrating a video call method according to another embodiment of the present invention.

First, when the first terminal 100 sends a video call connection request with the second terminal 200 in step S201, the mobile communication network 300 receives the request and receives a video call from the second terminal 200 in step S203. do.

Next, when the second terminal 200 transmits the acceptance signal for the video call incoming request to the mobile communication network 300 in step S205, the mobile communication network 300 is connected to the first terminal 100 and the second terminal (S207). 200) to form a video call channel. The first terminal 100 and the second terminal 200 exchange video call data obtained through the formed video call channel with each other and perform a video call.

Meanwhile, before performing step S201, the avatar providing server 400 may set an avatar corresponding to the other party and an avatar execution code for operating the set avatar in the avatar setting mode according to a user's selection signal through the first terminal 100. Can be. In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the counterpart likes or is interested in.

Next, in step S209, the first terminal 100 determines whether the avatar mode is selected from the user. In this case, the user may select the avatar mode through the input unit 120.

If the avatar mode is not selected as a result of the determination in step S209, the first terminal 100 maintains the normal video call mode.

When the avatar mode is selected as a result of step S209, in operation S211, the first terminal 100 transmits an avatar request signal corresponding to a user (relative side) of the second terminal 200 to the mobile communication network 300. In operation S213, the mobile communication network 300 transmits the received avatar request signal to the avatar providing server 400.

Next, in step S215, the avatar providing server 400 extracts the partner avatar set by the user of the first or second terminal 100 or 200 from the database unit 420 and transmits the avatar to the mobile communication network 300. In this case, the partner avatar may be an avatar set to correspond to the user of the second terminal 200. The avatar providing server 400 may transmit the avatar of the opposite party through a currently formed video call channel, a separate data communication channel, or a message service channel.

Next, in step S217, the mobile communication network 300 transmits the received video call data of the other party and the other party's avatar to the first terminal 100. Subsequently, in step S218, the first terminal 100 outputs video call data with the received counterpart avatar. That is, the first terminal 100 outputs the voice through the speaker SPK through the audio processor 130 in the received video call data. The first terminal 100 displays the counterpart image and the counterpart avatar together on the display unit 140 in the received video call data. In this case, when the first terminal 100 displays the other party's avatar, the other party's image and the other party's avatar may be displayed separately or may be displayed by overlaying the other party's avatar in the region where the other party's image is output.

Next, in step S219, the mobile communication network 300 transmits the video call data received from the second terminal 200 to the avatar providing server 400.

Next, in step S221, the avatar providing server 400 analyzes the received video call data and recognizes a specific gesture and voice taken by the other party. Subsequently, in step S223, the avatar providing server 400 generates an avatar execution code based on the recognized specific gesture and voice.

That is, the avatar providing server 400 extracts a counterpart image from the received image, tracks a region of interest in the extracted counterpart image, and recognizes an object gesture. In this case, the ROI includes at least one of a face, both hands, and both arms. The avatar providing server 400 recognizes the voice of the other party from the received voice whether the word or phrase expressing the other party's feelings is included and the height of the voice tone.

The avatar providing server 400 may generate an avatar execution code as follows through gesture recognition. That is, the avatar providing server 400 extracts a specific state and motion change from the image of the received video call data. The avatar providing server 400 compares whether the extracted specific state and motion change correspond to a preset specific gesture. When the comparison result corresponds to a specific gesture, the avatar providing server 400 extracts and generates an avatar execution code previously mapped to the specific gesture.

The avatar providing server 400 may generate an avatar execution code as follows through speech recognition. That is, the avatar providing server 400 recognizes the voice of the received video call data as described above. The avatar providing server 400 determines whether a predetermined voice is present in the recognized voice. As a result of the determination, when there is a specific voice, the avatar providing server 400 extracts and generates an avatar execution code previously mapped to the specific voice.

Meanwhile, the avatar providing server 400 may extract and generate an avatar execution code mapped to a specific gesture and a specific voice.

Next, in step S225, the avatar providing server 400 transmits the generated avatar execution code to the mobile communication network 300. In operation S227, the mobile communication network 300 transmits the received avatar execution code to the first terminal 100. In this case, the avatar providing server 400 may transmit the avatar execution code through a currently formed video call channel, a separate data communication channel, or a message service channel.

In operation S229, the first terminal 100 transforms and outputs the opponent avatar according to the received avatar execution code. That is, the first terminal 100 adjusts and outputs at least one of a state and an operation of the partner avatar according to the received avatar execution code. For example, when the first terminal 100 displays the opponent's avatar, the opponent's image and the opponent's avatar may be displayed separately or may be displayed by overlaying the opponent's avatar in a region where the opponent's image is output.

Meanwhile, in the video call method according to another embodiment of the present invention, the avatar providing server 400 discloses an example in which the avatar providing server 400 provides a counterpart avatar representing the emotional state of the user of the second terminal 200. On the contrary, a counterpart avatar may be provided to the second terminal 200 to express the emotional state of the user of the first terminal 100. Alternatively, the first and second terminals 100 and 200 may provide counterpart avatars representing the emotional state of the counterpart, respectively.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. As such, those of ordinary skill in the art will appreciate that various changes and modifications may be made according to equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

The present invention relates to an avatar-based video call method and system, and a terminal supporting the same. The present invention relates to controlling an expression or operation of an avatar outputted on a counterpart terminal or a display unit of a terminal based on a gesture and a voice of a user during a video call. By outputting, the terminal users can quickly and accurately recognize the emotion or expression that the other party wants to express through the other party's avatar while performing a video call.

1 is a view schematically showing the configuration of a video call system according to an embodiment of the present invention.

2 is a diagram schematically illustrating a configuration of a terminal according to an exemplary embodiment of the present invention.

FIG. 3 is a diagram illustrating in more detail the configuration of the controller in the terminal configuration of FIG. 2.

4 is a diagram illustrating an example of a screen interface configuring the display unit of FIG. 2.

5 is a flowchart illustrating an operation of a transmitting terminal in a video call method according to an exemplary embodiment of the present invention.

6 is a flowchart illustrating an operation of a receiving terminal in a video call method according to an exemplary embodiment of the present invention.

7 is a view schematically showing the configuration of a video call system according to another embodiment of the present invention.

8 is a flowchart illustrating a video call method according to another embodiment of the present invention.

Description of the Related Art [0002]

100: first terminal 110: communication unit

120: input unit 130: audio processing unit

140: Display unit 150:

160: control unit 161: avatar service module

163: video call module 165: gesture recognition unit

167: speech recognition unit 170: camera

200: second terminal 300: mobile communication network

400: avatar providing server 410: transceiver

420: database unit 430: server control unit

Claims (22)

When a video call channel is formed, the first terminal outputs an avatar corresponding to the second terminal, and adjusts and outputs at least one of the state and operation of the avatar based on the avatar execution code received from the second terminal. ; Recognizing a voice including any one of a specific word or a specific phrase expressing the height and emotion of the voice from a specific gesture and an audio signal collected by a microphone from an operation state and a change of a subject collected by a camera, and the recognized information The second terminal generating the avatar execution code for adjusting at least one of a state and an operation of the avatar, and transmitting the avatar execution code to the first terminal; Avatar-based video call system comprising a. A communication network controlling a video call between a first terminal and a second terminal; After receiving the avatar request signal from the first or second terminal after the video call channel is established through the communication network, the avatar corresponding to the other party is transmitted to the terminal that sent the avatar request signal, and the terminal receives the avatar. Receives the video call data from the other party's terminal and executes the avatar by recognizing a voice including any one of a specific word or a specific phrase expressing a specific gesture and a voice's height and emotion that the other party takes from the received video call data. An avatar providing server generating a code and transmitting the avatar execution code to a terminal receiving the avatar; Avatar-based video call system comprising a. A transceiver for communicating with a first terminal or a second terminal via a communication network; After receiving the avatar request signal from the first or second terminal after the video call channel is established through the communication network, the avatar corresponding to the other party is transmitted to the terminal that sent the avatar request signal, and the terminal receives the avatar. Receives the video call data from the other party's terminal and executes the avatar by recognizing a voice including any one of a specific word or a specific phrase expressing a specific gesture and a voice's height and emotion that the other party takes from the received video call data. A server controller which generates a code and transmits the avatar execution code to a terminal that has received the avatar; Avatar providing server of the avatar-based video call system comprising a. The method of claim 3, wherein A database unit for storing the avatar connected to the phone number of the counterpart terminal, the gesture recognition database for the gesture recognition, the voice recognition database for the voice recognition, the avatar execution code mapped to the recognized gesture and a specific voice; ; The avatar providing server of the avatar-based video call system, further comprising a. A camera for collecting an image to be transmitted to a counterpart terminal for a video call; A display unit for outputting an image of the opposite party received from the opposite terminal; When a video call channel is established with the counterpart terminal, the avatar corresponding to the counterpart terminal is output to the display unit, and any one of a specific word or a specific phrase expressing the height and emotion of the counterpart's gesture and voice is output from the counterpart terminal. A control unit configured to receive an avatar execution code generated by recognizing a voice including a voice, and to adjust at least one of a state and an operation of the avatar according to the avatar execution code to output to the display unit; A terminal of the avatar-based video call system comprising a. 6. The method of claim 5, A storage unit for storing an avatar connected to a phone number of the counterpart terminal, a gesture recognition database for gesture recognition, a voice recognition database for voice recognition, and an avatar execution code mapped to the recognized gesture and a specific voice; The terminal of the avatar-based video call system, further comprising. 6. The method of claim 5, The control unit A gesture recognition unit generating the avatar execution code through the gesture recognition; A speech recognition unit generating the avatar execution code through the speech recognition; An avatar service module for outputting the avatar and adjusting at least one of a state and an operation; A video call module for supporting the video call; Terminal of the avatar-based video call system comprising a. After receiving a avatar request signal from the first or second terminal after a video call channel is established between the first terminal and the second terminal, the avatar providing server transmits an avatar corresponding to the other party to the terminal that sent the avatar request signal. Transmitting an avatar; A receiving step of receiving, by the avatar providing server, video call data from a counterpart terminal of the terminal receiving the avatar; A step of generating, by the avatar providing server, an avatar execution code by recognizing a voice including one of a specific word and a specific phrase expressing a specific gesture and a voice level and an emotion of the other party in the received video call data; ; Transmitting, by the avatar providing server, the avatar execution code to the terminal receiving the avatar; Avatar-based video call method comprising a. 9. The method of claim 8, In the avatar transmission step or avatar execution code transmission step, The avatar providing server transmits the avatar or the avatar execution code through at least one of the video call channel, the data communication channel, and the message service channel. The method of claim 8, wherein the generating step, Extracting, by the avatar providing server, a specific state and motion change from the received video call data image; Comparing, by the avatar providing server, whether the extracted specific state and motion change correspond to a preset specific gesture; When the comparison result corresponds to the specific gesture, detecting, by the avatar providing server, an avatar execution code previously mapped to the specific gesture; Avatar-based video call method comprising a. The method of claim 8, wherein the generating step, Recognizing a speech by the avatar providing server extracting any one of a specific word or a specific phrase expressing the height and the emotion of the voice of the received video call data; Determining, by the avatar providing server, whether there is a predetermined voice in the recognized voice; Detecting, by the avatar providing server, an avatar execution code previously mapped to the specific voice if the specific voice is present as a result of the determination; Avatar-based video call method comprising a. The method of claim 8, wherein the step is performed before the avatar receiving step, Selecting and setting, by the avatar providing server, an avatar or an avatar execution code of a field that the user or the other party likes or is interested in in the avatar setting mode; Avatar-based video call method further comprising. A channel forming step of forming a video call channel between the first terminal and the second terminal; Outputting, by the first terminal, an avatar corresponding to the second terminal; Recognizing a voice including any one of a specific word or a specific phrase expressing the height and emotion of the voice from a specific gesture and an audio signal collected by a microphone from an operation state and a change of an image collected by the camera of the second terminal A receiving step of receiving, by the first terminal, the avatar execution code from the second terminal generating an avatar execution code capable of adjusting at least one of the state and operation of the avatar; An adjustment output step of the first terminal adjusting and outputting at least one of a state and an operation of the avatar according to the avatar execution code; Avatar-based video call method comprising a. The method of claim 13, Generating the avatar execution code in the receiving step Extracting, by the second terminal, a specific state and operation change of the image collected by the camera; Comparing, by the second terminal, whether the extracted specific state and operation change correspond to a preset specific gesture; When the comparison result corresponds to the specific gesture, detecting, by the second terminal, an avatar execution code previously mapped to the specific gesture; Avatar-based video call method comprising a. The method of claim 13, Generating the avatar execution code in the receiving step Extracting and recognizing, by the second terminal, a specific word or specific phrase that expresses the height and emotion of the voice collected by the microphone; Determining, by the second terminal, whether a predetermined voice is present in the recognized voice; Detecting, by the second terminal, an avatar execution code that is pre-mapped to the specific voice if the specific voice is present as a result of the determination; Avatar-based video call method comprising a. The method of claim 13, The output stage or the adjustment output stage Dividing the screen by the first terminal and dividing and outputting the image received from the avatar and the second terminal in the divided area; or Overlaying and outputting the avatar to an area in which the first terminal outputs an image received from the second terminal; Avatar-based video call method comprising a. The method of claim 13, The output step Detecting and outputting, by the first terminal, the stored avatar connected to the telephone number of the second terminal; Avatar-based video call method comprising a. The method of claim 13, The output step Receiving, by the first terminal, avatar data corresponding to the avatar from the second terminal, and outputting an avatar based on the received avatar data; or Designating, by the first terminal, a specific avatar previously stored in a storage unit as an avatar of the second terminal, and outputting the designated avatar; Avatar-based video call method comprising the step of any one of. The method of claim 18, The output step Storing, by the first terminal, the received avatar or the designated avatar in connection with the telephone number of the second terminal; Avatar-based video call method further comprising. The method of claim 13, In the receiving step And the first terminal receives the avatar or the avatar execution code through at least one of the video call channel, the data communication channel, and the message service channel. The method of claim 13, wherein the step is performed before the channel forming step. Selecting and setting, by the first terminal, an avatar or an avatar execution code of a field that the user or the other party likes or is interested in in the avatar setting mode; Avatar-based video call method further comprising. The method according to any one of claims 8 to 21, The avatar is a graphic content that represents the emotion state of the other party, and includes one of an animated character, a video, a still image, a user created content (UCC), an emoticon, a flash, and a haptic content in which a video and vibration are combined. Avatar-based video call method, characterized in that.

Images