KR20110020131A - System and method for delivering feeling during video call - Google Patents

Abstract

PURPOSE: A system and a method for expressing emotion by recognizing other party's expression are provided to exactly deliver other party's emotion through a video communication. CONSTITUTION: An avatar providing server(30) receives a video data from other party's terminal which receives an avatar. The avatar providing serve monitors other party's expression in the received video data. If a specific expression is detected through the monitoring, the avatar providing server transmits an avatar execution code capable of operating the avatar to the mobile terminal.

Description

System and method for delivering feeling during video call {a system and method for delivering feeling during video call}

The present invention relates to a video call technology, and more particularly, to detect a change in the facial expression of the other party displayed on the user terminal during a video call, and to operate the avatar according to the detected facial expression change to transmit the emotional state of the other party to the user. The present invention relates to an emotional delivery system and method through facial expression recognition of a counterpart during a call.

As high-speed information communication is realized in voice call-oriented call patterns, video call technology through communication terminals is attracting attention. In addition, as the spread of communication terminals capable of video calls (hereinafter referred to as 'video call terminals') is increasing, the frequency of video calls is increasing day by day.

Since the video call terminal displays the video of the other party on the screen and outputs the voice of the other party, the video call terminal can easily exchange each other's emotional states with each other as compared with simply making a voice call.

However, since the user is still receiving the emotional state of the other party only through the facial expression or voice during the video call, there is a limit in accurately transmitting the various emotional states of the other party to the user through the video call terminal. In addition, since the current video call is merely to perform the call while watching the video of the other party, there is a limit to expanding the use of the video call by inducing interest of the video call while satisfying various needs of the video caller.

Accordingly, it is an object of the present invention to provide an emotion delivery system and method through facial expression recognition of a counterpart during a video call capable of accurately transmitting various emotion states of the counterpart to a user through a video call.

Another object of the present invention is to provide an emotion delivery system and method through facial expression recognition of a counterpart during a video call that can induce interest of the video caller and thereby expand the use of the video call.

In order to achieve the above object, the present invention displays the avatar corresponding to the counterpart on the screen along with the video of the counterpart, detects the facial expression of the counterpart, operates the avatar according to the detected facial expression to control the emotional state of the counterpart. It provides a system and method for emotion delivery through the facial recognition of the other party during the video call to the user.

The present invention provides a system for transmitting emotions through facial expression recognition of a counterpart during a video call including a communication network and an avatar providing server. The communication network controls a video call between the calling terminal and the called terminal. When the video call is initiated between the calling terminal and the called terminal through the communication network, the avatar providing server transmits an avatar corresponding to the other party to the calling terminal or the called terminal through the communication network, and receives the avatar. Receiving the image data from the other terminal, and monitors whether the other party takes a specific facial expression from the received image data, and if the specific facial expression is detected during the monitoring can operate the avatar corresponding to the detected specific facial expression The avatar execution code is transmitted to the terminal that receives the avatar.

The present invention also provides an avatar providing server of the emotion delivery system through the facial recognition of the other party during a video call comprising a transceiver and a controller. The transceiver performs communication with a calling terminal or a called terminal via a communication network. When the video call is initiated between the calling terminal and the called terminal through the communication network, the controller transmits the avatar corresponding to the other party to the calling terminal or the called terminal through the communication network, and then receives the avatar. An avatar capable of receiving image data from a terminal, monitoring whether the counterpart takes a specific facial expression from the received image data, and if the specific facial expression is detected during the monitoring, operating the avatar corresponding to the detected specific facial expression The execution code is transmitted to the terminal that receives the avatar.

The present invention also provides a video call terminal including a communication unit, a camera unit, a display unit, and a terminal control unit. The communication unit performs a video call with the counterpart terminal. The camera unit photographs a user during a video call. The display unit displays the image data of the other party received through the communication unit. When the terminal controller initiates a video call with the counterpart terminal through the communication unit, the terminal controller displays the avatar along with the avatar corresponding to the counterpart and the image data received from the counterpart terminal, and displays the counterpart in the received image data. It monitors whether to take a specific facial expression, and if the specific facial expression is detected during the monitoring, the avatar is operated to correspond to the detected specific facial expression.

The present invention also establishes a video call path between a calling terminal and a called terminal through a communication network, and when a video call is initiated, the avatar providing server transmits an avatar corresponding to the other party to the calling terminal or the called terminal through the communication network. An avatar transmission step, the avatar providing server receiving image data from the counterpart terminal of the terminal that has received the avatar through the communication network, and whether the avatar providing server has a specific facial expression in the received image data; In the monitoring step of monitoring, when the avatar providing server detects a specific facial expression during the monitoring, an avatar execution code for transmitting an avatar execution code for operating the avatar corresponding to the detected specific facial expression to the terminal receiving the avatar Containing steps Of the call and provide the emotional delivery method through other's facial expression recognition.

The present invention also provides a display step of displaying the video data received from the counterpart terminal and the avatar corresponding to the counterpart when the user terminal sets up a video call path with the counterpart terminal. A monitoring step of monitoring whether the counterpart takes a specific facial expression in the received image data, and if the specific facial expression is detected during the monitoring, operating the avatar in response to the specific facial expression detected by the terminal; Provides a method of transmitting emotions through facial expression recognition of a video call.

According to the present invention, when the video call path with the counterpart terminal is set, the user terminal calls the avatar on the screen and displays the received video along with the received counterpart's image, and detects a specific facial expression related to the emotional state of the counterpart from the counterpart's video displayed on the screen. Since the avatar is operated according to the detected specific facial expression, the emotional state of the other party can be effectively transmitted to the user through the avatar.

Since the emotion delivery system according to the present invention can select and set an avatar or an avatar execution code of a field that the user or the other party likes or is interested in in the avatar setting mode, satisfying various needs of the video caller and causing interest of the video caller. Extend the use of video calls. In addition, the user can directly set the appearance or behavior of the avatar corresponding to the change of facial expression of the other party through the video call terminal by selecting the avatar execution code, so that the user can use the video call with the other party more interestingly while pursuing the user's personality. Can provide an environment that

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the present invention.

Here, 'Avatar' is a graphic content that represents the emotional state of the terminal user, such as animated characters, video, still images, UCC (User Created Contents), emoticons, flash, haptic content combined with video and vibration, etc. It includes.

Referring to FIGS. 1 and 2, the emotion delivery system 100 through the facial recognition of the other party during a video call according to the first embodiment of the present invention will be described. 1 is a block diagram illustrating an emotion transmission system 100 through facial expression recognition of a counterpart during a video call according to the first embodiment of the present invention. 2 is a block diagram showing the configuration of the originating terminal 20 of FIG.

The emotional delivery system 100 (hereinafter, referred to as an 'emotional delivery system') of a video call according to the first embodiment outputs an avatar corresponding to a specific facial expression taken by the other party during a video call and transmits the emotional state of the other party to the user. That is, since the facial expressions taken by the user during the video call are related to the emotional state of the user, the emotional delivery system 100 according to the first embodiment may provide an emotional state of the counterpart corresponding to the specific facial expression when the user takes a specific facial expression during the video call. By outputting an avatar capable of expressing on the screen of the user terminal can transmit the emotional state of the other party to the user.

Emotion delivery system 100 according to the first embodiment is configured to include a video call terminal (20, 40) for performing a video call connected via the communication network 10, the video call terminal (20, 40) Call originating terminal 20 and called terminal 40 are included.

The communication network 10 performs a series of data transmission and reception operations for data transmission and information exchange between the calling terminal 20 and the called terminal 40. The communication network 10 is an IP network providing a large-capacity data transmission / reception service and a seamless data service through IP, and may be an ALL-IP network that is an IP network structure integrating different networks based on IP. In addition, the communication network 10 may be one of a wired communication network, a mobile communication network, a wireless broadband (WiBro) network, a high-speed downlink packet access (HSDPA) network, and a satellite communication network.

The communication network 10 includes a call controller 14 and a location register 16.

The call controller 14 is a part in charge of functions related to call and session processing, and may be one of a call state control function (CSCF) or a mobile switching center (MSC). In particular, the call controller 14 implemented with CSCF performs a function as a gateway for incoming calls, a call control function, a SPD (Serving Profile Database) function, an address processing function, and the like.

The location register 16 is a master database of subscribers, and is responsible for tracking access means by maintaining a master list of services related to subscribers or performing management of subscriber location information. The location register 16 stores additional service subscription information subscribed to by the subscriber. The location register 16 may be one of a home location register (HLR) or a home subscriber server (HSS) that manages subscriber information in an All-IP network.

In addition, the calling terminal 20 and the called terminal 40 perform a video call with each other through the communication network 10. When the video call terminal 20 or 40 initiates a video call with the counterpart terminal, the video call terminal 20 or 40 displays an avatar which operates according to the video data received from the counterpart terminal and the facial expression taken by the counterpart. In this case, the video call terminals 20 and 40 are wired / wireless communication terminals capable of making video calls through the communication network 10, and include a mobile communication terminal, a wired communication terminal, and an internet telephone.

As such, the video call terminals 20 and 40 display an avatar operating on the screen along with the other party's image received from the other terminal, so that the user can know the emotional state of the other party through the avatar. have.

Since the calling terminal 20 and the called terminal 40 of the emotion transmitting system 100 according to the first embodiment have the same configuration, the following description will be given with reference to the configuration of the calling terminal 20 shown in FIG. same. The calling terminal 20 includes a communication unit 21, an input unit 22, a storage unit 23, a camera unit 24, an output unit 25 and 26, and a terminal control unit 27.

The communication unit 21 transmits and receives data for a video call with the destination terminal 40 through the communication network 10. The communication unit 21 receives the video data of the other party from the called terminal 40 under the control of the terminal controller 27, and transmits the video data of the user photographed by the camera unit 24 to the called terminal 40.

The input unit 22 provides a plurality of keys for the operation of the calling terminal 20, and generates a selection signal according to a user's key selection and transmits the selection signal to the terminal controller 27. The user may input a video call request signal through the input unit 22. The input unit 22 may be a keypad, a pointing device such as a touch pad, or an input device such as a touch screen.

The storage unit 23 stores a program necessary for controlling the operation of the calling terminal 20 and data generated during the execution of the program, and includes one or more volatile memory devices and nonvolatile memory devices. The storage unit 23 stores an execution program for operating the avatar according to a specific facial expression taken by the counterpart during the video call. The storage unit 23 stores the avatar set to correspond to the counterpart and the avatar execution code for operating the avatar. In this case, the storage unit 23 may store the avatar by matching the terminal information of the counterpart. The avatar includes an avatar set according to the terminal information of the counterpart according to the user's selection signal and a general avatar set by default. The general avatar set as a default may be used when conducting a video call with a person other than the user.

In this case, the specific facial expression is an expression related to the emotional state of the user among the facial expressions taken by the counterpart. For example, the specific facial expression may include a smiling facial expression, a crying facial expression, a thinking facial expression, a disgusting facial expression, a favorite facial expression, a surprised facial expression, and the like. The avatar executable code is mapped to a specific facial expression taken by the opponent. That is, the avatar execution code operates the avatar to express the emotion corresponding to the specific facial expression described above as an action. The storage unit 23 may match the specific expression and the avatar execution code and store the mapping table as a mapping table. For example, an avatar executable code that matches a favorite facial expression may cause the avatar to jump and smile.

In addition, the avatar executable code may be executable code that modifies the facial expression and behavior of the avatar to correspond to the facial expression taken by the counterpart. For example, when the smiling face of the other person is taken, the avatar also takes the same smiling face as the other face.

The camera unit 24 acquires image data of a user (sender) for transmission through the communication unit 21. That is, the camera unit 24 acquires image data by capturing the image of the user according to the acceptance of the video call through the input unit 22. In this case, the camera unit 24 may include an image sensor, a signal processor, and an image processor. The image sensor converts the optical signal of the captured image into an analog signal. The signal processor converts an analog signal into a digital signal. The image processing unit processes the image signal input through the signal processing unit to obtain image data, and outputs the obtained image data through the communication unit 21 or the display unit 25 or stores it in the storage unit 23. In addition, the image processing unit processes the image data of the other party (recipient) input through the communication unit 21 to output through the display unit 25 or to store in the storage unit (23).

The output units 25 and 26 include a display unit 25 and a sound source processing unit 26. The output units 25 and 26 may further include a vibration generator for generating vibration.

The display unit 25 displays information stored in the storage unit 23 including various function menus executed in the calling terminal 20. The display unit 25 displays an image or an avatar of the other party under the control of the terminal controller 27. As the display unit 25, a liquid crystal display (LCD) or a touch screen may be used. The touch screen simultaneously serves as a display device and an input device.

The sound source processor 26 digitizes the voice input to the microphone MIC under the control of the terminal controller 27 to obtain voice information, and demodulates the voice information received through the communication unit 21 to generate a speaker SPK. Output through

In addition, the terminal controller 27 is a microprocessor that performs overall control operations of the calling terminal 20. In particular, the terminal controller 27 controls the avatar output by recognizing the facial expression of the other party during the video call.

The terminal controller 27 may set an avatar corresponding to the other party and an avatar execution code for operating the set avatar in the avatar setting mode. The avatar and the avatar execution code may be variously classified and provided according to a user's interests or hobbies such as general, weather, sports (golf, basketball, baseball, etc.). In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the counterpart likes or is interested in.

For example, the avatar execution code mapped according to the specific expression of the counterpart may operate the avatar as shown in Table 1. In this case, the specific expression has been described as being limited to a smiling face, a crying face, or a surprised face, but is not limited thereto. In addition, the operation of the avatar according to each specific facial expression is not limited thereto.

Smiley expression Crying expression A surprised look Normal Avatar laughs Avatar crying
Turm Avatar eyes
Large moxibustion weather Avatar is sunny
Strolling in the weather Avatar rained Avatar head up
Lightning strikes golf The ball put by the avatar
I like the hole cup Avatar tee shot in water
I burst out and cry The tee shot is hole-in-one
Avatar is surprised basketball Avatar is on three point shot
Succeeded Avatar hit the ball
Deprived Avatar shot
Blocked

Meanwhile, if the avatar execution code is not separately set, the terminal controller 27 may use the avatar execution code set as a default. For example, a general item may be set as a default.

When the terminal controller 27 starts a video call with the called terminal 40, the terminal controller 27 displays the image data received from the called terminal 40 and the avatar corresponding to the counterpart on the display unit 25. The terminal controller 27 monitors whether the counterpart takes a specific facial expression from the received image data. When the terminal controller 27 detects a specific facial expression during monitoring, the terminal controller 27 operates the avatar corresponding to the specific facial expression.

At this time, the terminal controller 27 calls the avatar from the storage unit 23 or receives it from the calling terminal 40 and displays it on the display unit 25. Upon initiating a video call with the called terminal 40, the terminal controller 27 checks whether an avatar corresponding to the other party is set in the storage 23. If the check result is set, the terminal controller 40 calls the set avatar. If the check result is not set, the terminal controller 27 may call the general avatar stored in the storage 23 or request the avatar from the calling terminal 40 to receive the avatar. The terminal controller 27 displays the called or received avatar on the display unit 23.

The terminal controller 27 may monitor the facial expression of the other party in the received image data as follows. In other words, the terminal controller 27 analyzes the received image data, extracts a face image of the other party, and extracts a region of interest from the extracted face image. The terminal controller 27 detects the facial expression of the other party by tracking the position change of the extracted ROI. In this case, the terminal controller 27 determines whether the detected facial expression is included in a specific facial expression stored in the storage 23. The terminal controller 27 may extract the shape of the object as adaboost through, for example, a principal component analysis (PCA) in order to separate the counterpart (object) image from the background of the received image. The terminal controller 27 detects a continuous or discontinuous facial expression of the other party according to the tracked ROI. Here, adaboost is a learning algorithm for extracting the shape of an object. "A decision-theoretic generalization of on-line learning and an application to boosting" by Yoav Freund and Robert E. Schapire, In Computational Learning Theory: Eurocolt '95, pp. 23-37, Springer-Verlag, 1995.

The ROI is a face component that can convey the emotional state of the counterpart during a video call to the user, and includes at least one of an eye, an eyebrow, a brow, a nose, and a mouth. The terminal controller 27 can detect the expression of the other party by extracting the region of interest as follows. That is, the terminal controller 27 extracts the other party's face image from the received image data. For example, the face image may be extracted using one of PCA, Fisher Discriminarnt Analysis (FDA), and Independent Component Analysis (ICA). The terminal controller 27 extracts face components from the extracted face image. For example, facial components are extracted using adaboost or SVM (Support Vector Machine). In addition, the terminal controller 27 can detect a change in the facial expression of the other party from the position change of the extracted facial components.

When the terminal controller 27 detects a specific facial expression during monitoring, the terminal controller 27 extracts an avatar execution code corresponding to the detected specific facial expression from the storage 23. In addition, the terminal controller 27 may display the emotional state of the counterpart by operating the avatar according to the extracted avatar execution code. At this time, the terminal controller 27 compares the tracked region of interest with the position value of the region of interest of the specific facial expression and monitors whether there is a specific facial expression that matches.

Meanwhile, the terminal controller 27 may extract an avatar execution code capable of modifying the avatar's ROI corresponding to the tracked ROI, and display the emotional state of the other party by operating the avatar according to the extracted avatar execution code.

In addition, when the terminal controller 27 displays the counterpart image and the avatar on the display unit 25, the terminal controller 27 may overlap the counterpart image or display the picture in a PIP (Picture In Picture) form. For example, when the terminal controller 27 detects a favorite facial expression of the other party during monitoring, the terminal controller 27 extracts the avatar execution code matching the favorite facial expression to the storage 23. The terminal control unit 27 operates the avatar to make a jumping action while laughing according to the avatar execution code.

As described above, when the video call terminals 20 and 40 of the emotion delivery system 100 according to the first embodiment are set, a video call path is displayed on the screen by displaying an avatar along with the other party's image and then displaying the other party's emotion. Since it monitors whether a particular facial expression related to the state is taken and the avatar is operated according to the specific facial expression detected during monitoring, the emotional state of the other party can be effectively transmitted to the user through the avatar.

The emotion delivery method of the emotion delivery system 100 according to the first embodiment will be described with reference to FIGS. 1 to 5 as follows. 3 is a flowchart illustrating a method of transmitting emotions through facial expression recognition of a counterpart during a video call according to the first embodiment of the present invention. 4 is a detailed flowchart of the avatar display step of FIG. 3. 5 is a detailed flowchart according to the facial expression change detection step of FIG.

First, when the calling terminal 20 transmits a video call in step S51, the communication network 10 receives the request in step S53 and requests a video call to the called terminal 40. That is, when the calling terminal 20 sends a video call, the call controller 14 requests the subscriber information (location information and additional service information) of the calling and destination terminals 20 and 40 to the location register 16. The call controller 14 obtains routing information based on the received subscriber information. The call controller 14 requests video call reception to the called terminal 40 using the obtained routing information.

Meanwhile, before performing step S51, the calling terminal 20 may set an avatar corresponding to the other party and an avatar execution code for operating the set avatar in the avatar setting mode according to a user's selection signal through the input unit 22. In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the counterpart likes or is interested in.

Next, when the called terminal 40 transmits the acceptance of the video call to the communication network 10 in step S55, the communication network 10 transmits the video for the video call between the calling terminal 20 and the called terminal 40 in step S57. Set the call path. That is, the call controller 14 requests the incoming call by paging the called terminal 40, and when the called terminal 40 accepts the call for the incoming request, the call controller 14 calls the calling terminal 20 and the called terminal ( 40) to set the video call path.

As such, after the video call path is set between the calling terminal 20 and the called terminal 40, the calling terminal 20 and the called terminal 40 perform a video call through the video call path set in step S59. That is, the calling terminal 20 and the called terminal 40 perform a voice call while exchanging video data including their own video with each other.

When the video call is started in step S59, the calling terminal 20 displays the avatar corresponding to the user (the other party) of the called terminal 40 on the display unit 25 in step S70. The step S70 will be described in detail with reference to FIG. 4 as follows. First, in step S71, the calling terminal 20 checks whether the avatar is set in the storage unit 23 corresponding to the counterpart. If the result of the check in step S71 is set, the calling terminal 20 calls the set avatar in the storage unit 23 in step S72.

If the check result of step S71 is not set, in step S73 the calling terminal 20 determines again whether the avatar request signal is input. If it is not input as a result of the determination in step S73, the calling terminal 20 calls the general avatar stored in the storage unit 23 in step S74. When the determination result of step S73 is input, in step S75, the calling terminal 20 requests and receives an avatar from the called terminal 40. In this case, the avatar transmitted from the called terminal 40 is transmitted to the calling terminal 20 through the currently set video call path, or is sent through a message service channel such as a separate short message service (SMS) or multimedia message service (SMS). It may be transmitted to the terminal 20.

In operation S76, the calling terminal 20 displays the called or received avatar and the image data received from the called terminal 40 on the display unit 23. In this case, when the calling terminal 20 displays the image data and the avatar on the display unit 25, the calling terminal 20 may display the image data and the avatar in a superimposed manner or in a PIP form.

Meanwhile, in the first exemplary embodiment, an example in which steps S73 to S75 are performed according to a user input unless the check result of step S71 is set is not limited thereto. For example, if the check result of step S71 is not set, the calling terminal 20 may call a general avatar stored in the storage unit 23 in step S74. Alternatively, if the check result of step S71 is not set, the calling terminal 20 may request and receive an avatar from the called terminal 40 in step S75.

Next, in step S80, the calling terminal 20 monitors whether the counterpart takes a specific facial expression in the received image. Referring to FIG. 5, the following description will be made with reference to FIG. First, in step S81, the calling terminal 20 extracts the face image of the other party from the received image data. Next, in step S83, the calling terminal 20 extracts a region of interest from the extracted face image. In operation S85, the calling terminal 20 detects the expression of the other party by tracking the extracted change of the ROI. For example, the received image data includes a plurality of frame image data, and a predetermined number of image data is captured from the plurality of frame image data included in the image data received at a predetermined time, and the other image is extracted from the captured image data. The face image may be extracted. That is, the terminal controller 27 can extract the other party's face image from the received image data using one of PCA, FDA, and ICA. The terminal controller 27 extracts a region of interest such as an eye, an eyebrow, a mouth, or a nose from the extracted face image by using an adaboost or SVM. In addition, the terminal controller 27 can detect a change in the facial expression of the other party from the position change of the extracted region of interest.

Next, in step S87, the calling terminal 20 determines whether the detected facial expression is included in a specific facial expression previously stored in the storage unit 23. In other words, by monitoring the change in the region of interest extracted by the calling terminal 20 to monitor whether the other party takes a particular facial expression. For example, the calling terminal 20 may perform step S87 by comparing the region of interest included in the detected facial expression with the region of interest of the previously stored specific facial expression or not.

If the determination result of step S87 is not included in the pre-stored specific facial expression, the calling terminal 20 performs again from step S81.

If the determination result in step S87 is included in the pre-stored specific facial expression, in step S90 the calling terminal 20 operates the avatar corresponding to the detected facial expression change. That is, when a specific facial expression is detected during monitoring, the calling terminal 20 extracts the avatar execution code corresponding to the specific facial expression from the storage unit 23. Alternatively, the calling terminal 20 may extract the avatar execution code from the storage unit 23 that may transform the ROI of the avatar to correspond to the tracked ROI. The calling terminal 20 transmits the emotion of the other party to the user through the avatar by operating the avatar displayed on the screen according to the extracted avatar execution code. For example, when the avatar is displayed on the screen, the avatar may be superimposed on the counterpart image or displayed in a PIP form. Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar.

Meanwhile, in the emotion transmitting method according to the first embodiment, an example of transmitting an emotion state of the user of the called terminal 40 to the user of the calling terminal 20 by displaying an avatar on the calling terminal 20 is disclosed. ), An avatar may be displayed on the caller terminal 20 to transmit the emotional state of the caller terminal 20 user to the caller terminal 40 user.

Meanwhile, the emotion transmitting system 100 according to the first embodiment calls and displays the avatar stored in the storage unit 23 by the calling terminal 20 or the called terminal 40, and operates the avatar in response to the facial expression change of the other party. An example is disclosed, but is not limited thereto. That is, as shown in FIG. 6, the calling terminal 20 or the called terminal 40 may receive and display an avatar corresponding to the counterpart through the avatar providing server 30.

In the video call according to the second embodiment of the present invention, the emotion transmission system 200 through the facial recognition of the other party, as shown in Figure 6, the originating terminal 20 connected via the communication network 10, the avatar provided It comprises a server 30 and the destination terminal 40.

The calling terminal 20 and the called terminal 40 perform a video call with each other via the communication network 10 of the communication network 10. The calling terminal 20 or the called terminal 40 receives and displays the avatar reflecting the emotion according to the expression of the other party from the avatar providing server 30 through the communication network 10.

The communication network 10 performs a series of data transmission / reception operations for data transmission and information exchange between the calling terminal 20, the avatar providing server 30, and the called terminal 40 via the communication network 10. In particular, when the communication network 10 receives a video call from the calling terminal 20, the communication network 10 sets a video call path for a video call between the calling terminal 20 and the called terminal 30. When a video call is initiated through the set video call path, the communication network 10 receives an avatar reflecting the emotion according to the expression of the other party from the avatar providing server 30 and transmits the avatar to the video call terminals 20 and 40.

The avatar providing server 30 communicates with the calling terminal 20 and the called terminal 30 via the communication network 10. When the video call is started, the avatar providing server 30 transmits the avatar corresponding to the counterpart to the calling terminal 20 or the called terminal 40 through the communication network 10. The avatar providing server 30 receives image data from the counterpart terminal of the terminal that receives the avatar, monitors whether the other party takes a specific facial expression from the received image data, and operates the avatar corresponding to the specific facial expression detected during monitoring. . Therefore, the user can know the emotional state of the other party through the state of the facial expression or behavior of the avatar.

In particular, the avatar providing server 30 according to the second exemplary embodiment includes a transceiver 31, a database 33, and a controller 35.

The transceiver 31 communicates with the calling terminal 20 and the called terminal 40 via the communication network 10. The transceiver 31 receives an avatar request from the communication network 10 under the control of the controller 35, and transmits the requested avatar to the communication network 10. The transmitter / receiver 31 transmits the avatar execution code to the communication network 10 under the control of the controller 35.

The database unit 33 stores an avatar corresponding to a specific gesture taken by the user of the video call terminal including the calling terminal 20 or the called terminal 40. In addition, the database unit 33 stores the avatar set by the user of the video call terminals 20 and 40 and the avatar execution code for operating the avatar. The user may set the avatar to correspond to the counterpart. The avatar includes an avatar set according to the terminal information of the counterpart according to the user's selection signal and a general avatar set by default. The general avatar set as a default may be used when conducting a video call with a person other than the user.

In addition, the controller 35 performs an overall control operation of the avatar providing server 30. In particular, the controller 35 controls the transmission of the avatar to the user terminal by recognizing the facial expression of the other party during the video call.

In the avatar setting mode, the controller 35 may set an avatar corresponding to the counterpart according to a user's selection signal through the video call terminals 20 and 40, and set an avatar execution code for operating the set avatar. In this case, since the setting of the avatar and the avatar execution code by the control unit 35 may be performed in the same manner as the setting method by the terminal control unit 23 of FIG. 2, a detailed description thereof will be omitted.

When the video call is started between the calling terminal 20 and the called terminal 40 through the communication network 10, the controller 35 sends the calling terminal 20 or the called terminal 40 to the other party through the communication network 10. Send the corresponding avatar. The controller 35 receives image data from the counterpart terminal of the terminal that has received the avatar, and monitors whether the counterpart has a specific facial expression. If the control unit 35 detects a specific facial expression during monitoring, the controller 35 transmits an avatar execution code for operating the avatar corresponding to the specific facial expression to the counterpart terminal.

In this case, the avatar may operate in a set action manner corresponding to a specific facial expression taken by the counterpart. Different avatars may be set to correspond to specific facial expressions, or different behaviors may be set for the same avatar.

The controller 35 calls the avatar from the database unit 33 or receives the avatar from the counterpart terminal and transmits the avatar to the user terminal. That is, when the video call is started, the controller 35 checks whether the avatar corresponding to the counterpart is set in the database unit 33. If the check result is set, the control unit 35 calls the set avatar. If the check result is not set, the control unit 35 may call the general avatar stored in the database unit 33 or request and receive the avatar from the counterpart terminal. The controller 35 transmits the called or received avatar to the user terminal. In this case, the controller 35 transmits the avatar to the user terminal through a currently set video call path or a separate message service channel.

The controller 35 may monitor the facial expression taken by the other party in the received image data as follows. That is, the controller 35 analyzes the received image data, extracts a face image of the other party, and extracts a region of interest from the extracted face image. In addition, the controller 35 detects an expression of the other party by tracking the position change of the extracted ROI. For example, the controller 35 may extract the other party's face image from the received image data using one of PCA, FDA, and ICA. The controller 35 extracts a region of interest, such as an eye, an eyebrow, a mouth, or a nose, from an extracted face image by using an adaboost or SVM. In addition, the controller 35 may detect a change in the facial expression of the other party from the position change of the extracted ROI. The controller 35 detects a continuous or discontinuous facial expression of the other party according to the tracked ROI.

If the controller 35 detects a specific facial expression during monitoring, the controller 35 extracts the avatar execution code corresponding to the specific facial expression from the database 33. The controller 35 transmits the extracted avatar execution code to the user terminal to operate the avatar. In this case, the controller 35 may monitor whether there is a specific facial expression that matches by comparing the tracked ROI and the position value of the ROI of the specific facial expression. The controller 35 may transmit the avatar execution code through a currently set video call path or a separate message service channel.

On the other hand, the controller 35 may extract the avatar execution code for transforming the avatar's ROI corresponding to the tracked ROI and transmit it to the user terminal.

The emotion delivery method of the emotion delivery system 200 according to the second embodiment will now be described with reference to FIGS. 6 and 7. 7 is a flowchart illustrating a method of transmitting emotions through facial expression recognition of a counterpart during a video call according to the second embodiment of the present invention.

First, since steps S51 to S59 proceed in the same manner as in the emotion transmitting method of FIG. 3, a detailed description thereof will be omitted and will be described after step S59. Meanwhile, before performing step S51, the avatar providing server 30 may set an avatar corresponding to the other party and an avatar execution code for operating the set avatar in the avatar setting mode according to a user's selection signal through the calling terminal 20. have. In the avatar setting mode, the user may select and set an avatar or an avatar execution code of a field that the user or the counterpart likes or is interested in.

Next, when the video call is started in step S59, the communication network 10 requests the avatar providing server 30 to send the avatar to the calling terminal 20 in step S61. Next, in step S63, the avatar providing server 30 extracts the avatar set by the user of the calling terminal 20 or the called terminal 40 from the database unit 33 and transmits the avatar to the communication network 10. In this case, the avatar may be an avatar set to correspond to the user of the called terminal 40. The avatar providing server 30 may transmit the avatar through a currently set video call path or a separate message service channel.

In operation S65, the communication network 10 transmits the received counterpart image and the avatar to the calling terminal 20. In operation S76, the calling terminal 20 displays the received counterpart image and the avatar on the screen.

In operation S67, the communication network 10 transmits the counterpart image received from the called terminal 40 to the avatar providing server 30.

Next, in step S80, the avatar providing server 30 monitors whether the counterpart has a specific facial expression in the received image. The step S80 will be described in detail with reference to FIG. First, in step S81, the avatar providing server 30 extracts the face image of the other party from the received image data. Next, in step S83, the avatar providing server 30 extracts a region of interest from the extracted face image. In operation S85, the avatar providing server 30 detects the expression of the other party by tracking the extracted change of the ROI. For example, the received image data includes a plurality of frame image data, and a predetermined number of image data is captured from the plurality of frame image data included in the image data received at a predetermined time, and the other image is extracted from the captured image data. The face image may be extracted. That is, the controller 35 may extract the face image of the counterpart from the received image data using one of PCA, FDA, and ICA. The controller 35 extracts a region of interest, such as an eye, an eyebrow, a mouth, or a nose, from an extracted face image by using an adaboost or SVM. In addition, the controller 35 may detect a change in facial expression of the other party from the position change of the extracted ROI.

Next, in step S87, the avatar providing server 30 determines whether the detected facial expression is included in a specific facial expression previously stored in the database 33. That is, the avatar providing server 30 tracks the extracted region of interest and monitors whether the other person has a specific facial expression. For example, the avatar providing server 30 may perform step S87 by comparing the ROI included in the detected facial expression with the ROI of the specific facial expression stored in advance.

If the determination result in step S87 is not included in the pre-stored specific facial expression, the avatar providing server 30 performs again from step S81.

If the determination result in step S87 is included in the previously stored specific facial expression, in step S91, the avatar providing server 30 extracts the avatar execution code corresponding to the detected facial expression change from the database unit 33. Alternatively, the avatar providing server 30 may extract, from the database unit 33, the avatar execution code that can transform the ROI of the avatar to correspond to the tracked ROI.

Next, in step S93, the avatar providing server 30 transmits the extracted avatar execution code to the communication network 10. In operation S95, the communication network 10 transmits the received avatar execution code to the calling terminal 20. In this case, the avatar providing server 30 may transmit the avatar execution code to the calling terminal 20 through the currently set video call path or a separate message service channel.

In operation S97, the calling terminal 20 operates the avatar displayed on the screen according to the received avatar execution code to transmit the emotion of the other party to the user through the avatar. For example, when the avatar is displayed on the screen, the avatar may be superimposed on the counterpart image or displayed in a PIP form. Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar.

Meanwhile, in the emotion transmitting method according to the second embodiment, an example in which the avatar providing server 30 provides an avatar representing the emotional state of the user of the called terminal 40 to the calling terminal 20 is disclosed. ) May provide an avatar representing the emotional state of the user of the calling terminal 20. Alternatively, an avatar representing the emotional state of the other party may be provided to the called party and the calling terminal, respectively.

On the other hand, the embodiments of the present invention disclosed in the specification and drawings are merely presented specific examples to aid understanding, and are not intended to limit the scope of the present invention. In addition to the embodiments disclosed herein, it is apparent to those skilled in the art that other modifications based on the technical idea of the present invention may be implemented.

The present invention relates to a system and a method for transmitting emotions by recognizing a facial expression of a counterpart during a video call, wherein the video call terminal displays an avatar corresponding to the counterpart along with the video of the counterpart on a screen and receives an image received from the counterpart terminal. By operating the avatar in response to the specific facial expression of the counterpart by analyzing the data, the terminal user can quickly and accurately recognize the emotional state of the counterpart through the avatar while performing a video call. Since the user can directly set and use the avatar to be used during the video call, the user can expand the use of the video call by inducing interest of the video call while satisfying the various needs of the video caller. Since the method can be set directly, it can provide an environment in which a video call can be used more interestingly while pursuing the user's personality.

1 is a block diagram illustrating an emotion transmission system through facial recognition of a counterpart during a video call according to the first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration of a calling terminal of FIG. 1.

3 is a flowchart illustrating a method of transmitting emotions through facial expression recognition of a counterpart during a video call according to the first embodiment of the present invention.

4 is a detailed flowchart of the avatar display step of FIG. 3.

FIG. 5 is a detailed flowchart according to the facial expression change detection step of FIG. 3.

FIG. 6 is a block diagram illustrating an emotion transmission system through facial expression recognition of a counterpart during a video call according to the second embodiment of the present invention.

7 is a flowchart illustrating a method of transmitting emotions through facial expression recognition of a counterpart during a video call according to the second embodiment of the present invention.

Description of the Related Art [0002]

10: communication network

20: calling terminal

30: avatar providing server

40: called terminal

100, 200: emotional delivery system

Claims (23)

A communication network controlling a video call between a calling terminal and a called terminal; When a video call is initiated between the calling terminal and the called terminal through the communication network, an avatar corresponding to the other party is transmitted to the calling terminal or the called terminal through the communication network, and video data is received from the other terminal of the terminal receiving the avatar. Receiving a, and monitors whether the other party takes a particular facial expression in the received image data, and if the specific facial expression is detected during the monitoring, the avatar execution code that can operate the avatar corresponding to the detected specific facial expression An avatar providing server for transmitting the avatar to a terminal; Emotion delivery system through the recognition of the facial expression of the other side of the video call. A transceiver for performing communication with a calling terminal or a called terminal via a communication network; When a video call is initiated between the calling terminal and the called terminal through the communication network, an avatar corresponding to the other party is transmitted to the calling terminal or the called terminal through the communication network, and video data is received from the other terminal of the terminal receiving the avatar. And receiving an avatar from the received image data and monitoring whether the counterpart has a specific facial expression. If the specific facial expression is detected during the monitoring, the avatar execution code may be operated to correspond to the detected specific facial expression. A control unit for transmitting the avatar to the terminal; Avatar providing server of the emotion delivery system by recognizing the expression of the other side of the video call. The method of claim 2, A database unit for storing an avatar set by a user of the calling terminal or the called terminal and an avatar execution code for operating the avatar; More, The controller calls the avatar corresponding to the counterpart from the database to transmit the avatar to the calling terminal or the called terminal, extracts the avatar execution code corresponding to the detected facial expression from the database, and extracts the avatar. The avatar providing server of the emotion delivery system through the facial recognition of the other party during the video call characterized in that the transmission to the terminal. Communication unit for performing a video call with the other terminal; A camera unit for photographing a user during a video call; A display unit displaying image data of the other party received through the communication unit; When the video call is initiated with the counterpart terminal through the communication unit, the video display unit is displayed on the display unit together with the avatar corresponding to the counterpart and the video data received from the counterpart terminal, and the counterpart displays a specific facial expression in the received video data. A terminal control unit configured to monitor whether to take a gesture and to operate the avatar in response to the detected specific facial expression when the specific facial expression is detected during the monitoring; Video call terminal comprising a. The method of claim 4, wherein And a storage unit for storing the avatar set to correspond to the counterpart and the avatar execution code for operating the avatar. The controller calls the avatar corresponding to the counterpart from the storage unit and displays the avatar on the display unit. The controller extracts the avatar execution code corresponding to the detected facial expression from the storage unit and extracts the avatar execution code from the storage unit. And operating the avatar according to the video call terminal. Transmitting an avatar corresponding to the other party to the calling terminal or the called terminal through the communication network when the video call is started by setting a video call path between the calling terminal and the called terminal through a communication network; A receiving step of receiving, by the avatar providing server, image data from a counterpart terminal of a terminal that receives the avatar through the communication network; A monitoring step of monitoring whether the avatar providing server takes a specific expression related to the emotional state of the counterpart from the received image data; Transmitting the avatar execution code to the terminal receiving the avatar, when the avatar providing server detects the specific facial expression during the monitoring, an avatar execution code capable of operating the avatar corresponding to the detected specific facial expression; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 6, wherein the transmitting of the avatar comprises: And the avatar providing server calls the avatar from a database unit or receives the avatar from the counterpart terminal and transmits the avatar to the calling terminal or the called terminal. The method of claim 7, wherein the transmitting of the avatar execution code comprises: The avatar providing server extracts an avatar execution code corresponding to the specific facial expression detected during the monitoring from the database unit and transmits the avatar execution code to the terminal receiving the avatar. Way. The method of claim 6, wherein the transmitting the avatar and transmitting the avatar execution code comprise: The avatar providing server transmits the avatar and the avatar execution code through the set video call path or the message service channel. The method of claim 6, wherein the step is performed before the avatar transmission step, Selecting and setting, by the avatar providing server, an avatar or an avatar execution code of a field that the user or the other party likes or is interested in in the avatar setting mode; Emotion delivery method through the facial recognition of the other side of the video call. A display step of displaying an avatar corresponding to the counterpart and the image data received from the counterpart terminal when the user terminal sets up a video call path with the counterpart terminal; A monitoring step of monitoring whether the user terminal acquires a specific facial expression related to the emotional state of the counterpart from the received image data; Operating the avatar in response to the specific facial expression detected by the terminal when the specific facial expression is detected during the monitoring; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 11, wherein the monitoring step, Extracting, by the user terminal, a face image of the counterpart from the received image data, and extracting a region of interest from the extracted face image; An expression detection step of detecting, by the user terminal, a change of the extracted region of interest to detect an expression of the counterpart; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 12, wherein the region of interest is: At least one of an eye, an eyebrow, a brow, a nose and a mouth. The method of claim 12, wherein the operation step, If the specific facial expression is detected during the monitoring, extracting, by the user terminal, an avatar execution code corresponding to the detected specific facial expression from a storage unit; Operating the avatar in correspondence with the extracted avatar execution code by the user terminal; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 12, wherein the operation step, The method of claim 1, wherein the user terminal deforms the region of interest of the avatar corresponding to the tracked region of interest. The method of claim 12, wherein the facial expression detection step, And transmitting the emotion through the facial recognition of the other party during the video call, wherein the user terminal detects the continuous or discontinuous facial expression of the other party according to the tracked ROI. The method of claim 11, wherein the displaying step, And the user terminal calls the avatar from the storage unit or receives the avatar from the counterpart terminal. The method of claim 17, wherein the displaying step, A determination step of determining whether an avatar corresponding to the counterpart is set in the storage unit when the user terminal initiates a video call with the counterpart terminal; If the determination result is set, the user terminal calls the set avatar; if the determination result is not set, the user terminal calls a general avatar in the storage unit; Displaying, by the user terminal, together with the called avatar and the image data received from the counterpart terminal; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 17, wherein the displaying step, Determining whether an avatar corresponding to the counterpart is set in the storage unit when the user terminal initiates a video call with the counterpart terminal; If the determination result is set, the user terminal calls the set avatar; if the determination result is not set, the user requests and receives the avatar from the counterpart terminal; Displaying, by the user terminal, image data received from the counterpart terminal and the avatar together; Emotion transfer method through the facial expression recognition of the other side of the video call. The method of claim 19, wherein in the request and reception of the avatar, And the user terminal receives the avatar from the counterpart terminal through the set video call path or the message service channel. The method of claim 11, wherein in the displaying step, The avatar is superimposed on the received image data or displayed in the form of picture in picture (PIP) characterized in that the emotion transfer method through the facial expression recognition of the other party. The method of claim 11, wherein the step is performed before the avatar transmission step. Selecting and setting, by the user terminal, an avatar or an avatar execution code of a field that the user or the other party likes or is interested in in the avatar setting mode; Emotion delivery method through the facial recognition of the other side of the video call. The method according to any one of claims 6 to 22, The avatar is a graphic content that represents the emotion state of the other party, and includes one of an animated character, a video, a still image, a user created content (UCC), an emoticon, a flash, a haptic content in which a video and vibration are combined. Emotion transfer method through the facial recognition of the other party during the video call.

Images