KR20110025721A - System and method for delivering feeling during video call - Google Patents

Abstract

PURPOSE: A feeling delivery system during an image call and a method thereof are provided to exactly transmit various feeling states of an opponent to a user through an image call. CONSTITUTION: An avatar providing server(30) receives an image data of an opponent photographed from an opponent terminal of a terminal which sent an avatar providing request signal. The avatar providing service analyzes the received image data. The avatar providing server senses a gesture which the opponent poses. The avatar providing server extracts an avatar corresponding to the sensed gesture to a database unit. The avatar providing server transmits an image call traffic including the extracted avatar to a terminal which sent the avatar providing request signal.

Description

System and method for delivering feeling during video call}

The present invention relates to a video call technology, and more particularly, to analyze an image data received from a counterpart terminal during a video call, and when the counterpart takes a specific gesture, an avatar corresponding to the specific gesture is output to the user terminal to express the emotion of the counterpart. The present invention relates to an emotional delivery system and method for transmitting a video call to a user.

As high-speed information communication is realized in voice call-oriented call patterns, video call technology through communication terminals is attracting attention. In addition, as the spread of communication terminals capable of video calls (hereinafter referred to as 'video call terminals') is increasing, the frequency of video calls is increasing day by day.

Since the video call terminal displays the video of the other party on the screen and outputs the voice of the other party, the video call terminal can easily exchange each other's emotional states with each other as compared with simply making a voice call.

However, since the user is still receiving the emotional state of the other party only through the facial expression or voice during the video call, there is a limit in accurately transmitting the various emotional states of the other party to the user through the video call terminal. In addition, since the current video call is merely to perform the call while watching the video of the other party, there is a limit to expanding the use of the video call by inducing interest of the video call while satisfying various needs of the video caller.

Accordingly, it is an object of the present invention to provide an emotion delivery system and method during a video call that can accurately convey various emotional states of the counterpart to a user through a video call.

Another object of the present invention is to provide a system and method for transmitting emotions during a video call, which can induce interest of a video call to expand the use of the video call.

Another object of the present invention is to analyze the image data received from the counterpart terminal during the video call, and outputs the avatar corresponding to the gesture taken by the counterpart on the screen of the user terminal to transmit the emotion state of the counterpart to the user. A system and method are provided.

In order to achieve the above object, the present invention provides a system for transmitting emotions during a video call through gesture recognition including a communication network and an avatar providing server. When the communication network receives the avatar providing request signal from the calling terminal or the called terminal during the video call, the communication network establishes a session between the avatar providing server and the calling terminal and a session between the avatar providing server and the called terminal. The avatar providing server receives video data of the other party photographed from the other party's terminal of the terminal which has sent the avatar providing request signal, and analyzes the received video data to detect a gesture taken by the other party, and to the detected gesture. A corresponding avatar is extracted from a database unit, and video call traffic including the extracted avatar is transmitted to a terminal that has sent the avatar providing request signal.

In the emotion delivery system according to the present invention, the communication network sets a video call path between the calling terminal and the called terminal upon receiving a video call sent from the calling terminal, and when the avatar providing request signal is received, the video call path. Releases and sets a session between the avatar providing server and the calling terminal and a session between the avatar providing server and the called terminal.

The present invention also provides an avatar providing server of an emotion delivery system during a video call through gesture recognition including a transceiver, a database, and a controller. The transceiver performs communication with a communication network, a calling terminal and a called terminal. The database unit stores an avatar corresponding to a gesture of the calling terminal or the called terminal user. The controller sets a session with the calling terminal and the called terminal by the communication network according to the avatar providing request signal of the calling terminal or the called terminal during the video call, and from the counterpart terminal of the terminal sending the avatar providing request signal. Receives image data of the other party photographed, analyzes the received image data, detects a gesture taken by the other party, extracts an avatar corresponding to the detected gesture from a database unit, and includes a video call including the extracted avatar. The traffic is transmitted to the terminal that has sent the avatar providing request signal.

The present invention also provides a video call terminal of an emotion delivery system during a video call through gesture recognition including a communication unit, a camera unit, an output unit, a storage unit, and a terminal controller. The communication unit performs a video call with a counterpart terminal through a communication network. The camera unit photographs a user during a video call. The output unit outputs the video and audio data of the other party received through the communication unit. The storage unit stores an avatar corresponding to the gesture of the counterpart. The terminal controller analyzes the image data received from the counterpart terminal through the communication unit to detect a gesture taken by the counterpart, extracts an avatar corresponding to the detected gesture from the storage unit, and extracts the extracted avatar. Output to the output.

The present invention also provides a session setting step of establishing a session with the calling terminal and the called terminal by the communication network according to the avatar providing request signal of the calling terminal or the called terminal in which the avatar providing server is in a video call. Receiving step of receiving the image data of the other party photographed from the other party terminal of the terminal that has provided the provision request signal, the avatar providing server analyzes the received video data to detect the gesture taken by the other party, and to the detected gesture Extracting a corresponding avatar from the database unit; transmitting the video call traffic including the extracted avatar to the terminal that sent the avatar providing request signal by the avatar providing server; How to convey emotions Ball.

The present invention also provides a receiving step of receiving a video data from the counterpart terminal by a user terminal having a video call with the counterpart terminal via a communication network, and detecting the gesture of the counterpart by analyzing the received video data by the user terminal. And extracting an avatar corresponding to the detected gesture in a storage unit, and outputting the extracted avatar through an output unit by the user terminal. .

According to the present invention, the user terminal may analyze the image data received from the counterpart terminal during the video call, and output the avatar corresponding to the gesture taken by the counterpart along with the received image data to transmit the emotional state of the counterpart to the user. That is, the user terminal may effectively transmit the emotional state of the other party to the user by outputting the avatar according to the gesture of the other party as well as the other party's image and voice.

The video call terminal according to the present invention can perform a video call while outputting an avatar corresponding to the gesture of the other party, and the user can directly set and use an avatar to be used during the video call, so that various needs of the video caller can be met. While satisfying the video caller's interest, the use of the video call can be expanded. That is, since the user can directly set the appearance or behavior of the avatar corresponding to at least one gesture of the other party through the video call terminal, the user can provide an environment in which the video call can be used more interestingly while pursuing the user's personality. .

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the present invention.

Here, 'Avatar' is a graphic content that represents the emotional state of the terminal user, such as animated characters, video, still images, UCC (User Created Contents), emoticons, flash, haptic content combined with video and vibration, etc. It includes.

The emotion delivery system 100 during a video call according to the first embodiment of the present invention will be described with reference to FIGS. 1 and 2. 1 is a block diagram illustrating an emotion transmission system 100 during a video call according to a first embodiment of the present invention. 2 is a block diagram showing the configuration of the originating terminal 20 of FIG.

The emotional delivery system 100 (hereinafter, referred to as an 'emotional delivery system') of a video call according to the first embodiment outputs an avatar corresponding to a specific gesture taken by the other party during a video call and transmits the emotional state of the other party to the user. That is, since certain gestures such as facial expressions, hand gestures, head gestures, arm gestures, leg gestures, and combinations thereof, which are taken by the other party during the video call are related to the emotional state of the other party, the emotional delivery system 100 according to the first embodiment When the counterpart makes a specific gesture during the video call, the avatar may express the emotional state of the counterpart corresponding to the particular gesture on the screen of the user terminal to transmit the emotional state of the counterpart to the user.

The emotion transmission system 100 according to the first embodiment is configured to include a calling terminal 20 and a called terminal 40 connected to the communication network 10 to perform a video call.

The communication network 10 performs a series of data transmission and reception operations for data transmission and information exchange between the calling terminal 20 and the called terminal 40. The communication network 10 is an IP network providing a transmission / reception service and a seamless data service of a large amount of data through IP, and may be an ALL-IP network, which is an IP network structure integrating different networks based on IP. In addition, the communication network 10 may be one of a wired communication network, a mobile communication network, a wireless broadband (WiBro) network, a high-speed downlink packet access (HSDPA) network, and a satellite communication network.

Such a communication network 10 may include a call controller 14 and a location register 16.

The call controller 14 is a part in charge of functions related to call and session processing, and may be one of a call state control function (CSCF) or a mobile switching center (MSC). In particular, the call controller 14 implemented with CSCF performs a function as a gateway for incoming calls, a call control function, a SPD (Serving Profile Database) function, an address processing function, and the like. In particular, the call controller 14 transmits the video call traffic including the avatar to the terminal requesting to provide the avatar according to the request of the calling terminal 20 or the called terminal 40.

The location register 16 is a master database of subscribers, and is responsible for tracking access means by maintaining a master list of services related to subscribers or performing management of subscriber location information. The location register 16 stores additional service subscription information subscribed to by the subscriber. The location register 16 may be one of a home location register (HLR) or a home subscriber server (HSS) that manages subscriber information in an All-IP network.

In addition, the calling terminal 20 and the called terminal 40 perform a video call with each other through the communication network 10. When the avatar providing request signal is input from the calling terminal 20 or the called terminal 40, the terminal that sends the avatar providing request signal outputs an avatar corresponding to a specific gesture taken by the other party. The calling terminal 20 and the called terminal 40 are wired / wireless communication terminals capable of video calls through the communication network 10, and include a mobile communication terminal, a wired communication terminal, and an internet telephone. At this time, the user terminal displays the counterpart image including the avatar. Therefore, the user can know the emotional state of the other party through the output avatar.

Since the calling terminal 20 and the called terminal 40 of the emotional delivery system 100 during the video call according to the first embodiment have the same configuration, the description will be mainly given of the configuration of the calling terminal 20 shown in FIG. 2. Is as follows. The calling terminal 20 includes a communication unit 21, an input unit 22, a storage unit 23, a camera unit 24, an output unit 25 and 26, and a terminal control unit 27.

The communication unit 21 transmits and receives data for a video call with the destination terminal 40 through the communication network 10. The communication unit 21 outputs an avatar corresponding to the gesture taken by the counterpart to the output units 25 and 26 under the control of the terminal controller 27.

The input unit 22 provides a plurality of keys for the operation of the calling terminal 20, and generates a selection signal according to a user's key selection and transmits the selection signal to the terminal controller 27. The user may input a video call request signal and an avatar providing request signal during the video call through the input unit 22. The input unit 22 may be a keypad, a pointing device such as a touch pad, or an input device such as a touch screen.

The storage unit 23 stores a program necessary for controlling the operation of the calling terminal 20 and data generated during the execution of the program, and includes one or more volatile memory devices and nonvolatile memory devices. The storage unit 23 stores an execution program for outputting the avatar according to the gesture of the other party according to the avatar providing request signal during the video call. The storage unit 23 stores an avatar corresponding to a specific gesture of the counterpart or an avatar set by the user. At this time, the storage unit 23 matches the terminal information of the counterpart and stores the avatar. The storage unit 23 may map and store an avatar to operate according to a specific gesture of the counterpart.

Here, the specific gesture is a gesture related to the emotional state of the other of the gestures taken by the other party, and includes at least one of facial expressions, hand gestures, head gestures, arm gestures, leg gestures, and a combination thereof. For example, when the other person shakes his head from side to side, it is a gesture that is taken when expressing a negative emotional state such as rejection or dislike. When the other person shakes his head up and down, it is a gesture that he or she wants to express a positive emotional state such as acceptance or good. When the other party takes a heart-shaped gesture using both arms or both hands, the other party takes a gesture to express an emotional state using the user.

The avatar may operate in a set behavioral manner corresponding to a specific gesture taken by the counterpart (object). For example, when the other party makes a gesture of shaking his / her head from side to side, the avatar expresses the negative emotional state of the other party through a set facial expression or action. Alternatively, the avatar may operate in an action manner corresponding to a specific gesture taken by the counterpart. For example, when the other person shakes his head from side to side, the avatar also shakes his head from side to side in the same manner as the other person's gesture.

When the avatar is mapped to the specific gesture of the counterpart and stored, the avatar may be set to express the emotion state of the counterpart according to the specific gesture as it is, or the avatar may be set to be expressed in the opposite direction.

The camera unit 24 acquires image data of a user (sender) for transmission through the communication unit 21. That is, the camera unit 24 acquires image data by capturing the image of the user according to the acceptance of the video call through the input unit 22. In this case, the camera unit 24 may include an image sensor, a signal processor, and an image processor. The image sensor converts the optical signal of the captured image into an analog signal. The signal processor converts an analog signal into a digital signal. The image processing unit processes the image signal input through the signal processing unit to obtain image data, and outputs the obtained image data through the communication unit 21 or the display unit 25 or stores it in the storage unit 23. In addition, the image processing unit processes the image data of the other party (recipient) input through the communication unit 21 to output through the display unit 25 or to store in the storage unit (23).

The output units 25 and 26 include a display unit 25 and a sound source processing unit 26. The output units 25 and 26 may further include a vibration generator for generating vibration.

The display unit 25 displays information stored in the storage unit 23 including various function menus executed in the calling terminal 20. The display unit 25 displays an image or an avatar of the other party under the control of the terminal controller 27. As the display unit 25, a liquid crystal display (LCD) or a touch screen may be used. The touch screen simultaneously serves as a display device and an input device.

The sound source processor 26 digitizes the voice input to the microphone MIC under the control of the terminal controller 27 to obtain voice information, and demodulates the voice information received through the communication unit 21 to generate a speaker SPK. Output through

In addition, the terminal controller 27 is a microprocessor that performs overall control operations of the calling terminal 20. In particular, the terminal controller 27 analyzes the image data received from the counterpart terminal during the video call to monitor whether the counterpart takes a specific gesture, and controls the output of the avatar corresponding to the specific gesture taken by the counterpart during monitoring.

The terminal controller 27 may set an avatar corresponding to the specific gesture of the counterpart in the avatar setting mode. The avatar may be classified and provided in various ways according to the user's interests or hobbies such as general, weather, sports (golf, basketball, baseball, etc.). In the avatar setting mode, the user may select and set an avatar of a field that the user or the other party likes or is interested in.

For example, an avatar to operate as shown in Table 1 may be set according to a specific gesture of the other party. At this time, the emotional state is described as being limited to joy, sadness, surprise, but is not limited thereto. In addition, the motion of the avatar according to each emotional state is not limited thereto.

pleasure sadness surprised Normal Avatar laughs Avatar crying
Turm Avatar eyes
Large moxibustion weather Avatar is sunny
Strolling in the weather Avatar rained Avatar head up
Lightning strikes golf The ball put by the avatar
I like the hole cup Avatar tee shot in water
I burst out and cry The tee shot is hole-in-one
Avatar is surprised basketball Avatar is on three point shot
Succeeded Avatar hit the ball
Deprived Avatar shot
Blocked

On the other hand, if a separate avatar is not set, the terminal controller 27 may use an avatar set by default. For example, the general avatar may be set as a default.

The terminal controller 27 may output the avatar in the following manner when outputting the avatar through gesture recognition of the counterpart during the video call.

First, the terminal controller 27 monitors whether the other party takes a specific gesture in the video data received from the called terminal 40 during the video call through the communication unit 21. When the counterpart takes a specific gesture during monitoring, the terminal controller 27 extracts an avatar corresponding to the specific gesture from the storage unit 23 and outputs the extracted avatar to the output units 25 and 26.

In this case, the terminal controller 27 may perform monitoring as follows. That is, the terminal controller 27 extracts the counterpart image using an algorithm that recognizes the outline of the counterpart (object) from the image data received from the called terminal 40 during the video call. The terminal controller 27 recognizes the gesture taken by the counterpart from the extracted counterpart image. The terminal controller 27 determines whether the recognized gesture is included in a specific gesture stored in the storage 23. For example, the terminal controller 27 may use adaboost through PCA (principal component analysis) to extract a counterpart (object) image from the background of the received image and to extract and track a region of interest from the extracted counterpart image. Here, adaboost is a learning algorithm for extracting and tracking the shape and the area of interest of an object. "A decision-theoretic generalization of on-line learning and an application to boosting", In Computational Learning Theory by Yoav Freund and Robert E. Schapire: Eurocolt '95, pp. 23-37, Springer-Verlag, 1995.

The region of interest may be selected from a counterpart image capable of transmitting the emotional state of the counterpart in a video call to the user, and includes at least one of a face, two hands, both arms, and both feet of the counterpart.

The avatar operates in a set manner of action corresponding to a specific gesture taken by the opponent. Different avatars may be set to correspond to a specific gesture, or a different behavior may be set for the same avatar.

Next, the terminal controller 27 analyzes the image data received from the called terminal 40 during the video call through the communication unit 12 to extract and track the region of interest according to the gesture taken by the other party. The terminal controller 27 transforms the avatar to correspond to the tracked ROI. The terminal controller 27 outputs the modified avatar. That is, the avatar operates in a manner of action corresponding to a gesture taken by the counterpart under the control of the terminal controller 27.

In this case, the terminal controller 27 can extract and track the ROI as follows. That is, the terminal controller 27 extracts the counterpart image from the image data received from the called terminal 40 during the video call. The terminal controller 27 extracts a region of interest from the extracted counterpart image. The terminal controller 27 tracks the extracted change of the ROI. For example, the terminal controller 27 may use adaboost through the PCA to extract the other party's (object) image from the background of the received image and to extract and track the ROI from the extracted other party's image.

When the avatar controller 27 displays the avatar on the display unit 25, the terminal controller 27 may overlap the received image data or display the picture in a picture in picture (PIP) form.

As described above, the video call terminals 20 and 40 of the emotion transmitting system 100 according to the first embodiment select an avatar corresponding to the specific gesture when the other party takes a specific gesture in the image data received from the counterpart terminal during the video call. It can be output along with the received image data to convey the emotional state of the other party to the user. That is, the video call terminals 20 and 40 may effectively transmit the emotional state of the other party to the user by outputting the avatar according to the gestures of the other party as well as the other party's video and voice.

The video call terminals 20 and 40 may perform a video call while outputting an avatar corresponding to a specific gesture of the other party, and the user may directly set and use an avatar to be used during the video call. While satisfying various needs, it is possible to expand the use of video calls by inducing interest of the video callers. That is, since the user can directly set the appearance or behavior of the avatar corresponding to at least one specific gesture of the other party through the video call terminals 20 and 40, the user can use the video call more interestingly while pursuing the user's personality. It can provide an environment.

An example of an emotion delivery method of the emotion delivery system 100 during a video call according to the first embodiment will be described with reference to FIGS. 1 to 3. 3 is a flowchart illustrating an example of a method for transmitting emotion during a video call according to the first embodiment of the present invention.

First, when the calling terminal 20 transmits a video call in step S51, the communication network 10 receives the request in step S53 and requests a video call to the called terminal 40. That is, when the calling terminal 20 sends a video call, the call controller 14 requests the subscriber information (location information and additional service information) of the calling and destination terminals 20 and 40 to the location register 16. The call controller 14 obtains routing information based on the received subscriber information. The call controller 14 requests video call reception to the called terminal 40 using the obtained routing information.

Meanwhile, before performing step S51, the calling terminal 20 may set an avatar corresponding to the specific gesture of the other party in the avatar setting mode according to the user's selection signal through the input unit 22. In the avatar setting mode, the user may select and set an avatar of a field that the user or the other party likes or is interested in.

Next, when the called terminal 40 transmits the acceptance of the video call to the communication network 10 in step S55, the communication network 10 transmits the video for the video call between the calling terminal 20 and the called terminal 40 in step S57. Set the call path. That is, the call controller 14 requests the incoming call by paging the called terminal 40, and when the called terminal 40 accepts the call for the incoming request, the call controller 14 calls the calling terminal 20 and the called terminal ( 40) to set the video call path.

As such, after the video call path is set between the calling terminal 20 and the called terminal 40, the calling terminal 20 and the called terminal 40 perform a video call through the set video call path.

Next, in operation S59, the calling terminal 20 determines whether an avatar providing request signal is input from the user. In this case, the user may input an avatar providing request signal through the input unit 22.

If it is determined that the determination in step S59 is not input, the calling terminal 20 continuously performs a video call according to step S57.

When the determination result of step S59 is input, in steps S61 to S65, the calling terminal 20 monitors whether the user of the called terminal 20, that is, the counterpart, makes a specific gesture. The monitoring step is described in detail as follows.

In step S61, the calling terminal 20 extracts the other party's video from the image data received from the called terminal 40. The received image data includes a plurality of frame image data, and captures a predetermined number of image data from the plurality of frame image data included in the image data received at a predetermined time, and extracts the counterpart image from the captured image data. have.

Subsequently, in step S63, the calling terminal 20 recognizes the gesture of the other party from the extracted other party's image. In this case, the calling terminal 20 may recognize the gesture taken by the counterpart from the extracted counterpart image in the following manner. That is, the calling terminal 20 extracts and tracks a region of interest according to the gesture of the counterpart from the extracted counterpart image. The calling terminal 20 recognizes a gesture of the other party using the tracked ROI. For example, the terminal controller 27 may use adaboost through the PCA to extract the other party's (object) image from the background of the received image and to extract and track the ROI from the extracted other party's image.

Next, in step S65, the calling terminal 20 determines whether the recognized gesture is a specific gesture stored in the storage unit 23. For example, the calling terminal 20 may perform step S65 by comparing the region of interest included in the recognized gesture with the region of interest of the stored specific gesture or not.

If the result of the determination in step S65 is not a stored specific gesture, the calling terminal 20 performs again from step S61.

In the case of a specific gesture stored as a result of the determination in step S65, in operation S67, the calling terminal 20 extracts an avatar corresponding to the specific gesture from the storage unit 23. For example, when the user sets the golf avatar and the calling terminal 20 recognizes the other person's smile as a specific gesture, the avatar extracting the joy is extracted from the golf avatar.

In operation S69, the calling terminal 20 outputs the extracted avatar through the output units 25 and 26. At this time, the calling terminal 20 outputs the avatar together with the received image data. For example, when the extracted avatar is an avatar expressing joy in the golf avatar, the calling terminal 20 outputs a favorite figure in which the ball put by the avatar enters the hole cup and delivers a pleasant emotional state of the other party to the user. In this case, when the avatar is displayed on the display unit 25, the avatar may be displayed by being superimposed on the received image data or displayed in a PIP form.

Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar. For example, when the other person shakes his head from side to side, it is a gesture that is taken when expressing a negative emotional state such as rejection or dislike. When the calling terminal 20 recognizes the specific gesture of the counterpart according to the negative emotional state, the negative terminal expresses the negative emotional state through the facial expression or behavior of the avatar and transmits it to the user.

Meanwhile, in the above description, an example in which the calling terminal 20 transmits an avatar providing request signal is disclosed, but the calling terminal 20 or the called terminal 40 may send an avatar providing request signal.

Another example of an emotion delivery method of the emotion delivery system 100 during a video call according to the first embodiment will be described with reference to FIGS. 1, 2, and 4 as follows. 4 is a flowchart illustrating another example of a method for transmitting emotion during a video call according to the first embodiment of the present invention.

First, since steps S51 to S61 proceed in the same manner as the example of the emotion transmitting method of FIG. 3, a detailed description thereof will be omitted and the following description will be made after extracting the counterpart image from the received image according to step S61. .

Next, in step S62, the calling terminal 20 extracts and tracks a region of interest according to the gesture taken by the counterpart from the extracted counterpart image.

Next, in step S64, the calling terminal 20 deforms the avatar to correspond to the tracked ROI. That is, the calling terminal 20 calls the avatar set by the user from the storage 23. Subsequently, the calling terminal 20 deforms the region of interest of the avatar corresponding to the region of interest tracked to correspond to the region of interest tracked. For example, when the other party makes a heart-shaped gesture by using both arms or hands, the avatar providing server 30 performs a heart-shaped gesture by using both arms or hands of the avatar in the same manner as the other party's gesture. Transform.

Here, the avatar is an avatar set by the user of the calling terminal 20. The user may set the avatar differently according to the user of the called terminal 40.

In operation S66, the calling terminal 20 outputs the modified avatar to the output units 25 and 26. At this time, the calling terminal 20 outputs the avatar together with the received image data. For example, when the avatar is displayed on the display unit 25, the avatar may be displayed by being superimposed on the received image data or displayed in a PIP form.

Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar.

Meanwhile, in the above description, an example in which the calling terminal 20 transmits an avatar providing request signal is disclosed, but the calling terminal 20 or the called terminal 40 may send an avatar providing request signal.

In the video call according to the first embodiment, the emotion delivery system 100 outputs an avatar corresponding to a gesture of the other party to the screen by the video call terminals 20 and 40 according to a user's request for inputting an avatar providing request signal. An example has been disclosed but is not limited thereto. For example, the video call terminals 20 and 40 may extract the avatar according to the gesture of the other party and output the output to the output units 25 and 26 even when the user does not input the avatar providing request signal according to the user's setting.

Meanwhile, the emotion transmitting system 100 during the video call according to the first embodiment discloses an example in which the calling terminal 20 or the called terminal 40 outputs an avatar corresponding to a gesture of the other party, but is not limited thereto. That is, as shown in FIG. 5, the avatar may be received and output through the avatar providing server 30 at the request of the calling terminal 20 or the called terminal 40.

In the video call emotional delivery system 200 according to the second embodiment of the present invention, as shown in FIG. 5, the originating terminal 20, the avatar providing server 30, and the incoming call are connected via the communication network 10. It is configured to include a terminal 40.

The calling terminal 20 and the called terminal 40 perform a video call with each other via the communication network 10. When the avatar providing request signal is input from the calling terminal 20 or the called terminal 40, the terminal which sends the avatar providing request signal transmits the avatar providing request signal to the avatar providing server 30 through the communication network 10.

The communication network 10 performs a series of data transmission / reception operations for data transmission and information exchange between the calling terminal 20, the avatar providing server 30, and the called terminal 40. In particular, when the communication network 10 receives the avatar providing request signal from the calling terminal 20 or the called terminal 40 in the video call, the session providing server 30 and the avatar providing server 30 between the avatar providing server 30 and the calling terminal 20 are received. And establish a session between the called terminal 40. That is, when receiving a video call from the calling terminal 20, the communication network 10 sets the video call path for the video call between the calling terminal 20 and the called terminal 30. When the communication network 10 receives an avatar providing request signal from the calling terminal 20 or the called terminal 40 during a video call, the communication network 10 determines a video call path established between the calling terminal 20 and the called terminal 40. It releases and sets the session between the avatar providing server 30 and the calling terminal 20 and the session between the avatar providing server 30 and the called terminal 40.

The avatar providing server 30 communicates with the calling terminal 20 and the called terminal 30 via the communication network 10. When the avatar providing server 30 receives the avatar providing request signal from the calling terminal 20 or the called terminal 40, the avatar providing server 30 sends an avatar corresponding to a specific gesture taken by the other party of the terminal sending the avatar providing request signal to the video call traffic. Load it to the user terminal. Meanwhile, the avatar may be transmitted to the user terminal through a message service channel such as a short message service (SMS) or a multimedia message service (MMS) in addition to the currently set video call path.

The user terminal displays the counterpart image including the received avatar. In this case, when the user terminal displays the image data and the avatar on the display unit 25, the user terminal may display the image data and the avatar in a superimposed manner or in a PIP form. Therefore, the user can know the emotional state of the other party through the received avatar.

In particular, the avatar providing server 30 according to the second exemplary embodiment includes a transceiver 31, a database 33, and a controller 35.

The transceiver 31 communicates with the calling terminal 20 and the called terminal 40 via the communication network 10. The transceiver 31 receives the avatar providing request signal from the calling terminal 20 or the called terminal 40 under the control of the controller 35. The transmission / reception unit 31 loads the avatar corresponding to the specific gesture taken by the counterpart of the terminal that has sent the avatar providing request signal on the video call traffic to the user terminal under the control of the control unit 35.

The database unit 33 stores the avatar corresponding to the specific gesture of the video call terminal including the calling terminal 20 or the called terminal 40. In addition, the database unit 33 may store the avatar set by the user of the video call terminals 20 and 40. At this time, the database unit 33 matches the terminal information of the other party and stores the avatar.

In addition, the controller 35 performs an overall control operation of the avatar providing server 30. In particular, the controller 35 controls the transmission of the avatar to the user terminal through gesture recognition of the other party during the video call.

In the avatar setting mode, the controller 35 may set an avatar corresponding to a specific gesture of the counterpart according to a user's selection signal through the video call terminals 20 and 40. In this case, since the setting of the avatar by the control unit 35 may be performed in the same manner as the setting method by the terminal control unit 23 of FIG. 2, a detailed description thereof will be omitted.

The controller 35 analyzes the image data received from the counterpart terminal during the video call, monitors whether the counterpart takes a specific gesture, and controls the transmission of the avatar corresponding to the specific gesture taken by the counterpart during monitoring. In this case, the controller 35 may transmit the avatar in the following manner when transmitting the avatar through gesture recognition of the counterpart of the terminal that has sent the avatar providing request signal during the video call.

First, the control unit 35 establishes a session with the calling terminal 20 and the called terminal 40 by the communication network 10 according to the avatar providing request signal of the calling terminal 20 or the called terminal 40 during the video call. . The controller 35 receives image data including the other party's image from the other party's terminal of the terminal which has sent the avatar providing request signal, and analyzes the received image data to monitor whether the other party's specific gesture is performed. The controller 35 extracts an avatar corresponding to the specific gesture from the database unit 33 when the counterpart takes a specific gesture during monitoring. The controller 35 transmits the video call traffic including the extracted avatar to the user terminal.

In this case, the avatar operates in a set manner of action corresponding to a specific gesture taken by the other party. Different avatars may be set to correspond to a specific gesture, or a different behavior may be set for the same avatar.

Next, the control unit 35 sets up a session with the calling terminal 20 and the called terminal 40 by the communication network 10 according to the avatar providing request signal of the calling terminal 20 or the called terminal 40 during the video call. do. The controller 35 receives image data including an image of the counterpart photographed from the counterpart terminal of the terminal that has sent the avatar providing request signal, analyzes the image data, and extracts and tracks a region of interest according to a gesture taken by the counterpart. The controller 35 transforms the avatar to correspond to the tracked ROI. The controller 35 transmits video call traffic including the modified avatar to the user terminal. At this time, the control unit 35 calls the avatar set by the user in the database unit 33 and deforms the called avatar to operate in an action manner corresponding to the gesture taken by the counterpart.

In this case, the controller 35 may track the ROI as follows. That is, the controller 35 extracts the counterpart image from the received captured image. The controller 35 extracts a region of interest from the extracted counterpart image. The controller 35 tracks the change of the extracted ROI. For example, the controller 35 may use adaboost through the PCA to extract the counterpart (object) image from the background of the received image, and to extract and track the ROI from the extracted counterpart image. The region of interest may be selected from a counterpart image capable of transmitting the emotional state of the counterpart in a video call to the user, and includes at least one of a face, two hands, both arms, and both feet of the counterpart.

Meanwhile, in the above description, the avatar providing server 30 discloses an example of receiving the other party's image including the avatar requested by the calling terminal 20, but the other party's image including the avatar requested by the calling terminal 20 or the called terminal 40. Can be received.

An example of an emotion delivery method of the emotion delivery system 200 during a video call according to the second embodiment will be described with reference to FIGS. 5 and 6. 6 is a flowchart illustrating an example of a method for transmitting emotion during a video call according to a second embodiment of the present invention.

First, since steps S151 to S157 proceed in the same manner as steps S51 to S57 according to the example of the emotional delivery method of FIG. 3, a detailed description thereof will be omitted and will be described below from step S159 performed after step S157. Meanwhile, before performing step S151, the calling terminal 20 may set an avatar corresponding to the specific gesture of the other party in the avatar setting mode according to the user's selection signal through the input unit 22. In the avatar setting mode, the user may select and set an avatar of a field that the user or the other party likes or is interested in.

Next, in operation S159, the calling terminal 20 determines whether an avatar providing request signal is input from the user. In this case, the user may input an avatar providing request signal through the input unit 22.

If it is not input as a result of determination in step S159, the calling terminal 20 continues to perform the video call according to step S157.

When the determination result of step S159 is input, in step S161 to step S175 performs a step of monitoring whether or not the user of the called terminal 40 takes a specific gesture.

That is, in step S161, the calling terminal 20 transmits an avatar providing request signal to the communication network 10. In operation S 163, the communication network 10 transmits the received avatar providing request signal to the avatar providing server 30. At the same time in steps S165 and S167 the communication network 10 establishes a session between the avatar providing server 30 and the calling terminal 20 and a session between the avatar providing server 30 and the called terminal 40. At this time, the communication network 10 releases the video call path set in step S157. The originating terminal 20 and the terminating terminal 40 are each established with the avatar providing server 30.

Next, the receiver terminal 40 transmits the captured image data to the avatar providing server 30 in step S169.

Next, in step S171, the avatar providing server 30 extracts the counterpart image from the received image. At this time, the received image data includes a plurality of frame image data, and the avatar providing server 30 captures a predetermined number of image data from the plurality of frame image data included in the image data received at a predetermined time, and captures the captured image. The other party's image may be extracted from the data.

Subsequently, in step S173, the avatar providing server 30 recognizes a gesture taken by the counterpart from the extracted counterpart image.

In this case, the avatar providing server 30 may recognize the gesture taken by the counterpart from the extracted counterpart image in the following manner. That is, the avatar providing server 30 extracts and tracks a region of interest according to a gesture taken by the counterpart from the extracted counterpart image. The avatar providing server 30 recognizes a gesture of the other party using the tracked ROI. At this time, the avatar providing server 30 may use the adaboost through the PCA to extract the other party (object) image from the background of the received image, and to extract and track the region of interest from the extracted other party image.

In operation S175, the avatar providing server 30 determines whether the recognized gesture is a specific gesture stored in the database unit 33. For example, the avatar providing server 30 may compare the region of interest included in the recognized gesture with the region of interest of the stored specific gesture and perform step S175 to determine whether there is a specific gesture that matches.

If the result of the determination in step S175 is not a stored specific gesture, the avatar providing server 30 performs again from step S171.

In the case of a specific gesture stored as a result of the determination in step S175, in step S177, the avatar providing server 30 extracts an avatar corresponding to the specific gesture from the database unit 33. For example, when the user sets the golf avatar, when the avatar providing server 30 recognizes the other's smile as a specific gesture, the avatar providing server 20 extracts an avatar expressing joy from the golf avatar.

Next, in step S179, the avatar providing server 30 transmits video call traffic including the extracted avatar to the calling terminal 20.

In operation S181, the calling terminal 20 displays the counterpart image including the received avatar on the screen. For example, when the received avatar is an avatar expressing joy in the golf avatar, the calling terminal 20 outputs a favorite figure in which the ball put by the avatar enters the hole cup and transmits a pleasant emotional state of the other party to the user. In this case, when the avatar is displayed on the screen, the avatar may be overlapped with the counterpart image or may be displayed in a PIP form. Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar.

Meanwhile, in the above description, although the avatar providing server 30 transmits the counterpart image including the avatar to the calling terminal 20 at the request of the calling terminal 20, the calling terminal 20 or the called terminal 40 is disclosed. ), The avatar providing server 30 may transmit the counterpart image including the avatar to the user terminal.

Another example of an emotion delivery method of the emotion delivery system 200 during a video call according to the second embodiment will be described with reference to FIGS. 5 and 7. 7 is a flowchart illustrating another example of a method for transmitting emotion during a video call according to the second embodiment of the present invention.

First, since steps S151 to S171 proceed in the same manner as the example according to the emotion transmitting method of FIG. 6, a detailed description thereof will be omitted and described from step S172 performed after step S171 as follows. Meanwhile, before performing step S151, the calling terminal 20 may set an avatar corresponding to the specific gesture of the other party in the avatar setting mode according to the user's selection signal through the input unit 22. In the avatar setting mode, the user may select and set an avatar of a field that the user or the other party likes or is interested in.

Next, in step S172, the avatar providing server 30 extracts and tracks a region of interest according to a gesture taken by the counterpart from the extracted counterpart image. At this time, the avatar providing server 30 may use the adaboost through the PCA to extract the other party (object) image from the background of the received image, and to extract and track the region of interest from the extracted other party image.

Next, in step S174, the avatar providing server 30 deforms the avatar corresponding to the tracked ROI. That is, the avatar providing server 30 calls the avatar set by the user from the database unit 33. Subsequently, the avatar providing server 30 transforms the region of interest of the avatar corresponding to the region of interest tracked to correspond to the region of interest tracked. For example, when the other party makes a heart-shaped gesture by using both arms or hands, the avatar providing server 30 performs a heart-shaped gesture by using both arms or hands of the avatar in the same manner as the other party's gesture. Transform.

Here, the avatar is an avatar set by the user of the calling terminal 20. The user may set the avatar differently according to the user of the called terminal 40.

Next, the avatar providing server 30 transmits the video call traffic including the modified avatar to the calling terminal 20 in step S176. Meanwhile, the avatar providing server 30 may generate a separate data service channel in addition to the currently set video call path and transmit the modified avatar to the calling terminal 20.

In operation S178, the calling terminal 20 displays the counterpart image including the received avatar on the screen. For example, when the avatar is displayed on the screen, the avatar may be superimposed on the counterpart image or displayed in a PIP form. Therefore, the user of the calling terminal 20 can easily know the emotional state of the user of the called terminal 40 through the output avatar.

Meanwhile, in the above description, although the avatar providing server 30 transmits the counterpart image including the avatar to the calling terminal 20 at the request of the calling terminal 20, the calling terminal 20 or the called terminal 40 is disclosed. ), The avatar providing server may transmit the other party's image including the avatar to the user terminal.

On the other hand, the embodiments of the present invention disclosed in the specification and drawings are merely presented specific examples to aid understanding, and are not intended to limit the scope of the present invention. In addition to the embodiments disclosed herein, it is apparent to those skilled in the art that other modifications based on the technical idea of the present invention may be implemented.

The present invention relates to a system and method for transmitting emotions during a video call through gesture recognition, wherein the video call terminal analyzes the image data received from the counterpart terminal during the video call and outputs an avatar corresponding to the detected gesture of the counterpart. Since the user can directly set and use the avatar to be used in the video call, the user can expand the use of the video call by inducing the interest of the video call while satisfying various needs of the video caller. Since the user can directly set the appearance or behavior of the avatar corresponding to at least one gesture of the other party through the video call terminal, the user can provide an environment in which the video call can be used more interestingly while pursuing the user's personality.

1 is a block diagram illustrating an emotion transmission system during a video call according to a first embodiment of the present invention.

2 is a block diagram illustrating a configuration of a video call terminal of FIG. 1.

3 is a flowchart illustrating an example of a method for transmitting emotion during a video call according to the first embodiment of the present invention.

4 is a flowchart illustrating another example of a method for transmitting emotion during a video call according to the first embodiment of the present invention.

5 is a block diagram illustrating an emotion transmission system during a video call according to a second embodiment of the present invention.

6 is a flowchart illustrating an example of a method for transmitting emotion during a video call according to a second embodiment of the present invention.

7 is a flowchart illustrating another example of a method for transmitting emotion during a video call according to the second embodiment of the present invention.

Description of the Related Art [0002]

10: communication network

20: calling terminal

30: avatar providing server

40: called terminal

100, 200: emotional delivery system

Claims (23)

A communication network configured to establish a session between the avatar providing server and the calling terminal and a session between the avatar providing server and the called terminal when receiving an avatar providing request signal from a calling terminal or a called terminal in a video call; Receives the image data of the counterpart photographed from the counterpart terminal of the terminal that has sent the avatar providing request signal, analyzes the received video data to detect a gesture taken by the counterpart, and stores an avatar corresponding to the detected gesture. The avatar providing server extracting from the unit and transmitting the video call traffic including the extracted avatar to a terminal that has sent the avatar providing request signal; Emotion delivery system during a video call through gesture recognition, comprising a. The method of claim 1, wherein the communication network, Receiving a video call originated from the originating terminal, sets a video call path between the originating terminal and the called terminal, and when the avatar providing request signal is received, releases the video call route and sessions between the avatar providing server and the originating terminal; Emotion delivery system during a video call through the gesture recognition, characterized in that for setting the session between the avatar providing server and the called terminal. A transceiver for performing communication with a communication network, a calling terminal and a called terminal; A database unit for storing an avatar corresponding to a gesture of the calling terminal or the called terminal user; In response to the avatar providing request signal of the calling terminal or the called terminal during a video call, the communication network establishes a session with the calling terminal and the called terminal, and the other party photographed from the counterpart terminal of the terminal sending the avatar providing request signal. Receives image data, analyzes the received image data, detects a gesture taken by the counterpart, extracts an avatar corresponding to the detected gesture from a database unit, and extracts video call traffic including the extracted avatar from the avatar. A control unit which transmits a provision request signal to a terminal that has sent the provision request signal; Avatar providing server of the emotion transmission system during a video call through gesture recognition, comprising a. A communication unit for performing a video call with a counterpart terminal through a communication network; A camera unit for photographing a user during a video call; An output unit for outputting video and audio data of the other party received through the communication unit; A storage unit which stores an avatar corresponding to the gesture of the counterpart; Analyzing image data received from the counterpart terminal through the communication unit to detect a gesture taken by the counterpart, extracting an avatar corresponding to the detected gesture from the storage unit, and outputting the extracted avatar to the output unit. Terminal control unit; The video call terminal of the emotion delivery system of the video call through gesture recognition comprising a. A session setting step of establishing, by an avatar providing server, a session with the calling terminal and the called terminal by a communication network according to an avatar providing request signal of a calling terminal or a called terminal in a video call; A reception step of receiving, by the avatar providing server, image data of the other party photographed from the other party's terminal of the terminal which has sent the avatar providing request signal; An extraction step of the avatar providing server analyzing the received image data to detect a gesture taken by the counterpart and extracting an avatar corresponding to the detected gesture from a database unit; Transmitting, by the avatar providing server, video call traffic including the extracted avatar to a terminal that has sent the avatar providing request signal; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 5, wherein the avatar providing server, And an avatar corresponding to at least one specific gesture taken by the counterpart in the database unit. The method of claim 5, wherein the extracting step, A monitoring step of the avatar providing server analyzing the received image data to monitor whether the counterpart makes a specific gesture; If the avatar providing server detects a specific gesture taken by the counterpart during the monitoring, extracting an avatar corresponding to the detected specific gesture from the database unit; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 7, wherein the monitoring step, Extracting, by the avatar providing server, the counterpart image from the received image data; Recognizing, by the avatar providing server, a gesture taken by the counterpart by extracting and tracking a region of interest from the extracted counterpart image; Determining, by the avatar providing server, whether the recognized gesture is included in the stored specific gesture; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 8, wherein the avatar providing server, Emotion transfer method during a video call through gesture recognition, characterized in that for storing the avatar set by the calling terminal or the called terminal user in the database unit. The method of claim 5, wherein the extracting step, A tracking step of the avatar providing server analyzing the received image data and extracting and tracking a region of interest according to a gesture of the counterpart; And modifying, by the avatar providing server, an avatar set by a terminal user who has transmitted the avatar providing request signal in a database unit, and deforming the called avatar to correspond to the tracked specific area. In the transmitting step, the avatar providing server transmits the video call traffic including the modified avatar to the terminal that sent the avatar providing request signal, the emotion transmission method during the video call through gesture recognition. The method of claim 10, wherein the tracking step, Extracting, by the avatar providing server, the counterpart image from the received image data; Extracting, by the avatar providing server, a region of interest from the extracted counterpart image; Tracking, by the avatar providing server, the extracted change of the ROI; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 5, wherein the step is performed before the session establishment step, Selecting and setting, by the avatar providing server, an avatar of a field that the user or the other party likes or is interested in in the avatar setting mode; Emotional transmission method during a video call through gesture recognition further comprising a. Receiving a video data from the counterpart terminal by a user terminal in a video call with the counterpart terminal via a communication network; An extraction step of detecting, by the user terminal, the gesture taken by the counterpart by analyzing the received image data, and extracting an avatar corresponding to the detected gesture from the storage unit; An output step of the user terminal outputting the extracted avatar through an output unit; Emotion transfer method during a video call through gesture recognition, comprising a. The method according to claim 13, which is performed before the receiving step, A storage step of setting, by the user terminal, an avatar corresponding to at least one specific gesture taken by the counterpart and storing the avatar in the storage unit; Emotional transmission method during a video call through gesture recognition further comprising a. The method of claim 14, wherein in the storing step: And the set avatar operates in a behavioral manner corresponding to a specific gesture taken by the counterpart. The method of claim 13, wherein the extracting step, A monitoring step of monitoring, by the user terminal, whether the counterpart makes a specific gesture in the received image data; Extracting, from the storage unit, an avatar corresponding to the detected specific gesture by the user terminal when the specific gesture of the counterpart is detected during the monitoring; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 16, wherein the monitoring step, Extracting, by the user terminal, the counterpart image from the received image data; Recognizing, by the user terminal, a gesture taken by the counterpart by extracting and tracking a region of interest from the extracted counterpart image; Determining, by the user terminal, whether the recognized gesture is included in the stored specific gesture; Emotion transfer method during a video call through gesture recognition, comprising a. The method of claim 13, wherein the extracting step, A tracking step of the user terminal extracting and tracking a region of interest according to a gesture of the counterpart from the received image data; And modifying, by the user terminal, an avatar set by the counterpart in the storage unit, and modifying the called avatar to correspond to the tracked specific area. The emotion transmission method of the video call through the gesture recognition, characterized in that for outputting the modified avatar the user terminal in the output step. The method of claim 18, wherein the tracking step, Extracting, by the user terminal, the counterpart image from the received image data; Extracting, by the user terminal, a region of interest from the extracted counterpart image; Tracking, by the user terminal, a change of the extracted ROI; Emotion transfer method during a video call through gesture recognition, comprising a. The method according to claim 13, which is performed before the receiving step, Selecting and setting, by the user terminal, an avatar of a field that the user or the other party likes or is interested in in an avatar setting mode; Emotional transmission method during a video call through gesture recognition further comprising a. The method according to any one of claims 5 to 20, The avatar is a graphic content that represents the emotion state of the other party, and includes one of an animated character, a video, a still image, a user created content (UCC), an emoticon, a flash, and haptic content in which a video and vibration are combined. Emotion transfer method during a video call through gesture recognition, characterized in that. The method according to any one of claims 8 to 11, 17 to 19, The region of interest includes at least one of the user's face, both hands, both arms, and both feet included in the extracted user's image. The method according to any one of claims 6 to 11 and 14 to 17, The specific gesture may include at least one of a facial expression, a hand gesture, a head gesture, an arm gesture, a leg gesture, and a combination thereof related to the emotional state of the counterpart.

Images