KR100886489B1 - Method and system for inserting special effects during conversation by visual telephone - Google Patents

Abstract

A method and a system for synthesis of facial expression in reference with a template to fabricate in a video call according to the gesture of the face are provided to synthesize effect image designated in a dictionary according to facial expression at real time. A screen decoration unit(110) is comprised of a face detector(110a), a face tracking tracker(110b) and a face expression recognition part(110c). The face detection part detects the face range of individual indicated in the video call through the screen of the digital device. The face range about individual is detected by the face detection part. The face tracking portion performs the tracking of the detected face range to the periodic or non-periodic. The face expression recognition part analyzes the face range detected by the face detection part or the face range traced with the face tracking portion. The face verification technology is applied to the face of the individual included in the face range.

Description

METHOOD AND SYSTEM FOR INSERTING SPECIAL EFFECTS DURING CONVERSATION BY VISUAL TELEPHONE}

The present invention relates to a method and system for automatically synthesizing and displaying a predetermined decorating effect image according to a facial expression of a person making a call when making a video call using a digital device, and specifically, using a digital device. When making a video call, facial expression, face tracking, face recognition, etc. are applied to the person displayed on the screen of the digital device to recognize the expression of the person. And if the expression of the person is identical to the expression corresponding to the previously recorded expression decoration template, and if the same is the same, the decorating effect image corresponding to the expression of the person is synthesized in real time with the image of the person in the video call. A method and system for helping to make a video call.

In recent years, with the development of communication technology, mobile video telephony service is getting full-scale, and the market such as Internet video telephony and video conferencing system is attracting attention. Is appearing.

However, when the user continues to see the other's face during a video call, the situation may be awkward, or the unfamiliar user may be unnatural due to awkwardness. Thus, by adding various functions away from the monotonous video call, the need to help to create a more natural and fun atmosphere during the video call is increasing.

Accordingly, an object of the present invention is to detect, track, and recognize a face of a person with respect to a person displayed through a screen of the digital device when solving a problem of the prior art and making a video call using a digital device. If the facial expression of the person is the same as or similar to the facial expression of the model included in the facial decoration template picture previously stored by the user, the decorative effect image recorded corresponding to the template picture is automatically synthesized on the face of the person. . According to this, a video call can be made while expressing the user's emotions in abundance in real time, thereby enabling a more intimate and interesting video call.

In order to achieve the object of the present invention as described above, and to perform the characteristic functions of the present invention described below, the characteristic configuration of the present invention is as follows.

According to an aspect of the present invention, the user's face and the decorating effect image according to the expression of the user of the digital device with reference to the facial expression decoration template DB stored so that the template and the decorating effect image corresponding to each facial expression of the model A method of automatically synthesizing and providing a method, the method comprising: (a) detecting a face of a user photographed through a lens of the digital device using face detection technology; Searching for a specific template including the same or similar facial expressions, and (c) synthesizing the decorating effect image corresponding to the specific template with the detected face image.

According to the present invention, the user's emotions can be expressed in a more humorous and rich manner by synthesizing a predetermined decorating effect image in real time according to the facial expressions of a person appearing on the screen during a video call, thereby making a more intimate and interesting video call. It can help to expand the base of the video call market.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is according to an embodiment of the present invention, when making a video call using a digital device such as a mobile phone, PC cam, according to the facial expressions of the person in the conversation and automatically synthesizes the image corresponding to the decorating effect displayed on the screen It is a block diagram of the whole system 100 for the following.

Hereinafter, an example in which the present invention is mainly applied to a video call will be described. However, the present invention may be similarly applied to a case of generating a video using a camcorder, a mobile phone, or the like, or a still image such as a photo. to be.

Referring to FIG. 1, the entire system 100 may include a screen decoration unit 110, a facial expression decoration template database 120, an interface unit 130, a communication unit 140, and a controller 150.

According to the present invention, the screen decoration unit 110, the facial expression decoration template database 120, the interface unit 130, the communication unit 140, at least some of them are included in a user terminal device such as a mobile phone or communicate with the user terminal device The screen decoration unit 110, the template database 120, the interface unit 130, and the communication unit 140 are all included in the user terminal device. Such program modules may be included in the user terminal device in the form of an operating system, an application program module, and other program modules, and may also be physically stored on various known storage devices. Also, such program modules may be stored in a remote storage device that can communicate with a user terminal device. Such program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform particular tasks or execute particular abstract data types, described below, in accordance with the present invention.

The screen decoration unit 110 may include a face detector 110a, a face tracking unit 110b, an expression recognition unit 110c, and the like. Here, the face detector 110a, the face tracking unit 110b, and the facial expression recognizer 110c detect the face of the person displayed through the screen of the digital device, and detect the face of the face, such as the angle, eyes, nose, and mouth. It is classified as a convenience in performing the function of determining the expression of the person with reference to the shape and size of each part, etc., but is not necessarily limited thereto.

The face detector 110a detects face areas of people displayed through a screen of a digital device during a video call.

When the face tracking unit 110b detects the face area of the person by the face detection unit 110a, the face tracking unit 110b tracks the detected face area at regular intervals or at non-periodic time intervals. can do.

In addition, the facial expression recognition unit 110c analyzes the face area detected by the face detection unit 110a and / or the face area tracked by the face tracking unit 110b, and analyzes a face of a person included in the face area. A face recognition technology is applied to detect a face expression of the person.

Before describing the function of the facial expression recognition unit 110c in more detail, the facial expression decoration template DB 120 referred to by the facial expression recognition unit 110c to perform the function will be described.

The facial expression decoration template DB 120 is recorded such that the photograph of the facial expression of the model and the decorative effect image designated for each facial expression correspond to each other. When the user makes a video call and has the same facial expression as that of the model included in the specific template among the templates stored in the facial expression decoration template DB 120, the user's face is displayed in real time with the specific decorating effect image recorded corresponding to the specific template. The image may be combined with the image and displayed on the screen of the other terminal. Here, the decorating effect image can be synthesized not only by the item overlay method as shown in FIGS. 3A and 3B, but also by applying filters to apply special treatments such as brightness change, saturation change, and posashi treatment, and enlargement of the face part. Of course, it can also be synthesized by a method such as / crop (crop).

Meanwhile, various DBs, such as the facial expression decoration template DB 120 mentioned in the present invention, include not only a database of consultation, but also a database of a broad meaning including a data record based on a file system, and may be included in the system 100. It may also be present in the remote storage device capable of communicating with the system 100.

When the facial expression recognition unit 110c determines whether the facial expression of the user is the same as a specific facial expression among the facial expressions corresponding to a plurality of facial expression decorative templates stored in the facial expression decorative template DB 120, the specific facial expression is determined. The decorating effect image specified for the facial expression is synthesized in real time to the face image of the user to be displayed on the screen of the counterpart terminal through the interface unit 130.

In addition, the interface unit 130 may show through the screen of the digital device so that the user can reference or edit the data of the facial expression decoration template DB 120 recorded so that the decorative effect image corresponding to each facial expression of the person, It plays a role of displaying various images on the screens of both parties in the video call.

The communication unit 140 is responsible for transmitting and receiving signals between each component module in the system 100 or transmitting and receiving data with various external devices.

The controller 150 according to the present invention performs a function of controlling the flow of data between the screen decoration unit 110, the facial expression decoration template DB 120, the interface unit 130, the communication unit 140. That is, the controller 150 according to the present invention controls the signals transmitted and received between each component through the communication unit 140, the screen decoration unit 110, the template DB 120, the interface unit 130, each unique Control to perform functions.

Hereinafter, the overall process will be described in more detail with reference to FIGS. 2 and 3.

2 is a diagram illustrating an example of a face detection and face tracking method.

Referring to FIG. 2, if a face of a person is detected while performing a video call using a digital device, the face may be periodically or aperiodically redetected while tracking the detected face. The facial expression technology may be applied to the face of the person included in the detected face area and / or the tracked face area to determine the expression of the person. The grasped facial expression is displayed on the counterpart's terminal screen by being combined with a decorating effect image corresponding to each facial expression, as shown in FIGS. 3A and 3B to be described later.

Specifically, referring to FIG. 2, for example, tracking of the face area is performed every second during a video call, and tracking is performed at the second, third, and fourth seconds of the face area detected in the first second. You can see that it is losing. Then, the face area can be detected again at 5 seconds, and the tracking is performed based on the detected area at 5 seconds at 6 seconds. However, it should be noted that examples of face detection and tracking are not limited thereto.

Meanwhile, various face recognition techniques may be applied to determine which template the facial expression of the person included in the detected face area or the tracked face area corresponds to. For example, a principal component analysis method may be used. , Linear discriminant analysis, etc. can be assumed. W. Zhao, R. Chellappa, A. Rosenfeld, PJ Written by Phillips and published in the 2003 ACM Computing Surveys, “Face Recognition: A Literature Survey” or W.S. An example might be considered by Yambor and published in “Analysis of PCA-based and Fisher Discriminant-Based Image Recognition Algorithms,” an article published in the 2000 Technical Report of the Colorado State University School of Computer Science.

Specifically, in performing facial expression recognition, with the feature data of the eye, nose, mouth, face contour, etc., the feature data of each facial expression already stored in the facial expression decoration template DB 120 is referred to. By performing the facial expression matching process through comparison, the facial recognition rate can be increased.

As a technique related to face matching with reference to the characteristics of the face, "Lucas-Kanade 20 Years On: A" by Baker, S. et al., Published in the 2004 International Journal of Computer Vision (IJCV) in 2004 "Unifying Framework." The paper describes a method for efficiently detecting eye position from an image including a face of a person using a template matching method. However, it is not limited to this example.

The face detection unit 110a may estimate the position of the nose and the mouth based on the position of the eye detected by the above technique, and the face tracking unit 110b may periodically or aperiodically detect each part of the estimated face. The expression recognition unit 110c may track and match the detected part of the eye, nose, mouth, and the like, which is detected or tracked with the image of the eye, nose, mouth, etc. included in the facial expression decoration template DB 130, etc. from time to time. You can.

Here, the search method of each part such as eyes, nose and mouth is written by PN Belhumeur et al. Like the search method of the face, and published in IEEE TRANSACTIONS ON PATTERN ALAYSIS AND MACHINE INTELLIGENCE in 1997, "Eigenfaces vs. Fisherfaces" "Recognition Using Class Specific Linear Projection".

The similarity of the entire facial expression may be obtained by using a weighted sum or the like from the similarity of each part of the facial expressions performed in the above manner. Here, the weight of each site can be determined based on the importance in human perception.

However, other methods other than the method of calculating the similarity by combining the similarity for each site are of course possible. For example, rather than calculating the similarity for each part of the face and combining them, the similarity data may be determined by extracting feature data for the entire face and comparing the feature data extracted for the entire face.

FIG. 3 is a diagram illustrating a specific template when a facial expression of a model included in a specific template and a user's facial expression are the same among a plurality of templates recorded in the facial expression decoration template DB 120 according to an embodiment of the present invention. Is a diagram showing an example of synthesizing the recording effect image recorded in correspondence with the real time to provide to the terminal screen of the other party.

Referring to the left screens of FIGS. 3A and 3B, a user interface is provided so that a user may select and store a decorating effect image so as to correspond to a template including a specific facial expression in advance. In detail, FIG. 3A illustrates a case in which a decorating effect image of 'sweeping' is selected and stored so as to correspond to a template including an awkward mouth expression, and FIG. 3B corresponds to a template including a brightly smiling face. The case where a decorating effect image of 'Shabang' is selected and stored is shown.

Subsequently, when the user makes a facial expression corresponding to a specific template among a plurality of templates stored through the above operation while making a video call with the counterpart, for example, when the user makes an awkward facial expression, the face detection unit 110a ), The face tracking unit 110b, and / or the facial expression recognition unit 110c determine whether the facial expression belongs to a range similar to the facial expression of the person included in the template, and the right side of FIG. 3A. As can be seen on the screen, the image of the decorating effect called 'sweeping' stored in correspondence with the specific template is synthesized and provided to the terminal screen of the counterpart. In addition, when the user makes a facial expression corresponding to another template among a plurality of templates while making a video call with the other party, for example, when the user smiles brightly, the face detector 110a, the face tracking unit 110b, and / or The facial expression recognition unit 110c determines whether the facial expression belongs to a range similar to the facial expression of the person included in the template among the plurality of templates, and as shown in the right screen of FIG. 3B, a decorating effect of 'shabang' The image is synthesized and provided to the other party's terminal screen.

As mentioned above, the above-described process may be similarly applied not only to the video call but also to the shooting of still images such as moving pictures and pictures. In particular, in the case of taking a still image such as a photograph, if the facial expression displayed on the screen falls within the same or similar range as the facial expression of the person included in the specific template, after combining the decorating effect image corresponding to the specific template You can also have the photo taken automatically.

Embodiments according to the present invention can be implemented in the form of program instructions that can be executed by various computer means can be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In the present invention as described above has been described by the specific embodiments, such as specific components and limited embodiments and drawings, but this is provided to help a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations are possible from these descriptions.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and all the things that are equivalent to or equivalent to the claims as well as the following claims will belong to the scope of the present invention. .

1 is a block diagram of an entire system 100 for automatically synthesizing a decorating effect image corresponding to a facial expression of a person in a conversation and displaying it on a screen according to an embodiment of the present invention.

2 is a diagram illustrating an example of a face detection and face tracking method.

FIG. 3 is a diagram illustrating a specific template when a facial expression of a model included in a specific template and a user's facial expression are the same among a plurality of templates recorded in the facial expression decoration template DB 120 according to an embodiment of the present invention. Is a diagram showing an example of synthesizing the recording effect image recorded in correspondence with the real time to provide to the terminal screen of the other party.

 <Explanation of symbols for the main parts of the drawings>

110: screen decoration

110a: face detection unit

110b: face tracking unit

110c: facial expression recognition unit

120: facial expression decoration template DB

130: interface unit

140: communication unit

150: control unit

Claims (18)

delete delete delete delete A method of automatically synthesizing the user's face and the decorating effect image according to the user's facial expression of the digital device by referring to the facial expression decoration template DB stored so that the template and the decorating effect image correspond to each facial expression of the model. In (a) detecting a face of the user photographed through a lens of the digital device by using a face detection technique and tracking the detected face by using a face tracking technique; (b) searching for a specific template in the template that includes a facial expression that is the same as or similar to the expression of the detected face or the tracked face, and (c) synthesizing the decorating effect image corresponding to the specific template to the image of the detected face, Step (c) comprises the step of synthesizing the decorating effect image corresponding to the specific template to the image of the tracked face, And the facial expression decoration template DB can edit the correspondence between the template and the decorating effect image by the user. The method of claim 5, The decorating effect image is synthesized by at least one of an item overlay (item overlay), brightness change, saturation change, applying a filter for applying a special treatment such as posh processing, enlargement or crop of the face portion Way. The method of claim 6, In step (b), And matching each part of the face of the model included in the template with each part of the face of the user. The method of claim 7, wherein In step (b), And searching the position, shape, and size of each part of the face of the model included in the template by comparing the position, shape, and size of each part of the face of the user. The method of claim 8, Each part of the face comprises at least one of eyes, nose and mouth. The method of claim 5, The digital device is a device capable of video calling. The method of claim 10, and displaying the synthesized image on a screen of a terminal of a counterpart who makes a video call with the user. The method of claim 11, The tracking is performed using the digital device until the video call is terminated. The method of claim 12, In the step (c), And the decorating effect image is synthesized in real time during the video call. The method of claim 5, The digital device is a device for photographing still images. The method of claim 14, In step (c), Synthesizing the decorating effect image and automatically generating the still image when the detected facial expression or the facial expression of the tracked face falls within the same or similar range as the facial expression of the model included in the template. Method comprising a. The method of claim 5, The digital device is a device for shooting a video. The method of claim 16, In the step (c), And the decorating effect image is synthesized in real time while generating the moving picture digital data. A computer-readable medium for recording a computer program for executing the method according to any one of claims 5 to 17.

Images