KR20120095193A - Face recognition service system, method and apparatus for face recognition service thereof of - Google Patents

Abstract

PURPOSE: A face recognition service system, a face recognition service apparatus thereof, and a method thereof are provided to recognize a face in an image obtained through a video and to extract a feature point from the recognized face. CONSTITUTION: An image which a terminal device(100) obtains is transmitted to a face image recognition service unit(200). The terminal device receives a face recognition service according to function execution corresponding to an expression recognition result. The face image recognition service unit extracts a face area from the image. The face recognition service unit extracts feature points of a face from a normalized face area. The face recognition service unit expresses an expression variation value of the face.

Description

Face recognition service system, face recognition service apparatus and method in the system {face recognition service system, method and apparatus for face recognition service about of}

The present invention relates to a face recognition service system, and more particularly, a face recognition service system for recognizing an expression according to a change of a face by analyzing an image obtained by video recording, and thereby providing a service based on a change of expression, An apparatus and method for face recognition service in a system are provided.

In general, face recognition technology is a system that automatically identifies a user through a digital image by comparing the facial features extracted from the image with the facial database. Such face recognition technology has been mainly used as a security system.

The face recognition technology may be classified into a technology for recognizing the same person and a technology for analyzing the facial expression by comparing feature value values of the face. Facial expressions have various meanings, and these facial expressions are one of the important factors for communication. Recently, a technique for analyzing such facial expressions is being developed.

With the recent expansion of smartphones and web cams, it is expected that as image services are expanded, services for face or facial expression recognition will be provided even in video shooting rather than still images.

Therefore, there is a need for a technology capable of recognizing facial expression changes by analyzing facial images through video shooting that can be used in smart phones.

In addition, customers are expected to be provided with a change in facial expressions with various entertainment elements in connection with related services.

Accordingly, the present invention is proposed to solve the conventional inconvenience, to provide a face recognition service system, a face recognition service apparatus and method for recognizing a change in facial expression by analyzing a face image obtained through video recording. .

In addition, the present invention provides a facial recognition service system for providing a related service based on facial expression change by recognizing facial expression change values by analyzing facial images obtained through video shooting in a facial recognition service system, apparatus and method for face recognition service thereof. It is to provide.

The present invention is a means for solving the problem, a face recognition service according to the execution of a function to obtain a captured image through video recording and transmit it to the face recognition service device, and corresponding to the facial expression recognition result obtained from the captured image from the face recognition service device Terminal device for receiving and outputting to the user; And extracting and normalizing a facial region from the captured image, extracting feature points of the face from the normalized face region, expressing facial expression change values by tracking the extracted feature points, and recognizing facial expressions according to facial expression changes. A face recognition service system comprising a face recognition service device that executes a function corresponding to a recognition result and provides a face recognition service to a terminal device according to the execution of the function.

The face recognition service system according to the present invention may further include a template for preset face recognition information used for feature point extraction, and a content providing device for storing contents for executing a face recognition service.

In addition, the present invention provides a means for solving the above-mentioned problems, communication unit for receiving a captured image captured by a moving image from the terminal device while transmitting and receiving data with the terminal device, and transmits the face recognition service data to the terminal device; And recognizing the face by normalizing the face region in the captured image, extracting feature points from the recognized face pattern, tracking the extracted feature points, and recognizing facial expressions according to facial expression changes, and responding to facial expression recognition results or facial expression recognition results. It provides a face recognition service apparatus comprising a face recognition unit for providing a face recognition service to the terminal device according to the execution of the function.

The apparatus for face recognition service according to the present invention may further include a storage unit for storing data necessary for providing a face recognition service and a result of facial expression recognition.

In the face recognition service apparatus according to the present invention, the face recognition unit generates a normalized image based on a face region from a captured image, extracts a facial point from the generated normalized image, and extracts feature points from the extracted face pattern. Extraction unit; And an expression recognizing unit that tracks the feature points extracted from the face pattern and recognizes the expression according to the facial expression change.

In the face recognition service apparatus according to the present invention, the feature point extractor comprises: a face recognition core including a library and an algorithm for normalizing a face region in a captured image, recognizing a face, and processing mathematical calculations and images for face recognition; An engine module; And a feature point extraction module for extracting feature points from the face pattern.

In the facial recognition service apparatus according to the present invention, the facial expression recognition unit tracks the movement of the feature points extracted from the face pattern using at least one of a recognition library, a face recognition library, a feature point extraction library, and a library defining an application program interface. A face recognition core engine unit for obtaining a motion vector; And an expression recognition module that recognizes an expression according to a change of a face by using a feature point motion vector.

In the facial recognition service apparatus according to the present invention, the facial expression recognition unit expresses expression change values according to facial expression changes by applying a hidden Markov model using the feature point motion vector obtained by tracking the extracted feature points, The median value of the expression change value is detected statistically.

In addition, the present invention is another means for solving the above problems, Communication unit for transmitting and receiving data through the network; Transmitting the captured image captured by the video to the face recognition service device through the communication unit, and receives a face recognition service that executes a function corresponding to the facial expression recognition result recognized in the captured image from the face recognition service device to output to the user Control unit; And an output unit configured to output an execution result of the face recognition service to the user.

The terminal device according to the present invention may further include a capturing unit configured to execute capturing a video and provide a captured image obtained through capturing a video under the control of the controller.

The terminal device according to the present invention may further include a storage unit for storing data required for face recognition service and facial expression recognition results.

In the terminal device according to the present invention, the control unit includes a face recognition unit for recognizing a face according to a facial expression change by recognizing a face in the captured image, extracting a feature point from the recognized face area, and tracking the movement of the extracted feature point; And a service execution unit that provides a face recognition service to a user through execution of a function corresponding to a facial expression recognition result.

The terminal device according to the present invention stores one or more of a database storing preset content information on a function corresponding to a facial expression recognition result, a database storing template information used to extract feature points from a face region, and a statistical database. The storage unit may further include.

In addition, the present invention as another means for solving the above problems, the step of receiving a moving image captured image; Recognizing a face by extracting a face region from the captured image; Extracting feature points from the recognized face region; Recognizing an expression according to a face change by tracking the extracted feature points; Executing a function corresponding to a facial expression recognition result obtained through facial expression recognition; And providing a face recognition service to the terminal device according to the execution of the function.

In the face recognition service method according to the present invention, the step of recognizing a face comprises: extracting a face region from an image; Generating a normalized image in the extracted face region; And extracting a face pattern from the generated normalized image to recognize the face.

In the face recognition service method according to the present invention, extracting feature points from the recognized face region comprises: receiving a template for face recognition information preset from a content providing apparatus; Matching the received template to a face pattern extracted from the face area; And extracting feature points from the face area according to the matching result.

In the face recognition service method according to the present invention, the step of recognizing the expression according to the change of the face comprises: obtaining a feature point motion vector by tracking a change according to the movement of the extracted feature point; Expressing facial expression change values using the feature point motion vector as a parameter; And recognizing an expression using facial expression change values.

In the facial recognition service method according to the present invention, the step of recognizing the facial expression according to the change of the face may further include a step of recognizing the facial expression by statistically detecting an intermediate value of the facial expression change values.

In addition, the present invention provides a means for solving the above-mentioned problems, the method comprising the steps of obtaining a captured image through a moving image captured image; Transmitting a photographed image to a face recognition service device and receiving a facial expression recognition result according to a change in facial expression recognized by the photographed image; Executing a function corresponding to a facial expression recognition result; And providing a face recognition service to a user according to the execution of the function.

According to an aspect of the present invention, there is provided a face recognition service method comprising: recognizing a face by extracting a face region from a captured image; Extracting feature points from the recognized face region; And recognizing an expression according to a face change by tracking the extracted feature points.

The present invention can recognize a facial expression change by recognizing a face from an image acquired through video shooting, extracting a feature point from the recognized face, and tracking the movement of the extracted feature point, and applying the facial recognition result to various contents. Therefore, there is an effect that can be extended to provide a variety of services related to face recognition.

In particular, the present invention can provide a user with various application services including avatar messaging and multimedia MMS service by applying facial expression recognition results, thereby improving service differentiation and improving competitiveness.

1 is a block diagram illustrating a configuration of a face recognition service system according to an exemplary embodiment of the present invention.
2 is a block diagram illustrating a configuration of a terminal device in a face recognition service system according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating a configuration of a face recognition service apparatus in a face recognition service system according to an exemplary embodiment of the present invention.
4 is a block diagram illustrating a configuration of a feature point extracting unit in a face recognition unit of a face service apparatus according to an exemplary embodiment of the present invention.
5 is a block diagram showing the configuration of the facial expression recognition unit in the face recognition unit of the face service apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating a face recognition service process in the face recognition service system according to an exemplary embodiment.
7 is a flowchart illustrating a face recognition service method in a terminal device according to another embodiment of the present invention.
FIG. 8 is a diagram illustrating an aadb cascade method applied for face recognition according to an exemplary embodiment of the present invention.
9 is a diagram illustrating a method of extracting feature points according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, detailed description of well-known functions or constructions that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that like elements are denoted by the same reference numerals as much as possible throughout the drawings.

Then, the present invention analyzes an image (or image) obtained through video recording, recognizes a facial expression, and provides an embodiment of the present invention for providing a face recognition service for executing a set function according to the recognized facial expression recognition result. It will be described in detail with reference to.

1 is a block diagram illustrating a configuration of a face recognition service system according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the face recognition service system according to the present invention may include a terminal device 100, a face recognition service device 200, and a content providing device 300 connected through a network 10.

In the face recognition service system according to an exemplary embodiment, the terminal device 100 photographs a user's face and transmits an image including the photographed face to the face recognition service device 200. In addition, the terminal device 100 may extract the feature points of the photographed face and then transmit the extracted feature points to the face recognition service device 200. In this case, since the feature point information has a smaller amount of data than the image itself, the communication load at the time of transmission can be reduced, and since the face recognition service device 200 does not have to perform processing according to feature point extraction, the face recognition service device 200 is provided. Can reduce the operating load. The terminal device 100 may receive a specific service or function provided from the content providing device 300 according to the face recognition result of the face recognition service device 200 and output the received specific service or function to the user. The terminal device 100 may correspond to a desktop computer, a notebook computer, a mobile phone, a smart phone, a tablet PC, a PDA, and a PMP. In addition, the terminal device 100 may include all user devices that are currently developed and commercialized or may be developed in the future.

The face recognition service apparatus 200 receives feature point information of a face image or a captured image transmitted from the terminal apparatus 100, recognizes facial expressions of a user and a face, and provides the recognized facial expression recognition result to the terminal apparatus 100. Or, the corresponding function is executed according to the facial expression recognition result, and the service according to the execution of the function is provided to the terminal device 100. In particular, the facial recognition service apparatus 200 extracts a facial pattern by normalizing the face from the captured image, extracts a feature point of the face from the face pattern, and tracks the movement of the extracted feature point for facial expression recognition. By recognizing the change, it executes the function set according to the expression change and provides a face recognition service.

The content providing device 300 interworks with the terminal device 100 and the face recognition service device 200 through a repeater (web gateway), and preset facial expression recognition information for facial expression recognition in order to improve the performance of the face recognition service. It may be configured to include a template database for managing the content, a content database for providing a predetermined content after facial expression recognition, and a statistics database. The preset content may include a sound source download, an avatar action notification, a phone call, an e-mail, a multi-media message service (MMS), a tweeter, and the like. Various content that may provide a face recognition service by applying the may be included.

On the other hand, the network 10 may be any type as long as it is a network used for data transmission. For example, a wired / wireless personal area network (PAN), a local area network (LAN), a mobile communication network, and Wi- Any kind of network such as Fi network can be used.

Meanwhile, in another embodiment of the present disclosure, the face recognition service may be performed only by the operation of the terminal device 100 without the intervention of the face recognition service device 200. That is, the terminal device 100 includes a program for feature point extraction and facial expression recognition processing, extracts feature point information from an image photographed through the program, and recognizes a face expression change of a user based on the extracted face point information. The corresponding service may be provided by executing a corresponding function according to the facial expression recognition result.

2 is a block diagram illustrating a configuration of a terminal device in a face recognition service system according to an exemplary embodiment.

Referring to FIG. 2, the terminal device 100 may include a photographing unit 110, a communication unit 120, an input unit 130, a controller 140, and a storage unit 150.

The photographing unit 110 captures a video including a face of the photographing target according to a user input. The photographing unit 110 may be embedded in the terminal device 100. For example, an external photographing device such as a camera, a camcorder, a web cam, or the like may be connected to the terminal device 100.

The communication unit 120 transmits and receives data to and from the face recognition service apparatus 200 and the content providing apparatus 300 through various communication schemes as well as wired and wireless schemes. The extracted feature point information is transmitted, and the face recognition service apparatus 200 receives a face recognition service provided by executing a function corresponding to a facial expression recognition result and a facial expression recognition result of the face. In addition, the communication unit 120 receives the content of the preset function from the content providing device 300 according to the facial expression recognition result.

The input unit 130 generates a user input signal corresponding to a user's request or information according to a user's manipulation. In the present invention, for example, the user input signal may include one or more of a video photographing input signal and a face recognition service related input signal. The input unit 130 may be implemented by various input means that are currently commercialized or may be commercialized in the future. For example, a user's motion as well as a general input device such as a keyboard, a mouse, a joystick, a touch screen, a touch pad, and the like may be used. It may include a gesture input means for detecting and generating a specific input signal.

The controller 140 controls overall operations of the terminal device 100, and may include one or more of the face recognition unit 141 and the service execution unit 142 for the face recognition service according to the present invention. have.

The face recognition unit 141 recognizes a face region in the captured image, extracts feature points based on a face shape and a preset template from the recognized face region, and tracks the movement of the extracted feature points to change the facial expression. Perform facial expression recognition.

The service execution unit 142 executes a corresponding preset function according to the facial expression recognition result and provides a face recognition service thereto. This may be achieved by receiving a face recognition service according to the execution of a preset function corresponding to each facial expression recognition result from the face recognition service apparatus 200, and executing the provided face recognition service to the user. The service execution unit 142 may receive the facial expression recognition result from the face recognition unit 141 or the facial recognition service device 200 through the communication unit 120.

The storage unit 150 stores data necessary for face recognition service and facial expression recognition results, and includes photographed image information, facial recognition information, feature point extraction and tracking information (feature point motion vector), and facial expression recognition result information (expression recognition change value). ) Store one or more of The storage unit 150 is an optical recording medium such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, a compact disk read only memory (CD-ROM), and a digital video disk (DVD). It may be interpreted as a concept including a magneto-optical media such as a floppy disk, a ROM, a random access memory (RAM), and a flash memory.

Next, a configuration of the face recognition service apparatus in the face recognition service system according to an exemplary embodiment of the present invention will be described with reference to the accompanying drawings.

3 is a block diagram illustrating a configuration of a face recognition service apparatus in a face recognition service system according to another exemplary embodiment.

Referring to FIG. 3, the face recognition service apparatus 200 may include a communication unit 210, a face recognition unit 220, and a storage unit 230.

The communication unit 210 transmits and receives data to and from the terminal device 100 and the content providing device 300 through the network 10. The communication unit 210 may transmit and receive data through various communication methods as well as wired and wireless methods. More specifically, the communication unit 210 receives a video captured by the terminal device 100, recognizes a facial expression change in the received video, and executes a preset function (content) according to the recognized facial expression recognition result. The facial recognition service is provided to the terminal device 100. In addition, the communication unit 210 may receive content about a preset function corresponding to a facial expression recognition result from the content providing apparatus 300.

The face recognition unit 220 extracts a face region from a video received through the communication unit 210, extracts feature point information from the extracted face region, and then tracks the feature point information to recognize a change in facial expression. Extracts feature point information from the face region through normalization using the Adaboost Cascade and recognizes facial expression changes by tracking the change of the extracted feature point information. To this end, the face recognition unit 220 may include a feature point extractor 221 and an expression recognition unit 222.

As shown in FIG. 4, the feature point extractor 221 includes a face recognition core engine module for recognizing a face region and a feature point extraction module for detecting a feature point of a face, wherein the program is made of Java. Uses the Native Development Kit (NDK), an application development interface (e.g., Java Native Interface (JNI)) and development tools designed to access native code that can only run on that platform. It may include. In addition, the feature point extractor 221 extracts the feature point by applying a Principal Component Analysis (PCA) technique based on the shape of the face and the template information provided by the content providing device 300. That is, feature points are extracted by comparing the face shape and template information. In addition, the feature point extractor 221 estimates the movement of the feature point using an optical flow in order to recognize a change in facial expression based on a video. That is, since the position value of the face changes during video shooting, the feature point tracking result information is transmitted to the facial expression recognition unit 222 to be applied to facial expression recognition in consideration of the change value.

The face recognition core engine module of the feature point extractor 221 may include an image normalization module, a face recognition module, a numerical calculation library, an image processing library, and an image algorithm library. Here, the image normalization module generates a normalized image based on the face region in the captured image, and the face recognition module recognizes the face in the captured image. The numerical calculation library is a library for processing mathematical calculations for face recognition, and the image processing library is a library for image processing and image filtering.

The facial expression recognition unit 222 includes a facial recognition core engine unit and a facial expression recognition module, as shown in FIG. 5, which is an application program made to access native code executable only on a corresponding platform. Interfaces (e.g., JNI) and development tools NDK for application developers.

The face recognition core engine module of the facial expression recognition unit 222 recognizes a face in the captured image and tracks a change of a feature point in the recognized face. To this end, the face recognition core engine unit may include an optical floor library, a recognition library, an open computer vision (CV) library, a face recognition library, and a feature point extraction library. The optical floor library is a module that implements the optical floor for tracking the change of feature points on the face, and the face recognition library is a library for face recognition, feature extraction, and expression recognition (PCA / SVM (Support Vector Machine) / HMM (Hidden). Open CV library is a library that recognizes a face by comparing the eigenvector with the vector of the face to be recognized, and defines an integrated application program interface (API). The face recognition library recognizes a face from an input image or an image, and the feature point extraction library extracts feature point information from the recognized face. Here, the face recognition core engine unit may receive a preset template from the content providing apparatus 300 and extract feature point information by applying a PCA technique based on the preset template.

The facial expression recognition module of the facial expression recognition unit 222 recognizes a facial expression by using the feature point change value obtained by tracking the change of the feature point in the face recognition core engine module. In this case, the facial expression recognition module expresses facial expression change values by applying AAM (Active Appearance Model) and HMM (hidden Markov model) using the feature point motion vectors as parameters, and statistically detects the median of each change value Can increase the recognition rate.

The storage unit 230 stores data and facial expression recognition results necessary for providing a facial recognition service, and includes photographed image information, facial recognition information, feature point extraction and tracking information (feature point motion vector), and facial expression recognition result information (expression recognition change). Value). The storage unit 230 may include an optical media such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, a compact disk read only memory (CD-ROM), and a digital video disk (DVD). Magnetic-optical media such as floppy disks, and ROM, random access memory (RAM), and flash memory.

The face recognition service apparatus 200 and the content providing apparatus 300 configured as described above may be implemented as one or more servers operating based on cloud computing. Here, cloud computing refers to on-demand virtualized IT resources, for example, hardware (server, storage, network, etc.), software (database, security, web server, etc.), services, data, etc. using Internet technology. ) Means technology that serves in a way.

That is, the face recognition service apparatus 200 may provide a computing resource including one or more of hardware, software, services, and data necessary for face recognition, feature point extraction and tracking, facial expression recognition, and face recognition service.

Next, a method for providing a face recognition service by recognizing a facial expression in the face recognition service system according to the present invention will be described in detail with reference to the accompanying drawings.

6 is a flowchart illustrating a process of providing a face recognition service by recognizing a facial expression in a face recognition service system according to an exemplary embodiment of the present invention.

Referring to FIG. 6, the terminal device 100 obtains a captured image including a face of a user through video recording (S101). After that, when the terminal device 100 receives the captured image request message from the face recognition service apparatus 200 (S102), the terminal device 100 transmits a captured image message including the obtained captured image information to the face recognition service apparatus 200 (S103). ).

The face recognition service apparatus 200 extracts a face region from the received captured image and normalizes the extracted face region to recognize a face (S104). More specifically, as shown in the accompanying FIG. 8, the subwindow may be processed through several steps using the Adaboost Cascade method, in which several weak classifiers are linearly combined to form a single strong classifier. Finally, the face pattern is extracted while removing the sub window. In operation S105, the face recognition service apparatus 200 transmits a template information request message for requesting corresponding template information to be compared with a face shape to the content providing apparatus 300. Accordingly, the content providing device 300 transmits a template information message including the template information to the face recognition service apparatus 200 based on the information included in the received template information request message (S106).

The face recognition service apparatus 200 extracts a feature point by comparing the recognized face shape with the received template information, and tracks the movement of the feature point to recognize an expression according to a change in facial expression (S107). More specifically, as shown in FIG. 9, the feature point b of the face may be extracted by matching the received template information to the face shape a in the face pattern by applying the PCA technique. In addition, the movement of the extracted feature point is tracked using an optical flow to recognize a change in facial expression on a moving image basis. In other words, since the position value of the face changes in video shooting, it should be applied to facial expression recognition in consideration of the change value. The facial recognition service apparatus 200 recognizes an expression according to a facial expression change by applying AAM and HMM (hidden Markov model) using the feature point motion tracking result, that is, the feature point motion vector as a parameter. For example, the face recognition service apparatus 200 may express a change value, and statistically detect each change value intermediate value to increase a recognition rate of facial expression recognition. Here, the D value recognized as a dummy value has no meaning, and is automatically induced to a nearby value. That is, the statistical recognition rate can be increased by representing the dummy value. Such facial recognition can be recognized as surprise, ordinary, pleasure, anger, and the like.

Thereafter, when the face recognition service apparatus 200 receives the facial expression recognition result request message from the terminal apparatus 100 (S108), the facial recognition service apparatus 200 transmits the facial expression recognition result message including the facial expression recognition result information to the terminal apparatus 100 (S109). .

Accordingly, the terminal device 100 may provide a face recognition service to the user by executing a function corresponding to the facial expression recognition result and executing the face recognition service accordingly (S110).

In contrast, the terminal device 100 may receive and execute a face recognition service according to a function executed in the face recognition service device 200.

That is, after recognizing the facial expression in step S107, the facial recognition service apparatus 200 requests the content information for the function, that is, the content information request including the facial expression recognition result information, to execute a function corresponding to the facial expression recognition result. The message is transmitted to the content providing device 300 (S111). Accordingly, the content providing device 300 checks the facial expression recognition result information included in the content information request message and transmits the corresponding content information to the face recognition service device 200 by including the corresponding content information in the content information message (S112).

Then, the face recognition service apparatus 200 receives a content providing message (S112) and executes a function for the corresponding content (S113). For example, when a facial expression is recognized as a surprise expression, the MMS is executed by receiving an MMS indicating a content corresponding to the surprise, for example, a character or a shape of the surprise. In this case, new contents may be added to the executed MMS and executed. Accordingly, the face recognition service device 200 transmits the executed function (for example, MMS) to the terminal device 100 as a face recognition service (S114).

Then, the terminal device 100 receives the received face recognition service message and provides a face recognition service corresponding to the user (S115).

Meanwhile, in another embodiment of the present invention, the face recognition service according to the present invention may be performed only by the operation of the terminal device 100.

7 is a flowchart illustrating a method for providing a face recognition service by recognizing a facial expression in a terminal device according to an exemplary embodiment of the present invention.

Referring to FIG. 7, the terminal device 100 captures an image (or an image) including a face of a user according to a user input signal (S205).

The terminal device 100 normalizes the face area in the captured image (S210), extracts a face pattern from the normalized face image, and recognizes the face (S215). That is, as shown in FIG. 8, the terminal device 100 performs a plurality of steps using an Adaboost Cascade method, in which a plurality of weak classifiers are linearly combined to form one strong classifier. Finally, the face pattern is extracted while removing the sub window.

Thereafter, the terminal device 100 extracts a feature point from the extracted face pattern (S220). To this end, the terminal device 100 receives preset template information from the content providing device 300 and, as shown in FIG. 9, is applied to the face shape a in the face pattern by applying the PCA technique. The feature point b of the face may be extracted by matching the template information.

Then, the terminal device 100 tracks the movement of the extracted feature point using the optical flow in order to recognize the change in the expression on the basis of the video (S225). In other words, since the position value of the face changes in video shooting, it should be applied to facial expression recognition in consideration of the change value.

In operation S230, the terminal device 100 applies the AAM and the HMM (hidden Markov model) using the feature point motion tracking result, that is, the feature point motion vector as a parameter (S230). In this case, the terminal device 100 may express the change value of the feature point, and statistically detect each change value intermediate value to increase the recognition rate of facial expression recognition. Here, the D value recognized as a dummy value is meaningless and is automatically induced to a nearby value. That is, the statistical recognition rate can be increased by representing the dummy value. Such facial recognition can be recognized as surprise, ordinary, pleasure, anger, and the like.

Then, the terminal device 100 stores the facial expression recognition result, and extracts and executes a preset function corresponding to the facial expression recognition result (S235 and S240). In this case, the facial expression recognition result may be transmitted from the content providing apparatus 300 to receive a corresponding function, or the facial recognition service apparatus 200 may receive a facial expression recognition result to receive a service corresponding to the facial expression recognition result. .

Through the above-described method, the terminal device 100 may recognize a face in the captured image, detect a feature point of the recognized face, and perform facial expression recognition according to a change of face. By executing the function, it is possible to provide a face recognition service to the user.

In the above-described process, one or more of steps S210 to S230 may be executed through the face recognition service device 200 without directly executing the step in the terminal device 100. For example, the terminal device 100 recognizes a face area, extracts a feature point, transmits the feature point to the face recognition service device 200, and then outputs an expression recognition result based on tracking of the feature point information from the face recognition service device 200. The face recognition service method according to the present invention may be implemented in software form readable through various computer means and recorded on a computer readable recording medium. Here, the recording medium may include program commands, data files, data structures, and the like, alone or in combination. Program instructions recorded on the recording medium may be those specially designed and constructed for the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. For example, the recording medium may be magnetic media such as hard disks, floppy disks and magnetic tapes, optical disks such as Compact Disk Read Only Memory (CD-ROM), digital video disks (DVD), Magnetic-Optical Media, such as floppy disks, and hardware devices specially configured to store and execute program instructions, such as ROM, random access memory (RAM), flash memory, and the like. do. Examples of program instructions may include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be apparent to those skilled in the art. In addition, although specific terms are used in the specification and the drawings, they are only used in a general sense to easily explain the technical contents of the present invention and to help the understanding of the present invention, and are not intended to limit the scope of the present invention.

The present invention can be applied to a technique for face recognition, and recognizes a face according to a change in facial expression by recognizing a face from an image acquired through video recording, extracting feature points from the recognized face, and tracking feature point movements. As a result, the present invention may be extended to various services related to face recognition by applying to various contents through facial expression recognition results.

In addition, the present invention can apply various facial expression services, such as avatar messaging, multimedia MMS service by applying the facial expression recognition result, it is possible to promote the competitiveness and differentiation of the service.

100: terminal device 110: photographing unit 120: communication unit
130: input unit 140: control unit 150: storage unit
141: face recognition unit 142: service execution unit 200: face recognition service device
210: communication unit 220: face recognition unit 230: storage unit
221: feature point extractor 222: facial expression recognition unit 300: content providing device
310: repeater

Claims (20)

A terminal device which acquires a captured image through video recording and transmits the captured image to a face recognition service device, and receives and outputs a face recognition service according to a function execution corresponding to a facial expression recognition result obtained from the captured image from the face recognition service device to a user; And
The facial region is extracted from the captured image and normalized, the facial feature points are extracted from the normalized facial region, and the extracted feature points are traced to express facial expression change values, thereby recognizing facial expressions according to facial expression changes and facial expressions. A face recognition service device which executes a function corresponding to a recognition result and provides a face recognition service to the terminal device according to the execution of the function;
Face recognition system comprising a. The method of claim 1,
A content providing apparatus for storing a template for preset face recognition information used for extracting the feature point and content for executing a face recognition service;
Face recognition service system further comprises. A communication unit which receives a captured image captured by a moving image from the terminal device while transmitting and receiving data with the terminal device, and transmits face recognition service data to the terminal device; And
Recognize a face by normalizing a face region in the captured image, extract a feature point from a recognized face pattern, track a extracted feature point, recognize an expression according to a change in facial expression, and respond to an expression recognition result or the expression recognition result A face recognition unit providing a face recognition service to a terminal device according to a function execution;
Face recognition service apparatus comprising a. The method of claim 3,
A storage unit which stores data necessary for providing the face recognition service and a result of facial expression recognition;
Face recognition service device further comprises. The apparatus of claim 3, wherein the face recognition unit
A feature point extracting unit to generate a normalized image based on the face region in the captured image, and extract a feature point from the extracted face pattern by extracting a face pattern from the generated normalized image; And
An expression recognition unit recognizing an expression according to a facial expression change by tracking a feature point extracted from the face pattern;
Face recognition service apparatus comprising a. The method of claim 5, wherein the feature point extraction unit
A face recognition core engine module including a library and an algorithm for normalizing a face region in the captured image, recognizing a face, and processing mathematical calculations and images for face recognition; And
And a feature point extraction module for extracting feature points from the face pattern. The method of claim 5, wherein the facial expression recognition unit
A face recognition core engine unit which obtains a feature point motion vector by tracking the movement of the feature point extracted from the face pattern using at least one of a recognition library, a face recognition library, a feature point extraction library, and a library defining an application program interface; And
And a facial recognition module for recognizing an expression according to a change of face using the feature point motion vector. The method of claim 5, wherein the facial expression recognition unit
By using the feature mark motion vector obtained by tracking the extracted feature points as a parameter, a hidden Markov model is applied to express facial expression change values according to facial expression changes, and the median value of each facial expression change value is statistically detected. Face recognition service device. A communication unit for transmitting and receiving data through a network;
Transmitting the captured image captured by the video to the face recognition service device through the communication unit, and receives a face recognition service for executing a function corresponding to the facial expression recognition result recognized from the captured image from the face recognition service device to output to the user A control unit for controlling the control unit to perform the control; And
And an output unit for outputting an execution result of the face recognition service to a user. 10. The method of claim 9,
According to the control of the control unit, the image capturing unit for executing a video recording, and provides a captured image obtained through the video recording;
Terminal device characterized in that it further comprises. 10. The method of claim 9,
A storage unit which stores data necessary for the face recognition service and a result of facial expression recognition;
Terminal device characterized in that it further comprises. 10. The apparatus of claim 9, wherein the control unit
A face recognition unit recognizing a face in the captured image, extracting feature points from the recognized face region, and recognizing an expression according to a change in facial expression by tracking a movement of the extracted feature points; And
And at least one of a service execution unit which provides a face recognition service to a user through execution of a function corresponding to a facial expression recognition result. 10. The method of claim 9,
And a storage unit storing one or more of a database storing preset content information on a function corresponding to the facial expression recognition result, a database storing template information used to extract feature points from the face region, and a statistical database. Terminal device characterized in that. Receiving a moving image photographed image;
Recognizing a face by extracting a face region from the captured image;
Extracting feature points from the recognized face region;
Recognizing an expression according to a face change by tracking the extracted feature points;
Executing a function corresponding to a facial expression recognition result obtained through facial expression recognition; And
And providing a face recognition service to a terminal device according to the execution of the function. The method of claim 14, wherein recognizing the face comprises:
Extracting a face region from the image;
Generating a normalized image from the extracted face region; And
Recognizing a face by extracting a face pattern from the generated normalized image;
Face recognition service method in a face recognition apparatus comprising a. The method of claim 15, wherein the extracting the feature points from the recognized face area comprises:
Receiving a template for face recognition information preset from a content providing device;
Matching the received template to a face pattern extracted from the face area; And
Extracting feature points from the face area according to the matching result;
Face recognition service method in a face recognition apparatus comprising a. The method of claim 14, wherein the recognizing the facial expression according to the change of the face comprises:
Obtaining a feature point motion vector by tracking a change according to the movement of the extracted feature point;
Expressing facial expression change values using the feature point motion vector as a parameter; And
Recognizing an expression using the facial expression change values;
Face recognition service method in a face recognition apparatus comprising a. The method of claim 17, wherein recognizing the facial expression according to the change of the face comprises:
Recognizing an expression by statistically detecting a median value of the facial expression change values;
Face recognition service method in a face recognition apparatus further comprising a. Obtaining a captured image through a moving image captured image;
Transmitting the photographed image to a face recognition service device, and receiving a facial expression recognition result according to a facial expression change recognized in the photographed image;
Executing a function corresponding to a facial expression recognition result; And
Providing a face recognition service to a user according to the execution of the function;
Face recognition service method in a terminal device comprising a. 20. The method of claim 19,
Recognizing a face by extracting a face region from the captured image;
Extracting feature points from the recognized face region; And
And recognizing an expression according to a face change by tracking the extracted feature points.

Images