KR101957677B1 - System for learning based real time guidance through face recognition and the method thereof - Google Patents

Abstract

The present invention relates to a learning-type real-time guidance system and method for face recognition, and more particularly, to extracting and storing minutia information from a face image of a person to be managed, thereby anonymizing the personal information of a person to be stored and managed, The face information of the user who is photographed through the facial image of the user is compared with the stored minutia information of the management subject to recognize the face of the user so that the appearance of the management subject can be monitored in real time Type real time guidance system and a guidance method thereof.

Description

FIELD OF THE INVENTION [0001] The present invention relates to a learning-type real-

More particularly, the present invention relates to a learning-type real-time guidance system and method for recognizing personal information of a person to be stored and managed by extracting and storing minutia information from a face image of a person to be managed, At the same time, compares the minutia information extracted from the face image of the user photographed through the image photographing apparatus with the minutia information of the stored management object to recognize the face of the user so that the appearance of the managed object can be monitored in real time A learning-type real-time guidance system through recognition, and a guidance method thereof.

In general, where customers are selling to department stores, shops, convenience stores, hospitals or pharmacies, customer management must always be performed to provide the best service to customers.

Customer management can include in the customer management not only unusual customers, but also those who can commit a crime, such as customers who have been involved in riot, obstruction, or theft, including managing VIP customers. In other words, criminals such as robbers, thieves, and rapists who are officially under control can be included in the customer management subject.

In general, customer management is directly related to the sales to the sales office. Therefore, it recognizes the face of the management target including VIP customers, abnormal customers and criminals, grasps real-time access to the sales office, It is very important to prevent such acts as well as to provide optimal services for VIP customers and other excellent customers.

In order to solve such a problem, a technique of identifying the user's identity by comparing a facial image of a user entering and leaving a business office with a face image of a customer and a management subject stored in advance is developed and commercialized.

However, in the case of using the face image as in the related art, there are limitations in storing and managing faces of individuals due to the restriction of the personal information management method. In addition, there are many restrictions on the use of face images of anticipated criminals other than official offenders (for example, facial images of customers who have disturbed sales) for separate purposes (see the Korea Personal Information Protection Act).

In addition, since the conventional technique directly compares face images to identify a user's face, the face part is extracted from the image and compared with the previously stored face image, so that the user's face is identified and the identity is identified It takes a very long time and measures for customer management can not be performed immediately, and the calculation complexity required for the comparison is very high.

Therefore, according to the present invention, minutia information is extracted from facial images of a plurality of management subjects including an abnormal customer, an officially managed criminal and an excellent customer, and the minutiae information is grouped into one similar group according to the degree of similarity, The feature point information of the input face image of the user is similar to any of the similar groups among the plurality of similar groups that are grouped to compare the feature point information of the selected specific similar group and the feature point information of the user, A system and method are provided for instantly recognizing a face, confirming access to a designated management subject in real time, and notifying the manager of the entry in real time, so that appropriate measures can be taken immediately for the management subject.

Next, a brief description will be given of the prior arts that exist in the technical field of the present invention, and technical matters which the present invention intends to differentiate from the prior arts will be described.

Korean Patent Laid-Open Publication No. 1999-0050271 (May 05, 1999) discloses a method and apparatus for automatically searching for a criminal's face through face recognition. The method includes a database for storing feature information of a face including a photograph, Extracting feature information from an image, comparing the information with the information of the personality database to measure similarity, sorting and outputting the person information in the order of the weights of the measured similarities, and searching for information about a person And apparatus.

In the prior art, the feature point information of the face is extracted from the image input from the image photographing device, and the extracted feature point information is compared with the feature point information of the previously stored management object to recognize the input face. Similar technical features are described. However, in the prior art, there is a problem that time complexity and computational complexity are very high because a plurality of pieces of minutiae information are compared one by one and sorted and provided in order of degree of similarity. This means that you will instantly recognize those entering or leaving a particular business location, such as a sales office (such as a department store, a mart or clothing store) that sells the goods, or a sales office that provides a specific service (such as a hospital, restaurant or bank) There is a limit to enabling proper action to be performed in real time.

On the other hand, the present invention calculates similarities between minutia information acquired from facial images of a management target subject to be managed, such as criminals, abnormal customers, VIP customers, etc., and groups them into at least one group according to the calculated similarity, The number of feature point information to be subjected to the similarity calculation is remarkably reduced by judging to which group the feature vertex bone extracted from the user's image inputted through the image photographing apparatus belongs to which group, And the prior art is not described or suggested with respect to this technical construction of the present invention.

Korean Patent Registration No. 0904916 discloses a face recognition system and method for extracting a face region from an input facial image, and for each pixel of the extracted facial region, A facial recognition system for extracting feature information by multiplying the transformed image by a projection matrix and comparing the extracted feature information with feature information of a previously stored portrait image to recognize a face of the input image; ≪ / RTI >

The prior art is somewhat similar to the present invention in that feature information is extracted from a face image of a specific user and compared with a plurality of previously stored feature information to recognize a user's face. However, since the feature information stored for each individual is composed of a plurality of feature elements, the prior art that takes a long time to recognize the face of the user by matching both the feature information extracted from the face image of a specific user and the previously stored feature information There is a drawback that the computational complexity is very high.

On the other hand, according to the present invention, the similarity degree between the feature point information extracted from the facial image of the management subject is calculated, the feature point information is grouped according to the calculated degree of similarity, the representative feature point information for each group is generated and stored, Only the minutia information of the user's face extracted from the image and the generated minutia information are compared with each other to dramatically reduce the minutia information of the comparison target to recognize the user's face in real time and to perform the customer management for the user immediately .

However, according to the present invention, the prior art is limited to only recognizing the face of the user using only the feature information. As in the present invention, the feature point information of the face of the management subject is grouped according to the degree of similarity, The present invention and the prior art can easily identify the different technical configurations since the present invention does not provide a technical configuration for a method of promptly recognizing a user's face and thereby performing customer management promptly and efficiently have.

That is, although the above-mentioned prior arts disclose a configuration in which a user is recognized by simply comparing minutia information extracted from a user's facial image input from a radiographic apparatus and minutia information of a person image stored in a database, Or personal information about the person to be managed by extracting and storing only the minutia information from the facial image of the management subject collected from the image photographing apparatus and grouping them according to the minutia information according to the degree of similarity between the stored minutia information, Even when the user is judged whether the user is a management subject by using the minutia information of the face image of a specific user, minutia information about all previously stored face images is not considered However, It is possible to perform the similarity calculation only for the minutiae information of the user and the representative minutia information of each group to quickly determine whether or not the user is the management target so that the user can take appropriate measures in real time There is a clear difference from the technical structure of the present invention.

The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a system and a method for preventing unauthorized customers including aft, injury, assault, sex offense, It is possible to prevent the personal information exposed by the facial image from being distinguishable by extracting only the characteristic point information from the facial image including the superior customer and storing the extracted characteristic point information as numerical data, And to provide a system and method for recognizing a face of a user by comparing feature point information with the stored feature point information.

The present invention also relates to a method and apparatus for detecting abnormalities of a person to be managed including an abnormal customer such as aft, injury, assault, sex offense, and an abnormal customer having a career interruption, malicious customer, The feature point information of the specific user and the representative point information of each group are stored in advance in the process of grouping and storing a plurality of pieces of feature point information having a high similarity degree by calculating the similarity between the specific user and the representative point information, And performing similarity calculation only on the minutiae point information in the group selected by calculation so that the face recognition of the user can be performed at a high speed.

Another object of the present invention is to provide a system and method for more accurately recognizing a user's face by periodically regenerating or correcting representative minutia information for each group.

In addition, the present invention allows the minutia information that is not selected for the similarity comparison object to be removed in the database itself for a predetermined period of time, thereby reducing the number of minutia information to be subjected to the similarity calculation for the minutia information of the specific user, And a system and a method for enabling a user to recognize a face of a user.

Another object of the present invention is to provide a system and method for periodically updating minutia information for a management subject and storing the updated minutia information for the management target person so as to more accurately recognize the entrance and exit of the management target person so as to perform optimal customer management.

A learning-type real-time guidance system based on face recognition according to an embodiment of the present invention includes a feature point information extracting unit for extracting feature point information from a plurality of face images of a management subject and a user, And a user face recognizing unit for recognizing the face of the user in real time by calculating the similarity degree between the minutiae information of the extracted user and the minutia information of the management target person.

In addition, the grouping unit may calculate the similarities between the minutia information of the plurality of extracted subjects to be managed, group the minutia information according to the calculated similarity, or learn the minutia information of the to-be-managed person, Or a combination thereof, calculates a similarity to the pattern, and clusters the minutia information according to the calculated similarity to group the minutia information. .

The grouping unit may further include generating representative minutia information for each similar group, mapping the generated minutia information to the similar group, and storing the representative minutia information in a database, wherein the representative minutia information includes at least one of minutia information And when new or added minutiae information for the similar group is newly added or deleted, the new minutiae point information is replaced or corrected by new representative minutia information.

The user's face recognizing unit may calculate a similarity degree between the extracted minutiae information of the user and the representative minutiae information of the similar group, select a similar group for at least one representative minutia information according to the calculated similarity, The similarity degree between the minutiae point information of the management subject corresponding to the similar group and the minutiae point information of the user is calculated and the minutiae information of the management subject is selected according to the calculated similarity degree to recognize the face of the user.

In addition, the learning-type real-time guidance system based on face recognition may further include a notification unit that transmits a face image of the recognized user, a type of a management object to which the recognized user belongs, minutia information of the recognized user, Wherein the management object type includes a type of a criminal, an abnormal client type, or a type of superior client.

The learning-type real-time guidance system based on face recognition may automatically delete the management subject's minutia information in the database when the management subject's minutia information is not used for calculation of the similarity for recognizing the user's face for a predetermined period or more .

In addition, a learning-type real-time guidance method using face recognition according to an embodiment of the present invention includes a minutia information extraction step of extracting minutia information from a plurality of face images of a management subject and a user, a minutia information extraction step of extracting minutia information of the extracted minita- A grouping step of grouping the extracted feature points of the user into the groups, and a user face recognition step of recognizing the face of the user in real time by calculating a similarity degree between the minutiae information of the extracted user and the minutia information of the management object.

The grouping step may include grouping minutiae information according to the calculated degree of similarity, or learning minutia information of the to-be-managed person to calculate eye, nose, mouth, A jaw or a combination thereof, calculating a degree of similarity with respect to the pattern, and grouping the feature point information by clustering the feature point information according to the calculated degree of similarity .

Further, the grouping step may further include generating representative minutia information for each similar group, mapping the generated minutia information to the similar group, and storing the representative minutia information in a database, and the representative minutia information may include minutia information And when new or added minutia information for each similar group is newly added or deleted, it is replaced or corrected by new representative minutia information reflecting the minutia information.

Also, the user face recognizing step may include calculating similarity between the extracted feature point information of the user and representative feature point information of each similar group, selecting similar groups for at least one representative feature point information according to the calculated similarity, A similarity degree between the minutiae point information of the management subject corresponding to a similar group and the minutiae point information of the user is calculated and the minutiae information of the management subject is selected according to the calculated similarity to recognize the face of the user.

Also, the learning-type real-time guidance method using face recognition may include a notification step of transmitting the facial image of the recognized user, the type of the management object to which the recognized user belongs, the minutia information of the recognized user, Wherein the management subject type includes a type of a criminal, an abnormal customer type, or a type of superior customer.

The learning-based real-time method using face recognition is automatically deleted from the database when the management subject characteristic point information is not used for calculation of the similarity for recognizing a user's face for a predetermined period or more do.

The present invention relates to a learning-type real-time guidance system and method for recognizing a face of a specific user by extracting minutia information about a plurality of faces and applying a grouping technique to the extracted minutia information, It is possible to perform similarity calculation only for the minutia information in the selected group based on the minutia information about the face of the user without performing the similarity calculation for all minutia information in the process, The time can be remarkably reduced.

Also, according to the present invention, the representative minutiae information of each group is compared with the minutia information of the user to promptly determine whether the user belongs to the management subject, and notify the manager to promptly perform the customer management for the user .

FIG. 1 is a conceptual diagram for schematically explaining a learning-type real-time guidance system through face recognition and a guidance method thereof according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a method of grouping minutiae information of a management subject into a plurality of similar groups according to an embodiment of the present invention. Referring to FIG.
3 is a view for explaining a method of clustering minutiae information of a management subject to a plurality of similar clusters according to another embodiment of the present invention.
FIG. 4 is a block diagram illustrating a configuration of a guide device in a learning-type real-time guidance system using face recognition according to an embodiment of the present invention.
FIG. 5 is a flowchart illustrating a procedure of databaseing minutiae information of a management subject, which is reference data for recognizing a face of a user according to an embodiment of the present invention, and recognizing a user face using the minutiae information and transmitting a recognition result to the administrator terminal Describe the procedure in detail.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of a learning-type real-time guidance system using face recognition and a guidance method thereof according to the present invention will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements. Furthermore, specific structural and functional descriptions for embodiments of the present invention are presented for the purpose of describing an embodiment of the present invention only, and, unless otherwise defined, all terms used herein, including technical or scientific terms Have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as ideal or overly formal in the sense of the art unless explicitly defined herein .

FIG. 1 is a conceptual diagram for schematically explaining a learning-type real-time guidance system through face recognition and a guidance method thereof according to an embodiment of the present invention.

As shown in FIG. 1, the learning-based real-time guidance system 10 using facial recognition includes an abnormal client having a history of criminal activity including theft, injury, assault, sex crime, etc., and business interruptions, malicious customers, A real-time guidance device for learning based on face recognition for recognizing a face of a user based on minutia information about the face of the person to be managed, collecting minutia information about a face of a person to be managed including an excellent customer such as a VIP customer, At least one image capturing device 200 for capturing an image of a user, a database 300 for storing reference data serving as a reference for recognizing a user's face, a facial image for a criminal, A criminal database 400 for providing relevant information, an administrator terminal provided by the administrator, or a combination thereof.

In order to recognize a face of a user photographed from a plurality of image capturing apparatuses 200, a learning-type real-time guide apparatus (hereinafter, referred to as a guide apparatus) And collects the minutia information about the minutiae.

That is, the guide apparatus 100 extracts and stores minutia information including positions and shapes of eyes, noses, mouths, tits, etc., which are components of a face, from the face image of the management subject to be collected, Is used as reference data for recognizing a face of a specific user.

Further, the feature point information of the management subject is for recognizing the face of a specific user photographed by the image photographing apparatus 200, and the object is a face image of a criminal who has been disclosed, theft, Such as an abnormal customer or VIP customer who interferes with the user's face image.

Also, the minutia information about the face of the public criminal is extracted by detecting the face area from the image including the montage, photograph, image or a combination of the criminal's face collected from the criminal database 400.

In addition, the minutia information about the abnormal customer is detected by extracting the face area from the image photographed by the image photographing apparatus 200 that photographs the abnormal customer image, and the minutia information for the good customer such as the VIP customer is excellent The face region is extracted from the photographed image of the customer.

In addition, the face region can be detected using a method of segmenting the face by separating the background from the image such as the Ada boost algorithm (Algorithm).

Meanwhile, the AdaBoost algorithm is a typical machine learning method for detecting a region of interest (ROI) region, which is a region of interest, from a specific image and is a well-known technology, and thus a detailed description thereof will be omitted.

In addition, the minutia information for the facial image can be obtained by various methods for extracting minutia information from facial images such as a method using geometry information, a method using a predefined template, and a method using an appearance .

On the other hand, the method using the geometric information is performed by measuring the curvature of the iris, the eyes, the eyes, the eyes, and the like using the geometric information such as the relative position and size of the face element The method using the template extracts minutia information by defining a predetermined template, calculating an energy function, calculating a value of the template, and using the template having the minimum energy value as an optimal solution. In addition, the method using the outline is performed by performing two-dimensional pattern matching on each element (eg, eye, nose, mouth, etc.) of the face and using a Principal Component Analysis (PCA) technique and a mathematical operation using a Gabor filter, (Feature Vector), thereby extracting the minutia information. The techniques described above are general techniques for extracting feature point information (position, shape, etc. of eyes, nose, mouth, and the like) for a component of a face from a facial image, and a detailed description thereof will be omitted.

Also, the image of the criminal's face can be periodically collected from the criminal's database 400, and the image of the criminal's face stored in the criminal's database 400 uses the image disclosed to the public.

Also, the criminal database 400 is a repository for storing criminal information such as sexually assaultive officers, robbers, and thieves who are officially under control, and is capable of managing information on criminals such as police authorities, local governments, Lt; / RTI > database.

Also, the face image for the offender collected through the criminal database 400 includes a montage, a photograph, an image, or a combination thereof.

Also, the guiding device 100 segments the image of the criminal collected from the criminal database 400, the image of the abnormal customer and the superior customer photographed from the image photographing apparatus 200, and displays the image of the face part included in the corresponding image .

Further, the feature point information is extracted from the detected face image, and only the extracted feature point information is stored in the database 300. Since the feature point information is expressed by numerical data such as eyes, nose, mouth, jaw, etc., it is impossible to reproduce the face image by combining with other information only by the feature point information, impossible.

That is, the guide apparatus 100 does not store the face image of the individual but extracts and stores the minutia information from the face image, thereby anonymizing the face information of the individual with respect to the face image, To prevent exposure.

Also, the guide device 100 groups and stores the extracted minutia information together with similar minutia information, and the minutia information of each minutiae stored is mapped with a class label indicating minutia related information. The grouping will be described in detail with reference to FIG.

The minutiae point related information indicates which management target person represents the minutiae point information. That is, the minutia related information indicates the type of the management subject. For example, when the specific minutia information is a criminal, the type of the management target person may be a robbery, a rape or a thief. In the case of an abnormal client, Disturbance, theft, riot or intimidation. On the other hand, if the minutiae information is an excellent customer, the type of the person to be managed may be set to a level such as a VIP or an A class.

When the feature point information of a specific user is matched with the previously stored feature point information and the face of the user is recognized, the class label of the matching feature point information is confirmed to know the type of the management subject to which the user belongs.

Further, the guide device 100 acquires an image photographed in real time from the image photographing apparatus 200, and detects the user's face image from the acquired image. Also, the guide device 100 extracts feature point information of the user from the detected face image of the user to recognize the face of the user.

On the other hand, the image capturing apparatus 200 is preferably a security camera (for example, a closed circuit television (CCTV)) that captures a specific area (for example, in a business office) and transmits the captured image to the administrator. Or a mobile terminal provided by an employee of a sales office or the like. That is, the guide device 100 collects images of a user photographed from a mobile terminal equipped with a security camera patroling a specific area, a security manager responsible for security, or an employee of a business office, It can also be recognized.

Also, the guiding device 100 compares the minutiae information of the extracted user with the minutiae information of the grouped management subjects, which are stored and managed, and if they are within the similar range preset in the comparison result, The type of the management object to which the user belongs, the minutia information of the user, or a combination thereof, to the manager so that immediate action can be taken.

On the other hand, the manager confirms the facial image and the type of the management subject transmitted from the guide apparatus 100, and performs an active correspondence to the facial image and the type of the management subject. Even if the user is not a management subject, If it is necessary to store the minutia information, it can be added to the database 300.

FIG. 2 is a diagram illustrating a method of grouping minutiae information of a management subject into a plurality of similar groups according to an embodiment of the present invention. Referring to FIG.

As shown in FIG. 2, the minutia information for a plurality of management subjects is grouped into minutia information showing similar characteristics.

On the other hand, the grouping is performed to reduce the number of minutia information to be compared to recognize a face of a user, thereby shortening the time required for the recognition.

That is, the guide apparatus 100 groups the feature point information with high degree of similarity by calculating the similarity between the plurality of feature point information, and performs similarity calculation for recognizing the user's face only to the feature point information within the specific similar group Thus, the face of the user can be recognized quickly and efficiently.

To this end, the guide apparatus 100 generates representative minutia information for each group, calculates the similarity between the minutia information of the user extracted from the face image of the user and the generated minutiae representative point information, and calculates at least one similar group having high similarity And calculates the degree of similarity only for the minutia information in the selected similar group so that the face of the user can be recognized at high speed.

On the other hand, similarity can be calculated through various similarity calculation methods including Euclidean Distance, Manhattan Distance or Hamming Distance, and there is no limitation on the method of calculating the similarity.

Also, the representative feature point information may be generated by calculating an average value or an intermediate value of the feature point information within a specific similar group as minutia information representative of a specific similar group. That is, the representative feature point information can be generated by calculating an average value or an intermediate value of the position, shape, and size of the face component such as eyes, nose, mouth, or jaw based on all the feature point information within the similar group.

On the other hand, the representative minutia information may be used to select specific minutia information within the similar group, and use the selected minutia information as representative minutia information.

Also, the guide device 100 may periodically collect minutia information about the management subject and update the database 300 for the face recognition of the user.

In this case, the similarity between the feature point information collected periodically and the representative feature point information of each similar group may be calculated, and the result may be added to the similar group of the representative feature point information having the highest similarity degree and stored, and the same process of generating the representative feature point information The new representative minutia information can be generated and replaced or the existing minutia information can be corrected.

On the other hand, machine learning may be applied to the minutiae information of the to-be-administered person to group them into a plurality of similar clusters based on the minutia information of the plurality of to-be-managed persons. This will be described in detail with reference to FIG.

3 is a view for explaining a method of clustering minutiae information of a management subject to a plurality of similar clusters according to another embodiment of the present invention.

As shown in FIG. 3, the minutiae information for a plurality of subjects to be managed may be classified into similar minutia information by applying Unsupervised Machine Learning, such as a K-means method or a Gaussian Mixture Model method, The feature point information showing the pattern is clustered.

That is, the guiding device 100 generates a plurality of pseudo clusters by grouping patterns formed with minutiae points such as eyes, nose, mouth, etc., based on minutia information about a plurality of subjects to be managed.

Also, the guide device 100 groups a plurality of pieces of minutia information having similar patterns of eyes, nose, mouth, etc. to generate a plurality of pseudo clusters, and generates a representative pattern for each of the pseudo clusters generated . At this time, the generated representative pattern is the representative minutia information described with reference to FIG.

The representative pattern is used to identify a similar cluster for minutia information extracted from the image of the user when an image for recognizing the user's face input from the plurality of image capturing apparatuses 200 is input. Minutia information for a plurality of management subjects included in the similar cluster is represented.

On the other hand, the representative pattern is generated by selecting a pattern formed by specific minutia information in the similar clusters, or extracting a pattern formed of minutia information, which is generated through an average value or intermediate value calculation, of minutia information in the similar clusters, .

That is, the guide apparatus 100 learns minutia information and learns and detects a pattern for the minutiae on the basis of a linear distribution or a nonlinear distribution with respect to eyes, nose, mouth, and the like, By grouping the patterns, a pseudo cluster is created. Thus, the number of cases of the similarity calculation for recognizing the face of the user can be drastically reduced and the face of the user can be recognized in real time.

 FIG. 4 is a block diagram illustrating a configuration of a guide device in a learning-type real-time guidance system using face recognition according to an embodiment of the present invention.

As shown in FIG. 4, the learning-type real-time guidance apparatus 100 through face recognition includes an image collecting unit 110 for collecting a user's image photographed in real time from the image photographing apparatus 200, A feature point information extracting unit 130 for extracting feature point information from the detected face image, a feature point extracting unit 130 for extracting feature point information extracted from an image of the collected management subject, A user recognition unit 150 for recognizing a face of the user based on a user's image input from the image sensing device 200, a group recognition unit 150 for recognizing the user's face, A database update unit 170 for updating the database, and a control unit 180. The control unit 180 controls the operation of the database 180,

Also, the image collecting unit 110 periodically collects facial images of the management object through the network and the image capturing apparatus 200, and collects the images of the user captured through the image capturing apparatus 200 in real time.

On the other hand, the images of the management subjects collected are grouped by the grouping unit 150 and stored in the database 300 as reference images that serve as a reference for recognizing users.

In addition, the face image detection unit 120 separates the background from the collected image and detects only the image of the face.

The detection may be performed by applying various algorithms such as AdaBoost algorithm, Template Matching algorithm and the like to separate the face and the background from the image and detect the face part. In the present invention, There is no restriction on the method.

The feature point information extraction unit 130 extracts feature point information from the face image detected by the face image detection unit 120.

The minutia information is composed of a set of minutiae constituting constituent parts of the face such as eyes, nose, mouth, jaw or the like which is a constituent part of a face.

The minutia information extracting unit 130 maps the minutia information related to the minutiae point related information of the management subject (i.e., the management subject type) to the extracted minutia information when the face image belongs to the management subject, (140).

That is, the minutia related information allows the minutiae information for a specific management target person to know what type of management target person is.

When the face image detected by the face image detecting unit 120 is for recognizing the user, the feature point information extracting unit 130 provides the user's face recognizing unit 150 with the feature point information extracted from the face image .

Also, the grouping unit 140 groups the minutiae information about the plurality of management subjects into a plurality of similar groups, generates representative minutia information for each similar group, and stores the minutia information in the database 300. [ The minutiae information for each similar group stored at this time is reference data for recognizing the face of the user.

Also, the grouping is performed by classifying the pieces of feature point information having a high degree of similarity through calculation of similarities among pieces of feature point information. Meanwhile, as described above, the patterns formed by the minutia information are clustered by using the machine learning to classify the minutia information with high degree of similarity.

Also, the user's face recognizing unit 150 recognizes the user's face using the feature point information of the specific user provided from the feature point information extracting unit 130 and the feature point information of the pre-stored management object.

In order to recognize the face of the user, the user's face recognizer 150 first calculates at least one similar group having higher similarity than the preset value through calculation of similarity between the user's minutia information and the similar minutia information Select.

Next, at least one minutia information having the highest degree of similarity or higher than a predetermined value is selected through the 1: 1 similarity calculation of the minutia information belonging to the selected similar group and the minutia information of the user, Can be recognized. On the other hand, if the calculated degree of similarity is not higher than a preset value, the user may not provide or provide the recognition result to the manager together with information indicating that the user does not belong to the management subject.

In addition, the notification unit 160 may be configured to notify the user when the event occurs (that is, the management object appears) according to the recognition result through the user's face recognition unit 150, the face image of the recognized user, The user's type information (i.e., class label information), the minutiae information of the recognized user, or a combination thereof, to the administrator terminal so that the user can take immediate action.

That is, when the user is recognized as a thief, information about the recognition result including the face image of the user, class label information, or a combination of them is transmitted to the administrator terminal, whereby the manager confirms the face image, Can be performed. At this time, in addition to the information on the recognition result, coordinate information of the image pickup apparatus 200 photographed by the user may also be provided.

On the other hand, if the manager needs to store the minutia information in the database 300 (for example, when the user is recognized as a new abnormal client) even when the user is not a management target as a result of checking the user based on the received information, It is possible to control the guide apparatus 100 so that the minutiae information of the user can be added to the database 300 and stored.

At this time, the database update unit 170 of the guide device 100 may store the similarity information in the database 300 by adding the similarity information between the minutia information and the representative minutia information of each similar group into a specific similar group.

Also, it is preferable that the notification unit 160 transmits information on the final recognition result to the administrator terminal as described above. However, when the recognition unit 160 recognizes through the calculation of similarity between the minutiae information of the user and the representative minutia information of each similar group To the manager.

The information provided at this time includes the probability of belonging to the specific management object of the user and the facial image of the user.

That is, the notification unit 160 may calculate the probability of the management subject type of the similar group by referring to the minutia point related information of the plurality of minutia information items belonging to the specific similar group, and transmit the probability to the administrator terminal.

For example, if 10 pieces of characteristic point information belong to a specific similar group, 4 pieces of characteristic point information are for abnormal customers, 3 pieces of characteristic point information are for the thief, and the remaining 3 pieces of characteristic point information are about intensity, The user is calculated as the probability of abnormal customer (40%), the probability of being a thief (30%), the probability of a robbery (30%), the probability of an abnormal customer (40%) and the probability of a criminal can do. Thus, when the recognition result of the user's face is provided to the manager through the probability value, the appearance of the management subject can be recognized more quickly.

 The database update unit 170 updates the database 300 by adding the minutia information of the additional management target person to the database 300 and storing the added minutia information when the minutia information of the management target is newly added, (140) to be grouped into specific similar groups according to the degree of similarity between the added minutia information and representative minutia information for each similar group.

In a case where the new minutia information is added and grouped into a similar group, the database update unit 170 replaces the representative minutia information of the similar group with the new minutia information by reflecting the added minutia information, Information is corrected so that accurate face recognition can be performed.

In addition, the database update unit 170 removes minutia information that is not selected for the similarity comparison object for a predetermined period (e.g., 3 months, 5 months, 7 months, etc.) The feature point information of the management subject to be subjected to the similarity calculation to be performed can be reduced. This makes it possible to recognize the user's face more quickly and efficiently.

Meanwhile, each component of the guide device 100 described above may be divided into independent computing devices depending on the size of the system, and may be implemented as one image capturing device 200. [ At this time, the plurality of image capturing apparatuses 200 are operated in cooperation with a small number of databases and an event transmission system (i.e., an alarm system), and the database 300 is updated in real time at the request of the administrator.

Hereinafter, with reference to FIG. 5, a procedure of databaseing the minutiae information of the management subject and a procedure of the user face recognition and notification process will be described in detail.

FIG. 5 is a flowchart illustrating a procedure of databaseing minutiae information of a management subject, which is reference data for recognizing a face of a user according to an embodiment of the present invention, and recognizing a user face using the minutiae information and transmitting a recognition result to the administrator terminal Describe the procedure in detail.

As shown in FIG. 5, the procedure of converting the minutia information of the management subject, which is the reference data for recognizing the face of the user, into a database is performed by first acquiring facial images from the images of a plurality of management subjects collected through the guide device 100 (S110).

Next, the guiding device 100 extracts minutia information from the detected face image (S120).

At this time, the guide device 100 maps and stores a class label indicating a management object type for each minutia information, so that it is possible to easily determine which type of management target each minutia information is.

Next, the guiding device 100 groups the minutiae information of the extracted plurality of the management subjects by comparing them with each other, and stores the minutia information having similar characteristics in the database 300 (S130) The minutiae information is converted into a database.

Also, the process of recognizing the user's face from the image photographed by the image photographing apparatus 200 and transmitting the recognized result to the administrator terminal (i.e., the notification process) The face image of the user is detected from the image captured by the device 200 in real time, and the feature point information is extracted from the detected face image of the user (S210).

The feature point information of the face image of the user is extracted through the same method performed in step S120.

Next, the guiding device 100 loads representative minutia information for each similar group stored in the database 300 (S220), and calculates the similarity with the extracted minutia information of the user (S230).

The similarity calculation between the representative minutia information by similar group and the minutia information of the user performed in step S230 is performed in order to reduce the number of minutia information to be compared in order to recognize the user's face.

Next, the guide apparatus 100 selects at least one similar group for representative feature point information having a high similarity degree as a result of calculation of the similarity degree (S240). Then, the guide apparatus 100 selects 1: 1 similarity calculation is performed (S250), and at least one minutia information having a degree of similarity higher than a preset value is extracted.

That is, the guide device 100 does not perform similarity calculation with all the feature point information stored in the database 300 to recognize the face of the user photographed from the plurality of image capturing apparatuses 200, It is possible to dramatically reduce the time required for recognizing the face of the user by calculating the degree of similarity only with the feature point information within the group.

Next, the guide device 100 extracts information related to a specific event from the database 300 and transmits the extracted information to the administrator terminal in operation S260.

When the event occurs, the guide device 100 displays a face image of the user, a face image of the user to which the recognized user belongs The management subject type, the minutiae information of the user, or a combination thereof, from the database 300 and transmits the extracted information to the administrator terminal so as to prevent crime, prevent business interruption, or take immediate action for good customers.

As described above, in the learning-type real-time guidance system and method for guiding learning through face recognition of the present invention, a similar group is generated for each feature point information having similar characteristics based on feature point information extracted from face images of a plurality of subjects to be managed , A specific similar group is selected in a process of recognizing a face for a specific user, the face of the user is quickly recognized based only on the minutia information in the selected similar group, and the recognition result is provided to the manager, So that the user can immediately perform the action.

While the present invention has been described with reference to the preferred embodiments thereof, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. As will be understood by those skilled in the art.

10: Learning-based real-time guidance system through face recognition
100: guide device 110: image collecting part
120: Facial image detecting unit 130: Feature point information extracting unit
140: Grouping unit 150: User's face recognition unit
160: notification unit 170: DB update unit
200: image capturing apparatus 300: database
400: Criminal database

Claims (12)

A minutia information extracting unit for extracting minutia information from face images of a plurality of management subjects and users;
Grouping the minutiae information of the extracted management subject into a plurality of similar groups, generating representative minutia information for each of the plurality of similar groups by the grouping, mapping the generated minutia information to the similar groups, ; And
And a user face recognition unit for calculating a similarity degree between the extracted feature point information of the user and the feature point information of the management subject and recognizing the face of the user in real time,
The user's face recognizing unit,
Calculating similarity between the minutiae information of the extracted user and the representative minutia information of each similar group, selecting a similar group for at least one representative minutia information according to the calculated similarity, And the feature point information of the user, thereby performing the face recognition of the user at a high speed. The method according to claim 1,
The grouping unit,
Calculating similarities between minutiae information about the extracted plurality of management subjects, grouping minutiae information according to the calculated similarities,
Detecting a pattern of a linear or non-linear distribution of facial elements including eye, nose, mouth, jaw or a combination thereof by learning minutia information of the person to be supervised, calculating a degree of similarity to the pattern, And grouping feature point information by clustering feature point information according to a similarity. The method according to claim 1,
The grouping unit,
Generating the representative minutia information by calculating an average value or an intermediate value of the minutia information for each similar group,
And when the minutiae information for each similar group is newly added or deleted, the minutiae information is replaced or corrected with new representative minutia information reflecting the minutia information. delete The method according to claim 1,
The learning-type real-
And a notification unit for transmitting a face image of the recognized user, a type of a management object to which the recognized user belongs, minutia information of the recognized user, or a combination thereof to the administrator terminal,
Wherein the management target type includes a type of a criminal, a type of an abnormal customer, or a type of an excellent customer. The method according to claim 1,
The learning-type real-
Wherein the feature point information of the management subject is automatically deleted from the database when the feature point information of the management subject is not used for calculation of the degree of similarity for recognizing the face of the user for a predetermined period or more. A minutia information extracting step of extracting minutia information from face images of a plurality of management subjects and a user in a learning type real time guidance device through face recognition;
The learning-type real-time guidance apparatus according to claim 1, further comprising: grouping minutia information of the extracted management subjects into a plurality of similar groups, generating representative minutia information for each of the plurality of similar groups by grouping, A grouping step of mapping each of the similar groups and storing the same in a database; And
And a user face recognizing step of recognizing a face of the user in real time by calculating the similarity degree between the minutiae information of the extracted user and the minutia information of the management subject in the learning type real time guidance device through the face recognition,
The user face recognition step may include:
Calculating similarity between the minutiae information of the extracted user and the representative minutia information of each similar group, selecting a similar group for at least one representative minutia information according to the calculated similarity, And the feature point information of the user, thereby performing the face recognition of the user at a high speed. The method of claim 7,
Wherein the grouping comprises:
In the learning-type real-time guidance apparatus through the face recognition, the similarity degree between the extracted minutia information about the plurality of management subjects is calculated, the minutia information is grouped according to the calculated similarity degree,
Detecting a pattern of a linear or non-linear distribution of facial elements including eye, nose, mouth, jaw or a combination thereof by learning minutia information of the person to be supervised, calculating a degree of similarity to the pattern, And grouping feature point information by clustering feature point information according to a degree of similarity. The method of claim 7,
Wherein the grouping comprises:
In the learning-type real-time guidance apparatus based on face recognition, the representative minutia information is generated by calculating an average value or an intermediate value of minutia information for each similar group,
Wherein when the minutia information of each similar group is newly added or deleted, the representative minutia information is replaced or corrected with new representative minutia information reflecting the same. delete The method of claim 7,
A learning-type real-time guidance method using face recognition,
An informing step of transmitting the facial image of the recognized user, the type of the management object to which the recognized user belongs, the minutiae information of the recognized user, or a combination thereof to the administrator terminal in the learning-type real-time guidance device through face recognition; Further comprising:
Wherein the management object type includes a type of a criminal, a type of an abnormal customer, or a type of a superior customer. The method of claim 7,
A learning-type real-time method using face recognition,
In the learning-type real-time guidance apparatus based on face recognition, when the minutia information of the management subject is not used for the calculation of the similarity for recognizing the user's face for a predetermined period or more, Learning - based real - time guidance method using face recognition.

Images