KR20220087908A - Method for management entrance and exit having a face recognition while wearing mask - Google Patents

Abstract

The present invention relates to an access management system for performing access control through face recognition, and to a technology that allows a previously registered person to be easily recognized as a registered person even when wearing a mask.
In the access control method having a mask wearing face recognition function according to the present invention, the access control device determines the pre-registered entrants based on the face images of persons included in the captured images received from a plurality of image photographing devices, and In the access control method having a mask wearing face recognition function for performing access control, each frame image of the same person is tracked in a way that the access control device tracks the trajectory of a person existing in a captured image received from each video recording device A first step of extracting an original face image, a second step of determining whether the face image extracted in the first step is a mask wearing image, and a second step of determining whether the mask wearing image is a mask of the corresponding mask wearing image The third step of generating a plurality of mask change images for the person by changing to the registered form, applying the mask wearing image and mask changing image to the learning model that classifies the entrants corresponding to the face image to determine whether or not they are registered entrants In the fourth step of determining, if it is determined as a registered person in the fourth step, the mask wearing image and a plurality of mask change images on the corresponding trajectory are stored as the learning data of the person, and the learning model is updated using this learning data. It is characterized in that it is configured to include a fifth step.

Description

Method for management entrance and exit having a face recognition while wearing mask}

The present invention relates to an access management system for performing access control through face recognition, and to a technology that allows a previously registered person to be easily recognized as a registered person even when wearing a mask.

Recently, face recognition technology has been used in various fields, including door control devices, and has an active advantage compared to other biometric recognition technologies such as fingerprint or iris recognition, and thus is further increasing.

In the case of fingerprint or iris recognition, in order for a user to obtain authentication, it is inconvenient to take an action to bring a finger into contact with the fingerprint recognition sensor or bring an eye closer to the iris recognition sensor.

However, because the face recognition technology can perform face recognition when the user's face is captured by the camera image, there is no need for the user to take an unnatural action for authentication, and it recognizes the user's face even when the user is not aware of it. There are advantages to being authenticated.

In particular, an access management system that performs access management using the facial recognition of users allowed to enter has been introduced.

The occupant recognition method using face recognition extracts the face image of a person included in the captured image taken by the camera of the access management system, classifies it by person, and then applies it as learning data to the face recognition algorithm to create a learning model for face classification. It is a method of classifying the occupants by generating the occupants and then inputting the entrants' face images extracted from the captured images into a pre-created learning model, which has the advantage of high accuracy and fast recognition speed.

On the other hand, recently, the frequency of wearing a mask for health reasons of users is increasing due to the occurrence of fine dust or Corona 19.

Accordingly, there may be a method of pre-registering a face image of a registered entrant wearing a mask, but this is inconvenient in which the registrant entrants additionally records and registers a mask wearing image.

In addition, in the case of classification of a person wearing a mask, there is a difference in face recognition performance depending on the shape or color of the mask worn by the user. .

1. Domestic Registered Patent No. 10-1363017 (Title of the invention: face image shooting and classification system and method) 2. Domestic Registered Patent No. 10-2039277 (Title of Invention: Pedestrian Face Recognition System and Method)

Accordingly, the present invention was created in consideration of the above circumstances, and the face image is managed as a tracking list according to the trajectory. Classify the registered entrants by applying the learning model, but by registering the mask wearing image on the trajectory of the person and the mask change image as the learning data of the registered entrant, and updating the learning model, It has a technical purpose to provide an access control method having a mask-wearing face recognition function that can easily recognize the registered person and perform access management even in the case of the user.

According to one aspect of the present invention for achieving the above object, the access control device determines the pre-registered occupants based on the face images of persons included in the captured images received from a plurality of image capturing devices, and manages access to the occupants. In the access control method having a mask wearing face recognition function to perform A first step of extracting an image, a second step of determining whether the face image extracted in the first step is a mask wearing image, and a second step of determining whether the mask wearing image is a mask of the corresponding mask wearing image The third step of generating a plurality of mask change images for the person by changing the shape, applying the mask wearing image and the mask changing image to the learning model that classifies the person corresponding to the face image to determine whether the person is a registered person In the fourth and fourth steps, if it is determined that a registered person is a registered person, the mask wearing image and a plurality of mask change images on the trajectory are stored as the learning data of the person, and the learning model is updated using the learning data. There is provided an access control method having a mask wearing face recognition function, characterized in that it comprises the steps of.

In addition, the third step provides an access control method having a mask wearing face recognition function, characterized in that generating a mask change image in which at least one of the shape and color of the mask is changed.

In addition, if it is determined in the fourth step that the mask wearing image and the mask changing image are not registered visitors, acquiring a face image without a mask from the captured image of the trajectory of the person and applying it to the learning model, determining whether or not a registered entrant An access control method having a mask wearing face recognition function is provided.

In addition, in the fourth step, the access control device extracts the face shape including the eyes and eyebrows and the lower part of the mask except for the facial feature points covered by the mask with respect to the mask wearing image and the mask change image as facial feature points, and learns them There is provided an access management method having a face recognition function wearing a mask, characterized in that the person classification is applied to the model.

There is provided an access management method having a mask wearing face recognition function, characterized in that the eyes, eyebrows, and upper face line excluding the facial feature points are extracted as facial feature points, and the person is classified by applying this to a learning model.

In addition, in the fourth step, the access control device additionally extracts the length of the mask line from the beginning of the mask line to the ear part as a facial feature point for the mask wearing image and the mask change image, and applies it to the learning model to perform person classification. There is provided an access control method having a mask wearing face recognition function, characterized in that.

According to the present invention, by managing the face image as a tracking list according to the trajectory and applying the mask-wearing face image to the learning model to classify the registered entrants, the registered entrant does not register the mask-wearing face image in advance. Access management can be performed by easily recognizing the entrants.

In addition, when the face image wearing a mask is classified as a registered person, the mask wearing image on the trajectory of the person and the mask change image are registered as the learning data of the registered person and the learning model is updated to update the mask wearing image of the registered person can further improve the person classification performance of

In addition, for the mask wearing image and mask change image, the length of the mask line from the beginning of the mask line to the ear is additionally extracted as facial feature points and applied to the learning model. Thus, the person classification performance can be improved.

1 is a view showing a schematic configuration of an access management system having a mask wearing face recognition function to which the present invention is applied.
FIG. 2 is a block diagram showing the functionally separated internal configuration of the access control device 300 shown in FIG. 1 .
Fig. 3 is a diagram showing data stored in the data storage 340 shown in Fig. 2;
4 is a flowchart for explaining an access control method having a mask wearing face recognition function according to the present invention.
Fig. 5 is a diagram illustrating a mask change image generated in Fig. 4;
6 and 7 are diagrams for explaining a person classification method for the mask image in FIG. 4;

The configuration shown in the embodiments and drawings described in the present invention is only a preferred embodiment of the present invention, and does not express all the technical ideas of the present invention, so the scope of the present invention is the embodiment and drawings described in the text should not be construed as being limited by That is, since the embodiment may have various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless otherwise defined. Terms defined in the dictionary should be interpreted as being consistent with the meaning of the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning not explicitly defined in the present invention.

1 is a diagram showing a schematic configuration of an access management system having a mask wearing face recognition function to which the present invention is applied.

Referring to FIG. 1 , an access management system having a mask wearing face recognition function to which the present invention is applied includes an access detection device 100 , an image photographing device 200 , and an access management device 300 . Here, the access detection device 100 and the image photographing device 200 are installed at an access management target place.

The access detection device 100 is installed at the access management target location, that is, on the door side, detects whether a person enters or not, and transmits the access detection information to the access management device 300 upon entry.

The image photographing apparatus 200 is installed in an access management target place, that is, around an entrance door, captures an image around the entrance door, and transmits the photographed image to the access management apparatus 300 . In this case, the image photographing apparatus 200 is arranged so that a plurality of image photographing apparatuses 200 photograph different areas when the photographing range is set so that an area from a location spaced apart from the entrance door by a predetermined distance is photographed.

The access management device 300 receives the captured image provided from each image capturing device 200, manages the face image of a person included in the received captured image in a way that tracks the person trajectory, and based on the face image It determines the previously registered entrants and performs access management for the entrants.

In particular, in the present invention, the access control device 300 determines whether the person is a registered person using the face image of the person wearing a mask, and performs access control for the person wearing the mask. At this time, the access control device 300 generates a plurality of mask change images by changing at least one of the shape or color of the mask in the mask wearing image extracted from the photographed image, and when it is determined as a registered person, the mask image of the trace tracking list is displayed. The learning model is updated by registering it as the learning data of the previously registered visitor.

FIG. 2 is a block diagram showing the functionally separated internal configuration of the access control device 300 shown in FIG. 1 .

Referring to FIG. 2 , the access control device 300 includes a captured image collecting unit 310 , a person classifying unit 320 , an access management unit 330 , and a data storage 340 .

The captured image collecting unit 310 stores the captured image received from each image capturing apparatus 200 and the face image extracted from the captured image in the data storage 340 .

The person classifying unit 320 determines whether or not a registered person is a registered person using the face image obtained from the photographed image, determines the face image of the image tracked along the trajectory as the face image of the same person, and updates the learning model through this. .

In addition, the person classifying unit 320 additionally generates various mask change images with respect to the mask wearing image, determines it as a learning face image of a registered person, and updates the learning model through this.

The person classification unit 320 includes a face image preprocessing module 321 , an object tracking module 322 , and a person classification module 323 .

The face image pre-processing module 321 adds masks of various characteristics to the face image extracted from the photographed image to generate a mask modified image, and stores it in the data storage unit 340 .

The object tracking module 322 assigns a person code to a person in a face image extracted from a photographed image, and tracks the same person in consecutive frame images after recognizing the person to the person to whom the person code is assigned, so as to face each movement time. Create an object tracking list of images.

In addition, the object tracking module 322 performs a tracking operation for the object tracking list set as an unregistered person until an image without a mask in which the mask is removed is obtained from the captured image on the trajectory of the person, and when the image without a mask is obtained This is stored in the data storage unit 340 .

At this time, the object tracking module 322 assigns a different person code to each recognized person when a person of a plurality of mask-wearing face images is recognized in one shot image, and provides a face for the person by each person code. The object tracking process is performed until the entire image is acquired.

The person classification module 323 classifies a person corresponding to a face image of a photographed image or a full face image obtained by the object tracking module 322 using a learning model registered in the data storage 340 .

In addition, the person classification module 323 applies the learning data accumulated so far in the data storage 340, the original face image, and the pre-registered registered face image to the face recognition algorithm to learn to classify the entrants corresponding to the face image. A model is created and the previous learning model stored in the data storage 340 is updated based on this. Here, as the face recognition algorithm, various types of well-known algorithms for classifying a person using a plurality of learning data may be applied.

That is, in the initial state of the system, the person classification module 323 generates a learning model by applying pre-registered face images (eg, a predetermined number of face images including front face images) to the face recognition algorithm. Then, after the learning data is accumulated and stored, when a preset update condition is satisfied, the learning model is newly updated using the face image of the occupant including the accumulated learning data until now.

Accordingly, in the present invention, as time elapses, the learning data applied to the face recognition algorithm, including the learning data accumulated for the previously registered entrants, increases, so that the reliability of the learning model for occupant classification is further improved. In particular, since mask wearing images of various characteristics are included as learning data, it is possible to smoothly classify registered visitors even when wearing a mask.

The entrant management unit 330 performs a series of entrant registration processing by storing the resident registration information including the personal information of the entrant and the registered face image including the front face image of the entrant in the data storage 340 .

In addition, the access management unit 330 generates access management information including the access date and access time for the registered person and stores it in the data storage 340 to correspond to the corresponding person.

As shown in FIG. 3 , the data storage 340 includes an image collection device meta information storage unit 341 including an installation location for each identification code and a shooting radius of the image collection device 200 , and the image collection device 200 , as shown in FIG. 3 . A map data storage unit 342 for storing map data corresponding to the installed access management service area, a photographed image storage unit 343 for storing photographed images, a registered face image for each visitor and a face image extracted from the photographed image are stored. Face image storage unit 344, access information storage unit 345 for storing access management information for each visitor, personal information storage unit 346 for storing visitor personal information including names and contact information for each visitor, person classification unit ( 320) includes a learning data storage unit 347 for storing the face image determined as training data, and a learning model storage unit 348 for storing learning model information generated by the face recognition algorithm.

Next, an access control method having a mask wearing face recognition function according to the present invention will be described with reference to the flowchart shown in FIG. 4 .

Referring to FIG. 4 , the access control device 300 collects accessor registration information and stores and registers it in the data storage 340 (ST110). At this time, an occupant identification code is assigned to each entrant, resident registration information includes resident personal information and a registered face image, and the registered face image includes a front image of the entrant.

In addition, the access management device 300 generates a learning model using the registered face image of the person who has been previously registered in the data storage 340 and stores it in the data storage 340 (ST120). That is, the initial learning model information based on the resident registration information is generated by using the resident registration face image as learning data of the face recognition algorithm.

In the above state, the access control device 300 continuously collects captured images from the image capturing device 200 . Then, a face image of the same person is extracted from the collected captured images and stored, and an object tracking list for the person is generated using the extracted face image (ST130). At this time, the access control device 300 searches for a person in the photographed image, assigns a person code to the searched person, and stores the photographed image consisting of a plurality of consecutive frame images, and the person disappears from the photographed image or access is detected Until the access detection information is received from the device 100, the captured image for a predetermined time including the corresponding person is stored. In addition, the access control device 300 extracts a corresponding person image for each frame image from consecutive captured images collected for a certain period of time for each person code, and extracts a face image for the person from each extracted person image to obtain an object Save it as a tracking list. In this case, the face image is extracted and stored in real time along with the image frame reception.

Next, the access control device 300 determines in real time whether the face image extracted in step ST130 is a mask worn image (ST140).

When it is determined that the face image currently extracted in step ST140 is an image without a mask, the access control device 300 applies the face image to the learning model and, when classified as a pre-registered specific person, stored in the object tracking list of the person The face image is registered in the data storage 340 as learning data of a corresponding person, that is, a registered visitor (ST150). In this case, the learning model for the mask wearing image may be generated and used separately from the learning model for the mask not wearing image.

In addition, when it is determined that the face image currently extracted in step ST140 is a mask wearing image, the access control device 300 generates various mask change images in which only the mask characteristics are changed from the mask wearing image (ST160). In this case, various mask deformation images in which at least one or more of the shape and color of the mask are changed may be generated, and the mask deformation image is stored in the object tracking list of the person. 5 exemplifies mask deformation images in which various types of masks are worn for the same person.

Next, the access control device 300 performs a face classification function by applying the mask wearing image and the mask change image, that is, the mask image, extracted from the captured image to the learning model (ST170).

In step ST170, the access control device 300 performs the face-shaped face including the eyes and eyebrows excluding the facial feature points covered by the mask 2 in the face image, and the lower part of the mask, as shown in FIG. 6 (A). Classification of people is performed using feature points.

In addition, the access control device 300 learns the mask wearing image by additionally extracting the length of the mask line 3 from the beginning of the mask line 3 to the ear part as facial feature point information as shown in Fig. 6 (B). By applying it to the model, person classification can be performed. This is attribute information that can infer the size of the ears of the person entering, and this person classification attribute information is applied to a learning model generation algorithm in advance to create a learning model.

On the other hand, in step ST170, if one of the mask images of the same person is classified as a specific person, that is, a registered person with a probability value greater than or equal to a set threshold (ST180), the access control device 300 returns to the object tracking list of the person registered so far. The stored mask image is registered in the data storage 340 as learning data of the registered person (ST190).

Next, the access control device 300 updates the learning model using the learning data registered in the data storage 340 (ST200).

Thereafter, the access management device 300 performs the operation below step ST130, that is, the occupant classification processing for the occupant's face image, using the recently updated learning model, and this operation is repeated at a cycle satisfying the preset update condition. is carried out

In addition, the access control device 300 opens the door to the entrant recognized as a pre-registered entrant to allow the occupant to enter.

On the other hand, if the mask image is not classified as a pre-registered specific person in step ST180, the access control device 300 classifies the object tracking list of the person as an unclassified person, and after adding the mask image to the object tracking list , the above-described operation after ST130 is performed on the face image of the next frame (ST210).

A process of classifying a person in an image wearing a mask will be exemplified and described with reference to FIG. 7 .

First, a specific person moves while wearing a mask at time “t1” and is detected for the first time at time “t2” by the first imaging device (device 1), then moves while wearing a mask and moves to “t3” “It disappears in time. In this case, the access management device 300 classifies the face image (the mask wearing image of FIG. 7 ) at each time using the learning model, but if the face image is not classified as a registered person, it is set as an unclassified person.

Here, the access control device 300 uses the image recording device meta information storage 341 and the map information storage 342 of the data storage 340 to track the trajectory of a specific person for each photographed image device, and Creates an object tracking list including a face image extracted from the captured image of the trajectory.

Next, for the specific person, the face with the mask removed at time “t4” is detected by the second imaging device (device 2) in the physical space 1 in the object tracking list for the same person code, and the mask is detected at time “t5” Disappears while wearing At this time, the access control device 300 does not perform the person classification processing on the mask wearing image for the person classified as an unclassified person, and does not wear a mask photographed at time "t4" by the second image capturing device (device 2). When the image is obtained, the person classification process can be performed by applying the mask-free image to the learning model.

Thereafter, in the object tracking list for the corresponding person code, trajectory tracking may be continuously performed until the specific person leaves a certain spatial area in a state of being classified as a person. That is, in FIG. 7 , the specific person is extracted as a mask wearing image by the third imaging device (device 3) at time “t6 to t7” and stored in the object tracking list, so the movement of the person can be managed. have. For example, it can be confirmed that the specific person of FIG. 7 entered and exited the physical space 1 in which the first and second image photographing apparatuses are installed during times t1 to t5.

100: access detection device, 200: video recording device,
300: access control device, 310: captured image collection unit,
320: person classification unit, 321: face image pre-processing module,
322: object tracking module, 323: person classification module,
330: access management unit, 340: data storage,
341: image recording device meta information storage unit,
342: map data storage unit, 343: captured image storage unit,
344: face image storage unit, 345: access information storage unit,
346: personal information storage unit, 347: learning data storage unit,
348: learning model storage unit.

Claims (5)

In an access control method having a mask wearing face recognition function, the access control device determines the pre-registered occupants based on the face images of persons included in the captured images received from multiple video recording devices and performs access management for the occupants. in,
A first step of extracting an original face image for each frame image of the same person in a way that the access control device tracks the trajectory of a person existing in the captured image received from each video recording device;
A second step of determining whether the face image extracted in the first step is a mask wearing image,
A third step of generating a plurality of mask change images for the person by changing the mask of the corresponding mask wearing image to a previously registered form if it is determined in the second step as a mask wearing image;
A fourth step of determining whether a person is a registered person by applying a mask wearing image and a mask change image to a learning model for classifying a person corresponding to a face image;
If it is determined that the person is a registered person in the fourth step, including a fifth step of storing the mask wearing image and a plurality of mask change images on the trajectory as the learning data of the person, and updating the learning model using the learning data Access control method having a mask wearing face recognition function, characterized in that configured.
According to claim 1,
The third step is an access control method having a mask wearing face recognition function, characterized in that generating a mask change image in which at least one of the shape and color of the mask is changed.
According to claim 1,
In the fourth step, if it is determined that the mask wearing image and the mask change image are not registered visitors, it is characterized in that by acquiring a face image without a mask from the photographed image of the trajectory of the person and applying it to the learning model, it is determined whether or not a registered visitor is Access control method with face recognition function wearing a mask.
According to claim 1,
In the fourth step, the access control device extracts the face shape including the eyes and eyebrows and the lower part of the mask except for the facial feature points covered by the mask for the mask wearing image and the mask change image as facial feature points, and uses them in the learning model. An access control method having a face recognition function wearing a mask, characterized in that applying to perform a person classification.
5. The method of claim 4,
In the fourth step, the access management device additionally extracts the length of the mask line from the beginning of the mask line to the ear part as a facial feature point for the mask wearing image and the mask change image, and applies it to the learning model to perform person classification An access control method having a face recognition function wearing a mask.

Images