KR20230016470A - Method and apparatus for detecting mask wearing - Google Patents

Abstract

Provided is a method for detecting mask wearing from an entire image inputted from a mask wearing detection device. The method for detecting mask wearing comprises: a step of normalizing and pre-processing an image of a facial area detected from the entire image; and a step of detecting a mask normal wearing state and a mask abnormal wearing state wherein a mask completely covers the nose and mouth corresponding to the respiratory organ based on a deep learning network from the normalized and preprocessed image of the facial area.

Description

Mask wearing detection method and device {METHOD AND APPARATUS FOR DETECTING MASK WEARING}

The present invention relates to a mask wearing detection method and device, and more particularly, to a mask wearing detection method and device capable of detecting whether a mask is normally worn using a lightweight neural network.

Wearing a mask is mandatory to prevent the spread of the virus. Conventionally, since wearing a mask was not compulsory in daily life, the need for a non-mask detection function has not been highlighted. However, the need for a non-mask detection function that can check whether the nose and mouth are completely covered to prevent infectious diseases has increased. In particular, as demand for devices with a mask wearing detection function is increasing in public facilities and large marts, it is necessary to have a mask wearing detection function in accordance with the government's mask wearing guidelines.

With the recent development of deep learning-based image recognition technology, image recognition technology is being used in various places, such as intelligent CCTV using AI technology and face recognition access control.

Deep learning generally improves performance as the depth of the network increases. Recently, with the development of deep learning-based image processing technology, many lightweight deep learning network structures with high accuracy, such as SqueezeNet, MobileNet, and ShuffleNet, have been proposed. However, even this lightweight network requires a computation time of hundreds of milliseconds to several seconds when operated on a low-performance processor such as a low-end processor or an older processor. In an environment such as a door with a large number of operations, long computation time causes inconvenience.

In addition, in order to detect wearing a mask using deep learning, a process of detecting a person's face and then determining whether the mask is worn normally is necessary. For mask wearing detection, if a series of processes are operated in one new network, functions that are already in use, such as face detection, are duplicated, and the amount of computation required for the entire system increases. Therefore, for smooth and rapid mask wearing detection, it is necessary to develop a deep learning network for lightweight mask wearing detection that guarantees fast operation even with low-cost and old processors and can be flexibly used on various existing platforms.

The problem to be solved by the present invention is to provide a mask wearing detection method and device capable of detecting whether or not the mask is normally worn.

In addition, the problem to be solved by the present invention is to provide a mask wearing detection method and device that can reduce the amount of calculation and ensure fast operation by utilizing the deep learning function currently in use.

According to one embodiment of the present invention, a method of detecting wearing of a mask from an entire image input by a mask wearing detection device is provided. The mask wearing detection method includes the steps of normalizing and pre-processing the image of the face region detected from the entire image, and using a deep learning network from the normalized and pre-processed image of the face region so that the mask completely covers the nose and mouth corresponding to the respiratory tract. and detecting the normal wearing state of the covered mask and the abnormal wearing state of the mask. In this case, the deep learning network includes at least one convolution layer and a fully connected layer generating one output value from data output from the at least one convolution layer, and the at least one convolution layer determines the resolution. It includes at least one inverse block for extracting a feature while reducing it, and at least one inverse residual block for extracting a feature by increasing the number of channels whenever it passes through the inverse block.

The normalizing and preprocessing may include acquiring positions of the eyes, nose, and mouth in the face region detected from the entire image, and moving the positions of the eyes, nose, and mouth in the face region to designated positions. can

The normalizing and preprocessing may include cropping only the face region image from the entire image and converting the face region image into a black and white image.

The detecting may include classifying the image of the face region as a normal mask wearing state when the output value of the deep learning network is greater than a set threshold, and the face when the output value of the deep learning network is less than or equal to the threshold. A step of classifying the image of the region as a mask abnormal wearing state may be included.

The step of classifying the mask into an abnormal wearing state is a step of classifying into one of a wearing state in which the nose or mouth is exposed, a mask not wearing a mask state, and a state covered by a hand or object based on the output value of the deep learning network. can include

The normalizing and preprocessing may include receiving a face detection result from a pre-installed face detector, and detecting a face region from the entire image based on the face detection result.

According to another embodiment of the present invention, a mask wearing detection device for detecting mask wearing is provided. The mask wearing detection device cuts an image of a face region from an entire image based on a face detection result, a normalization processing unit that sets positions of eyes, nose, and mouth in the face region, and a face for which the positions of the eyes, nose, and mouth are set. A deep learning detection unit that uses the image of the region as input data of the deep learning network to detect a normal wearing state of the mask and an abnormal wearing state of the mask in which the mask completely covers the nose and mouth corresponding to the respiratory tract, and the deep learning network includes at least One convolution layer and a fully connected layer generating one output value from data output from the at least one convolution layer, wherein the at least one convolution layer extracts features while reducing resolution. It may include one inverse block and at least one inverse residual block for extracting a feature by increasing the number of channels each time it passes through the inverse block.

The deep learning detection unit may include a classification unit configured to classify the image of the face region into a mask normal wearing state and the mask abnormal wearing state by comparing an output value of the deep learning network with a set threshold. there is.

The mask wearing detection device may further include an image pre-processing unit that converts the image of the face region where the positions of the eyes, nose, and mouth are set to a black-and-white image of a set size and outputs the converted image to the deep learning network.

The mask wearing detection device may further include a face detector that detects a face from an entire input image, generates a face detection result, and transmits the face detection result to the normalization processor.

According to an embodiment of the present invention, it is possible to determine whether a mask is worn correctly to prevent respiratory infections, and even though a mask is worn, abnormal wearing of a mask, such as partial exposure of the nose and mouth, which are respiratory organs, can be detected. It can be used in public facilities that require prevention.

In addition, by separating the mask wearing detection, face detection, and face alignment functions, the infrastructure of the place where the face detection function is used can be used as it is.

In addition, if the face detection function uses the existing infrastructure, the deep learning network for mask wearing detection only needs to determine whether or not the mask is worn, so it is possible to implement the mask wearing detection function even with a low-cost processor, and if a high-performance processor is used, a shorter A mask wearing detection operation may be possible within a certain amount of time.

1 is a diagram schematically showing a mask wearing detection device according to an embodiment of the present invention.
FIG. 2 is a view showing the mask wearing determination unit shown in FIG. 1 .
FIG. 3 is a diagram showing an example of a structure of a deep learning network of the deep learning detection unit shown in FIG. 2 .
FIG. 4 is a diagram illustrating an example of an inverse block of the convolution layer shown in FIG. 3 .
FIG. 5 is a diagram illustrating an example of an inverse residual block of the convolution layer shown in FIG. 3 .
6 is a flowchart illustrating a mask wearing detection method according to an embodiment of the present invention.
7 is a diagram illustrating a device for detecting wearing a mask according to another embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification and claims, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

Now, a mask wearing detection method and apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a diagram schematically showing a mask wearing detection device according to an embodiment of the present invention.

First, a mask can be defined as a cloth (cotton) covering made to cover the mouth and nose, which are respiratory organs, by hanging it over the ears or wearing it on the face in the form of a belt.

Referring to FIG. 1 , the mask wearing detection device 100 may include a face detector 110 and a mask wearing determining unit 120 .

The face detector 110 detects a face from an input image. The face detector 110 may detect a face from an input image using deep learning. The face detector 110 transfers the positions of the two eyes, the tip of the nose, and the ends of the lips to the mask wearing determination unit 120 as a face detection result.

In general, the face detector 110 is used in many areas of video surveillance, such as face recognition access control or intelligent CCTV. In the case of an environment in which the face detector 110 is being used, the mask wearing detection device 100 may use the face detection function of the already used face detector 110 .

When the face detection result is received from the face detector 110, the mask wearing determination unit 120 determines whether the mask is normally worn using the face detection result. The mask wearing determination unit 120 cuts the face area using the positions of the eyes, the tip of the nose, and both ends of the lips from the face detector 110, and determines whether the mask is normally worn using only the image of the face area. At this time, the mask wearing determination unit 120 may use a deep learning network in determining whether or not the mask is normally worn.

At this time, if the existing face detector 110 is used to detect wearing a mask, the deep learning network for detecting wearing a mask only needs to determine whether or not the mask is worn, so the learning difficulty can be reduced and the learning problem can be simplified. Therefore, computation efficiency and mask wearing detection performance can be improved.

FIG. 2 is a view showing the mask wearing determination unit shown in FIG. 1 .

Referring to FIG. 2 , the mask wearing determination unit 120 includes a normalization processing unit 122, an image pre-processing unit 124, a deep learning detection unit 126, and a determination unit 128.

The face detector 110 detects whether there is a face in the image of the entire area, detects the positions of the eyes, nose, and mouth of the face in the image, and outputs the position of the face including the positions of the eyes, nose, and mouth of the face in the image.

The normalization processing unit 122 receives the positions of both end points of the eyes, nose tip, and lips from the face detector 110, and uses the positions of both end points of the eyes, nose tip, and lips received from the face detector 110 to determine the entire area. A cropping process is performed to cut out only the face area from the image. In this case, the positions of the eyes, nose, and mouth in the image of the face region that is cut out may be inaccurate depending on the posture of the face. The normalization processor 122 performs a normalization process of moving the positions of the eyes, nose, and mouth in the face region to designated positions in order to reduce performance degradation of the deep learning detection unit 126 due to positional errors of the eyes, nose, and mouth. The normalization processing unit 122 may align the face by performing similarity transformation so that the positions of the eyes, nose, and mouth of the face are at designated positions. By aligning the faces in this way, it is possible to obtain a face image with a certain shape to a certain extent, which reduces the factor that may cause the deep learning network for mask discrimination to malfunction.

Also, the normalization processing unit 122 converts the image of the face area to a set size. For example, the normalization processing unit 122 converts an image of the face region into an image having a size of 128x128.

The image preprocessor 124 performs a preprocessing process of converting the image of the face region output from the normalization processor 122 into a 1-channel black and white image so as to be robust against changes in brightness and color due to ambient lighting. The image of the face region of the set size changed to black and white by the image pre-processing unit 124 is input to the deep learning detection unit 126 .

The deep learning detection unit 126 detects whether or not the mask is normally worn based on the deep learning network from the input black and white image of the face area. The face wearing a mask can be largely divided into wearing the mask normally, exposing the nose and mouth, and covering the nose and mouth with an object other than a mask. Wearing a mask is when the nose and mouth are completely covered with a mask, and not wearing is when the nose and mouth are exposed. Nose exposure is when a mask covers the mouth but exposes the nose. Lastly, the scheme is a case where the nose and mouth are covered with a hand or smartphone, not a mask in the video. The deep learning detection unit 126 may detect normal wearing of the mask and abnormal wearing of the mask from the input image of the face region. Abnormal wearing of a mask is divided into not wearing a mask, poor wearing such as nose exposure, and planning, and the deep learning detection unit 126 can detect whether the mask is not wearing, poor wearing, or a plan when the mask is worn abnormally. In the case of a plan, it is possible to detect whether it is covered by a hand or a small item such as a smart phone.

The determination unit 128 determines whether or not the mask is normally worn based on the detection result of the deep learning detection unit 126, and outputs a result of whether or not the mask is normally worn.

FIG. 3 is a diagram showing an example of a structure of a deep learning network of the deep learning detection unit shown in FIG. 2 .

Referring to FIG. 3 , the deep learning detection unit 126 includes a deep learning network 1262 and a classification unit 1264.

The deep learning network 1262 includes a plurality of convolution layers 10, 20, and 30 and a fully connected layer 40. 3 shows three convolutional layers 10, 20, and 30.

At least one of the convolution layers 10 and 20 of the convolution layers 10, 20 and 30 may include at least one inverted block and at least one inverted residual block. The convolution layer 30 located at the end of the convolution layers 10, 20, and 30 may be composed of at least one inverted block. The inverse residual block skips layers with a small number of channels.

For example, the first convolution layer 10 to which an image of a face region of a set size changed to black and white is input has a structure in which one inverse block and two inverse residual blocks are sequentially connected. The feature map, which is the output data of the first convolution layer 10, is input to the second convolution layer 20, and the second convolution layer 20 has one inverse block and three inverse residual blocks. It has a sequentially connected structure. Next, the third convolution layer 30 to which the feature map, which is the output data of the second convolution layer 20, is input has a structure in which two inverse blocks are sequentially connected. The feature map, which is the output data of the third convolution layer 30, is input to the fully connected layer 40.

The fully connected layer 40 generates and outputs one value (mask score) through fully connected output data obtained from the previous convolution layer 30.

At this time, the number of filters doubles each time it passes through the inverse block of the convolution layer (10, 20), and the feature map that passes both the inverse block and the inverse residual block of the convolution layer (10, 20, 30) is completely It is converted into one output value (mask score) through the connection layer 40.

In this way, the deep learning network 1262 has a structure in which an inverse residual block is mixed with an inverse block in order to increase the depth of the network.

The inverse block and the inverse residual block are in the form of a depthwise separable convolution composed of a connection between a depthwise convolution layer and a pointwise convolution layer. Convolution by depth is performed by dividing the convolution operation by channel of the feature map, and convolution by point is performed by dividing the space of the feature map. As the channel and space are divided and calculated, the number of parameters to be calculated is reduced, reducing the amount of calculation and maintaining the expressive power of information.

The classification unit 1264 detects whether or not the mask is normally worn from the face region image based on the output value of the fully connected layer 40 . Based on the output value (mask score) of the fully connected layer 40, the classification unit 1264 converts the image of the face region into one of a normal mask wearing state, a bad wearing state, a mask not wearing state, a hand covered state, and a prop covered state. can be classified as In particular, the classifier 1264 compares the output value (Mask score) of the fully connected layer 40 with a set threshold, and determines whether the output value (Mask score) of the fully connected layer 40 is the threshold. ), it is detected that the mask is in a normal wearing state. The classification unit 1264 detects that the mask is worn abnormally when the output value (Mask score) of the fully connected layer 40 is less than or equal to a threshold, and outputs the output value (Mask score) of the fully connected layer 40. Based on this, it can be classified into one of a state of poor wearing, a state of not wearing a mask, a state of covering hands, and a state of covering small items.

Based on the state value output from the deep learning detection unit 126, the decision unit 128 may output a normal mask when the mask is worn normally, and may output a result as an abnormal situation in other states.

FIG. 4 is a diagram illustrating an example of an inverse block of the convolution layer shown in FIG. 3 .

Referring to FIG. 4, the inverse blocks of the convolution layers 10, 20, and 30 are sequentially composed of a pointwise convolution operation block, a depthwise convolution operation block, and a pointwise convolution operation block. has a structure connected to Here, stride 2 is a block used for down sampling. The resolution is reduced by a stride of 2.

This inverse block is a structure that extracts meaningful features from the previous feature map while reducing the processing resolution.

FIG. 5 is a diagram illustrating an example of an inverse residual block of the convolution layer shown in FIG. 3 .

Referring to FIG. 5, in the inverse residual blocks of the convolution layers 10 and 20, a point-by-point convolution operation block, a depth-based convolution operation block, and a point-by-point convolution operation block are sequentially connected, and in this case, the number of channels is small. It has a structure in which point-by-point convolution operation blocks are connected.

This inverse residual block is a structure for extracting meaningful features at a processing resolution.

The deep learning network 1262 using the inverse block and the inverse residual block shown in FIGS. 4 and 5 processes the input image with 16 channels and then passes through the inverse block, increasing the number of channels to 2 each time the resolution is reduced by half. The feature value is output by increasing the number of channels by a factor of 64 and finally increasing to 64 channels.

Next, the feature value output from the last convolution layer 30 of the plurality of convolution layers 10, 20, and 30 passes through the fully connected layer 40 and then wears a mask through comparison with a threshold. It is judged whether

The Rectified Linear Unit (ReLU) function used as an activation function in deep learning causes information loss due to nonlinearity. In order to reduce the information loss caused by the activation function, the data is expanded (expansion) and then compressed to the original size through convolution by depth and convolution by point.

Meanwhile, the deep learning network 1262 using the inverse block and inverse residual block shown in FIGS. 4 and 5 uses the inverse block and inverse residual block shown in FIGS. 4 and 5, as shown in FIG. By having a structure, it is generated by stacking inverse blocks and inverse residual blocks to a depth that can reduce operation time by reducing the amount of calculation as much as possible while satisfying the performance for discriminating wearing a mask. In this way, a more lightweight network can be configured than the structure of the existing MobileNet V2.

6 is a flowchart illustrating a mask wearing detection method according to an embodiment of the present invention.

Referring to FIG. 6 , the mask wearing detection device 100 detects a face from an input image (S610). The mask wearing detection device 100 may use the already used face detector 110 for face detection.

The mask wearing detection device 100 cuts out the face region from the entire image using the face detection result (S620), and then performs face normalization by moving the positions of the eyes, nose, and mouth to designated positions in the face region (S630).

The mask wearing detection device 100 converts the image of the face area into a set size (S640) and converts the image of the face area into a black and white image (S650).

The mask wearing detection device 100 detects whether the mask is normally worn based on the deep learning network from the black and white image of the face area (S660). The mask wearing detection device 100 uses the black and white image of the face area as an input image of the deep learning network, and from the value output from the deep learning network, the image of the face area is classified as a normal mask wearing state, a bad wearing state, a mask not wearing state, a hand It can be classified into one of the covered state and small item covered state, output as normal when the mask is normally worn based on the classified state value, and output as abnormal in other states.

7 is a diagram illustrating a mask wearing detection device according to another embodiment of the present invention.

Referring to FIG. 7 , the mask wearing detection device 700 may represent a computing device in which the mask wearing detection method described above is implemented.

The mask wearing detection device 700 may include at least one of a processor 710, a memory 720, an input interface device 730, an output interface device 740, and a storage device 750. Each component may be connected by a common bus 760 to communicate with each other. In addition, each component may be connected through an individual interface or individual bus centered on the processor 710 instead of the common bus 760 .

The processor 710 may be implemented in various types such as an Application Processor (AP), a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), and the like, and executes commands stored in the memory 720 or the storage device 750. It may be any semiconductor device that The processor 710 may execute a program command stored in at least one of the memory 720 and the storage device 750 . The processor 710 stores program instructions for implementing at least some functions of the normalization processing unit 122, the image preprocessing unit 124, the deep learning detection unit 126, and the determination unit 128 described in FIG. 2 in the memory 720. By storing, it can be controlled so that the operation described based on FIGS. 1 to 6 is performed.

The memory 720 and the storage device 750 may include various types of volatile or non-volatile storage media. For example, the memory 720 may include read-only memory (ROM) 721 and random access memory (RAM) 722 . The memory 720 may be located inside or outside the processor 710 and may be connected to the processor 710 through various known means.

Input interface device 730 is configured to provide data to processor 710 .

Output interface device 740 is configured to output data from processor 710 .

At least some of the method for detecting wearing a mask according to an embodiment of the present invention may be implemented as a program or software running on a computing device, and the program or software may be stored in a computer-readable medium.

In addition, at least some of the mask wearing detection methods according to embodiments of the present invention may be implemented as hardware that can be electrically connected to a computing device.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. that fall within the scope of the right.

Claims (10)

In the method of detecting mask wearing from the entire image input from the mask wearing detection device,
Normalizing and pre-processing the image of the face region detected from the entire image, and
Detecting a mask normal wearing state and a mask abnormal wearing state in which the mask completely covers the nose and mouth corresponding to the respiratory organ based on a deep learning network from the normalized and preprocessed image of the face region
Including,
The deep learning network is
at least one convolutional layer; and
A fully connected layer generating one output value from data output from the at least one convolution layer,
The at least one convolution layer includes at least one inverse block for extracting features while reducing resolution, and at least one inverse residual block for extracting features by increasing the number of channels each time it passes through the inverse block. How to detect wearing a mask. In paragraph 1,
The normalization and preprocessing steps
Acquiring positions of eyes, noses, and mouths in the face region detected from the entire image, and
A mask wearing detection method comprising the step of moving the positions of the eyes, nose and mouth to designated positions in the face area. In paragraph 2,
The normalization and preprocessing steps
Cropping only the image of the face region from the entire image, and
and converting the image of the face region into a black and white image. In paragraph 1,
The step of detecting
classifying the image of the face region as a normal mask wearing state when the output value of the deep learning network is greater than a set threshold; and
and classifying the image of the face region as a mask abnormal wearing state when the output value of the deep learning network is less than or equal to the threshold value. In paragraph 1,
The step of classifying the mask into an abnormal wearing state is a step of classifying into one of a wearing state in which the nose or mouth is exposed, a mask not wearing a mask state, and a state covered by a hand or object based on the output value of the deep learning network. A mask wearing detection method comprising: In paragraph 1,
The normalization and preprocessing steps
Receiving a face detection result from a pre-installed face detector, and
and detecting a face region from the entire image based on the face detection result. In a mask wearing detection device that detects wearing a mask,
A normalization processing unit for cropping an image of a face region from an entire image based on a face detection result and setting positions of eyes, nose, and mouth in the face region; and
Using the image of the face region where the positions of the eyes, nose, and mouth are set as input data of the deep learning network, the mask completely covers the nose and mouth corresponding to the respiratory organ. Deep learning to detect normal wearing and abnormal wearing of the mask detection unit
Including,
The deep learning network is
at least one convolutional layer; and
A fully connected layer generating one output value from data output from the at least one convolution layer,
The at least one convolution layer includes at least one inverse block for extracting features while reducing resolution, and at least one inverse residual block for extracting features by increasing the number of channels each time it passes through the inverse block. Mask wearing detection device. In paragraph 7,
The deep learning detection unit includes a classification unit configured to classify the image of the face region into a mask normal wearing state and the mask abnormal wearing state by comparing the output value of the deep learning network with a set threshold. wear detection device. In paragraph 7,
An image pre-processing unit that converts the image of the face region where the positions of the eyes, nose, and mouth are set to a set size and black-and-white image and outputs the image to the deep learning network.
A mask wearing detection device further comprising a. In paragraph 7,
A face detector for detecting a face from the entire input image, generating the face detection result, and passing the face detection result to the normalization processing unit
A mask wearing detection device further comprising a.

Images