KR20230166568A - Hair follicle classification and hair loss severity estimation system and method - Google Patents

Abstract

This relates to a system and method for classifying hair follicles and estimating hair loss severity based on deep learning for image region segmentation. An image collection module that collects images of the scalp, and an area of the hair follicles is divided into a circle for the images collected in the image collection module. An extraction module for extracting, a discrimination module for determining the state of hair follicles in the region of hair follicles extracted from the extraction module, hair loss for each image according to the region of each hair follicle extracted from the extraction module and the state of the hair follicles determined in the discrimination module A configuration is provided that includes an estimation module that estimates the severity of hair loss, and a hair loss status determination module that visualizes, quantifies, and displays the hair loss severity of the entire scalp for the severity of hair loss estimated in the estimation module, using a deep learning algorithm. Local and scalp level hair loss severity can be estimated.

Description

Hair follicle classification and hair loss severity estimation system and method}

The present invention relates to a system and method for classifying hair follicles and estimating the severity of hair loss based on deep learning for image region segmentation. In particular, the pore region is extracted into a circular shape similar to the shape of the pore, and Mask is actively applied to image region segmentation. It relates to a hair follicle classification and hair loss severity estimation system and method that classifies hair follicles and estimates the severity of hair loss by applying the R-CNN framework and multitasking deep learning method.

In general, hair loss is a condition in which there is no hair in the area of the scalp where hair should be present. It means that the hair on the scalp gradually falls out. As hair loss progresses, the person becomes 10 to 20 years older than a normal person. Hair loss has a huge impact on a person's appearance because it makes them look thinner.

This kind of hair loss is generally caused by aging or genetic causes, but recently, not only due to aging or genetic factors, but also acquired causes such as environmental pollution, stress, abnormal hormonal secretion due to changes in diet, etc. It occurs in all age groups, and the rate of progression is accelerating.

Hair loss as described above is a disease with a high prevalence rate, and when it occurs, it can cause negative psychological effects on the individual, such as stress, lack of confidence, social phobia, and depression, and in severe cases, it can lead to mental illness. You can.

An example of a technology for detecting such a hair loss state is disclosed in Patent Documents 1 to 3 below.

For example, in Patent Document 1 below, a pore image generating unit extracts pores from at least one scalp image taken of the scalp and generates a pore image for the scalp image, and pores for the user's scalp from the generated pore image. Disclosed is a hair loss status monitoring device through pore matching, including a pore map generator that generates a map and a hair loss status monitor that generates the pore map in time series and monitors the hair loss status by matching the generated pore maps with each other. It is done.

In addition, Patent Document 2 below discloses a measurement unit that measures the hair thickness value and hair circumference value multiple times and outputs hair information including the hair thickness value and hair circumference value, and hair information is measured from the hair information output from the measurement unit. A portable device including a hair loss determination unit that calculates the average value of the thickness value and the hair circumference value and then compares it with a preset reference range to determine and output hair loss, and a display unit that displays the operation status of the measurement unit and the hair loss determination unit. A hair loss measuring device is disclosed.

In addition, Patent Document 3 below discloses a step of obtaining a head image of a user, extracting a first group of feature points and a second group of feature points from the head image, and corresponding to the boundary line of the user's hair and forehead in the head image. extracting a plurality of boundary points, selecting a matching boundary point corresponding to a feature point of the first group from among the plurality of boundary points, and matching the feature point of the first group and the corresponding feature point based on a first axis on the head image. Obtaining a calculated value of the upper facial region determined based on the distance between boundary points, calculating a calculated value of the middle and lower facial region determined based on the distance between the feature points of the first group and the feature points of the second group with respect to the first axis. A method of providing hair loss status information including the step of obtaining and providing hair loss status information of the user based on a ratio of the calculated value of the upper facial portion and the calculated value of the middle or lower facial portion is disclosed.

Meanwhile, in Non-Patent Document 1 below, hair density measurement (HDM) is a process used to detect the severity of hair loss by counting the number of hairs present in the occipital donor area for transplantation, in which the area of pores is extracted as a square, The accuracy of HDM is analyzed by applying deep learning technology to object detection and the possibility of HDM automation is disclosed, and the data set for training and evaluation exists in 4,492 enlarged hair scalp RGB images and images acquired from male hair loss patients. It consists of corresponding annotation data including location information of hair follicles and hair follicle type information according to the number of hairs, and EfficientDet, YOLOv4, and DetectoRS are disclosed for technologies used as object detection algorithms for performance comparison.

Republic of Korea Patent Publication No. 10-2303429 (registered on September 13, 2021) Republic of Korea Patent Publication No. 2018-0110856 (published on October 11, 2018) Republic of Korea Patent Publication No. 10-2272468 (registered on June 28, 2021)

M. Kim, S. Kang, and B.-D. Lee, “Evaluation of Automated Measurement of Hair Density Using Deep Neural Networks,” Sensors, vol. 22, no.2, p.650, 14 January 2022.

Patent Document 1 as described above discloses a technology that can confirm and monitor the progress of hair loss by tracking and comparing changes in pore patterns in time series based on the matched pore map, but the severity of hair loss can be measured. I couldn't judge. That is, in Patent Document 1, the progress of hair loss can be monitored from a plurality of collected scalp images, but the severity of hair loss cannot be determined because a standard for determining the type of hair loss has not been established.

In addition, Patent Document 2 discloses a technology that can diagnose the state of hair loss by measuring the hair thickness value and the hair circumference value. However, in order to measure the hair thickness, some hair must be cut off, and the circumference In order to measure length, all hair must be collected and measured, so there was a problem that it could not be measured in the case of people with short hair.

In addition, in Patent Document 3, matching boundary points are selected based on the boundary points and feature points of the head image, and hair loss status is determined based on the boundary points of the user's hair and forehead, so hair loss occurring in the central part of the scalp cannot be measured. There was a problem.

Meanwhile, the non-patent document 1 is a technology for extracting the pore area in a rectangular shape, identifying the pores, calculating the number of pores, and estimating the hair density, but the problem is that the extraction performance is reduced and the state of hair loss cannot be determined. There was.

As described above, the prior art did not disclose a technology capable of determining the severity of hair loss depending on the state of hair loss.

The purpose of the present invention is to solve the problems described above, and provide a system and method for classifying hair follicles and estimating the severity of hair loss that can visually display the severity of hair loss and determine the severity of hair loss according to the state of hair loss. It is done.

Another object of the present invention is to provide a system and method for classifying hair follicles and estimating hair loss severity that can determine the type of hair loss according to the state of hair loss.

Another object of the present invention is to provide a system and method for classifying hair follicles and estimating hair loss severity that can accurately determine hair loss at high speed.

In order to achieve the above object, the hair follicle classification and hair loss severity estimation system according to the present invention includes an image collection module that collects images of the scalp, an extraction module that extracts a circular area of the hair follicle for the images collected in the image collection module, A discrimination module that determines the state of hair follicles in the area of the hair follicles extracted from the extraction module, and an estimation that estimates the severity of hair loss for each image according to the area of each hair follicle extracted from the extraction module and the state of the hair follicles determined in the discrimination module. It is characterized in that it includes a hair loss status determination module that visualizes, quantifies, and displays the hair loss severity of the entire scalp with respect to the severity of hair loss estimated in the estimation module.

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the extraction of the hair follicle area in the extraction module and the determination of the state of the hair follicle in the discrimination module are performed simultaneously.

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the extraction module is characterized by extracting the hair follicle area in a circular shape by applying Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101).

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the condition of the hair follicles in the determination module is determined as good, normal, or severe depending on the number of hair follicles, the thickness of the hair, and the number of hairs in each hair follicle. do.

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the severity estimation in the estimation module is performed by estimating the hair loss severity index (P) according to the following equation,

Here, k refers to the 12 locations, n refers to the number of hair follicles, and i refers to the three conditions: i={1 (serious), 2 (normal), and 3 (good)}. α refers to the weight for the state of the hair follicles, and β refers to the weight for the total number of hair follicles. And by normalizing the results for the entire value, the hair loss severity of 12 areas where the scalp was photographed is calculated as a value between 0 and 1 through equation (1).

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the severity is judged when the radius of the hair follicle is less than 10 pixels, the number of hairs per follicle is 1, and the thickness of the hair is less than 3 pixels, and the normal judgment is It is judged when the radius of the hair follicle is between 10 pixels and 20 pixels, the number of hairs per follicle is 1, and the thickness of the hair is between 3 pixels and 6 pixels, and the above judgment of good is when the radius of the hair follicle is greater than 20 pixels and the number of hairs per follicle is It is characterized in that it is judged when the number is 2 or more and the thickness of the hair exceeds 6 pixels.

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, the visualization of the hair loss severity in the hair loss status determination module is displayed as a heat map to determine the hair loss type, and the hair loss severity is quantified as a value between 0 and 1. It is characterized by

In addition, in order to achieve the above object, the method of classifying hair follicles and estimating hair loss severity according to the present invention includes (a) photographing the scalp using an imaging device consisting of a micro microscope (Dino-Lite) and collecting scalp images at 200 times magnification; (b) extracting the hair follicle area as a circle in the extraction module for the image collected in step (a) and simultaneously determining the state of the hair follicle in the discrimination module, (c) in step (b) After classifying the hair follicles and determining the state of the hair follicles, estimating the severity of hair loss for each image in an estimation module according to the area of each hair follicle and the state of the hair follicles, (d) the severity of hair loss estimated in step (c) It is characterized in that it includes the step of executing visualization and quantification in the hair loss status determination module.

In addition, in the hair follicle classification and hair loss severity estimation method according to the present invention, the extraction in step (b) involves extracting the hair follicle area in a circular shape by applying Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101). It is characterized by

In addition, in the hair follicle classification and hair loss severity estimation method according to the present invention, the condition of the hair follicles in step (b) is determined as good, normal, or severe depending on the number of hair follicles, the thickness of the hair, and the number of hairs in each hair follicle, and In step (c), severity estimation is performed by estimating the hair loss severity index (P) according to the following equation,

Here, k refers to the 12 locations, n refers to the number of hair follicles, and i refers to the three conditions: i={1 (serious), 2 (normal), and 3 (good)}. α refers to the weight for the state of the hair follicles, and β refers to the weight for the total number of hair follicles. And by normalizing the results for the entire value, the hair loss severity of 12 areas where the scalp was photographed is calculated as a value between 0 and 1 through equation (1).

As described above, according to the hair follicle classification and hair loss severity estimation system and method according to the present invention, the extraction module for extracting the hair follicle area in a circular shape, the discrimination module for determining the state of the hair follicle in the hair follicle area, and the severity of hair loss for each image By providing an estimation module that estimates , an efficient and accurate method for detecting and classifying hair follicles and a deep learning algorithm can be used to estimate local and scalp level hair loss severity.

In addition, according to the system and method for classifying hair follicles and estimating hair loss severity according to the present invention, once scalp imaging is completed, the individual's hair loss status can be accurately and efficiently informed, giving the effect of giving clinical significance.

In addition, according to the hair follicle classification and hair loss severity estimation system and method according to the present invention, the effect of being able to visually express the overall hair loss severity estimate of hair loss using a heat map is obtained.

1 is a block diagram of a hair follicle classification and hair loss severity estimation system according to the present invention;
Figure 2 is a diagram showing the results of hair follicle area extraction using Mask R-CNN in the extraction module according to the present invention;
Figure 3 is a diagram showing the labeling results using VIA (VGG Image Annotation);
Figure 4 is a photo showing the results of extracting the hair follicle area and determining its status by applying Mask R-CNN based on ResNet 101 to the image taken by the image collection module;
Figure 5 is a diagram showing the results of visualizing and quantifying the severity of hair loss on the entire scalp according to the present invention;
Figure 6 is a process chart for explaining the process of classifying hair follicles and estimating hair loss severity according to the present invention;
Figure 7 shows the model architecture of Mask-RCNN applied to the present invention;
Figure 8 is a diagram showing the hair follicle classification results of the training data set (top) and test data set (bottom) comparing ResNet 101 and ResNet 50;
Figure 9 is a diagram showing the confusion matrix results of hair follicle classification.

The above and other objects and new features of the present invention will become more clear by the description of this specification and the accompanying drawings.

The term “module” or “unit” used herein performs at least one function or operation, and may be implemented as hardware or software consisting of a mechanical or electrical or electronic configuration, or as a combination of hardware and software, and may be implemented as a plurality of functions or operations. A “module” or a plurality of “units” may be integrated into at least one module and implemented with at least one processor, except for a “module” or “unit” that needs to be implemented with specific hardware.

First, hair loss for hair follicle classification and hair loss severity estimation applied to the present invention will be described.

Androgenetic alopecia, or hair loss mediated by the presence of androgen dihydrotestosterone, refers to hereditary hair loss and is the most common form of hair loss (alopecia) in men and women, affecting up to 50% of the world's population. It affects. In particular, in Asian men, the incidence of male pattern baldness at all ages is 14.1%, and it steadily increases with age. For example, alopecia areata is an autoimmune disease in which hair falls out in one or more small clumps, and is defined as a lifelong, non-scarring form of hair loss that begins before the age of 30. The condition is unknown, but is more common in people with other autoimmune diseases, and the same process can lead to complete loss of hair on the scalp (alopecia totalis).

Telogen effluvium is the second most common form of hair loss, and is a change in the number of hair follicles from which hair grows. It is associated with thinning of the hair on the scalp and occurs more frequently at the top of the scalp. Anagen effluvium is another form of hair loss that results in more rapid hair loss. Traumatic hair loss is alopecia caused by external damage, such as hair pulling, hair exposure to extreme heat, or strong chemicals that damage the hair (bleaching, hair dye, permanent).

In addition, major causes of hair loss include genetics, hormonal changes, diseases such as diabetes and lupus, medications and supplements, and a normal part of aging. It can also be caused by severe stress or emotional trauma. Hair loss is most commonly caused by heredity and becomes more common with age, and an accurate and timely diagnosis of the type, severity and progression of hair loss can help develop the right treatment and intervention methods and delay the onset or further progression of hair loss, leading to a more fundamental It is essential for reducing costs associated with medical treatment or surgical procedures.

Common hair loss diagnostic methods widely adopted and practiced in clinical settings are hair pluck testing, pluck testing, scalp biopsy, daily hair counting, or trichoscopy. Among these, hair colposcopy uses computer measuring tools to analyze the characteristics of the scalp and hair, and generally uses a small device with a high-precision microscope camera to take pictures of the scalp and hair, and objectively diagnose and monitor the progression of hair loss. It provides the means to do so and provides information about the hair condition, hair follicles, and scalp, including the total number of hairs on the scalp and the diameter of each hair strand. Therefore, although hair colposcopy is currently considered the most effective method for the diagnosis and prognosis of hair loss, this method relies heavily on visual inspection of digital images in the hair colposcope, is prone to human error, and produces different results from clinician to clinician. There is a limitation in that it takes a lot of money and time to educate and train individuals.

Due to these limitations, advanced tools and techniques are being developed that leverage computer-based processing and analysis, especially deep learning, and multilayer feed-forward neural networks, a type of artificial neural network, are being developed based on gender, age, and genetic factors. , used a backpropagation algorithm to predict hair loss based on eight attributes: surgery, pregnancy, zinc deficiency, iron deficiency, anemia, and use of cosmetics. This method showed good accuracy in a two-layer neural network used with the Levenberg-Marquardt algorithm. A difference of Gaussian (DoG) filter was applied for follicle detection using a Support Vector Machine (SVM) classifier. However, this method relies heavily on features selected from microscopic scalp hair images, which are prone to errors. An autonomous hair segmentation and counting method was also proposed, which showed an accuracy of 95.3% and a recall of 88.6% when classifying wavy and curly hair, but similarly its main drawback was manual parameter selection.

Other classification methods, such as scalp stain detection and hair condition assessment using microscopic images, deep learning methods to classify patterned baldness and hair loss levels in facial images, or deep learning to classify four common scalp hair symptoms (dandruff). A method was also developed. However, more accurate, efficient, and robust methods are needed to more comprehensively infer hair loss severity at the local and scalp level, such as by counting the number of hairs or measuring the diameter of each strand of hair.

Recently, a technology has been developed to extract hair loss features by combining scalp images taken with a microscope using grid line selection and eigenvalues and identifying the progress of hair loss. Additionally, a machine learning model was developed to classify healthy hair and alopecia areata using SVM (Support Vector Machine) and KNN (k-nearest neighbor). We tested the possibility of automating the hair density assessment process by measuring the number and type of hairs in hair follicles using different methods such as EfficientDet, YOLOv4, and deep learning-based object detection. Common drawbacks of these approaches are the prevalence of false detection of hair counts, inaccuracy in hair follicle detection, and lack of full scalp-level hair loss severity estimation.

In order to solve the limitations of the hair loss detection algorithm as described above, the present invention applies the Mask R-CNN (Convolutional Neural Network) framework and multitasking deep learning method to provide an intelligent system that classifies hair follicles and estimates the severity of hair loss. did.

The present invention collects images of 12 areas of the scalp through a 200x magnification micro microscope (Dino-Lite Microscope) that photographs the scalp, and applies the hair follicle region segmentation technology based on Mask R-CNN developed in the present invention to determine the size of hair follicles. It determines the number, hair thickness, and number of hairs in each hair follicle, and applies multitasking deep learning techniques to classify the condition of each hair follicle into three types (good, normal, and serious). Based on the information obtained in the previous step, the hair loss severity in 12 areas of the scalp was calculated as a normalized value of 0 to 1, and based on this, the hair loss severity of the entire scalp was visualized and quantified as a value between 0 and 1.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

Figure 1 is a block diagram of a hair follicle classification and hair loss severity estimation system according to the present invention.

As shown in FIG. 1, the hair follicle classification and hair loss severity estimation system according to the present invention includes an image collection module 100 that collects images of the scalp, and an area of the hair follicle for the images collected by the image collection module 100. An extraction module 200 for extracting a circular shape, a determination module 300 for determining the state of a hair follicle in the area of the hair follicle extracted from the extraction module 200, an area of each hair follicle extracted from the extraction module 200, and An estimation module 400 that estimates the severity of hair loss for each image according to the state of the hair follicles determined in the determination module 300, and visualizes the severity of hair loss estimated in the estimation module 400 as the severity of hair loss on the entire scalp, , may include a hair loss status determination module 500 that quantifies and displays the hair loss status.

For example, the image collection module 100 may include an image capture device consisting of a micro microscope (Dino-Lite), and image information from 12 locations on the scalp is collected as a scalp image with a magnification of 200 times. Images collected in the image collection module 100 can be connected to the extraction module 200 through, for example, USB, and the images are transmitted and processed in real time by applying OpenCV and Python technology. A total of 600 collected images applied to the present invention were of Korean men aged 25 to 55, and were collected from various scalp locations.

The extraction module 200 applies Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101) to extract the hair follicle area in a circular shape, as shown in FIG. 2. For this purpose, the area of each hair follicle was labeled using VIA (VGG Image Anotation Tool) as shown in Figure 3. The number of labeling was 24,012, and Mask R-CNN learning was conducted by dividing them into learning and verification purposes at a ratio of 3:1.

Figure 2 is a diagram showing the results of hair follicle area extraction using Mask R-CNN in the extraction module according to the present invention, and Figure 3 is a diagram showing the labeling results using VIA (VGG Image Annotation). Figure 3 shows the labeling scheme for the three classifications using the VIA tool, where 0 and the red circle represent “severe,” 1 and the blue circle represent “moderate,” and 2 and the green circle represent “good.”

The determination module 300 determines the condition of the hair follicles shown in FIGS. 2 and 3, and the condition of the hair follicles is classified into three types: serious, normal, and good. These three types were classified based on hair thickness and number of hairs in hair follicles. For example, in severe cases, the radius of the hair follicle is less than 10 pixels, the number of hairs per follicle is 1, and the thickness of the hair is less than 3 pixels. In a normal case, the radius of the hair follicle is between 10 pixels and 20 pixels, the number of hairs per follicle is 1, and the thickness of the hair is between 3 pixels and 6 pixels. In the case of good, the radius of the hair follicle is more than 20 pixels, the number of hairs per follicle is 2 or more, and the thickness of the hair is more than 6 pixels.

The determination module 300 applies multitasking deep learning technology based on the thickness of the hair and the number of hairs in the hair follicle in the area extracted by the extraction module 200 to determine the state of each hair follicle, as shown in FIG. It is classified into three levels: good, normal, and serious.

Figure 4 is a photograph showing the results of extracting the hair follicle area and determining its condition by applying Mask R-CNN based on ResNet 101 to the image captured by the image collection module. Figures 4 (a) to (f) show good to severe hair loss grades, respectively. In Figure 4, the red circle represents the severe label, the blue circle represents the normal label, the green circle represents the good label, and the purple circle appears because the classification of two very adjacent hair follicles (severe and normal) is different. Additionally, the bounding box with the corresponding predicted label in each image is indicated with a confidence probability.

The estimation module 400 is provided to estimate the hair loss severity for each image, and on the basis of determining the area of each hair follicle and the state of the hair follicle, the hair loss severity index (P ) is estimated. The severity of hair loss is calculated based on the number of hair follicles for each of the three hair follicle conditions and the total number of hair follicles.

… (One)

Here, k refers to the 12 locations, n refers to the number of hair follicles, and i refers to the three conditions: i={1 (serious), 2 (normal), and 3 (good)}. α refers to the weight for the state of the hair follicles, and β refers to the weight for the total number of hair follicles. And by normalizing the results for the entire value, the hair loss severity of 12 areas where the scalp was photographed can be calculated as a value between 0 and 1 through equation (1).

The hair loss status determination module 500 estimates the severity of hair loss in 12 areas of the entire scalp collected by the image collection module 100 as a value between 0 and 1 and visualizes it as shown in FIG. 5.

Figure 5 is a diagram showing the results of visualizing and quantifying the severity of hair loss on the entire scalp according to the present invention. Figures 5 (a) and (d) show an overall good condition, Figures 5 (b) and (e) show a subject with moderate hair loss, and Figures 5 (c) and (f) show a subject with severe hair loss. Representative top view head photos and heatmaps of each hair loss severity. The upper part of Figure 5 displays the local hair loss severity index (P) along with images taken at various locations on the scalp. The hair loss severity estimates P _avg for good, moderate, and severe hair loss are 0.879, 0.565, and 0.160, respectively, indicating values that gradually decrease as hair loss progresses.

As shown in Figure 5, visualization in the hair loss status determination module 500 according to the present invention assumes the entire scalp as one round rectangle and applies two-dimensional interpolation to the severity value for each location and the values between them. estimate. Based on these results, a heat map of the severity of hair loss is output. Additionally, the current hair loss severity can be quantified as a value between 0 and 1 through the average of the severity values for each 12 locations. The heat map as described above can be displayed on a display member provided in the system.

Meanwhile, in the above description, the extraction module 200, the determination module 300, the estimation module 400, and the hair loss status determination module 500 are described separately, but are not limited thereto and are continuously operated by a program provided in the computer system, etc. It can be executed with . That is, the image collected by the image collection module 100 is transmitted in real time to the computer system via USB, and the above-described extraction module 200, determination module 300, estimation module 400, and hair loss status determination module ( In 500), the hair follicle area segmentation technology based on Mask R-CNN is applied to display the severity of hair loss as shown in FIG. 5 on the display member.

In addition, in the hair follicle classification and hair loss severity estimation system according to the present invention, hair loss can be determined at high speed and accurately by simultaneously extracting the hair follicle area as shown in FIG. 2 and determining the condition of the hair follicle as shown in FIG. 3. . Additionally, as shown in Figure 5, the type of hair loss can be determined by displaying the severity of hair loss on a display member as a heat map.

On the other hand, the hair follicle classification and hair loss severity estimation system according to the present invention is configured to be provided by individual users through a user terminal such as a smart phone equipped with a camera function, or can be provided by a hair care shop such as a plastic surgery clinic, beauty salon, or Lee Yong-won. It can be installed and used in various hair care institutions or service shops.

Next, a method of estimating the severity of hair loss by applying the hair follicle classification and hair loss severity estimation system shown in FIG. 1 will be described with reference to FIGS. 6 and 7.

Figure 6 is a process diagram for explaining the process of classifying hair follicles and estimating hair loss severity according to the present invention, and Figure 7 is the model architecture of Mask-RCNN applied to the present invention.

In order to classify hair follicles and estimate hair loss severity according to the present invention, first, as shown in FIGS. 6 and 7, the scalp 12 is captured using an imaging device consisting of a micro microscope (Dino-Lite) in the image collection module 100. Photograph the area and collect scalp images at 200x magnification (S10).

Next, the extraction module 200 applies Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101) to extract the hair follicle area in a circular shape from images taken of 12 scalp locations, as shown in Figure 2. At the same time, the status of the hair follicle is determined in the determination module 300 (S20).

Mask R-CNN, applied in step S20, is one of the object detection and localization algorithms that uses deep learning and image segmentation and is used in various fields such as medical and industrial applications. This Mask R-CNN is an extension of Faster R-CNN and consists of two stages. The first step checks the image and then generates suggestions, and the second step classifies the suggestions and creates masks and bounding boxes. In addition, we perform region proposal of the object through box regression and classification, and classify the class to which the image corresponds. Mask R-CNN's backbone network adopted ResNet 50 and ResNet 101. ResNet 50 operates with 50 network layers and 25.6 million parameters, and has the advantage of faster training speed and faster algorithm execution than ResNet 101. On the other hand, ResNet 101 operates on 44.5 million parameters with 101 network layers, so the training and algorithm execution speed is relatively slower than ResNet 50 at the expense of increasing accuracy.

Therefore, in the present invention, after resizing the input image, the data is passed to a feature extractor using ResNet 101 pre-trained on the COCO data set. Next, after obtaining the feature maps, the regions of interest (ROIs) are sorted to find relevant regions in the feature maps. It allows multiple layers of a neural network to learn features at different scales. It includes three branches: head to mask, bounding box regression, and classification. For training, the head is fine-tuned N times, the pre-trained weights are fixed and all trainable layers are subsequently trained N times.

In the process shown in Figure 7, the ResNet model is first used as a feature extractor of the input image, and the extracted features are used for bounding box regression. Next, the trained model implements the process of localizing and classifying hair follicles by class and extracting feature factors. Each grade was classified according to the condition of the hair follicles, that is, the number of hairs per unit area, the results were obtained, and the average value was taken. After going through the above process, the distribution of the extracted feature factors is analyzed and classified into three levels: good, normal, and severe. The score for each hair follicle is calculated and normalized to a range between 0 and 1.

Labeling standards are determined by three factors: number of hair follicles, hair thickness, and number of hairs in each follicle. The serious label is when the radius of the hair follicle is less than 10 pixels, the number of hairs per follicle is 1, and the thickness of the hair is less than 3 pixels. Usually, the label is used when the radius of the hair follicle is between 10 pixels and 20 pixels, the number of hairs per follicle is 1, and the thickness of the hair is between 3 pixels and 6 pixels. A good label indicates that the radius of the hair follicle is more than 20 pixels, the number of hairs per follicle is 2 or more, and the thickness of the hair is more than 6 pixels. Additionally, because the training model in the present invention focused on the degree of hair loss, learning was conducted without considering other factors such as red spots, oil, and dead skin cells.

Meanwhile, because hair loss in women is generally less severe than in men, we only collected images of men (all Asian) across age groups and at various locations on the scalp. Additionally, at the time of data acquisition, hair loss on the head and sideburns were not prevalent, so data were mainly obtained from the upper part of the scalp. A total of 600 images were collected from 10 men (aged between 25 and 55 years), and data labeling for each image was performed by a clinician, resulting in 10 to 45 region annotations per image. The total number of labels was 24,012, of which 3,836 were serious, 11,262 were normal, and 8,914 were good.

The training to test ratio of the dataset was set from 0.75 to 0.25, VGG Image Annotation (VIA) tool was used for mask annotation (circular border around each detected hair follicle) and the exact labeling performed for each hair follicle is shown in Figure 3 As shown, it was validated by several clinicians.

In addition, training to determine the status of hair follicles in step S20 was performed using the TensorFlow v2 framework running on an AMD Ryzen 9 3950X 16-core processor equipped with Nvidia Titan x3 and 32GB RAM in an Ubuntu 18.04 5 LTS environment. Meanwhile, using ADAM (Adaptive Moment Estimation) as the optimizer, the batch size between processes was 25 and the learning rate was 1x10 ^-4 . Model training was performed by fine-tuning a pre-trained ResNet model on the COCO dataset. After training the head for 50 epochs and the rest of the body for 50 epochs, the bbox loss converged and training was terminated at that point. 450 images were used for training (75% of the data), while 150 images were used for testing (25% of the data). Additionally, the present invention performed instance segmentation, which can detect and classify the health status of hair follicles. It is worth noting that training was performed on randomly selected images from the entire dataset without considering the local context, i.e. the part of the scalp from which the images were taken.

In addition, data augmentation was performed to improve the accuracy and quality of the test image used for model training and testing to determine the state of the hair follicle in step S20. First, adjust brightness and contrast to compensate for differences in ambient light during image capture for consistency in test images. Next, if the test image was flipped or tilted due to the camera angle in the image acquisition module 100, appropriate correction was made to improve the normality of the data. Lastly, Gaussian Blur was used to improve the model's image recognition performance. These steps improved the usefulness and accuracy of the images, improving the performance and efficiency of model training and testing. The bounding box of the hair follicle was created based on the width and height of the hair follicle according to equations (2) and (3) below.

… (2)

… (3)

In addition, the number of hairs using the number of hair follicles along with the partial least squares (PLS) method of equation (3) below was used as an indicator to measure the severity of hair loss.

… (4)

In the above equations (2) to (4), m and n represent the number of hair follicles and the number of images in the training data set, respectively, Area _ij is calculated as the area of the bounding box for each hair follicle, and y _ij is for each hair follicle. It is a prediction label. Therefore, PLS is calculated as the weighted sum of each image considering the area of the bounding box.

The multitasking loss function of Mask R-CNN is a combination of the losses of classification, localization, and segmentation masks. That is, L = L _cls + L _box + L _mask . Here, L _cls and L _box are the same as Faster R-CNN. Mask branching creates a mask of dimension m×m for each ROI and each class (k classes in total). Therefore, the total output is of size Km ² . Because the model learns masks for each class, there is no competition between classes to generate masks. L _mask is defined as the average binary cross-entropy loss and contains only the kth mask if the region is associated with the ground truth class k. The detailed loss used in each mask calculation is given in Equation (5) below.

… (5)

where y _ij is the label (i, j) of the cell in the actual mask for an area of size m×m, is the predicted value of the same cell in the mask learned for ground true class k.

The bounding box adaptively changes its size according to the size of the hair follicle, as shown in Figure 4, which is a photo of the test result of Mask R-CNN trained using ResNet 101 as a feature extractor. As shown in Figure 4, because the size of each image is slightly different, all images were processed after being resized to a consistent width of 1,024 pixels, and the same scaling was applied to the annotation mask for training. Mask R-CNN was applied in the next process. First, the health status of the hair was classified into 3 levels (good, normal, serious), and then training was conducted based on the corresponding label information. Next, the degree of hair loss was evaluated based on the number of hair follicles and hair condition through hair follicle detection. Because hair distribution and hair loss vary in different areas of the scalp, data were collected from 12 different locations on the scalp. During the model inference phase, each image took approximately 2 minutes to process and output the results, including detection, localization, and classification of hair follicles, which is reasonably fast and efficient.

Also, in step S20, two models, ResNet 50 and ResNet 101, were used to classify hair follicles using Mask R-CNN. The performance of these two models was evaluated on training and test datasets using four utility metrics commonly used to evaluate the performance of deep learning models: precision, recall, f1 score, and accuracy. Overall, in all utilization metrics evaluated for both training and test datasets, ResNet 101 outperformed ResNet50 due to having 2x more layers and 1.7x more parameters compared to ResNet 50.

Additionally, the performance of each model was evaluated separately for each label on the training and test data sets, as shown in Figure 8. Figure 8 is a diagram showing the hair follicle classification results of the training data set (top) and test data set (bottom) comparing ResNet 101 and ResNet 50.

As can be seen in Figure 8, the results confirmed the superior performance of ResNet 101 over ResNet 50 in all three labels for both training and test data sets. On the training dataset, ResNet 50 reached 66% to 74%, while ResNet 101 reached 82% to 87%. On the test dataset, ResNet 50 reached 55% to 72%, while ResNet 101 reached 76% to 86%. Relatively low performance was observed for the severe label of 66% to 70% in ResNet 101 and 55% to 67% in ResNet 50 compared to both normal and healthy labels. This may be due to the smaller sample of severe (3,836) than moderate (11,262) and good (8,914).

Additionally, a confusion matrix as shown in Figure 9 was created to compare the misclassification rate between the two models for both the training and test data sets. Figure 9 is a confusion matrix result of hair follicle classification, where the upper left shows the ResNet 50 training data set, the upper right shows the ResNet 101 training data set, the lower left shows the ResNet 50 test data set, and the lower right shows the ResNet 101 test data set.

The ResNet 50 training dataset misclassified 633 and 308 severe labels into moderate and good labels, respectively, which is larger than the ResNet 101 training dataset (429 and 83), respectively. Both models incorrectly classified severe cases as moderate more than twice as often as good cases. Similarly, the ResNet 50 training dataset showed 600 and 1,491 misclassifications of moderate labels as severe and good, respectively, which is more than the ResNet 101 training dataset (347 and 985), respectively. Both models incorrectly classified moderate cases as good more than 2.5 times more than severe cases. Finally, the ResNet 50 training dataset showed 198 and 1,491 misclassifications of good labels as severe and moderate, respectively, which are larger than the ResNet 101 training dataset (0 and 1,012), respectively. Both models misclassified good as moderate, but not as much as severe. Overall, most misclassifications occurred between severe and moderate and between moderate and good, as expected. In the training data set, the misclassification rates of ResNet 50 and RestNet 101 were 0.274 and 0.165, respectively, showing that supporting RestNet 101 performed better than ResNet 50. A similar trend was observed in the test dataset between ResNet 50 and RestNet 101, with misclassification rates of 0.316 and 0.207, respectively. In particular, the misclassification rate between “severe” and “good” was more than five times higher in ResNet 50 (0.071 in training, 0.114 in test data set) than in ResNet 101 (0.010 in training, 0.025 in test data set). These results support the rationale for selecting ResNet 101 over ResNet 50 as the system architecture of Mask R-CNN in the present invention.

To further verify the performance of the ResNet 101 model compared to other alternatives, we performed a benchmark comparison similarly applied to hair follicle detection classification with other models (EfficientDet and YOLOv4), as shown in Table 2. Table 2 presents a performance comparison between deep learning models applied to hair follicle classification.

As can be seen in Table 2, ResNet 50 showed better performance compared to EfficientDet. However, performance was inferior compared to YOLOv4. On the other hand, ResNet 101 showed superior performance compared to all other models compared. Additionally, in the present invention, a prototype was applied to mask the region of interest (ROI) to train the model. This may be the reason why the mAP value of ResNet 101 is higher compared to other models because it is more similar to the shape of an actual hair follicle, which can improve Intersection over Union (IoU).

Next, after classifying the hair follicles and determining the state of the hair follicles in step S20, the severity of hair loss for each image is estimated in the estimation module 400 according to the area and state of each hair follicle (S30).

As described above, the results of Mask R-CNN using ResNet 101 in step S20 classified all detected hair follicles as good, normal, and severe. Using this information, the Hair Loss Severity Index (P) is calculated by applying Equation (1) described above based on the number of hair follicles for each of the three hair follicle states and the total number of hair follicles, normalized to a range between 0 and 1. It is estimated to indicate the severity of hair loss.

Meanwhile, in the present invention, visualization and quantification are performed in the hair loss status determination module 500 according to the severity of hair loss estimated in step S30 (S40).

That is, as shown in Figure 5, the upper region of the scalp is divided into 12 sub-regions, specifically the left, center, and right sides of the scalp, and each is divided into 4 regions from front to back according to the ratio, The overall severity and type of hair loss was identified. The hair loss severity estimate P _avg was calculated as the average of the P values obtained from 12 locations on the scalp, as shown in Equation (6).

… (6)

P _avg is a scale of 0 to 1 that qualitatively indicates the severity of hair loss in an individual. Here m is the number of images, which in this case is 12.

Mapping of the localized hair loss severity index to a graphical representation of the scalp was performed using a heatmap that visualizes the overall hair loss severity of the entire scalp. Figure 5 shows data on good hair loss, moderate hair loss, and severe hair loss from three representative people to show the process from calculating the local hair loss severity index to estimating and mapping hair loss severity.

In other words, mapping for visualization and quantification of the severity index first presents a representative image of the main area of the scalp and the hair loss severity index P of the corresponding area. P _avg was then calculated for each case after processing ^{the k} P values of all 12 positions, which corresponds to hair loss severity of 0.879 for good, 0.565 for moderate hair loss, and 0.160 for severe hair loss. Hair loss severity mapping using a heatmap indicates areas of good to severe loss, as shown in Figures 5 (d)-(f).

Therefore, the hair follicle classification and hair loss severity estimation method according to the present invention can provide an intuitive method to visualize various types of hair loss such as M type, U type, or O type as well as the hair loss area and condition. Additionally, this visual mapping of hair loss severity can also be used as a diagnostic and/or predictive tool or to evaluate treatment results if mapping is performed periodically.

Although the invention made by the present inventor has been described in detail according to the above-mentioned embodiments, the present invention is not limited to the above-described embodiments and can of course be changed in various ways without departing from the gist of the invention.

By using the hair follicle classification and hair loss severity estimation system and method according to the present invention, local and scalp level hair loss severity can be estimated using deep learning algorithms.

100: Image acquisition module
200: extraction module
300: Discrimination module
400: Estimation module
500: Hair loss status determination module

Claims (10)

An image collection module that collects images of the scalp,
An extraction module that extracts a circular area of the hair follicle from the image collected in the image collection module,
A determination module that determines the status of hair follicles in the area of hair follicles extracted from the extraction module,
An estimation module that estimates the severity of hair loss for each image according to the area of each hair follicle extracted in the extraction module and the state of the hair follicle determined in the determination module,
A hair follicle classification and hair loss severity estimation system comprising a hair loss status determination module that visualizes, quantifies, and displays the hair loss severity of the entire scalp with respect to the hair loss severity estimated in the estimation module. In paragraph 1:
A hair follicle classification and hair loss severity estimation system, characterized in that the extraction of the hair follicle area in the extraction module and the determination of the state of the hair follicle in the determination module are performed simultaneously. In paragraph 2,
The extraction module is a hair follicle classification and hair loss severity estimation system characterized in that the hair follicle area is extracted in a circular shape by applying Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101). In paragraph 3,
A hair follicle classification and hair loss severity estimation system characterized in that the condition of the hair follicles in the determination module is judged as good, normal, or severe depending on the number of hair follicles, hair thickness, and the number of hairs in each hair follicle. In paragraph 4,
In the estimation module, severity estimation is performed by estimating the hair loss severity index (P) according to the following equation,

Here, k refers to the 12 locations, n refers to the number of hair follicles, and i refers to the three conditions: i={1 (serious), 2 (normal), and 3 (good)}. α refers to the weight for the state of the hair follicles, and β refers to the weight for the total number of hair follicles. And a hair follicle classification and hair loss severity estimation system characterized by calculating the hair loss severity of 12 areas where the scalp was photographed as a value between 0 and 1 through equation (1) by normalizing the results for the entire value. In paragraph 4,
The severity is judged when the radius of the hair follicle is less than 10 pixels, the number of hairs per follicle is 1, and the thickness of the hair is less than 3 pixels,
The above normal judgment is made when the radius of the hair follicle is between 10 pixels and 20 pixels, the number of hairs per follicle is 1, and the thickness of the hair is between 3 pixels and 6 pixels,
The hair follicle classification and hair loss severity estimation system is characterized in that the judgment of the good condition is made when the radius of the hair follicle is more than 20 pixels, the number of hairs per follicle is 2 or more, and the thickness of the hair is more than 6 pixels. In paragraph 2,
In the hair loss status determination module, the visualization of the hair loss severity is displayed as a heat map to determine the hair loss type, and the hair loss severity is quantified as a value between 0 and 1. A hair follicle classification and hair loss severity estimation system. (a) photographing the scalp using an imaging device consisting of a micromicroscope (Dino-Lite) and collecting scalp images at 200x magnification,
(b) extracting the hair follicle area as a circle in the extraction module for the image collected in step (a) and simultaneously determining the state of the hair follicle in the discrimination module;
(c) classifying the hair follicles in step (b) to determine the state of the hair follicles, and then estimating the severity of hair loss for each image in an estimation module according to the area of each hair follicle and the state of the hair follicle,
(d) A method for classifying hair follicles and estimating hair loss severity, comprising the step of performing visualization and quantification in a hair loss status determination module according to the severity of hair loss estimated in step (c). In paragraph 8:
The extraction in step (b) is a hair follicle classification and hair loss severity estimation method characterized in that the hair follicle area is extracted in a circular shape by applying Mask R-CNN technology based on ResNet (ResNet 50 or ResNet 101). In paragraph 9:
In step (b), the condition of the hair follicles is determined as good, normal, or serious depending on the number of hair follicles, hair thickness, and the number of hairs in each hair follicle,
In step (c), severity estimation is performed by estimating the hair loss severity index (P) according to the following equation,

Here, k refers to the 12 locations, n refers to the number of hair follicles, and i refers to the three conditions: i={1 (serious), 2 (normal), and 3 (good)}. α refers to the weight for the state of the hair follicles, and β refers to the weight for the total number of hair follicles. And a hair follicle classification and hair loss severity estimation method characterized by calculating the hair loss severity of 12 areas where the scalp was photographed as a value between 0 and 1 through equation (1) by normalizing the results for the entire value.