KR102567671B1 - Efficient hair damage detection method and system using sem image based on convolutional neural network - Google Patents

Abstract

An object of the present invention is to automatically identify and segment damage based on an artificial intelligence algorithm used for hair surface image damage by scanning electron microscopy, while constructing a new hair damage data set based on scanning electron microscopy image data. It is to provide an efficient hair damage detection method and system using neural network-based scanning electron microscope images.
In order to achieve the above object, an efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention includes a first step of collecting, by a collecting unit, hair data sample images taken by a scanning electron microscope; a second step of generating a data set from the collected hair data sample images by a construction unit; a third step of extracting features of the hair data sample image from the generated data set by an extractor; and a fourth step of classifying the extracted features by a classification unit.

Description

Efficient hair damage detection method and system using scanning electron microscope image based on convolutional neural network

The present invention relates to an efficient hair damage detection method and system using a runner electron microscope image based on a convolutional neural network, and more particularly, to a runner electron microscope based on a convolutional neural network capable of recognizing and determining the degree of hair damage from a hair image It relates to an efficient hair damage detection method and system using images.

Hair is an important part of human body image.

Modern people's aesthetic desire for beauty, continuous growth and diversification of hair, and continuous development of hair shape, color, and texture are trying to change, and they can change their image using their individuality to follow the trend.

Hair consists of 1 to 8% outer hydrophobic lipid epidermis, 80 to 90% α-helical or β-sheet form of parallel polypeptide chains, containing water insoluble keratin, less than 3% melanin pigment and 0.6 to 1.0% trace elements, moisture. form 10-15%, etc.

A normal cuticle has a smooth appearance that reflects light and limits friction between the hair shafts.

It is responsible for the shine and texture of the hair.

The keratin layer of hair is weakened and split under the influence of external environment, temperature, humidity, chemical and physical treatment, which affects hair quality.

Most people's hair is inconvenient to observe, so it is easy to cause various damage problems, but there is a problem that detailed analysis is impossible and there are few studies on hair damage related to it.

With the widespread use of light microscopy in the field of hair microstructure analysis, it has become easier to observe hair details that were difficult to observe clearly with conventional microscopes.

Clinically, microscopic analysis can be used as an indicator of health status as a tool to evaluate hair damage, and can be qualitatively analyzed by identifying morphological characteristics of hair damage.

Exposure of hair to various chemical and physical influences such as detergents, dyes, combing, ultraviolet light, etc. affects the hair.

However, few studies have quantified the degree of hair damage according to the morphological characteristics of hair.

Traditionally, it is difficult to distinguish hair damage in detail only by manual observation with an optical microscope.

With the development of microscopy technology and the application of scanning electron microscopy (SEM) in various fields, fine parts of hair can be found more easily.

For example, energy-dispersive X-ray spectroscopy (EDX) using scanning electron microscopy and other physical and chemical methods to determine the effect of alopecia areata on the structure and composition of human hair. did

In addition, hair was heated using a heating iron and hair images were observed with a scanning electron microscope to evaluate damage to the epidermis and cortex of hair fibers due to heat loss.

However, this manual method uses scanning electron microscopy imaging techniques to analyze hair microscopic images, but does not perform detailed quantitative classification of hair damage.

Korean Patent Publication No. 10-2019-0049222

An object of the present invention to solve the conventional problems as described above is to automatically identify and segment damage based on an artificial intelligence algorithm used for hair surface image damage by scanning electron microscopy, and at the same time based on scanning electron microscopy image data To provide an efficient hair damage detection method and system using scanning electron microscope images based on a convolutional neural network that constructs a new hair damage data set.

In order to achieve the above object, an efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention includes a first step of collecting, by a collecting unit, hair data sample images taken by a scanning electron microscope; a second step of generating a data set from the collected hair data sample images by a construction unit; a third step of extracting features of the hair data sample image from the generated data set by an extractor; and a fourth step of classifying the extracted features by a classification unit.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the data set equally divides the collected hair data sample image into 3 equal parts, and divides the central 5 mm segment into It is characterized by selecting for observation and generating weak damage, medium damage, and high damage images as the final sample data set.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the total amount of the final sample data set is expanded through a data enhancement algorithm, the data enhancement algorithm comprising an image inversion algorithm, Characterized in that it is at least one of a rotation algorithm, a cropping algorithm, and a Gaussian noise addition algorithm.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the extraction unit is a convolutional neural network (CNN), a residual channel spatial attention network (RCSAN-Net: Residual Channel Spatial Attention Network) is used.

In addition, in the efficient hair damage detection method using a convolutional neural network-based scanning electron microscope image according to the present invention, the residual channel spatial attention network stacks multiple residual blocks and multiple residual attentional blocks formed by combining attentional blocks, It is characterized by building a network framework.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the residual channel spatial attention network connects the residual block and the attention block using skip connection, characterized in that do.

In addition, in the convolutional neural network-based efficient hair damage detection method using scanning electron microscopy images according to the present invention, the residual channel spatial attention network generates a feature map using standard convolution calculation in an initial stage, and the attention mechanism It is characterized by entering a module to extract multi-channel and spatial information for features, and entering a calculation round to generate a state feature map.

In addition, in the efficient hair damage detection method using a convolutional neural network-based scanning electron microscope image according to the present invention, the residual channel spatial attention network includes a channel attention module and an attention mechanism module having a spatial attention module. do.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the channel attention module is characterized in that the interdependence between feature channels is improved.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the spatial attention module is a specific region information part of a feature map supplementing the channel attention module, and the spatial attention map is Characterized in that it is created using spatial relationships between elements.

In addition, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the spatial attention module combines maximum pooling and average pooling to maintain feature information at maximum in the channel attention module It is characterized in that two two-dimensional feature maps (Fa and Fm) are generated by adopting a double pooling method.

In addition, in the efficient hair damage detection method using a convolutional neural network-based scanning electron microscope image according to the present invention, the residual channel spatial attention network collects hair features to improve detection and recognition accuracy. It is characterized by designing a channelism model and a spatialism model in the sensing model.

In addition, in order to achieve the above object, an efficient hair damage detection system using a scanning electron microscope image based on a convolutional neural network according to the present invention is detected by an efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network. do.

On the other hand, in order to achieve the above object, an efficient hair damage detection system using a scanning electron microscope image based on a convolutional neural network according to the present invention includes a collection unit for collecting hair data sample images taken by a scanning electron microscope; a configuration unit generating a data set from the collected hair data sample images; an extraction unit extracting features of the hair data sample image from the generated data set; and a classification unit for classifying the extracted features.

In addition, in the efficient hair damage detection system using a scanning electron microscope image based on a convolutional neural network according to the present invention, the extraction unit is a convolutional neural network (CNN), a residual channel spatial attention network (RCSAN-Net: Residual Channel Spatial Attention Network) is used.

In addition, in the efficient hair damage detection system using a convolutional neural network-based scanning electron microscope image according to the present invention, the residual channel spatial attention network collects hair features to improve detection and recognition accuracy. It is characterized by designing a channelism model and a spatialism model in the sensing model.

Details of other embodiments are included in the "specific details for carrying out the invention" and the accompanying "drawings".

Advantages and/or features of the present invention, and methods of achieving them, will become apparent with reference to the various embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited only to the configuration of each embodiment disclosed below, but may also be implemented in various other forms, and each embodiment disclosed herein only makes the disclosure of the present invention complete, and the present invention It is provided to completely inform those skilled in the art of the scope of the present invention, and it should be noted that the present invention is only defined by the scope of each claim of the claims.

According to the present invention, there is an effect of automatically identifying and segmenting damage based on an artificial intelligence algorithm used for hair surface image damage by scanning electron microscopy and at the same time constructing a new hair damage data set based on scanning electron microscopy image data. there is.

1 is a scanning electron microscope (SEM) photograph for observing a hair magnification 800× micrograph.
Fig. 2 shows various residual block frameworks of ResNet;
3 is a diagram illustrating a Residual Channel Spatialism Network (RCSAN);
Fig. 4 is an attentional mechanism, a diagram illustrating interactions between features and attentional masks;
Fig. 5 shows the structure of a channel attention module;
6 is a diagram showing the structure of a spatial attention module;
7 is a diagram showing classification of hair damage.
8 is a diagram showing enhancement of hair SEM microscopic image data.
Fig. 9 is a diagram showing attention characteristics of a hair damaged area;
10 is a diagram showing a comparison of fine details of hair at various magnifications.
Fig. 11 is a diagram illustrating attentional function extraction of an attentional mechanism module;

Before explaining the present invention in detail, the terms or words used in this specification should not be construed unconditionally in a conventional or dictionary sense, and in order for the inventor of the present invention to explain his/her invention in the best way It should be noted that concepts of various terms may be appropriately defined and used, and furthermore, these terms or words should be interpreted as meanings and concepts corresponding to the technical idea of the present invention.

That is, the terms used in this specification are only used to describe preferred embodiments of the present invention, and are not intended to specifically limit the contents of the present invention, and these terms represent various possibilities of the present invention. It should be noted that it is a defined term.

In addition, it should be noted that in this specification, singular expressions may include plural expressions unless the context clearly indicates otherwise, and similarly, even if they are expressed in plural numbers, they may include singular meanings. .

Throughout this specification, when a component is described as "including" another component, it does not exclude any other component, but further includes any other component, unless otherwise stated. It can mean you can do it.

Furthermore, when a component is described as “existing inside or connected to and installed” of another component, this component may be directly connected to or installed in contact with the other component, and a certain It may be installed at a distance, and when it is installed at a certain distance, a third component or means for fixing or connecting the corresponding component to another component may exist, and now It should be noted that the description of the components or means of 3 may be omitted.

On the other hand, when it is described that a certain element is "directly connected" to another element, or is "directly connected", it should be understood that no third element or means exists.

Similarly, other expressions describing the relationship between components, such as "between" and "directly between", or "adjacent to" and "directly adjacent to" have the same meaning. should be interpreted as

In addition, in this specification, the terms "one side", "the other side", "one side", "the other side", "first", "second", etc., if used, refer to one component It is used to be clearly distinguished from other components, and it should be noted that the meaning of the corresponding component is not limitedly used by such a term.

In addition, in this specification, terms related to positions such as "top", "bottom", "left", and "right", if used, should be understood as indicating a relative position in the drawing with respect to the corresponding component, Unless an absolute position is specified for these positions, these positional terms should not be understood as referring to an absolute position.

In addition, in this specification, in specifying the reference numerals for each component of each drawing, for the same component, even if the component is displayed in different drawings, it has the same reference numeral, that is, the same reference throughout the specification. Symbols indicate identical components.

In the drawings accompanying this specification, the size, position, coupling relationship, etc. of each component constituting the present invention is partially exaggerated, reduced, or omitted in order to sufficiently clearly convey the spirit of the present invention or for convenience of explanation. may be described, and therefore the proportions or scale may not be exact.

In addition, in the following description of the present invention, a detailed description of a configuration that is determined to unnecessarily obscure the subject matter of the present invention, for example, a known technology including the prior art, may be omitted.

Hereinafter, embodiments of the present invention will be described in detail with reference to related drawings.

An efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention includes four steps.

In a first step, a hair data sample image taken by a scanning electron microscope is collected by a collection unit.

In the second step, the collected hair data sample images are created as a data set by the construction unit.

In the third step, the feature of the hair data sample image is extracted from the generated data set by the extraction unit.

In the fourth step, the extracted features are classified by a classification unit.

In addition, the efficient hair damage detection system using a scanning electron microscope image based on a convolutional neural network according to the present invention detects hair damage using an efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network.

More specifically, an efficient hair damage detection system using a convolutional neural network-based scanning electron microscope image according to the present invention includes a collection unit, a configuration unit, an extraction unit, and a classification unit.

The collection unit collects hair data sample images taken with a scanning electron microscope.

The construction unit generates the collected hair data sample images as a data set.

The extraction unit extracts features of the hair data sample image from the generated data set.

The classification unit classifies the extracted features.

In the present invention, a novel hair damage detection network is provided (see FIG. 3).

It automatically identifies and segments the damage based on the artificial intelligence algorithm used for image damage to the hair surface by scanning electron microscopy (see Figure 1).

1 is a scanning electron microscope (SEM) photograph for observing a hair magnification 800× micrograph.

At the same time, a new hair damage data set should be constructed based on the scanning electron microscopy image data.

In summary, the present invention is as follows.

A new hair microscopy data set is created based on the scanning electron microscopy image data, and quantitative analysis is performed to classify the degree of hair damage into weak, medium, and high damage.

We present RCSAN-Net (Residual Channel Spatial Attention Network), a novel and effective convolutional neural network model for hair damage detection.

To improve the accuracy of detection and recognition, a channel attention mechanism and a spatial attention mechanism are designed and introduced into a hair damage detection model that collects hair features.

In other words, in the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the extraction unit is a convolutional neural network (CNN), a residual channel spatial attention network (RCSAN-Net: Residual Channel Spatial Attention Network) is used.

In this residual channel spatial attention network, a channel attention model and a spatial attention model are designed in a hair damage detection model that collects hair features to improve detection and recognition accuracy.

- Related Tasks -

The detection and recognition of fine details in hair can be used in many fields such as medicine and forensics.

In the context of forensic science, microscopic hair analysis (a qualitative method) has shown effective discrimination, and detailed examination of hair under a microscope aids in forensic identification.

Clinically, microscopic analysis can be used as a tool to assess hair damage and as an indicator of health status, but the analysis must be performed under specific conditions.

Hair is exposed to various physical and chemical agents, including detergents, dyes, combing, and ultraviolet light, which change the structure of the hair.

Morphological characteristics of damaged hair can be detected by qualitative methods for clinical studies.

Most studies have focused on the identification of hair damage, and few studies have quantified the extent of hair damage based on morphological characteristics.

In one existing example, a classification system for hair damage was provided.

This system is divided into 5 damage levels, and the influence of external factors on hair damage is explained.

After that, there is an example of extending the damage level to 12.

In this example, hair microscopic images of a scanning electron microscope were observed to subjectively evaluate hair structure irregularities, and the degree of hair damage was subdivided into 12 stages.

In addition, visual evaluation was used as the main qualitative technique for hair damage.

There are few inventions of more objective hair damage analysis and identification systems and most inventions focus on manual analysis of morphological features detected by light microscopy and analysis using commercial software, whereas inventions on automatic and efficient detection of hair damage is almost non-existent.

Detecting and recognizing minute features of hair is a difficult task.

In recent decades, with the continuous development of microscopy technology, the microscopic structure of hair that can be observed is clearer and more accurate, and sample preparation is simple and high-resolution, so more research on microscopic morphology of hair morphology in greater depth is being conducted. there is.

Manual detection methods are still the most important practical detection methods, but it usually takes at least one year to train qualified professional detection personnel.

This takes a lot of time and effort because the person using the microscope has to observe the shape of the fiber for a long time with the microscope, and subjective accuracy and detection accuracy vary depending on the person using the microscope.

Therefore, over the years, fast and accurate methods for automatically detecting the extent of hair damage have been developed and many useful trials have been made.

These methods mainly include methods for autonomously and efficiently detecting the degree of hair damage by applying light microscope and scanning electron microscope observation, hair microscope image analysis, image processing, and computer vision technology.

In recent years, there has been the emergence and many applications of convolutional neural networks.

Methods based on image processing and computer vision are current research hotspots, with more and more inventors focusing on the field of hair detection.

For example, there are convolutional models that use convolutional neural network algorithms from an unconstrained perspective and achieve complete hair analysis (detection, segmentation, and hairstyle classification) using only texture information.

In addition, as part of scalp health management, there is a deep learning-based smart scalp inspection and diagnosis system that detects and diagnoses four common scalp and hair symptoms (dandruff, folliculitis, hair loss, and oily hair). and hair physical therapy.

In the early days, people mainly used image processing techniques to measure geometrical parameters of hair such as hair diameter and hair uniformity.

Recently, in order to recognize abstract detailed features in hair microscopic images, computer vision technology has begun to extract abstract detailed features.

For example, there is XI-Net, an artificial intelligence-based convolutional neural network algorithm that performs feature analysis, extraction, and classification of scanning electron microscope images to handle forensic crime investigation cases.

According to this algorithm, forensics can quickly extract the desired hair detail features, and the investigation efficiency can be greatly improved, which can speed up the detection of incidents.

- Materials and Methods -

- Convolutional Neural Networks

In the field of handwriting recognition, convolutional neural networks have opened a new chapter in automatic image recognition.

Since then, convolutional neural networks have been applied to image classification.

Following this trend, the trend has shifted from engineering manual functions to engineering network architectures.

For example, VGG-Net proposed a modular network design method to stack strategies of network blocks of the same type, which simplifies the workflow of network design and simplifies the transfer learning of downstream applications.

VGG-Net provides a modular network design method for stacking strategies of network blocks of the same type that simplifies the workflow of network design and transfer learning for downstream applications.

Inception acquires more features by extending the depth and width of the convolutional layer and extracting multiple scale information from the image.

At the same time, the amount of computation is reduced by using a 1 × 1 convolutional layer for dimensionality reduction.

ResNet introduces skip connections, which greatly alleviates the problem of vanishing gradients in deep neural networks and allows the network to learn enhanced feature representations.

In the remaining blocks (see FIG. 2), network performance can be improved by appropriately changing the type and number of layers.

Figure 2 is a diagram showing various residual block frameworks of ResNet.

ResNet is one of the most successful CNN architectures and has been adopted for various computer vision applications.

- RCSAN-Net: Residual Channel Spatialism Network -

3 is a diagram illustrating a residual channel spatial attention network (RCSAN).

In the convolutional neural network-based efficient hair damage detection method using scanning electron microscope images according to the present invention, the residual channel spatial attention network stacks multiple residual blocks and multiple residual attentional blocks formed by combining the attentional blocks to form a network framework. build

In addition, the residual channel spatial attentional network connects the residual block and the attentional block using a skip connection.

As shown in Figure 3, the RCSAN network is based on the residual network.

We build an efficient network framework for hair damage detection by stacking multiple residual blocks formed by combining multiple residual blocks and attention blocks to process complex and difficult-to-detect hair damage feature regions.

The RCSAN network is constructed by stacking several residual attention modules.

Previously, we generally focused on only one type of attention, such as scale attention or spatial attention, and could not extract accurate features of the target region.

In the attention mechanism model, each module is divided into two parts: a residual block and a channel-space connective attention block.

The residual block convolutional network extracts the global features of the image and then enters the attention module to pay attention to the local detailed features.

However, adding the number of models of interest does not improve the overall network performance.

In the network model of the present invention, the residual block and the attention module are connected using skip connection.

In addition, in the convolutional neural network-based efficient hair damage detection method using scanning electron microscopy images according to the present invention, the residual channel spatial attention network generates a feature map using standard convolution calculation at an early stage, and as an attention mechanism module Enter to extract multi-channel and spatial information for features, and enter a computational round to generate state feature maps.

After receiving the image, this model generates a feature map using standard convolutional calculations in the initial step, enters the Attention Mechanisms module to extract multi-channel and spatial information about features, and enters the next round of computation to generate the Attention Feature Map. generate

Figure 4 is a diagram illustrating the interaction between attentional mechanisms, features and attentional masks.

A note on the feature extraction process is shown in FIG. 4 .

- Attention Mechanism Module -

In the efficient method for detecting hair damage using a scanning electron microscope image based on a convolutional neural network according to the present invention, the residual channel spatial attention network includes a channel attention module and an attention mechanism module having a spatial attention module.

Attention plays an important role in human perception.

To better capture the visual structure, humans use a series of parts to glance at and selectively focus on a prominent part.

Adding attention mechanisms in convolutional neural networks can effectively improve the performance of CNNs on large-scale classification tasks.

When processing hair image data from scanning electron microscopy, general algorithmic models cannot accurately discern subtle differences in hair damage due to small differences in hair surface properties.

In the present invention, an attention mechanism module suitable for hair microscopic imaging is designed.

Improve the accuracy of detection of hair images of scanning electron microscopy and identification of damage features.

Attention mechanism modules include a channel attention module and a spatial attention module.

- Channel attention module -

In the method for efficient hair damage detection using scanning electron microscopy images based on a convolutional neural network according to the present invention, the channel attention module improves interdependence between feature channels.

Each channel graph of a feature can be regarded as a response by class, and different semantic responses are related to each other.

Interdependencies between channel mappings can be exploited to emphasize interdependent feature mappings and improve specific semantic functional representations.

Therefore, a channel attention module used to improve the interdependency between feature channels is added.

The structure of the channel attention module is shown in FIG. 5 .

5 is a diagram showing the structure of a channel attention module.

Referring to Figure 5, cast transformed into and After performing matrix multiplication between the transpositions of , and then passing the softmax function to generate

In addition, matrix multiplication is performed between the transpositions of K and A, and the result is transform into

Characteristic map of the final output channel Mc(F)∈RC×H×W

Function map of final output channel attention is represented by Equation 1.

[Equation 1]

scale parameter if measures the effect of the jth channel on the ith channel learns the weights gradually from 0.

The final feature of each channel is a weighted sum of the features of all channels and the original features modeling the remote semantic dependencies between the feature maps.

- Spatial attention module -

In the efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to the present invention, the spatial attention module is a specific area information part of a feature map that complements the channel attention module, and the spatial attention map determines the spatial relationship between elements. created using

This spatial attention module generates two 2-dimensional feature maps (Fa and Fm) by adopting a double pooling method combining maximum pooling and average pooling to keep feature information at maximum in the channelism module.

In more detail, unlike channelism, the focus of spatialism is a specific region information part of a feature map that complements channelism.

A spatial attention map is created using spatial relationships between elements.

The structure of the spatial attention module is as shown in FIG. 6 .

6 is a diagram showing the structure of a spatial attention module.

Referring to FIG. 6, two two-dimensional feature maps (Fa and Fm) are generated by adopting a double pooling method combining maximum pooling and average pooling in order to maximize feature information in channelism when calculating spatial attention.

These are concatenated to create effective feature descriptors, and after the computation of standard convolutional layers, the feature maps of the spatial domain. generate

This can be represented by Equation 2.

[Equation 2]

here denotes the sigmoid function, is the filter size of the convolution operation.

- result -

- data set

Hair samples of 50 men and 50 women aged 20 to 60 are collected, and hair microscopic images are generated using a Hitachi S-4700 scanning electron microscope.

First, sample hair is sorted, sorted and cut.

The hair section is attached to a conductive tape coated with carbon on both sides and fixed to a metal aluminum rod, and platinum is used as a metal target.

The spray coater performs sputtering coating in a high vacuum condition.

Next, the coated hair sample was scanned at 15 kV using a scanning electron microscope (model name: Hitachi S-4700).

Multiple sections of each axis are scanned to ensure that the observed changes in the surface of the hair sample are uniform and not separated.

7 is a diagram showing classification of hair damage.

In a scanning electron microscope, hair shaft damage is classified into three stages: weak damage, medium damage, and high damage (see FIG. 7).

In the efficient hair damage detection method using a convolutional neural network-based scanning electron microscope image according to the present invention, the data set equally divides the collected hair data sample image into 3 equal parts, and selects the central 5 mm segment for observation to generate weak, medium and high damage images as the final sample data set.

The degree of cuticle damage is used as a criterion for determining the degree of hair damage.

Samples without obvious cracks in the cuticle of the hair surface are defined as mild damage.

Samples with severe cracks and distortions of the skin are defined as moderate damage.

When the epidermis has almost completely disappeared from the sample, this is defined as high damage.

Data of 500 hair samples from 100 testers are collected, each hair data sample is equally divided into 3 parts and the central 5 mm segment is selected for observation to generate weak damage, medium damage and high damage images.

500 images for each hair damage category are generated.

The collection of scanning electron microscopy hair images is complex and expensive, making large-scale data collection impossible.

8 is a diagram showing enhancement of hair SEM microscopic image data.

In the efficient hair damage detection method using scanning electron microscope images based on convolutional neural networks according to the present invention, the total amount of the final sample data set is expanded through a data enhancement algorithm, and the data enhancement algorithm includes an image inversion algorithm, a rotation algorithm, and a cropping algorithm. algorithm, one or more of the Gaussian noise addition algorithms.

To this end, a data enhancement algorithm is used to enhance the collected scanning electron microscope hair images as shown in FIG. 8 .

Examples include image inversion, rotation, cropping, Gaussian noise addition, and other algorithms.

A runner electron microscopy hair data set was created, which contains a total of 15,000 scanning electron microscopy hair images of various types.

This is as follows.

A dataset of 5000 lightly damaged scanning electron microscopy hair images, 5000 moderately damaged scanning electron microscopy hair images, and 5000 highly damaged scanning electron microscopy hair images were created.

- detailed implementation -

In our experiment, we split the data into training set and test set, 80% for training set and 20% for test set.

Crop the image size to 224 × 224 using center crop.

The network is trained using Stochastic Gradient Descent Algorithm (SGD).

The initial learning rate is set to 0.1 and gradually decreased.

A Nesterov momentum of 0.9 and 10-4 weight reduction without damping were used for all weights.

These model parameters are shown in Table 1.

[Table 1]

Table 1 is the configuration parameters of SACN-Net.

- Comparison with other methods -

Different network models, such as the classic AlexNet algorithm, the popular VGG network model, the lightweight CNN networks MobileNet, ResNet, and other state-of-the-art methods were selected and compared with the provided model algorithms.

This model is based on ResNet50.

In the performance test comparison, 34-layer and 50-layer ResNet networks were compared.

As shown in Table 2, it shows that this SACN-Net has better accuracy than other models.

[Table 2]

Table 2 shows the accuracy comparison of different models.

In the model test, the test of the attention module was added.

Fig. 9 is a diagram showing attention characteristics of a hair damage region.

9 is a comparative test result of a dedicated channel attention module, a one-time use spatial attention module, and a channel spatial attention module combined with a position change method for the overall performance of the model.

If you choose to use different attention modules in your model, there is a big difference in performance.

A single attentional module cannot accurately identify an object.

[Table 3]

Table 3 shows the classification errors in the scanning electron microscopy hair data set.

In a benchmark test such as Top-1 in Table 3, it was found that the performance was the best based on the channel first and then on the spatial attention module.

The present invention is a method for autonomously and efficiently detecting hair damage.

By building an artificial intelligence convolutional neural network-based system that can autonomously detect the degree of hair damage, it can be used for personal daily care and maintenance, medical health diagnosis and other directions.

By developing a handheld machine that can detect the degree of hair damage and combining it with a hair damage detection algorithm, hair damage can be quickly diagnosed.

When collecting hair samples, only adults aged 20 to 60 years are considered, and collections of hair samples from minors and those over 60 years of age are not included.

The reason is that people between the ages of 20 and 60 pay more attention to daily hair care and have more hair damage.

Thus, it is helpful in collecting hair damage samples.

Brown, brown, and red hair samples were partially collected by observing the scanning electron microscope to consider the possibility of the effect of hair color on the degree of damage to the hair.

As a result of observation and analysis, it was found that the relationship between hair color and hair damage was weak.

10 is a diagram illustrating a comparison of fine details of hair at various magnifications.

When creating the data set, the microscopic properties of the hair were compared at 400× and 800× (see FIG. 10).

First, a 400× hair image was used in observation with a scanning electron microscope.

The display effect was unsatisfactory, the hair surface texture could not be accurately expressed, and it was difficult to use for model learning.

Finally, we chose 800× hair images as the standard for our data set and created our own Scanning Electron Microscopy hair data set.

In the model, the proposed attentional mechanism model is discussed to find the best performance of the model.

After many modifications, the dual attention mechanism module, especially the channel-priority attention method, was found to have the highest efficiency in identifying hair damage features.

11 is a diagram illustrating attention function extraction of an attention mechanism module.

- conclusion -

In the present invention, a microscopic hair data set consisting of 50 males and 50 females aged 20 to 60 years old was created based on a scanning electron microscope, and data were classified into weak, medium, and high damage according to the degree of cuticle damage on the hair surface. separated.

The focus was mainly on the degree of hair damage.

A random collection method was adopted when collecting hair samples.

Samples include hair samples such as dyed hair, curly hair, and straight hair.

When observing under a microscope, the middle part of each hair is used as an observation area.

Reduce sample variability.

In addition, a new residual convolution network based on the spatial attention module is provided and verified with a data set.

Experiments show that the network model is effective and robust.

As described above, according to the present invention, based on an artificial intelligence algorithm used for hair surface image damage by scanning electron microscopy, damage is automatically identified and segmented, while constructing a new hair damage data set based on scanning electron microscopy image data. It works.

In the above, various preferred embodiments of the present invention have been described with some examples, but the description of various embodiments described in the "Specific Contents for Carrying Out the Invention" section is only exemplary, and the present invention Those skilled in the art will understand from the above description that the present invention can be practiced with various modifications or equivalent implementations of the present invention can be performed.

In addition, since the present invention can be implemented in various other forms, the present invention is not limited by the above description, and the above description is intended to complete the disclosure of the present invention and is common in the technical field to which the present invention belongs. It is only provided to completely inform those skilled in the art of the scope of the present invention, and it should be noted that the present invention is only defined by each claim of the claims.

Claims (16)

A first step of collecting, by a collecting unit, hair data sample images taken with a scanning electron microscope;
a second step of generating a data set from the collected hair data sample images by a construction unit;
a third step of extracting features of the hair data sample image from the generated data set by an extractor; and
A fourth step of classifying the extracted features by a classification unit; includes,
The extraction part,
Using a residual channel spatial attention network (RCSAN-Net), which is a convolutional neural network (CNN),
The residual channel spatialism network,
Build a network framework by stacking multiple residual blocks formed by combining multiple residual blocks and attentional blocks;
Connecting the residual block and the attentional block using skip connection;
In the initial stage, standard convolutional calculations are used to generate feature maps, the attention mechanism module is entered to extract multi-channel and spatial information for features, the calculation round is entered to generate attentional feature maps,
The feature map of the state is,
Characterized in that the following Equation 1, which is a feature map of the channel principle, and Equation 2 below, which is the feature map of the spatial principle,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
[Formula 1]


[Formula 2]

According to claim 1,
The data set is
Characterized in that the collected hair data sample image is equally divided into 3 equal parts and the central 5 mm segment is selected for observation to generate weak damage, medium damage and high damage images as the final sample data set,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
According to claim 2,
Expanding the total amount of the final sample data set through a data enhancement algorithm;
Characterized in that the data enhancement algorithm is one or more of an image inversion algorithm, a rotation algorithm, a cropping algorithm, and a Gaussian noise addition algorithm.
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
delete delete delete delete According to claim 1,
The residual channel spatialism network,
Characterized in that it comprises an attention mechanism module having a channel attention module and a spatial attention module,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
According to claim 8,
The channel attention module,
Characterized in improving the interdependence between feature channels,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
According to claim 9,
The spatial attention module,
A specific area information part of a feature map that complements the channel attention module,
Characterized in that the spatial attention map is generated using the spatial relationship between elements,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
According to claim 10,
The spatial attention module,
Characterized in that, in the channel attention module, two two-dimensional feature maps (Fa and Fm) are generated by adopting a double pooling method combining maximum pooling and average pooling to maintain the maximum feature information.
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
According to claim 1,
The residual channel spatialism network,
In order to improve the accuracy of detection and recognition, a channel attention model and a spatial attention model are designed in a hair damage detection model that collects hair features,
Efficient hair damage detection method using scanning electron microscopy images based on convolutional neural networks.
A scanning electron microscope image based on a convolutional neural network detected by an efficient hair damage detection method using a scanning electron microscope image based on a convolutional neural network according to one of claims 1 to 3 and 8 to 12 Efficient hair damage detection system using
a collection unit that collects hair data sample images taken with a scanning electron microscope;
a configuration unit generating a data set from the collected hair data sample images;
an extraction unit extracting features of the hair data sample image from the generated data set; and
A classification unit for classifying the extracted features; includes,
The extraction part,
Using a residual channel spatial attention network (RCSAN-Net), which is a convolutional neural network (CNN),
The residual channel spatialism network,
Build a network framework by stacking multiple residual blocks formed by combining multiple residual blocks and attentional blocks;
Connecting the residual block and the attentional block using a skip connection;
In the initial stage, standard convolutional calculations are used to generate feature maps, the Attention Mechanism module is entered to extract multi-channel and spatial information for features, the calculation round is entered to generate Attention Feature Maps,
The feature map of the state is,
Characterized in that the following Equation 1, which is a feature map of channel principals, and Equation 2 below, which is a feature map of spatial principals,
An efficient hair damage detection system using scanning electron microscopy images based on convolutional neural networks.
[Equation 1]


[Equation 2]

delete 15. The method of claim 14,
The residual channel spatialism network,
In order to improve the accuracy of detection and recognition, a channel attention model and a spatial attention model are designed in a hair damage detection model that collects hair features,
An efficient hair damage detection system using scanning electron microscopy images based on convolutional neural networks.