KR102458614B1 - System for diagnosing skin using non-contact type rgb camera and skin diagnosis method thereof - Google Patents

Abstract

A skin diagnosis system and method using a non-contact RGB camera according to an embodiment are disclosed. The skin diagnosis method includes: acquiring an image by photographing a face for which a skin condition is to be diagnosed; generating a masking image by pre-processing the obtained image and calculating a wrinkle score quantifying the degree of wrinkles from the generated masking image; and diagnosing the skin condition of the skin region of the face based on the detected wrinkle score, and generating skin diagnosis information as a result of the diagnosis.

Description

Skin diagnosis system and method using non-contact RGB camera

The embodiment relates to a skin diagnosis system and method using a non-contact RGB camera.

The global healthcare AI market is growing rapidly every year and is expected to grow even more in the future. Among them, the smart beauty field is evaluated as a high-growth industry with great marketability. Smart beauty technology is largely divided into digital skin diagnosis technology and treatment technology. The digital skin diagnosis technology measures and analyzes the condition of the skin by analyzing the image captured by the camera, and a contact type RGB camera or a non-contact UV camera is mainly used.

Skin diagnosis technology using a non-contact RGB camera has a higher market value than a technology using a contact camera or a UV camera because it has a wider facial area that can be analyzed and has a lower unit price. However, the resolution of the skin region in the image is relatively low and the location thereof is also varied, so the accuracy of a diagnosis technique using the same is very poor. In other words, it is essential to develop a low-resolution skin diagnosis technology for skin diagnosis on a wide face area at a low unit cost.

(Patent 1) Japanese Patent Laid-Open No. 2019-058641 (Patent 2) US Patent Application Publication US2017/0270350 (Patent 3) Patent Publication No. 10-2016-0018033

The embodiment may provide a skin diagnosis system and method using a non-contact RGB camera.

A skin diagnosis method according to an embodiment includes: acquiring an image by photographing a face for diagnosing a skin condition; preprocessing the image to generate a masking image; calculating a wrinkle score quantifying the degree of wrinkles from the masking image; and diagnosing the skin condition of the skin region of the face based on the detected wrinkle score, and generating skin diagnosis information as a result of the diagnosis.

The generating may include detecting a face region from the image, detecting a predetermined skin region among the detected face regions, and generating a masking image including the detected skin region.

The calculating may include converting the color space of the image into an HSV color space, extracting brightness information using V of the HSV color space, and detecting an edge from the image based on the extracted brightness information to generate an edge image. And, a contour line is detected from the generated edge image, the number of the detected edges and contour lines is calculated, and a wrinkle score obtained by quantifying the degree of the wrinkles based on the calculated number of edges and contour lines can be calculated. .

The calculating may include performing histogram smoothing on the image based on the extracted brightness information, but dividing the image into a plurality of regions to perform histogram smoothing.

The calculating includes blurring the image on which the histogram smoothing has been performed in horizontal and vertical directions, respectively, to remove noise edges to generate a horizontal edge image and a vertical edge image, and the generated horizontal edge image and vertical edge image An edge gradient is obtained from each of the directional edge images using a sobel edge detector, and an edge having a maximum magnitude is obtained using a Non Maximum Suppression (NMS) algorithm based on the edge gradient. is a candidate edge, it is determined whether it is an actual edge by applying a threshold to the determined candidate edge, a morphology open operation is performed on the horizontal edge image, and the edge determined to be the actual edge The edge image may be generated by combining each included horizontal edge image and vertical edge image.

According to an exemplary embodiment, a skin diagnosis system includes: a skin diagnosis device that acquires an image by photographing a face for diagnosing a skin condition; and pre-processing the obtained image to generate a masking image, calculate a wrinkle score quantifying the degree of wrinkles from the generated masking image, and diagnose the skin condition for the skin region of the face based on the calculated wrinkle score Thus, the user terminal may include a user terminal that generates skin diagnosis information as a result of the diagnosis.

The user terminal may detect a face region from the image, detect a predetermined skin region among the detected face regions, and generate a masking image including the detected skin region.

The user terminal converts the color space of the image into an HSV color space, extracts brightness information using V of the HSV color space, detects an edge from the image based on the extracted brightness information, and generates an edge image; , a contour line is detected from the generated edge image, the number of the detected edges and contour lines is calculated, and a wrinkle score obtained by quantifying the degree of the wrinkles based on the calculated number of edges and contour lines can be calculated.

The user terminal may perform histogram smoothing on the image based on the extracted brightness information, but divide the image into a plurality of regions to perform histogram smoothing.

The user terminal removes noise edges by blurring the image on which the histogram smoothing has been performed in the horizontal and vertical directions, respectively, to generate a horizontal edge image and a vertical edge image, and the generated horizontal edge image and vertical direction An edge gradient is obtained from each of the edge images using a Sobel edge detector, and an edge having a maximum magnitude is obtained using a Non Maximum Suppression (NMS) algorithm based on the edge gradient. It is determined as a candidate edge, it is determined whether or not it is an actual edge by applying a threshold to the determined candidate edge, a morphology open operation is performed on the horizontal edge image, and the edge determined to be the actual edge is applied to each of the edges. The edge image may be generated by combining the included horizontal edge image and the vertical edge image.

According to the embodiment, a wrinkle is detected in a pre-classified skin region within the face using an RGB image taken of the face, a wrinkle score is calculated that quantifies the degree of wrinkle, and the skin condition is diagnosed based on the calculated wrinkle score. By doing so, it is possible to accurately diagnose a skin condition for each skin region and provide a diagnosis result visually.

According to the embodiment, the reliability of the product may be improved by accurately diagnosing the skin condition.

1 is a diagram illustrating a skin diagnosis method according to an embodiment of the present invention.
2A to 2C are diagrams for explaining a skin region detection process according to an embodiment.
3 is a view showing a process of detecting and digitizing wrinkles according to an embodiment of the present invention.
4 is a diagram illustrating the configuration of a skin diagnosis system according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a detailed configuration of the user terminal shown in FIG. 4 .
6 is a view for explaining a skin diagnosis process according to an embodiment of the present invention.
7 is a diagram illustrating a screen displaying a skin diagnosis result through a mobile app.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

However, the technical spirit of the present invention is not limited to some embodiments described, but may be implemented in various different forms, and within the scope of the technical spirit of the present invention, one or more of the components may be selected between the embodiments. It can be combined and substituted for use.

In addition, terms (including technical and scientific terms) used in the embodiments of the present invention may be generally understood by those of ordinary skill in the art to which the present invention pertains, unless specifically defined and described explicitly. It may be interpreted as a meaning, and generally used terms such as terms defined in advance may be interpreted in consideration of the contextual meaning of the related art.

In addition, the terms used in the embodiments of the present invention are for describing the embodiments and are not intended to limit the present invention.

In this specification, the singular form may also include the plural form unless otherwise specified in the phrase, and when it is described as “at least one (or more than one) of A and (and) B, C”, it is combined with A, B, and C It may include one or more of all possible combinations.

In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used.

These terms are only for distinguishing the component from other components, and are not limited to the essence, order, or order of the component by the term.

And, when it is described that a component is 'connected', 'coupled' or 'connected' to another component, the component is not only directly connected, coupled or connected to the other component, but also with the component It may also include the case of 'connected', 'coupled' or 'connected' due to another element between the other elements.

In addition, when it is described as being formed or disposed on “above (above) or under (below)” of each component, the top (above) or bottom (below) is one as well as when two components are in direct contact with each other. Also includes a case in which another component as described above is formed or disposed between two components. In addition, when expressed as “up (up) or down (down)”, the meaning of not only the upward direction but also the downward direction based on one component may be included.

In the embodiment, a wrinkle is detected in a pre-divided skin region within the face using the RGB image taken of the face, a wrinkle score is calculated that quantifies the degree of wrinkle, and the skin condition is diagnosed based on the calculated wrinkle score We propose a new way to do this.

Here, the wrinkle means a dent in the skin that occurs between parts having a certain thickness.

1 is a diagram illustrating a skin diagnosis method according to an embodiment of the present invention.

Referring to FIG. 1 , the skin diagnosis apparatus according to the first embodiment of the present invention may acquire an RGB image by photographing a user's face using an RGB camera for skin diagnosis ( S110 ).

Next, the skin diagnosis apparatus may provide the obtained RGB image to the user terminal.

Next, the user terminal may detect the face region from the RGB image provided through the mobile application (S120). In this case, the size or angle of the face in the captured image may vary according to the image capturing environment or individual face characteristics. If the size or angle of the face is different for each image, the accuracy of detecting the skin region is lowered and the performance of the system is deteriorated. Therefore, in order to detect a uniform skin region, it is necessary to precisely find the face region even if the photographing environment or facial characteristics change. As a method for detecting a face region, various known techniques can be used, for example, a Harr-like feature-based method, an AlexNet-based method, a Cascade-based CNN method, a method using an AutoEncoder, a FAN (Face Alignment Algorithm) etc.

Next, the user terminal may detect a predetermined skin region from among the detected face regions ( S130 ). In the embodiment, the diagnosis is performed by limiting the diagnosis to a specific skin region rather than the entire face skin, but the skin region of interest for each diagnosis object may be different.

For example, for wrinkles, both cheeks and sides of both eyes are skin regions of interest.

2A to 2C are diagrams for explaining a skin region detection process according to an embodiment.

Referring to FIG. 2A , in the case of both cheeks, when a rectangular area is detected, it does not match the actual cheek area, so a partial nose boundary or lip boundary other than the cheek may be included, or a narrower area than the actual cheek may be detected. Since such a narrow area causes performance degradation in wrinkle detection, a convex polygon should be configured to include the entire cheek area as much as possible.

Referring to FIG. 2B , on the sides of both eyes, noises in a straight line such as eyebrows, eyelids, and eyes may be erroneously detected as wrinkles. Therefore, by using the eyebrow and eye coordinates from the detected face area coordinates, the area next to both eyes is designated in the form of a triangle and noise is not included.

Referring to FIG. 2C , a masking operation is performed on the area between the forehead, both cheeks, and the sides of the eyes, and it can be stored as a separate image, that is, a masking image, and used when detecting wrinkles.

Next, the user terminal may quantify and quantify the degree of the detected wrinkles by detecting wrinkles in the detected skin region (S160).

Next, the user terminal may diagnose the skin condition for each skin region based on the result of detecting the wrinkles ( S170 ), and display the diagnosed skin condition on the screen ( S180 ).

3 is a view showing a process of detecting and digitizing wrinkles according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal according to the embodiment may extract brightness information by converting the color space of the RGB image. In other words, since the color difference with the surrounding skin is insignificant for wrinkles, brightness information, not color information, must be used. Here, HSV represents H(Hue), S(Saturation), and V(Value, lightness). That is, in the HSV color space, since colors are expressed by H and S values and brightness is expressed by V, brightness information is extracted using V (S410).

Next, the user terminal may perform histogram equalization on the image based on the extracted brightness information (S412). When histogram smoothing is applied, the pixel values of the image are evenly distributed within the range of 0 to 255 and the contrast of the image is improved. Therefore, in the embodiment, Contrast Limited Adaptive Histogram Equalization (CLAHE) is intended to be used. In this case, CLAHE is a regional contrast enhancement technique, and CLHE can be performed by dividing an image into several blocks.

Next, the user terminal may generate an edge image by detecting an edge from the smoothed RGB image using a predetermined edge detection algorithm ( S420 ). Here, as the edge detection algorithm, various known algorithms may be used, for example, canny edge detection.

Specifically, the user terminal may remove the noise edge by blurring the image using a Gaussian filter in order to improve edge detection performance.

At this time, the user terminal blurs the image in the horizontal direction and the vertical direction, respectively, to remove the noise edge to generate a shape suitable for separating and analyzing horizontal wrinkles and vertical wrinkles, that is, a horizontal edge image and a vertical edge image. .

Next, the user terminal may obtain an edge gradient from each of the horizontal edge image and the vertical edge image from which the noise edge is removed using a Sobel edge detector. At this time, assuming that the horizontal and vertical gradients are Gx and Gy, respectively, the edge gradient is expressed by Equation 1 below.

[Equation 1]

,

,

Next, the user terminal may determine an edge having a maximum magnitude as a candidate edge using a Non Maximum Suppression (NMS) algorithm based on the edge gradient. In this case, the search direction is determined by considering the direction of the edge, rather than determining whether it is the maximum for all neighbors.

Next, the user terminal may determine whether the determined candidate edge is an actual edge. That is, if the strength of an edge is greater than an upper threshold, it is designated as an edge, and if it is smaller than a lower thereshold, the edge is used when synthetically connecting the edges.

Next, the user terminal may perform a morphology open operation on the horizontal edge image. Morphology is a technique to analyze the geometric shape of an image. The basic morphological operations are erosion and dilation operations. Erosion has the effect of removing slight noise at the cost of reducing the size of the region of interest. Dilation increases the size of the region of interest, but the size of noise may also increase. In this case, if the expansion calculation is performed after the erosion calculation, the shape and size of the object are preserved and at the same time, the slight noise can be removed. The process of performing erosion and dilation operations in this order is called an open operation and is often used to remove small noise. Since wrinkles in the horizontal direction are particularly noisy, in the embodiment, a monopolistic open operation is performed only on horizontal edge images.

Next, the user terminal may generate an edge image by combining a horizontal edge image and a vertical edge image each including edges determined to be actual edges.

Next, the user terminal may detect the outline from the edge image using a predetermined outline detection algorithm ( S430 ). A contour is a curve connecting all consecutive points with the same intensity and is used as a useful tool for shape analysis and object detection and recognition. In this case, various known algorithms may be used as the Yun Kwan-seok detection algorithm.

Next, the user terminal may calculate a quantified value based on the detected edge and contour (S440). That is, the skin diagnosis apparatus may calculate the number of detected edges and contours, and may calculate a number or score obtained by quantifying the degree of wrinkles based on the calculated number of edges and contours.

First, the number of each detected edge and contour is expressed by Equations 2 and 3 below.

[Equation 2]

,

,

[Equation 3]

where w is the width of the image, h is the height of the image, n is the number of detected contours, (x _k,1 , y _k,1 ) is the starting point of the kth contour, (x _k,2 , y _k,2 ) denotes the end point of the kth contour.

Line-shaped wrinkles are analyzed by all wrinkle lines, and can be quantified by counting the number of detected edges as in Equation 2, and thick wrinkles (Rhytid) are analysis of the area where wrinkles are formed, as shown in Equation 3 It can be quantified by calculating the total sum of the detected contour lengths.

When the wrinkle score is calculated using Equations 2 and 3, it is expressed as Equation 4 below.

[Equation 4]

,

,

Here, p _m represents a mask at the pixel position (i, j).

As in Equation 4 above, the area of the mask is obtained, log is taken to correct the intensity of the value, and the relative values of Score _w-line and Score _w-rhytid are calculated with respect to the mask area.

4 is a diagram illustrating the configuration of a skin diagnosis system according to an embodiment of the present invention.

Referring to FIG. 4 , a skin diagnosis system according to an embodiment of the present invention may include a skin diagnosis apparatus 100 , a user terminal 200 , a service server 300 , and a database 400 .

The skin diagnosis apparatus 100 may obtain an RGB image by photographing a face using a non-contact RGB camera, and may provide the obtained RGB image to the user terminal 200 .

The user terminal 200 detects a face region using the provided RGB image and detects wrinkles in a skin region divided in advance within the face region, and calculates a numerical value quantifying the degree of wrinkles to diagnose the skin condition.

The user terminal 200 may check the history of the user's facial skin condition by interworking with the service server 300 and uploading and managing the diagnosis result to the service server.

The service server 300 may interwork with the skin diagnosis apparatus 100 , and may receive and manage a numerical value obtained by quantifying the degree of wrinkles and information on a skin condition from the skin diagnosis apparatus 100 .

The service server 300 may provide information so that the user can check whether the user's facial skin condition is improved by comparing the image of the current date with the image of the previous date.

The database 400 may store information on skin conditions provided for each user. The database 400 also stores information about a numerical value quantifying the degree of wrinkles as well as a skin condition, and such information may be sequentially stored by date or time.

FIG. 5 is a diagram illustrating a detailed configuration of the user terminal shown in FIG. 4 .

Referring to FIG. 5 , the user terminal 200 according to an embodiment of the present invention includes a first communication unit 111 , a second communication unit 112 , an input unit 113 , a control unit 114 , an output unit 115 , It may include a storage unit 116 and a display unit 117 .

The first communication unit 111 may interwork with the skin diagnosis apparatus 100 and receive an RGB image.

The second communication unit 112 may transmit/receive various data in conjunction with the service server 300 .

The input unit 113 may receive information according to a user's menu or key manipulation.

The controller 114 may detect a face region using the RGB image, divide the detected face region into a plurality of predetermined skin regions, and detect wrinkles in the plurality of divided skin regions.

The controller 114 may diagnose a skin condition by calculating a numerical value quantifying the degree of wrinkles in a plurality of skin regions.

The output unit 115 may output audio to guide skin diagnosis and skin condition.

The storage unit 116 may store information related to skin diagnosis, such as RGB images by date, skin regions, orthogonalized values of wrinkles, and skin conditions.

The display unit 117 may display an RGB image of the user's face, a numerical value obtained by orthogonalizing the degree of wrinkles using the RGB image, a skin condition, and the like on the screen.

6 is a view for explaining a skin diagnosis process according to an embodiment of the present invention, and FIG. 7 is a view showing a screen displaying a skin diagnosis result through a mobile app.

Referring to FIG. 6 , the skin diagnosis apparatus according to an embodiment of the present invention may acquire an RGB image by photographing a face using a non-contact RGB camera (S610).

Next, the skin diagnosis apparatus may provide the obtained RGB image to the user terminal (S620).

Next, the user terminal may analyze the obtained RGB image, detect a face region as a result of the analysis, and detect a wrinkle in a skin region divided in advance within the face region (S630).

Next, the user terminal may calculate a numerical value quantifying the degree of wrinkles, diagnose the skin condition based on this, and generate skin diagnosis information ( S640 ).

Next, the user terminal may display the generated skin diagnosis information on the screen (S650). As shown in (a) of FIG. 7 , the user terminal visually displays skin diagnosis information through a mobile application, but shows wrinkle scores for each skin region as shown in (b) and (c).

Next, when the skin diagnosis apparatus provides the skin diagnosis information to the service server ( S660 ), the service server may store the received skin diagnosis information in the database ( S670 ).

The term '~ unit' used in this embodiment means software or hardware components such as field-programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. '~unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

100: skin diagnosis device
200: user terminal
300: service server
400: database

Claims (10)

acquiring, by a skin diagnosis apparatus, an image by photographing a face for which a skin condition is to be diagnosed;
generating, by the user terminal, a masking image by pre-processing the image;
calculating, by the user terminal, a wrinkle score quantifying the degree of wrinkles from the masking image; and
and diagnosing, by the user terminal, a skin condition for a skin region of the face based on the calculated wrinkle score, and generating skin diagnosis information as a result of the diagnosis,
The calculating step is
performing histogram smoothing on the masking image,
By blurring the masking image on which the histogram smoothing has been performed in the horizontal and vertical directions, respectively, to remove noise edges to generate a horizontal edge image and a vertical edge image,
Obtaining an edge gradient using a Sobel edge detector from each of the generated horizontal edge image and vertical edge image,
Determining an edge having a maximum magnitude as a candidate edge using a Non Maximum Suppression (NMS) algorithm based on the edge gradient,
It is determined whether or not it is an actual edge by applying a threshold to the determined candidate edge,
After performing a morphology open operation on the horizontal edge image, the edge image is generated by combining the horizontal edge image including the edge determined to be the actual edge and the vertical edge image,
A skin diagnosis method for calculating a wrinkle score quantifying the degree of the wrinkle from the generated edge image. According to claim 1,
The generating step is
detecting a face region from the image, and detecting a predetermined skin region among the detected face regions;
and generating a masking image including the detected skin region. The method of claim 1,
The calculating step is
Converting the color space of the masking image into an HSV color space and extracting brightness information using V of the HSV color space,
and performing histogram smoothing on the masking image based on the extracted brightness information. 4. The method of claim 3,
The calculating step is
Histogram smoothing is performed on the masking image based on the extracted brightness information, and the histogram smoothing is performed by dividing the masking image into a plurality of regions. 5. The method of claim 4,
The calculating step is
Detecting an outline from the generated edge image,
calculating the number of the detected edges and contours,
A skin diagnosis method for calculating a wrinkle score quantifying the degree of the wrinkle based on the calculated number of edges and contours. a skin diagnosis apparatus for acquiring an image by photographing a face for which a skin condition is to be diagnosed; and
Pre-processing the obtained image to generate a masking image, calculate a wrinkle score that quantifies the degree of wrinkles from the generated masking image, and diagnose the skin condition for the skin region of the face based on the calculated wrinkle score and a user terminal that generates skin diagnosis information as a result of the diagnosis,
The user terminal is
performing histogram smoothing on the masking image,
By blurring the masking image on which the histogram smoothing has been performed in the horizontal and vertical directions, respectively, to remove noise edges to generate a horizontal edge image and a vertical edge image,
Obtaining an edge gradient using a Sobel edge detector from each of the generated horizontal edge image and vertical edge image,
Determining an edge having a maximum magnitude as a candidate edge using a Non Maximum Suppression (NMS) algorithm based on the edge gradient,
It is determined whether or not it is an actual edge by applying a threshold to the determined candidate edge,
After performing a morphology open operation on the horizontal edge image, the edge image is generated by combining the horizontal edge image including the edge determined to be the actual edge and the vertical edge image,
and calculating a wrinkle score quantifying the degree of the wrinkle from the generated edge image. 7. The method of claim 6,
The user terminal is
detecting a face region from the image, and detecting a predetermined skin region among the detected face regions;
and generating a masking image including the detected skin region. 7. The method of claim 6,
The user terminal is
Converting the color space of the masking image into an HSV color space and extracting brightness information using V of the HSV color space,
and histogram smoothing on the masking image based on the extracted brightness information. 9. The method of claim 8,
The user terminal is
The histogram smoothing is performed on the masking image based on the extracted brightness information, and the histogram smoothing is performed by dividing the masking image into a plurality of regions. 10. The method of claim 9,
The user terminal is
Detecting an outline from the generated edge image,
calculating the number of the detected edges and contours,
and calculating a wrinkle score quantifying the degree of the wrinkle based on the calculated number of edges and contours.

Images