KR102151710B1 - Method for evaluating condition of skin based on image analysis result and skin recognizer and program for executing the method - Google Patents

Abstract

A method of evaluating a skin condition based on an image analysis result and a learned skin recognizer according to an embodiment of the present invention includes at least one evaluation target image satisfying a predetermined photographing condition among a plurality of skin images of the skin. Determining an image to be evaluated; Determine at least one of a first attribute value corresponding to the color of the skin, a second attribute value corresponding to the degree of wrinkles of the skin, and a third attribute value corresponding to the flatness of the skin from the at least one evaluation target image Determining an attribute value; A first feature vector generation step of generating a first feature vector based on at least one of the first, second, and third attribute values; And a second feature vector determining step of determining a second feature vector corresponding to the grade of the skin based on the first feature vector and the skin recognizer. In this case, the skin recognizer may be a data set representing a correlation between a first feature vector corresponding to one or more attribute values of each of the plurality of skins and a second feature vector corresponding to each grade of the plurality of skins.

Description

A method and program for evaluating the condition of the skin based on the results of image analysis and the learned skin recognizer {METHOD FOR EVALUATING CONDITION OF SKIN BASED ON IMAGE ANALYSIS RESULT AND SKIN RECOGNIZER AND PROGRAM FOR EXECUTING THE METHOD}

Embodiments of the present invention relate to a method and program for evaluating a condition of a skin based on an image analysis result and a learned skin recognizer.

The skin is the largest organ of the integument system, including the skin accessory tissues such as hair, nails, hair and sweat glands. The skin contains multiple layers of epithelial tissue that protects the underlying muscles and organs. Since the skin is constantly attacked by a number of external and internal factors, the skin can be infected with numerous diseases. Therefore, monitoring the skin condition is important from a health point of view and from a beauty point of view.

There are many methods of monitoring the condition of the skin, but most of the methods require access to a dermatologist or a dedicated device provided in the dermatologist. However, access to such a doctor or dedicated device is accompanied by difficulties, inconvenience, and excessive expenditure.

Therefore, there is a need for a simpler solution that allows general users to easily determine the condition of the skin without having to face a dermatologist or a dedicated device provided in the dermatologist.

An object of the present invention is to provide a method and program for evaluating a skin condition capable of confirming an attribute value and a grade of skin for one or more skin attributes from a skin image.

In addition, the present invention is to provide a method and program for evaluating a skin condition in which an attribute value for each attribute of the skin is checked, and an objective grade of the skin is determined based on this, so that appropriate measures can be taken.

In addition, an object of the present invention is to provide a method and program capable of checking the attribute value of skin based on image analysis.

In addition, an object of the present invention is to provide a method and a program for confirming an attribute value for a characteristic in a three-dimensional space of a skin based on a two-dimensional image analysis.

Furthermore, the present invention aims to provide a skin condition evaluation method and program that can integrate and provide solutions based on each condition in addition to analyzing the condition of the skin.

A method of evaluating a skin condition based on an image analysis result and a learned skin recognizer according to an embodiment of the present invention includes at least one evaluation target image satisfying a predetermined photographing condition among a plurality of skin images of the skin. Determining an image to be evaluated; Determine at least one of a first attribute value corresponding to the color of the skin, a second attribute value corresponding to the degree of wrinkles of the skin, and a third attribute value corresponding to the flatness of the skin from the at least one evaluation target image Determining an attribute value; A first feature vector generation step of generating a first feature vector based on at least one of the first, second, and third attribute values; And a second feature vector determining step of determining a second feature vector corresponding to the grade of the skin based on the first feature vector and the skin recognizer. In this case, the skin recognizer may be a data set representing a correlation between a first feature vector corresponding to one or more attribute values of each of the plurality of skins and a second feature vector corresponding to each grade of the plurality of skins.

In the determining of the attribute value, in any one of the evaluation target images, a reference region for skin color evaluation is set, and from the at least one evaluation target image based on the color of the reference region. Determining a first attribute value corresponding to the color of the skin; And a second attribute value corresponding to the degree of wrinkles of the skin and a third attribute value corresponding to the flatness of the skin through mutual comparison between the images taken of the skin from different angles among the at least one evaluation target image. It may include; determining at least one.

The skin is the skin of the face, and the determining of the first attribute value includes a difference between a reference color, which is a color of the reference area, and a color in an evaluation target image including a front image of the face among the one or more evaluation target images. Selecting at least one point where is equal to or greater than a predetermined threshold difference; And the first attribute value based on at least one of a quantity of the selected at least one point, a degree of color difference between the selected at least one point and the reference color, and a degree of dispersion between the selected at least one point. It may include; determining step.

The determining of at least one of the second attribute value and the third attribute value may include determining a first bending area in a first evaluation target image, and determining a first degree of curvature that is a degree of curvature of the first bending area. ; Determining a second degree of curvature, which is a degree of curvature of an area corresponding to the first curvature area in a second evaluation target image different from the first evaluation target image; And determining at least one of the second attribute value and the third attribute value based on the first curvature and the second curvature.

The skin is the skin of the face, and the predetermined photographing condition is whether the image includes a front image of the face, whether the image includes an image of the left side of the face taken at a predetermined first angle, and the image is predetermined. Whether the image includes a right side image of the face taken at a second angle of, whether the image includes a top image of the face taken at a predetermined third angle, and the image of the face taken at a predetermined fourth angle At least one of whether to include an image may be included.

A method of evaluating a skin condition based on an image analysis result and a learned skin recognizer according to an embodiment of the present invention includes, after the step of determining the second feature vector, the grade of the skin based on the second feature vector. It may further include a step of determining; In this case, the grade of the skin may include a grade 0 that does not require special management, a grade that requires general management, a grade that requires professional management, and a grade that requires treatment, according to the necessity of managing the skin. have.

In the generating of the first feature vector, a first feature vector is generated based on at least one of the first attribute value, the second attribute value, and the third attribute value and at least one physical quantity of the skin, and the at least one The physical quantity may evaluate the condition of the skin based on an image analysis result and a learned skin recognizer, which is at least one of the elasticity of the skin, the moisture content of the skin, the keratin thickness of the skin, and the thickness of the makeup painted on the skin. .

In the method of evaluating the skin condition based on the image analysis result and the learned skin recognizer according to an embodiment of the present invention, before the evaluation target image determination step, a second feature vector corresponding to the skin grade is marked And training the skin recognizer based on one or more training skin information including a first feature vector corresponding to an attribute value for at least one attribute of the skin.

According to the present invention, it is possible to implement a skin condition evaluation method and program capable of checking attribute values and skin grades for one or more skin attributes from a skin image.

In addition, it is possible to implement a skin condition evaluation method and program in which an attribute value for each attribute of the skin is checked and an objective grade of the skin is determined based on this, so that appropriate measures can be taken for the corresponding skin.

In addition, it is possible to implement a method and program that can check the attribute value of the skin based on image analysis.

In addition, a method and a program capable of checking attribute values for characteristics in a three-dimensional space of the skin may be implemented based on the analysis of a two-dimensional image.

Furthermore, the present invention can implement a skin condition evaluation method and program that can provide an integrated solution for each condition in addition to analyzing the condition of the skin.

1 schematically shows a system for evaluating a skin condition according to an embodiment of the present invention.
2 schematically shows the configuration of a skin condition evaluation apparatus according to an embodiment of the present invention.
3 is an example of a predetermined photographing condition for a face when a person's facial skin is to be evaluated.
4 is a diagram illustrating only an evaluation target image that satisfies a predetermined shooting condition among a plurality of images received from a user terminal by a control unit according to an embodiment of the present invention.
5A to 5C are examples in which the controller sets a reference area from various evaluation target images.
6 is a diagram illustrating a process of determining a first attribute value by a controller according to an embodiment of the present invention.
7A and 7B are diagrams for explaining a process of determining a third attribute value by a controller according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating a method of generating/updating a skin recognizer by a skin condition evaluation apparatus according to an embodiment of the present invention.
9 illustrates a method of determining, by the skin condition evaluation apparatus according to an embodiment of the present invention, an attribute value and a grade of an evaluation target skin based on a skin recognizer.
10 is a flowchart illustrating a method of evaluating a skin condition performed by the apparatus for evaluating a skin condition according to an embodiment of the present invention.
11 is an example of a screen displayed on a user terminal according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will be apparent with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

In the following embodiments, terms such as first and second are not used in a limiting meaning, but are used for the purpose of distinguishing one component from another component. In the following examples, the singular expression includes the plural expression unless the context clearly indicates otherwise. In the following embodiments, terms such as include or have means that the features or elements described in the specification are present, and do not preclude the possibility that one or more other features or elements may be added. In the drawings, components may be exaggerated or reduced in size for convenience of description. For example, the size and shape of each component shown in the drawings are arbitrarily shown for convenience of description, and thus the present invention is not necessarily limited to what is shown.

1 schematically shows a system for evaluating a skin condition according to an embodiment of the present invention.

Referring to FIG. 1, a skin condition evaluation system according to an embodiment of the present invention may include a server 100, a user terminal 200, an external device 300, and a communication network 400 connecting them.

In the skin condition evaluation system according to an embodiment of the present invention, the server 100 receives an image of the skin from the user terminal 200 and/or the external device 300 and evaluates the skin condition as an attribute value, or The status of can be evaluated by grade. In addition, the skin condition evaluation system allows the server 100 to obtain one or more training skin information marked with attribute values and grades from the user terminal 200 and/or the external device 300, and train the skin recognizer based on this. have. A more detailed description will be given later.

In the present invention,'skin' is a tissue of the human body that makes up the surface of the human body, and may be an object for which an attribute value or grade is desired. In this case, the skin may be the skin of the face or the skin of various parts of the human body. However, in the following description, for convenience of explanation, it is assumed that the skin is the skin of the face.

The degree of redness of the skin refers to the degree of redness of the skin, and the redness of the skin may indicate a higher degree of redness. The degree of redness of the skin may have meaning as an element that determines the color of the skin, and may be evaluated as a first attribute value described later.

The degree of wrinkles on the skin may mean the degree to which the skin is finely folded due to the decrease in elasticity of the skin. In this case, the high degree of wrinkles on the skin may mean a case in which the skin is folded finely or the finely folded degree is large, and may be evaluated as a second attribute value described later.

On the other hand, the flatness of the skin may mean the degree to which the skin is uneven due to other factors excluding the curvature caused by the wrinkles of the skin. For example, the flatness of the skin may mean the degree of unevenness of the skin due to pores, or it can mean the degree of unevenness of the skin due to wounds, spots, scars, etc. in a specific area, and will be evaluated as a third attribute value described later. I can.

However, the above-described skin attribute values are illustrative, and the spirit of the present invention is not limited thereto, and it is obvious that any one capable of representing the condition of the skin can be used as an attribute of the skin of the present invention.

Meanwhile,'attribute values' such as the above-described first attribute value, second attribute value, and third attribute value may include components corresponding to each of at least one evaluation item. In other words, each attribute value may be a single value such as 10, 20, 0.1, and despite the name'value', each attribute value is a vector composed of multiple components such as [0.1, 0.3, 0.7] It may be a work or a set.

In the present invention, the'level of skin' may be a value determined according to the necessity of skin care based on the above-described skin attribute value. The grade is, for example, four grades, and may include a grade 0 that does not require special management, a first grade that requires general management, a second grade that requires professional management, and a third grade that requires treatment. However, the classification of such grades is exemplary, and skin may be classified into more or fewer grades. At this time, the grade of the skin may be calculated by considering the above-described properties of the skin in various ways.

In the present invention, the user terminal 200 may refer to various devices capable of transmitting an image of the skin to the server 100 and receiving an attribute value for the skin or a grade of the skin from the server 100. In this case, the terminal may be a portable terminal 201 or a personal computer 202.

The user terminal 200 may include a display means for displaying content or the like, and an input means for acquiring a user's input for such content. In this case, the input means and the display means may be configured in various ways. For example, the input means may include a keyboard, a mouse, a trackball, a microphone, a button, and a touch panel, but are not limited thereto.

Although not shown in the drawing, the user terminal 200 may be used together with a skin diagnosis apparatus (not shown) that obtains various information on the skin.

The skin diagnosis apparatus (not shown) may irradiate the skin with light of at least one or more wavelengths to obtain an image of the skin for each wavelength of light, and transmit the image of the skin to the server 100 through the user terminal 200. In addition, the skin diagnosis apparatus (not shown) may acquire skin images from various angles or predetermined angles according to a request received by the user terminal 200 from the server 100 and transmit the skin images to the server 100.

In addition to the image of the skin, the skin diagnosis apparatus (not shown) may acquire information on an environment in which the skin is exposed, and transmit the information to the server 100. For example, the skin diagnosis apparatus (not shown) may measure the temperature, humidity, illuminance, color temperature, and degree of ultraviolet rays at the time of photographing a skin image and transmit the same to the server 100.

Of course, the skin diagnosis apparatus (not shown) may measure the physical quantity of the skin itself and transmit it to the server 100. For example, a skin diagnosis device (not shown) measures the elasticity of the skin, the degree of moisture in the skin, the keratin thickness of the skin, and the thickness of the makeup painted on the skin, based on various sensors provided in the skin diagnosis device (not shown). This may be transmitted to the server 100.

Such a skin diagnosis apparatus (not shown) may be a device separate from the user terminal 200 as described above. For example, the skin diagnosis device (not shown) is a device that is connected to the user terminal 200 through a wired/wireless network. In this case, the user terminal 200 is a device that mediates the skin diagnosis device (not shown) and the server 100 Can be

Meanwhile, the skin diagnosis apparatus (not shown) may be a device included in the user terminal 200. In other words, the user terminal 200 according to an embodiment of the present invention may operate as the aforementioned skin diagnosis apparatus.

In the present invention, the external device 300 may refer to various devices that transmit and receive data through the server 100 and the user terminal 200 and the communication network 400.

According to an embodiment of the present invention, the external device 300 may be a device that provides training skin information for training a skin recognizer provided in the server 100. For example, the external device 300 is a server of a skin hospital, and may be a device that provides an image of a skin and diagnostic information about the skin (especially, the above-described skin attribute value or evaluation information on the skin grade). The number of such external devices 300 may be singular or plural.

According to another embodiment of the present invention, the external device 300 may be a device for checking the attribute value or grade of the skin image acquired by the external device 300 itself. For example, the external device 300 may be a device that is provided in a dermatology or the like to provide skin information of a patient. However, both cases described above are exemplary, and the spirit of the present invention is not limited thereto.

In the present invention, the communication network 400 serves to connect the server 100, the user terminal 200, and the external device 300. For example, the communication network 400 provides an access path so that the user terminal 200 can transmit and receive packet data after accessing the server 100. The communication network 400 is a wired network such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, satellite communication, etc. It may cover wireless networks of, but the scope of the present invention is not limited thereto.

In the present invention, the server 100 may receive an image of a skin for which the attribute value or grade is to be known from the user terminal 200 and/or the external device 300 and determine the attribute value or grade of the corresponding skin. Of course, prior to determining the attribute value or grade for the skin, the server 100 may acquire training skin information from the user terminal 200 and/or the external device 300, and train the skin recognizer based thereon. . To this end, the server 100 may include a skin condition evaluation device according to an embodiment of the present invention.

2 schematically shows the configuration of the skin condition evaluation apparatus 110 according to an embodiment of the present invention.

Referring to FIG. 2, the skin condition evaluation apparatus 110 according to an embodiment of the present invention may include a communication unit 111, a control unit 112, and a memory 113. In addition, although not shown in the drawings, the skin condition evaluation apparatus 110 according to an embodiment of the present invention may further include an input/output unit, a program storage unit, and the like.

The communication unit 111 is a hardware necessary for the skin condition evaluation device 110 to transmit and receive signals such as control signals or data signals through wired/wireless connection with other network devices such as the user terminal 200 and/or the external device 300 And a device including software.

The controller 112 may include all kinds of devices capable of processing data, such as a processor. Here, the'processor' may refer to a data processing device embedded in hardware having a circuit physically structured to perform a function represented by a code or instruction included in a program. As an example of such a data processing device built into the hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an ASIC (Application-Specific Integrated) Circuit) and processing devices such as a Field Programmable Gate Array (FPGA) may be covered, but the scope of the present invention is not limited thereto.

The memory 113 temporarily or permanently stores data processed by the skin condition evaluation apparatus 110. The memory may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto.

Hereinafter, a description is made on the premise that the skin condition evaluation apparatus 110 is provided in the server 100, but the skin condition evaluation apparatus 110 may be provided separately from the server 100 according to role allocation.

Meanwhile, as described above, the server 100, that is, the skin condition evaluation apparatus 110, receives an image of the skin from the user terminal 200 and/or the external device 300, and the attribute value and/or grade for the corresponding skin Can be determined. In addition, the skin condition evaluation apparatus 110 may acquire one or more training skin information on which attribute values and grades are marked from the user terminal 200 and/or the external device 300, and train the skin recognizer based thereon.

In the following, the skin condition evaluation apparatus 110 is in the order of 1) determining one or more attribute values for the skin, 2) training the skin recognizer, and 3) determining the skin level based on the recognized skin recognizer. Explained as.

1) Method of determining attribute values for skin by the skin condition evaluation device 110

Hereinafter, the description will focus on the content of the skin condition evaluation apparatus 110 determining the attribute value of the skin based on the skin image received from the user terminal 200 or the external device 300.

The controller 112 of the skin condition evaluation apparatus 110 according to an embodiment of the present invention may obtain an image of the skin from the user terminal 200. In this case, the controller 112 may acquire a plurality of images of the skin from the user terminal 200. The plurality of images may be images obtained by different photographing conditions such as a photographing angle for the same subject (eg, a face) and a wavelength of light irradiated during photographing. Such photographing conditions may be tagged in the form of metadata on the skin image, or may be acquired from the image itself through image analysis by the controller 112.

Subsequently, the controller 112 according to an embodiment of the present invention may determine at least one evaluation target image that satisfies a predetermined shooting condition from among a plurality of images received from the user terminal 200.

In the present invention, the'evaluation target image' is an image used in skin evaluation, and may be an image selected according to different criteria (ie, the above-described predetermined photographing conditions) for each part to be evaluated for a condition.

3 is an example of a predetermined photographing condition for a face when evaluating the facial skin of the person 500.

Referring to FIG. 3, when evaluating the facial skin of a person 500, a predetermined photographing condition is whether the image includes a front image of the face (for example, an image obtained from the 623 direction), and the image is a predetermined first Whether to include an image of the left side of the face taken at an angle (e.g., an image obtained from the 624 or 625 direction), an image of the right side of the face taken from a second predetermined angle (e.g., an image obtained from the 621 or 622 direction) Whether or not the image includes a top image of a face photographed at a predetermined third angle (eg, an image acquired from the 613 direction), and whether the image includes a bottom image of a face photographed at a predetermined fourth angle (eg, 633 It may be at least one of whether to include an image acquired from a direction).

Of course, in addition to the above-described shooting conditions, the shooting conditions include whether the image includes an image of the upper and left side of the face taken at a predetermined fifth angle (for example, an image obtained from the 614 or 615 direction), and whether the image is taken at a predetermined sixth angle. Whether to include an image of the upper right side of the face (e.g., an image obtained from the 611 or 612 direction), whether the image contains an image of the lower left side of the face taken from a predetermined seventh angle (e.g., an image obtained from the 634 or 635 direction) It may further include at least one of whether or not the image includes a lower right side image of the face photographed at a predetermined eighth angle (eg, an image acquired in the 631 or 632 direction). However, this is exemplary, and the spirit of the present invention is not limited thereto, and the shooting conditions may be set more variously or simpler than those described above.

The control unit 112 according to an embodiment of the present invention extracts an image that satisfies the above-described shooting condition among a plurality of images received from the user terminal 200 as an evaluation target image, and determines an attribute value for the skin to be described later. Can be used.

In another embodiment of the present invention, the control unit 112 according to an embodiment of the present invention provides the above-described shooting conditions to the user terminal 200, and the user terminal 200 acquires only images meeting the above-described conditions. Thus, it can be transmitted to the server 100. In other words, the user terminal 200 may acquire an image according to the provided shooting condition and transmit it to the server 100.

In this case, the controller 112 may immediately determine the image received from the user terminal 200 as the image to be evaluated without determining an image to be evaluated that satisfies a predetermined shooting condition among the plurality of images. In this case, the control unit 112 may provide an interface to the user terminal 200 to enable photographing according to photographing conditions. For example, the controller 112 may provide the user terminal 200 with a photographing interface that displays a sample image according to the photographing condition, information on the photographing condition, and whether the currently photographed image satisfies the photographing condition.

In another embodiment of the present invention, the controller 112 may determine the above-described evaluation target image from the video received from the user terminal 200. That is, the control unit 112 may determine at least one frame that satisfies a predetermined shooting condition among a plurality of frames included in the video as the evaluation target image. Of course, this is based on the premise that the user terminal 200 transmits to the server 100 a video photographed from various angles of an object for which the skin condition is to be evaluated. In this case, the process in which the control unit 112 provides a shooting condition and a shooting interface to the user terminal 200 in advance, or the control unit 112 determines an image to be evaluated from a plurality of images received from the user terminal 200 will be omitted. I can.

However, in the following description, for convenience of explanation, it is assumed that the controller 112 determines an evaluation target image from a plurality of images received from the user terminal 200.

4 is an evaluation target image 511 to 515, 521 to 525, and 531 to 535 that satisfy a predetermined shooting condition among a plurality of images received from the user terminal 200 by the control unit 112 according to an embodiment of the present invention. ) Only.

For convenience of explanation, it is assumed that a predetermined photographing condition for determining an evaluation target image includes images acquired in directions 611 to 615, 621 to 625, and 631 to 635 as described in FIG. 3.

In this case, the controller 112 may determine 15 images as shown from the images (eg, 100 images) received from the user terminal 200 as the evaluation target images. The determined evaluation target image may be used to determine one or more attribute values for the skin, or to determine a grade of the skin.

The control unit 112 according to an embodiment of the present invention sets a reference area for skin color evaluation in any one of the evaluation target images, and at least one evaluation target image based on the color of the reference area. From, a first attribute value corresponding to the color of the skin may be determined.

As described above, in the present invention, the'attribute value' of the skin may refer to various factors that are the basis for evaluating the skin, and the first attribute value may refer to an evaluation value for the color of the skin.

In this case, the evaluation of the color of the skin may mean various evaluations determined by color such as homogeneity of the skin color, the size of the redness of the skin, and the degree of redness of the skin. Meanwhile, the reference region is a region used as a reference for determining the color of the skin, and the color of the reference region may mean a reference color used for determining the first attribute value.

5A to 5C are examples in which the control unit 112 sets a reference area from various evaluation target images 523 and 525.

For example, as illustrated in FIG. 5A, the controller 112 may set an area 523-1 corresponding to the eyebrow in the front image 523 of the face as a reference area. In addition, as shown in FIG. 5B, the controller 112 may set an area 525-1 corresponding to a peripheral portion of the ear in the left side image 525 of the face as a reference area. In addition, as shown in FIG. 5C, the controller 112 may set an area 523-2 corresponding to the rest of the body (eg, neck, etc.) except for the face in the front image 523 of the face as the reference area.

The setting of the reference area can be set to various areas of the image to be evaluated according to the evaluation items of the color to be expressed as the first attribute value (for example, the degree of redness, the degree of burns, the degree of acne, etc.). Which area should be set can be obtained experimentally.

For example, in the case of indicating the degree of redness of the skin as the first attribute value, the reference region may be determined as the region 523-1 corresponding to the eyebrows, as shown in FIG. 5A. Of course, this can be premised on the fact that the glabellar is the region where redness is the least expressed experimentally.

In addition, when trying to indicate the degree of skin burn (that is, the degree of sunburn) as the first attribute value, the reference region is an area 525-1 corresponding to the ear periphery as shown in FIG. 5B, or shown in FIG. 5C. As described above, it may be determined as a region 523-2 corresponding to the rest of the body (eg, neck, etc.) excluding the face. Of course, it can also be assumed that it has been experimentally confirmed that the affected areas are the areas that are the least sunburned.

However, this is merely an example, and the spirit of the present invention is not limited thereto. Hereinafter, as an example, a case where the degree of redness of the skin is to be expressed as a first attribute value is determined, the step of determining the first attribute value by the controller 112 Look at in more detail.

6 is a diagram for explaining a process in which the controller 112 determines a first attribute value according to an embodiment of the present invention.

Referring to FIG. 6, the controller 112 according to an embodiment of the present invention includes an evaluation including a front image of a face among one or more evaluation target images (eg, 511 to 515, 521 to 525 and 531 to 535 of FIG. 4). The face area 523-5 may be recognized from the target image 523. In addition, the control unit 112 selects an exclusion area including the eye areas 523-6 and 523-7, the nose area 523-8, and the mouth area 523-9 from the recognized face area 523-5. I can recognize it. That is, since the eyes, nose, and mouth in the image 523 are not subject to skin condition evaluation, a more meaningful evaluation result can be derived by excluding parts of the image, such as eyes, nose, and mouth having different colors in advance.

In the face area 523-5, the control unit 112 determines the reference color and color in the remaining areas except for the selection exclusion areas 523-6, 523-7, 523-8, and 523-9 recognized by the above-described process. At least one point 523-3 and 523-4 whose difference is greater than or equal to a predetermined threshold difference may be selected. In this case, the reference color may be the color of the reference area 523-1 set by the above-described process.

Subsequently, the controller 112 determines the quantity of the selected at least one point (523-3, 523-4), the degree of color difference from the reference color of the selected at least one point (523-3, 523-4), and the selected The first attribute value may be determined based on at least one of the degree of dispersion between the at least one point 523-3 and 523-4.

At this time, the determined first attribute value may include information on each of the above-described evaluation elements as one or more components for each. For example, the first attribute value is a component for the number of selected points 523-3 and 523-4, a component representing the degree of color difference from the reference color of the selected points 523-3 and 523-4, and selection A component indicating the degree of dispersion of the points 523-3 and 523-4 can be included in the form of components such as [2, 0.4, 0.7].

The control unit 112 according to an embodiment of the present invention compares the images taken from different angles of the skin among the images to be evaluated (for example, 511 to 515, 521 to 525 and 531 to 535 in FIG. 4) (for example, 522 and 523). (Comparison of) to determine at least one of the second attribute value and the third attribute value.

Since an image represents an object in a three-dimensional space (for example, a person's face) in a two-dimensional space, it may be difficult to grasp the characteristics of the object in a three-dimensional space with only one image. For example, it may be difficult to accurately grasp the degree of wrinkles or curvature of the face from the front image of the face (523 in FIG. 4). Accordingly, the control unit 112 can more accurately grasp the characteristics of the object in the three-dimensional space by comparing images captured from various angles with respect to the same part.

Hereinafter, with reference to FIGS. 7A and 7B, contents of the controller 112 determining the third attribute value will be described on the assumption that there is a protruding point on the face.

7A and 7B are diagrams for explaining a process in which the controller 112 determines a third attribute value according to an embodiment of the present invention.

For convenience of explanation, it is assumed that there is a point 523-10 on a person's face, and the point 523-10 protrudes as shown to affect the third attribute value. Meanwhile, as described above, the third attribute value may be a value indicating a degree of unevenness of the skin due to other factors other than the curvature caused by the wrinkles of the skin.

Under the above-described assumption, the control unit 112 according to an embodiment of the present invention determines the first bending area (the point 523-10) from the first evaluation target image 523, and the first bending area (the point ( 523-10) The first degree of curvature, which is the degree of curvature of the region) may be determined. In this case, the controller 112 may determine a first curved area (a point 523-10) based on a color in the first evaluation target image 523. In other words, the control unit 112 may determine a point in the first evaluation target image 523 where the difference between the reference color and the color is greater than or equal to the threshold difference as the first curved area (the point 523-10). On the other hand, the first degree of curvature determined based on the first evaluation target image 523 may be determined to correspond to no curvature according to the type of the first curvature area, or may be determined to reflect the degree of curvature. For example, as shown in FIG. 7A, when the area of the point 523-10 is determined as the first curved area, the degree of curvature of the first curved area cannot be grasped from the first evaluation target image 523 (because the point 523 Since the front image of -10)), the first degree of curvature may be determined to correspond to that without curvature. However, in contrast, when the degree of curvature can be determined from the first evaluation target image 523 (for example, when determining the second attribute value, when the first evaluation target image 523 contains wrinkles), the first curvature degree May be determined to reflect the degree of curvature in question.

Subsequently, the control unit 112 determines the area corresponding to the first curved area (the point 523-10) in the second evaluation target image 522 different from the first evaluation target image 523 (that is, the point 522-1). The second degree of curvature, which is the degree of curvature of the region), may be determined. In other words, based on the image of the same object taken from different angles, the degree of curvature can be accurately grasped even without information about the corresponding area in the 3D space.

The controller 112 may determine a third attribute value based on at least one of the first and second degrees of curvature determined by the above-described process. For example, the control unit 112 may determine a third attribute value based on the average of the first and second bend degrees, and the third attribute value based on the larger of the first and second bends. You can decide.

Meanwhile, since the process described in FIGS. 7A and 7B may be similarly repeated in determining the second attribute value, a detailed description of the process of determining the second attribute value will be omitted.

Accordingly, the present invention can evaluate the color of the skin and evaluate the characteristics in a three-dimensional space based on the analysis of the image of the skin.

The control unit 112 according to an embodiment of the present invention may perform various operations based on at least one of a first attribute value, a second attribute value, and a third attribute value determined according to the above-described process. For example, the control unit 112 may provide the determined first attribute value, second attribute value, and third attribute value to the user terminal 200, or may be used to determine a skin grade as described later. Hereinafter, a description will be made focusing on the contents of the control unit 112 determining the skin level from the determined attribute values.

2) How the skin condition evaluation device 110 creates/updates the skin recognizer

Before the skin condition evaluation apparatus 110 determines the grade of the skin based on the attribute values determined by the above-described process, generation and/or learning of a skin recognizer that determines the grade of the skin needs to be preceded. Therefore, hereinafter, a method of generating/updating a skin recognizer of the skin condition evaluation apparatus 110 will be described first.

In the present invention,'creating' the skin recognizer may mean not only configuring the skin recognizer for the first time, but also updating the configured skin recognizer.

The skin recognizer according to an embodiment of the present invention may be a data set expressing a correlation between a first feature vector corresponding to one or more attribute values of each of a plurality of skins and a second feature vector corresponding to each grade of a plurality of skins. have. In this case, the one or more attribute values may mean at least one of a first attribute value, a second attribute value, and a third attribute value determined by the above-described process.

Such a skin recognizer includes a correlation between a first feature vector corresponding to one or more attribute values of each of a plurality of skins generated by a machine learning technique and a second feature vector corresponding to each grade of the plurality of skins, That is, it may be mapping information of both.

The skin condition evaluation apparatus 110 includes Logistic regression, Decision tree, Nearest-neighbor classifier, Kernel discriminate analysis, Neural network, Support Vector Machine, Random forest, Boosted tree to classify a plurality of training skin images and/or training skin information. Algorithms and/or methods (techniques) such as, etc. can be used.

The skin condition evaluation apparatus 110 may use algorithms and/or methods (techniques) such as linear regression, regression tree, kernel regression, support vector regression, and deep learning to predict an attribute value or grade of a certain skin.

In addition, the skin condition evaluation apparatus 110 may use algorithms and/or methods (techniques) such as principal component analysis, non-negative matrix factorization, independent component analysis, manifold learning, and SVD to calculate vectors.

The skin condition evaluation apparatus 110 may use algorithms and/or methods (techniques) such as k-means, hierarchical clustering, mean-shift, and self-organizing maps (SOMs) for grouping a plurality of skin images.

The skin condition evaluation apparatus 110 may use algorithms and/or methods (techniques) such as Bipartite cross-matching, n-point correlation two-sample testing, and minimum spanning tree for data comparison.

However, the above-described algorithm and/or method (technique) is exemplary, and the spirit of the present invention is not limited thereto.

Meanwhile, the skin recognizer may be a data set as described above. In other words, the skin recognizer may be a data set composed of a plurality of numbers (or coefficients) such as a matrix.

FIG. 8 is a diagram for explaining a method of creating/updating the skin recognizer 821 by the skin condition evaluation apparatus 110 according to an embodiment of the present invention.

Referring to FIG. 8, the controller 112 according to an embodiment of the present invention may obtain one or more training skin information 810 to generate the skin recognizer 821. The training skin information 810 includes a skin image, and an attribute value for at least one attribute of the skin and a skin level may be labeled. That is, the training skin information 810 is information on skin that already knows its attribute value and grade, and may be information used to generate and/or update (or learn) a skin recognizer.

For example, in the case of the first training skin information 811, an attribute value 812 including a degree of redness, a degree of wrinkles, and a flatness of the skin and a grade 813 of the skin may be labeled. At this time, each of the attribute values 812 such as the degree of redness, the degree of wrinkles, and the flatness of the skin and the grade 813 of the skin may be labeled in the form of a vector. In other words, in the first training skin information 811, a first feature vector corresponding to an attribute value for at least one attribute of the skin and a second feature vector corresponding to a grade of the skin may be labeled.

In an optional embodiment, each of the attribute value 812 and the grade 813 may be marked on the training skin information 811 in the form of a number. For example, the attribute value 812 may be labeled in the form of a numerical value indicating the strength of each attribute, such as 0.1 and 0.8. In addition, the grade of the skin may be labeled with any one of the above four grades, such as'first grade' or'third grade'. In this way, when each attribute value is labeled as a numerical value, the control unit 112 according to an embodiment of the present invention converts the numerical values into a vector corresponding to each numerical value in the process of generating or updating the recognizer 821. can do.

Meanwhile, the attribute value 812 is an attribute value for skin age (such as 30 years old), gender (such as male), race (such as Asian), disease status (such as chronic dermatitis), in addition to attribute values for the above three attributes. It may further include. However, this is merely an example and the spirit of the present invention is not limited thereto.

The controller 112 according to an embodiment of the present invention may obtain such training skin information 810 from the external device 300 or from the memory 113 of the skin condition evaluation apparatus 110. . For example, the control unit 112 may obtain training skin information 810 from a treatment history stored in a server of a specific dermatology department. At this time, the treatment history stored in the dermatology server may be created by a specialist (eg, a dermatologist). In other words, the training skin image 310 may be based on data evaluated by an expert. In addition, the control unit 112 may obtain training skin information 810 from a history of determining attribute values or grades of other skin images in the past by itself.

The controller 112 according to an embodiment of the present invention may update the skin recognizer 821 based on the attribute value and the grade included in the training skin information 810. In this case, each of the attribute value and the grade may be in the form of a vector as described above. Accordingly, the control unit 112 according to an embodiment of the present invention uses the skin recognizer 821 based on the first feature vector (corresponding to the attribute value) and the second feature vector (corresponding to the grade) included in the training skin information 810. You can update (Update). At this time, the skin recognizer 821 includes a first feature vector corresponding to one or more attribute values of each of the plurality of skins generated by machine learning as described above, and a second feature vector corresponding to each grade of the plurality of skins. It may be correlation between feature vectors, that is, mapping information of both.

The control unit 112 according to an embodiment of the present invention updates the data set of the skin recognizer 821 so that the first feature vector and the second feature vector can be mapped to each other. ) Can be updated. For example, when the data set of the skin recognizer 821 is composed of a plurality of coefficients as shown in FIG. 8, the control unit 112 may be configured with a skin recognizer 821 based on the first feature vector and the second feature vector. You can adjust the coefficients that make up

The control unit 112 is a skin recognizer 821 that repeatedly obtains training skin information labeled with the first feature vector and the second feature vector, and repeatedly updates coefficients based on this, thereby providing a grade of skin with improved accuracy. Can be obtained.

3) How the skin condition evaluation device 110 determines the grade of the skin

In the above, a method of generating the skin recognizer 821 by the skin condition evaluation apparatus 110 has been described with reference to FIG. 8. Hereinafter, a method of determining the grade of the skin to be evaluated using the skin recognizer 821 generated by the skin condition evaluation apparatus 110 through the above-described process will be described with reference to FIG. 9.

9 illustrates a method in which the skin condition evaluation apparatus 110 according to an embodiment of the present invention determines an attribute value and a grade of the skin to be evaluated based on the skin recognizer 821.

For convenience of explanation, the controller 112 determines the attribute values 921 from the skin image 911 through image analysis according to the process described in FIGS. 3 to 7B, and the skin recognizer according to the process described in FIG. Assume that (821) is in the created state.

The control unit 112 according to an embodiment of the present invention may generate the first feature vector 931 based on at least one of the first attribute value, the second attribute value, and the third attribute value 921. For example, the controller 112 may generate a first feature vector 931 having each attribute value (a first attribute value, a second attribute value, and a third attribute value) as a component. In this case, when each attribute value has a plurality of components, the controller 112 may cause at least some of the plurality of components of each attribute value to be included in the first feature vector 931. However, this is merely an example and the spirit of the present invention is not limited thereto.

In an optional embodiment, in generating the first feature vector 931, the controller 112 may generate the first feature vector 931 by further considering at least one physical quantity of the skin or a physical quantity of the environment in which the skin is exposed. I can.

In the case of images, only visual information is involved. Therefore, it may be difficult to obtain information other than visual information about the skin from the image (for example, the elasticity of the skin, the degree of moisture in the skin, the keratin thickness of the skin, the thickness of the makeup applied to the skin, etc.). In generating the first feature vector 931, the controller 112 according to an embodiment of the present invention may generate the first feature vector 931 by further considering at least one physical quantity of the skin. For example, the control unit 112 according to an embodiment of the present invention comprises at least one of the aforementioned first attribute value, second attribute value, and third attribute value, and the first feature vector 931 based on at least one physical quantity of the skin. Can be created. At this time, the at least one physical quantity may be at least one of a skin elasticity, a moisture content of the skin, a keratin thickness of the skin, and a thickness of a makeup applied to the skin.

In addition, the result of the image may be different depending on the acquisition environment. For example, when an image is acquired under lighting conditions of a specific color, the color of the skin on the image may be different from the color of the original skin. In generating the first feature vector 931, the control unit 112 according to an embodiment of the present invention may generate the first feature vector 931 by further considering a physical quantity of an environment in which the skin is exposed.

Meanwhile, one or more physical quantities for the skin or physical quantities for the environment in which the skin is exposed may be obtained and transmitted by the aforementioned skin diagnosis apparatus (not shown) and/or the user terminal 200.

Accordingly, the present invention can more accurately evaluate the condition of the skin in consideration of various environmental factors.

The control unit 112 according to an embodiment of the present invention may determine a second feature vector 941 corresponding to the skin level based on the generated first feature vector 931 and the skin recognizer 821.

As described above, the skin recognizer 821 includes a first feature vector corresponding to one or more attribute values of each of a plurality of skins generated by a machine learning technique and a second feature corresponding to each grade of the plurality of skins. It may be correlation between vectors, that is, mapping information of both.

Therefore, the control unit 112 according to an embodiment of the present invention inputs the first feature vector 931 generated by the above-described process into the skin recognizer 821 and obtains the second feature vector 941 as an output result. I can. In this case, the second feature vector 941 may have a meaning as a vector indicating the grade of the skin. Of course, the second feature vector 941 may include a vector component corresponding to the skin level.

Accordingly, in the present invention, the attribute value for each attribute of the skin can be checked, and an objective grade of the skin can be determined based on this.

The control unit 112 according to an embodiment of the present invention may determine a skin level based on the second feature vector 941. Also, the control unit 112 may provide the determined skin level to the user terminal 200. In this case, the control unit 112 may provide the attribute value 921 determined by the above-described process to the user terminal 200 together with the determined skin level.

In an optional embodiment, the controller 112 may provide at least some of the skin grade and skin attribute values determined by the above-described process to the external device 300.

At this time, the grade of skin provided to the user terminal 200 or the external device 300 may have a meaning as the overall condition of the skin, and the attribute value of the skin indicates which part of the skin needs to be improved in a specific treatment. It can have meaning as individual information to inform.

In addition, the controller 112 according to an embodiment of the present invention may determine a recommended management product based on the above-described attribute value and skin grade for each attribute of the skin, and further provide the recommended management product to the user terminal 200. In this case, the recommended management product may include at least one of a treatment service for skin management, a product for skin management, and educational content for skin management.

Accordingly, the present invention can provide an integrated solution based on each condition in addition to analyzing the condition of the skin.

Meanwhile, the controller 112 receives the user's selection information for the recommended management product provided to the user terminal 200 by the above-described process, and based on the user's selection information, the attribute value and skin The grade can be transmitted to a third server (not shown). In this case, the third server (not shown) may be a server of the operating entity of the recommended management product.

For example, when receiving selection information on'treatment service for skin management' from the user terminal 200, the control unit 112 is a server of a medical institution that provides treatment services, and the attribute value and skin grade for each attribute of the skin Can be transmitted. At this time, the control unit 112 may also transmit the skin image. The transmitted information can be used in a series of procedures for treatment of the skin (eg, the assignment of a specialist).

In addition, when receiving selection information on'product or content for skin management' from the user terminal 200, the control unit 112 is a server that provides the product or content, and the attribute value and skin grade for each attribute of the skin Can also be transmitted.

Accordingly, the present invention not only enables accurate skin diagnosis from an image of the skin, but also provides an appropriate treatment method according to the skin condition in connection with a subsequent procedure.

10 is a flowchart illustrating a method of evaluating a skin condition performed by the apparatus 110 for evaluating a skin condition according to an embodiment of the present invention. Hereinafter, descriptions of contents overlapping with those described with reference to FIGS. 1 to 9 will be omitted.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may generate a skin recognizer. (S101) In the present invention,'generating' the skin recognizer may mean not only configuring the skin recognizer for the first time, but also updating the configured skin recognizer. The skin recognizer according to an embodiment of the present invention may be a data set expressing a correlation between a first feature vector corresponding to one or more attribute values of each of a plurality of skins and a second feature vector corresponding to each grade of a plurality of skins. have.

Such a skin recognizer includes a correlation between a first feature vector corresponding to one or more attribute values of each of a plurality of skins generated by a machine learning technique and a second feature vector corresponding to each grade of the plurality of skins, That is, it may be mapping information of both.

Meanwhile, the skin recognizer may be a data set as described above. In other words, the skin recognizer may be a data set composed of a plurality of numbers, such as a matrix.

Referring back to FIG. 8, the apparatus 110 for evaluating a skin condition according to an embodiment of the present invention may acquire one or more training skin information 810 to generate the skin recognizer 821. The training skin information 810 includes a skin image, and an attribute value for at least one attribute of the skin and a skin level may be labeled. Accordingly, the training skin information 810 is information on skin that already knows its attribute value and grade, and may be information used to generate and/or update the skin recognizer.

For example, in the case of the first training skin information 811, an attribute value 812 including a degree of redness, a degree of wrinkles, and a flatness of the skin and a grade 813 of the skin may be labeled. At this time, each of the attribute values 812 such as the degree of redness, the degree of wrinkles, and the flatness of the skin and the grade 813 of the skin may be labeled in the form of a vector. In other words, in the first training skin information 811, a first feature vector corresponding to an attribute value for at least one attribute of the skin and a second feature vector corresponding to a grade of the skin may be labeled.

In an optional embodiment, each of the attribute value 812 and the grade 813 may be marked on the training skin information 811 in the form of a number. For example, the attribute value 812 may be labeled in the form of a numerical value indicating the strength of each attribute, such as 0.1 and 0.8. In addition, the grade of the skin may be labeled with any one of the above four grades, such as'first grade' or'third grade'. In this way, when each attribute value is labeled as a numerical value, the skin condition evaluation apparatus 110 according to an embodiment of the present invention converts the numerical values into a vector corresponding to each numerical value in the process of generating or updating the recognizer 821. Can be converted to form.

Meanwhile, the attribute value 812 is an attribute value for skin age (such as 30 years old), gender (such as male), race (such as Asian), disease status (such as chronic dermatitis), in addition to attribute values for the above three attributes. It may further include. However, this is merely an example and the spirit of the present invention is not limited thereto.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may obtain such training skin information 810 from the external device 300 or from the memory 113 of the skin condition evaluation apparatus 110 You may. For example, the skin condition evaluation apparatus 110 may acquire training skin information 810 from a treatment history stored in a server of a specific dermatology department. At this time, the treatment history stored in the dermatology server may be created by a specialist (eg, a dermatologist). In other words, the training skin image 310 may be based on data evaluated by an expert. In addition, the skin condition evaluation apparatus 110 may obtain training skin information 810 from a history of determining attribute values or grades of other skin images in the past by itself.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may update the skin recognizer 821 based on attribute values and grades included in the training skin information 810. In this case, each of the attribute value and the grade may be in the form of a vector as described above. Accordingly, the skin condition evaluation apparatus 110 according to an embodiment of the present invention uses a skin recognizer based on a first feature vector (corresponding to an attribute value) and a second feature vector (corresponding to a grade) included in the training skin information 810. (821) can be updated. At this time, the skin recognizer 821 includes a first feature vector corresponding to one or more attribute values of each of the plurality of skins generated by machine learning as described above, and a second feature vector corresponding to each grade of the plurality of skins. It may be correlation between feature vectors, that is, mapping information of both.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention updates the data set of the skin recognizer 821 so that the first feature vector and the second feature vector can be mapped to each other. The recognizer 821 can be updated. For example, when the data set of the skin recognizer 821 is composed of a plurality of coefficients as shown in FIG. 8, the skin condition evaluation apparatus 110 is based on the first feature vector and the second feature vector. The coefficients that make up (821) can be adjusted.

The skin condition evaluation apparatus 110 is a skin recognizer that repeatedly acquires training skin information labeled with a first feature vector and a second feature vector, and repeatedly updates coefficients based on this, thereby providing a skin grade with improved accuracy. (821) can be obtained.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may obtain an image of the skin from the user terminal 200 (S102). At this time, the skin condition evaluation apparatus 110 may obtain an image of the skin from the user terminal 200. Multiple images of the skin can be acquired. The plurality of images may be images obtained by different photographing conditions such as a photographing angle for the same subject (eg, a face) and a wavelength of light irradiated during photographing. Such photographing conditions may be tagged in the form of metadata on the skin image, or may be acquired from the image itself through image analysis by the skin condition evaluation apparatus 110.

Subsequently, the skin condition evaluation apparatus 110 according to an embodiment of the present invention may determine at least one evaluation target image that satisfies a predetermined shooting condition among a plurality of images received from the user terminal 200 (S103).

In the present invention, the'evaluation target image' is an image used in skin evaluation, and may be an image selected according to different criteria (ie, the above-described predetermined photographing conditions) for each part to be evaluated for a condition.

Referring back to FIG. 3, when it is desired to determine the skin of the face of the person 500, the predetermined shooting condition is whether the image includes a front image of the face (for example, an image obtained from the 623 direction), and the image is Whether to include an image of the left side of the face taken from a first angle (for example, an image obtained from the 624 or 625 direction), the image of the right side of the face taken from a second predetermined angle (e.g., a 621 or 622 direction) Image), whether the image includes a top image of a face taken at a predetermined third angle (for example, an image obtained from the 613 direction), and whether the image includes a bottom image of a face taken at a predetermined fourth angle ( For example, it may be at least one of whether to include an image acquired in the direction 633).

Of course, in addition to the above-described shooting conditions, the shooting conditions include whether the image includes an image of the upper and left side of the face taken at a predetermined fifth angle (for example, an image obtained from the 614 or 615 direction), and whether the image is taken at a predetermined sixth angle. Whether to include an image of the upper right side of the face (e.g., an image obtained from the 611 or 612 direction), whether the image contains an image of the lower left side of the face taken from a predetermined seventh angle (e.g., an image obtained from the 634 or 635 direction) It may further include at least one of whether or not the image includes a lower right side image of the face photographed at a predetermined eighth angle (eg, an image acquired in the 631 or 632 direction). However, this is exemplary, and the spirit of the present invention is not limited thereto, and the shooting conditions may be set more variously or simpler than those described above.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention extracts an image that satisfies the above-described shooting condition among a plurality of images received from the user terminal 200 as an evaluation target image, Can be used to determine the value.

In another embodiment of the present invention, the skin condition evaluation apparatus 110 according to an embodiment of the present invention provides the above-described shooting conditions to the user terminal 200, and the user terminal 200 meets the above-described conditions. Only the image can be acquired and transmitted to the server 100. In other words, the user terminal 200 may acquire an image according to the provided shooting condition and transmit it to the server 100.

In this case, the skin condition evaluation apparatus 110 may immediately determine the image received from the user terminal 200 as the evaluation target image without determining an evaluation target image that satisfies a predetermined photographing condition among a plurality of images. In this case, the skin condition evaluation apparatus 110 may provide the user terminal 200 with an interface that enables photographing according to photographing conditions. For example, the skin condition evaluation apparatus 110 may provide the user terminal 200 with a photographing interface that displays a sample image according to the photographing condition, information on the photographing condition, and whether the photographing condition of the currently photographed image is satisfied.

In another embodiment of the present invention, the skin condition evaluation apparatus 110 may determine the above-described evaluation target image from the video received from the user terminal 200. That is, the skin condition evaluation apparatus 110 may determine at least one frame that satisfies a predetermined shooting condition among a plurality of frames included in the video as the evaluation target image. Of course, this is based on the premise that the user terminal 200 transmits to the server 100 a video photographed from various angles of the object for which the skin condition is to be determined. In this case, the skin condition evaluation apparatus 110 provides a photographing condition and a photographing interface to the user terminal 200 in advance, or the skin condition evaluation apparatus 110 selects an evaluation target image from a plurality of images received from the user terminal 200. The decision-making process can be omitted.

However, in the following description, for convenience of explanation, it is assumed that the skin condition evaluation apparatus 110 determines an evaluation target image from a plurality of images received from the user terminal 200.

Referring back to FIG. 4, the skin condition evaluation apparatus 110 may determine 15 images as shown from images (eg, 100 images) received from the user terminal 200 as an evaluation target image. The determined evaluation target image may be used to determine one or more attribute values for the skin, or to determine a grade of the skin.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may determine attribute values from any one evaluation target image among the evaluation target images (S104).

First, the skin condition evaluation apparatus 110 according to an embodiment of the present invention sets a reference area for evaluating the color of the skin in any one of the evaluation target images, and at least based on the color of the reference area. A first attribute value corresponding to the color of the skin may be determined from one evaluation target image.

As described above, in the present invention, the'attribute value' of the skin may refer to various factors that are the basis for evaluating the skin, and the first attribute value may refer to an evaluation value for the color of the skin.

In this case, the evaluation of the color of the skin may mean various evaluations determined by color such as homogeneity of the skin color, the size of the redness of the skin, and the degree of redness of the skin. Meanwhile, the reference region is a region used as a reference for determining the color of the skin, and the color of the reference region may mean a reference color used for determining the first attribute value.

An example in which the skin condition evaluation apparatus 110 sets a reference region from various evaluation target images 523 and 525 will be described again with reference to FIGS. 5A to 5C.

For example, the skin condition evaluation apparatus 110 may set an area 523-1 corresponding to the eyebrow in the front image 523 of the face as a reference area as shown in FIG. 5A. In addition, as shown in FIG. 5B, the skin condition evaluation apparatus 110 may set an area 525-1 corresponding to the periphery of the ear in the left side image 525 of the face as a reference area. In addition, as shown in FIG. 5C, the skin condition evaluation apparatus 110 sets an area 523-2 corresponding to the rest of the body (eg, neck, etc.) excluding the face from the front image 523 of the face as a reference area. I can.

The setting of the reference area can be set to various areas of the image to be evaluated according to the evaluation items of the color to be expressed as the first attribute value (for example, the degree of redness, the degree of burns, the degree of acne, etc.). Which area should be set can be obtained experimentally.

For example, in the case of indicating the degree of redness of the skin as the first attribute value, the reference region may be determined as the region 523-1 corresponding to the eyebrows, as shown in FIG. 5A. Of course, this can be premised on the fact that the glabellar is the region where redness is the least expressed experimentally.

In addition, when trying to indicate the degree of skin burn (that is, the degree of sunburn) as the first attribute value, the reference region is an area 525-1 corresponding to the ear periphery as shown in FIG. 5B, or shown in FIG. 5C. As described above, it may be determined as a region 523-2 corresponding to the rest of the body (eg, neck, etc.) excluding the face. Of course, it can also be assumed that it has been experimentally confirmed that the affected areas are the areas that are the least sunburned. However, this is merely an example, and the spirit of the present invention is not limited thereto. Hereinafter, as an example, a case where the degree of redness of the skin is to be expressed as a first attribute value, the skin condition evaluation apparatus 110 Let's take a closer look at the steps to make.

Referring back to FIG. 6, the skin condition evaluation apparatus 110 according to an embodiment of the present invention includes a front image of a face among one or more evaluation target images (eg, 511 to 515, 521 to 525, and 531 to 535 of FIG. 4). The face area 523-4 may be recognized from the evaluation target image 523 including. In addition, the skin condition evaluation device 110 selects from the recognized face area 523-4 to include the eye areas 523-5 and 523-6, the nose area 523-7, and the mouth area 523-8. Excluded areas can be recognized. That is, since the eyes, nose, and mouth in the image 523 are not subject to skin condition evaluation, a more meaningful evaluation result can be derived by excluding parts of the image, such as eyes, nose, and mouth having different colors in advance.

The skin condition evaluation apparatus 110 is a reference color in the remaining areas except for the selection exclusion areas 523-5, 523-6, 523-7, and 523-8 recognized by the above-described process in the face area 523-4. At least one point 523-2 and 523-3 in which the difference between the color and the color is equal to or greater than a predetermined threshold difference may be selected. In this case, the reference color may be the color of the reference area 523-1 set by the above-described process.

Subsequently, the skin condition evaluation apparatus 110 determines the number of the selected at least one point (523-2, 523-3) and the degree of color difference between the selected at least one point (523-2, 523-3) from the reference color. And the first attribute value may be determined based on at least one of the degree of dispersion between the selected at least one point 523-2 and 523-3.

At this time, the determined first attribute value may include information on each of the above-described evaluation elements as one or more components for each. For example, the first attribute value is a component for the number of selected points 523-2 and 523-3, a component representing the degree of color difference from the reference color of the selected points 523-2 and 523-3, and selection Components indicating the degree of dispersion of the points 523-2 and 523-3 may be included, such as [2, 0.4, 0.7].

The skin condition evaluation apparatus 110 according to an embodiment of the present invention compares the images taken of the skin from different angles among the images to be evaluated (e.g., 511 to 515, 521 to 525 and 531 to 535 in FIG. 4). 522 and 523), at least one of the second attribute value and the third attribute value may be determined.

Since an image represents an object in a three-dimensional space (for example, a person's face) in a two-dimensional space, it may be difficult to grasp the characteristics of the object in a three-dimensional space with only one image. For example, it may be difficult to accurately grasp the degree of wrinkles or curvature of the face from the front image of the face (523 in FIG. 4). Accordingly, the skin condition evaluation apparatus 110 may more accurately grasp the characteristics of the object in a three-dimensional space by comparing images taken from various angles with respect to the same region.

Hereinafter, with reference to FIGS. 7A and 7B, on the assumption that there is a protruding point on the face, the content of the skin condition evaluation apparatus 110 determining the third attribute value will be described.

For convenience of explanation, it is assumed that there is a point 523-8 on a person's face, and the point 523-8 has a predetermined height as shown to affect the third attribute value. Meanwhile, as described above, the third attribute value may be a value indicating a degree of unevenness of the skin due to other factors other than the curvature caused by the wrinkles of the skin.

Under the above-described assumption, the skin condition evaluation apparatus 110 according to an embodiment of the present invention determines a first curved area (a point 523-8) from the first evaluation target image 523, and the first curved area The first degree of curvature, which is the degree of curvature of the (area of point 523-8), may be determined.

The skin condition evaluation apparatus 110 may determine a first curved region (a point 523-8) based on a color in the first evaluation target image 523. In other words, the skin condition evaluation apparatus 110 may determine a point in the first evaluation target image 523 where the difference between the reference color and the color is greater than or equal to the threshold difference as the first curved area (the point 523-8). On the other hand, the first degree of curvature determined based on the first evaluation target image 523 may be determined to correspond to no curvature according to the type of the first curvature area, or may be determined to reflect the degree of curvature. For example, as shown in FIG. 7A, when the area of the point 523-8 is determined as the first curved area, the degree of curvature of the first curved area cannot be grasped from the first evaluation target image 523 (because the point 523 Since the front image for -8)), the first degree of curvature may be determined to correspond to that without curvature. However, in contrast, when the degree of curvature can be determined from the first evaluation target image 523 (for example, when determining the second attribute value, when the first evaluation target image 523 contains wrinkles), the first curvature degree May be determined to reflect the degree of curvature in question.

Subsequently, the skin condition evaluation device 110 is a second evaluation target image 522 different from the first evaluation target image 523, which is the degree of curvature of the area corresponding to the first curved area (point 523-8). 2 Can determine the degree of curvature. In other words, based on the image of the same object captured from different angles, it is possible to accurately grasp the degree of curvature without 3D information on the corresponding area.

The skin condition evaluation apparatus 110 may determine a third attribute value based on at least one of the first degree of curvature and the second degree of curvature determined by the above-described process. For example, the skin condition evaluation apparatus 110 may determine a third attribute value based on the average of the first and second degrees of curvature, and the third property value may be determined based on the larger of the first and second degrees of curvature. You can also determine the attribute value.

Meanwhile, since the process described in FIGS. 7A and 7B may be similarly repeated in determining the second attribute value, a detailed description of the process of determining the second attribute value will be omitted.

Accordingly, the present invention can evaluate the color of the skin and evaluate the characteristics in a three-dimensional space based on the analysis of the image of the skin.

Referring back to FIG. 9, the apparatus 110 for evaluating a skin condition according to an embodiment of the present invention includes a first feature vector based on at least one of a first attribute value, a second attribute value, and a third attribute value 921. 931 may be generated. (S105) For example, the skin condition evaluation apparatus 110 includes a first feature vector 931 having each attribute value (a first attribute value, a second attribute value, and a third attribute value) as a component. ) Can be created. In this case, when each attribute value has a plurality of components, the skin condition evaluation apparatus 110 may allow at least some of the plurality of components of each attribute value to be included in the first feature vector 931. However, this is merely an example and the spirit of the present invention is not limited thereto.

In an optional embodiment, in generating the first feature vector 931, the skin condition evaluation apparatus 110 further considers at least one physical quantity of the skin or a physical quantity for an environment in which the skin is exposed, and the first feature vector 931 Can be created.

In the case of images, only visual information is involved. Therefore, it may be difficult to obtain information other than visual information about the skin from the image (for example, the elasticity of the skin, the degree of moisture in the skin, the keratin thickness of the skin, the thickness of the makeup applied to the skin, etc.). In generating the first feature vector 931, the skin condition evaluation apparatus 110 according to an embodiment of the present invention may generate the first feature vector 931 by further considering at least one physical quantity of the skin. For example, the skin condition evaluation apparatus 110 according to an embodiment of the present invention includes a first feature vector based on at least one of the aforementioned first attribute value, second attribute value, and third attribute value and at least one physical quantity of the skin. (931) can be created. At this time, the at least one physical quantity may be at least one of a skin elasticity, a moisture content of the skin, a keratin thickness of the skin, and a thickness of a makeup applied to the skin.

In addition, the result of the image may be different depending on the acquisition environment. For example, when an image is acquired under lighting conditions of a specific color, the color of the skin on the image may be different from the color of the original skin. In generating the first feature vector 931, the skin condition evaluation apparatus 110 according to an embodiment of the present invention may generate the first feature vector by further considering a physical quantity for an environment in which the skin is exposed.

Meanwhile, one or more physical quantities for the skin or physical quantities for the environment in which the skin is exposed may be obtained and transmitted by the aforementioned skin diagnosis apparatus (not shown) and/or the user terminal 200.

Accordingly, the present invention can more accurately evaluate the condition of the skin in consideration of various environmental factors.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may determine a second feature vector 941 corresponding to the grade of the skin based on the generated first feature vector 931 and the skin recognizer 821. Yes. (S106)

As described above, the skin recognizer 821 includes a first feature vector corresponding to one or more attribute values of each of a plurality of skins generated by a machine learning technique and a second feature corresponding to each grade of the plurality of skins. It may be correlation between vectors, that is, mapping information of both.

Accordingly, the skin condition evaluation apparatus 110 according to an embodiment of the present invention inputs the first feature vector 931 generated by the above-described process to the skin recognizer 821 and outputs the second feature vector 941 as a result. Can be obtained. In this case, the second feature vector 941 may have a meaning as a vector indicating the grade of the skin. Of course, the second feature vector 941 may include a vector component corresponding to the skin level.

Accordingly, in the present invention, the attribute value for each attribute of the skin can be checked, and an objective grade of the skin can be determined based on this.

The skin condition evaluation apparatus 110 according to an embodiment of the present invention may determine a skin level based on the second feature vector 941 (S107).

In addition, the skin condition evaluation apparatus 110 may provide the determined skin level to the user terminal 200 (S108). At this time, the skin condition evaluation apparatus 110 determines the attribute value 921 determined by the above-described process. It may be provided to the user terminal 200 together with the skin level.

In an optional embodiment, the skin condition evaluation apparatus 110 may provide at least some of the skin grade and skin attribute values determined by the above-described process to the external device 300.

At this time, the grade of the skin may have a meaning as the overall condition of the skin, and the attribute value of the skin may have a meaning as individual information indicating which areas need improvement in a specific treatment for the skin.

In addition, the skin condition evaluation apparatus 110 according to an embodiment of the present invention determines a recommended management product based on the above-described attribute value and skin grade for each attribute of the skin, and further provides this to the user terminal 200. I can. In this case, the recommended management product may include at least one of a treatment service for skin management, a product for skin management, and educational content for skin management.

Accordingly, the present invention can provide an integrated solution based on each condition in addition to analyzing the condition of the skin.

Meanwhile, the skin condition evaluation device 110 receives the user's selection information for the recommended management product provided to the user terminal 200 through the above-described process, and attributes for each attribute of the skin based on the user's selection information. The value and skin level may be transmitted to a third server (not shown). In this case, the third server (not shown) may be a server of the operating entity of the recommended management product.

For example, in the case of receiving selection information on the'treatment service for skin management' from the user terminal 200, the skin condition evaluation device 110 is a server of a medical institution providing the treatment service, and attribute values for each attribute of the skin And the skin grade can be transmitted. In this case, the skin condition evaluation device 110 may transmit the skin image together. The transmitted information can be used in a series of procedures for treatment of the skin (eg, the assignment of a specialist).

In addition, in the case of receiving selection information on'product or content for skin management' from the user terminal 200, the skin condition evaluation device 110 is a server that provides the product or content and provides attribute values for each attribute of the skin. And the skin grade may be transmitted.

Accordingly, the present invention not only enables accurate skin diagnosis from an image of the skin, but also provides an appropriate treatment method according to the skin condition in connection with a subsequent procedure.

11 is an example of a screen 1100 displayed on the user terminal 200 according to an embodiment of the present invention.

Referring to FIG. 11, a screen 1100 is an area 1110 in which an image of skin to be evaluated by a user is displayed, an area 1120 in which simple information and grades on the skin are displayed, and attribute values for each attribute of the skin The displayed area 1130 and areas 1140 and 1150 in which recommended management products are displayed may be included.

The grade displayed in the area 1120 in which brief information and grades are displayed may be determined by the controller 112 based on the skin recognizer. Meanwhile, the gender and the estimated age displayed together may be derived from image analysis or may be input from the user terminal 200.

One or more attributes displayed in the area 1130 in which attribute values for each attribute of the skin are displayed may be determined by the controller 112 through image analysis.

The recommended management products and services displayed in the areas 1140 and 1150 in which the recommended management products are displayed may be determined by the controller 112 based on attribute values and skin grades for each attribute of the skin.

Accordingly, the user can easily grasp the condition of his/her own skin through the screen 1100, and can determine and purchase an appropriate product suitable for his/her skin condition.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may store a program executable by a computer. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And a ROM, RAM, flash memory, and the like, and may be configured to store program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of the computer program may include not only machine language codes produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

The specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections that can be replaced or additionally It may be referred to as a connection, or circuit connections. In addition, if there is no specific mention such as "essential", "important", etc., it may not be an essential component for the application of the present invention.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

100: server
110: skin condition evaluation device
111: communication department
112: control unit
113: memory
200: user terminal
300: external device
400: communication network

Claims (9)

In a method for the skin condition evaluation device to evaluate the condition of the skin based on the image analysis result and the learned skin recognizer,
Training the skin recognizer on the basis of one or more training skin information including a first feature vector corresponding to an attribute value of at least one attribute of the skin and labeled with a second feature vector corresponding to the skin level;
An evaluation target image determining step of determining at least one evaluation target image that satisfies a predetermined photographing condition among a plurality of skin images of the skin;
Determine at least one of a first attribute value corresponding to the color of the skin, a second attribute value corresponding to the degree of wrinkles of the skin, and a third attribute value corresponding to the flatness of the skin from the at least one evaluation target image Determining an attribute value;
A first feature vector generation step of generating a first feature vector based on at least one of the first, second, and third attribute values; And
A second feature vector determining step of determining a second feature vector corresponding to the grade of the skin based on the first feature vector and a skin recognizer; and
The skin recognizer
A data set expressing a correlation between a first feature vector corresponding to one or more attribute values of each of a plurality of skins and a second feature vector corresponding to a grade of each of the plurality of skins,
The step of determining the attribute value
In any one evaluation target image among the at least one evaluation target image, a reference area for skin color evaluation is set, and the color of the skin is matched from the at least one evaluation target image based on the color of the reference area. Determining a first attribute value to be used; And
At least one of a second attribute value corresponding to the degree of wrinkles of the skin and a third attribute value corresponding to the flatness of the skin through mutual comparison between images of the skin photographed from different angles among the at least one evaluation target image Determining one; comprising, a method for evaluating the condition of the skin based on the image analysis result and the learned skin recognizer.
delete The method of claim 1
The skin is the skin of the face,
The step of determining the first attribute value
Selecting at least one point in which a difference between a reference color, which is a color of the reference area, and a color of the reference region from among the at least one evaluation target image including a front image of the face is greater than or equal to a predetermined threshold difference; And
The first attribute value is determined based on at least one of the quantity of the selected at least one point, the degree of color difference between the selected at least one point and the reference color, and the degree of dispersion between the selected at least one point A method for evaluating a condition of the skin based on the image analysis result and the learned skin recognizer, including.
The method of claim 1
The step of determining at least one of the second attribute value and the third attribute value
Determining a first curved area in the first evaluation target image, and determining a first degree of curvature, which is a degree of curvature of the first curved area;
Determining a second degree of curvature, which is a degree of curvature of an area corresponding to the first curvature area in a second evaluation target image different from the first evaluation target image; And
Determining at least one of the second attribute value and the third attribute value based on the first degree of curvature and the second degree of curvature; including, based on the image analysis result and the learned skin recognizer, of the skin How to evaluate the condition.
The method of claim 1
The skin is the skin of the face,
The predetermined shooting conditions are
Whether the image includes a front image of the face, whether the image includes a left side image of the face taken at a predetermined first angle, and the image includes a right side image of the face taken at a predetermined second angle An image comprising at least one of whether the image includes a top image of the face taken at a predetermined third angle, and whether the image includes a bottom image of the face taken at a predetermined fourth angle A method of evaluating the condition of the skin based on the results of the analysis and the learned skin recognizer.
The method of claim 1
The method of evaluating the condition of the skin based on the image analysis result and the learned skin recognizer
After the step of determining the second feature vector
Determining a grade of the skin based on the second feature vector; further comprising,
The skin grade is
According to the need for the above skin care
The condition of the skin based on the results of image analysis and learned skin recognizers, including grade 0 requiring no special care, grade 1 requiring general care, grade 2 requiring professional care, and grade 3 urgently requiring treatment. How to rate it.
The method of claim 1
The step of generating the first feature vector
Generating a first feature vector based on at least one of the first attribute value, the second attribute value, and the third attribute value and at least one physical quantity of the skin,
The at least one physical quantity is
A method of evaluating a skin condition based on an image analysis result and a learned skin recognizer, which is at least one of the elasticity of the skin, the moisture content of the skin, the keratin thickness of the skin, and the thickness of the makeup applied to the skin.
delete A computer program stored on a medium to execute the method of any one of claims 1 and 3 to 7 using a computer.

Images