KR102356465B1 - Method and server for face registration and face analysis - Google Patents

Abstract

A user facial skin analysis method according to an aspect of the present application is to generate a first three-dimensional face model and a second three-dimensional face model including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal, respectively step; specifying a second region of interest matching the first region of interest in a second three-dimensional face model based on the three-dimensional coordinate data of the first region of interest designated by a user in the first three-dimensional face model; matching the first region of interest and the second region of interest based on three-dimensional coordinate data and individual feature data of the first region of interest of the first three-dimensional face model and the second region of interest of the second three-dimensional face model; extracting change data from the matched first region of interest and the second region of interest to generate comparative analysis data, generating skin change analysis result data including individual feature data and comparative analysis data, and providing it to a user terminal; include

Description

User facial skin analysis method and server

The present invention relates to a user's facial skin analysis method, and more particularly, a 3D model of the face is estimated by acquiring a before and after face photo of the same person obtained through a camera, and then a selected area is performed after performing area matching between the two photos. It relates to a method of conducting skin analysis and comparison for

Recently, as interest in health care and beauty increases, various systems for diagnosing and managing skin conditions are emerging. In the past, hospitals used the method of monitoring patients on a regular basis and providing direct management and solutions. developed and used.

In the case of skin analysis and solution provision using conventional captured images, when comparing the before and after situations, it tells how much improvement or deterioration has occurred within the entire face or a large pre-specified area rather than showing changes in a specific area. It does not show whether the skin has been damaged or improved. Accordingly, the user can view only the overall change rather than obtain change information on the region of interest. That is, since information on the user's ROI is not included and the changed part is not separately shown, reliability may be lowered.

In order to solve this problem, it is necessary to inform more about the condition before and after skin change. For example, it would be much more helpful for consumers to say that the acne on the right cheek and the area around the mouth is 60% gone, rather than simply improving the skin from 3 to 4 points.

Republic of Korea Patent Publication No. 10-2019-0084771 (Title of the invention: module for skin condition measurement)

The present invention is to solve the above problem, and by performing accurate region matching through a pair of before and after face images, accurate skin condition analysis for a specific face region and comparative analysis of before and after photos in a specific region can be performed. It is a technical task to provide a system that has

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for solving the above-described technical problem, the user facial skin analysis method according to the first aspect of the present disclosure includes a first face image and a second face image received from a user terminal, including three-dimensional coordinate data based on the first face image and the second face image. generating a first three-dimensional face model and a second three-dimensional face model, respectively; specifying a second region of interest matching the first region of interest in a second three-dimensional face model based on the three-dimensional coordinate data of the first region of interest designated by a user in the first three-dimensional face model; matching the first region of interest and the second region of interest based on three-dimensional coordinate data and individual feature data of the first region of interest of the first three-dimensional face model and the second region of interest of the second three-dimensional face model; extracting change data from the matched first region of interest and the second region of interest to generate comparative analysis data, generating skin change analysis result data including individual feature data and comparative analysis data, and providing it to a user terminal; include

In addition, the user facial skin analysis server according to the second aspect of the present disclosure includes a memory in which a user facial skin analysis program is stored; and a processor executing the program stored in the memory. The processor generates a first three-dimensional face model and a second three-dimensional face model each including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal by executing the program, and specifying a second region of interest matching the first region of interest in a second three-dimensional face model based on the three-dimensional coordinate data of the first region of interest designated by the user in the first three-dimensional face model; matching the first region of interest with a second region of interest based on three-dimensional coordinate data and individual feature data of the first region of interest and the second region of interest of the second three-dimensional face model, and matching the first region of interest with the second region of interest; Comparative analysis data is generated by extracting change data from the region of interest, and skin change analysis result data including the individual characteristic data and the comparative analysis data is generated and provided to the user terminal.

According to any one of the above-described problem solving means of the present application, it is possible to easily and quickly compare and analyze the skin condition in any place with a pair of face images.

In addition, the skin comparative analysis method for a region of interest according to the present invention can perform a more effective and meaningful comparative analysis by performing comparative analysis on a specific region desired by the user or set in advance.

1 is a block diagram illustrating a configuration of a user facial skin analysis server according to an embodiment of the present invention.
2 is a block diagram illustrating the configuration of a user's facial skin analysis system according to an embodiment of the present invention.
3 is a flowchart illustrating a process step of a user facial skin analysis method according to an embodiment of the present invention.
4 is a view for explaining a user's facial skin analysis method according to an embodiment of the present invention.
5 is a view for explaining a user's facial skin analysis method according to an embodiment of the present invention.
6 is a view for explaining a user's facial skin analysis method according to an embodiment of the present invention.
7 is a view for explaining matching and matching in a user facial skin analysis method according to an embodiment of the present invention.
8 is a view for explaining an analysis result in a method for analyzing a user's face skin according to an embodiment of the present invention.
9 is a view for explaining a histogram in a method for analyzing user's face skin according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is said to be located “on” another member, this includes not only a case in which a member is in contact with another member but also a case in which another member is present between the two members.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of a user facial skin analysis server according to an embodiment of the present invention.

As shown, the user facial skin analysis server 100 may include a communication module 110 , a memory 120 , a processor 130 , a database 140 , and an input module 150 .

The communication module 110 may transmit/receive data to and from the connected user terminal 200 . The communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 120 stores a user facial skin analysis program. The user face skin analysis program generates a first three-dimensional face model and a second three-dimensional face model including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal, respectively, and the first three-dimensional face A second region of interest that matches the first region of interest in a second three-dimensional face model is specified based on the three-dimensional coordinate data of the first region of interest specified by the user in the model, and the first region of interest of the first three-dimensional face model and Matching the first region of interest with a second region of interest based on three-dimensional coordinate data and individual feature data of a second region of interest of the second three-dimensional face model, and changing the matched first region of interest and second region of interest The data is extracted to generate comparative analysis data, and skin change analysis result data including the individual characteristic data and the comparative analysis data is generated and provided to the user terminal.

The memory 120 stores various types of data generated during the execution of an operating system for driving the user's facial skin analysis server 100 or a user's facial skin analysis program.

In this case, the memory 120 collectively refers to a non-volatile storage device that continuously maintains stored information even when power is not supplied, and a volatile storage device that requires power to maintain the stored information.

In addition, the memory 120 may perform a function of temporarily or permanently storing data processed by the processor 130 . Here, the memory 120 may include magnetic storage media or flash storage media in addition to the volatile storage device that requires power to maintain stored information, but the scope of the present invention is limited thereto. it is not going to be

The processor 130 executes the program stored in the memory 120, but controls the entire process according to the execution of the user's facial skin analysis program. Each operation performed by the processor 130 will be described in more detail later.

The processor 130 may include all kinds of devices capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as code or instructions included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific (ASIC) An integrated circuit) and a processing device such as a field programmable gate array (FPGA) may be included, but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for the user's facial skin analysis system under the control of the processor 130 . The database 140 may be included as a component separate from the memory 120 , or may be built in a partial area of the memory 120 .

2 is a block diagram illustrating the configuration of a user's facial skin analysis system according to an embodiment of the present invention.

The user facial skin analysis method may be implemented in the server 100 or the user terminal 200 . In the user's facial skin analysis method, data collection such as a face image may be performed in the user terminal 200 . The collected data may be analyzed in the server 100 , and skin analysis result data may be provided to the user terminal 200 . A plurality of users may each perform a user facial skin analysis method through their user terminal 200 .

3 is a flowchart illustrating a process step of a user facial skin analysis method according to an embodiment of the present invention.

The processor 130 generates a first three-dimensional face model and a second three-dimensional face model each including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal (S110).

For example, the first face image before skin change received from the user terminal to analyze the skin change may be the same as the image shown in FIG. 4( a ), and the second face image after the skin change may be the same as the image shown in FIG. At this time, the user terminal 200 shoots the first face image of Fig. 4 (a), which is the comparison target, and then shoots the second face image of Fig. 4 (b) changed through skin care, etc., and sends the image to the server ( 100) can be sent.

The processor 130 may use a facial skin analysis model based on deep learning technology using an artificial neural network to generate a three-dimensional face model based on the received face image. The processor 130 receives the first face image and the second face image, which are two-dimensional images, as input through the facial skin analysis model, generates three-dimensional face models, respectively, while obtaining three-dimensional coordinate data, extracting the vertices of the three-dimensional face model, The depth information of the 3D face model can be estimated. FIG. 5 is a result of estimating depth information of the 3D face model, and FIG. 6 is a view showing 3D coordinate data including vertices extracted from the 3D face model.

Referring back to FIG. 3 , the processor 130 selects a second region of interest matching the first region of interest in the second three-dimensional face model based on the three-dimensional coordinate data of the first region of interest specified by the user in the first three-dimensional face model. It is specified (S120). In this case, vertices extracted from the first three-dimensional face model and the second three-dimensional face model may be used to specify a second region of interest that matches the first region of interest. For example, the same area as the area designated in the first 3D face model may be found based on the eye, which is one of the vertices commonly extracted from the face model.

As the first region of interest, the user may directly designate the region of interest in the face image, and skin features such as acne and dark circles may be detected using object detection or localization technology, which is a kind of artificial intelligence technology. It is also possible to detect or designate the area using simple computer vision technology (red area detection, pore detection) using openCV. That is, the processor 130 learns one or more features of acne, dark circles, red regions, or pore regions through a facial skin analysis model, which is a deep learning model based on a convolutional neural network, based on the three-dimensional face model input in advance to learn the features of the skin feature region. may be extracted, and the skin feature region may be designated as the first region of interest.

Then, a region corresponding to the first region of interest selected from the first three-dimensional face model may be found in the second three-dimensional face model by using the vertex coordinate values. Deep learning-based convolutional neural networks can be used to detect facial vertices and landmarks. Like the 68 landmarks commonly used in computer vision, 40,000 to 60,000 dots can each cover a specific area on the face. For example, if a vertex with ID: 1 corresponds to an eye, it can always be positioned toward the eye during face alignment. In this way, when a user or a professional designates a specific area within a face area, the same area can be found in other photos based on the vertex area with the same ID.

The processor 130 matches the first region of interest with the second region of interest based on the three-dimensional coordinate data and individual feature data of the first region of interest of the first three-dimensional face model and the second region of interest of the second three-dimensional face model ( S130).

The processor 130 may select a vertex for registration between the matched first ROI and the second ROI, and may convert the matched second ROI based on the selected vertex into a matched second ROI.

Referring to FIG. 7 , when the user designates a first region of interest c as in (a), as shown in (b), a second region of interest d corresponding to the first region of interest may be matched. . Also, for comparative analysis, the matching region (d) may be converted into a matching region (e) having the same shape corresponding to the region of interest (c). Since the transformed matching region (e) has a one-to-one correspondence with the region of interest (c) in coordinates, comparative analysis is easy.

The processor 130 generates comparative analysis data by extracting change data from the matched first region of interest and the second region of interest, and generates skin change analysis result data including individual feature data and comparative analysis data to the user terminal. provided (S140).

In the present invention, skin analysis can be performed separately for a designated or a region of interest rather than an analysis of the entire facial skin region, and the corresponding region in the photo taken before the change is accurately detected in the photo after the change to perform a precise comparative analysis and analyzed data can be visualized. The skin change analysis result data may include visualized data, and as a specific example, may include the contents shown in FIGS. 8 and 9 .

8 is a view for explaining an analysis result in a method for analyzing a user's face skin according to an embodiment of the present invention.

(a) is a first face image, (b) is a second face image, the first region of interest is (c1), and the second region of interest is (c2). (d1) and (e1) are images transformed from (c1) for analysis, and (d2) and (e2) are images transformed from (c2) for analysis. Since (c1) and (c2) extracted from the original image may not have uniform contrast, they are converted into the forms (d1) and (d2) through histogram equalization, which is an operation to redistribute the distribution of contrast values, and then again It may be normalized in the form of (e1) and (e2). A histogram graph as shown in FIG. 9 may be generated through smoothing and normalization. (f) is an image showing the difference between (e1) and (e2).

The processor 130 may include the histogram graphs generated through the previous process in the first region of interest c1 and the second region of interest c2 in the individual feature data, respectively. Additionally, the processor 130 performs a histogram comparison technique such as correlation, intersection, or chi-square to obtain a difference in distribution between the histogram graph of the region set as normal skin and the histogram graph of the region of interest. It can be used, and as a result of the comparison, the lower the similarity, the worse the skin condition is determined, and the corresponding data can be included in each individual characteristic data. The processor 130 may objectively determine the skin condition of the region of interest by comparing the histogram between the region set as normal skin and the region of interest, and determine the degree of change in the skin condition through the histogram comparison between the first region of interest and the second region of interest. It is also possible to

9 is a view for explaining a histogram in a method for analyzing user's face skin according to an embodiment of the present invention.

A histogram is a graph showing the distribution of image brightness values, and is an important tool for image analysis. By analyzing the histogram of the input image, information on the brightness composition of the image and the contrast of light and dark can be known, and this can be used for skin analysis.

In the case of the visualization process, a score for each skin feature or histogram analysis can be used. The histogram is a two-dimensional graph showing the distribution of pixel values as shown in FIG. 9 . (a) is a skin image of the first region of interest before the change, (b) is a skin image of the second region of interest after the change, and (c) is an image of normal skin without skin troubles. The distribution of pixel values of (a), (b), and (c) is displayed together on the graph of (d). Through this, the histogram of normal skin, the histogram of skin troubles, and the histogram of the skin after skin care can be compared. By calculating how far it deviates from the histogram distribution of normal skin, the difference value can be defined as an abnormal degree. The higher the difference value, the worse the skin condition.

The user can easily access the analyzed results without having to directly listen to the opinion of a specialist or memorizing past skin information one by one. Therefore, rather than simply measuring the skin condition in the entire area, using the present invention, it effectively records what area, how, and when such analysis was conducted and utilizes it sufficiently to provide more precise and reliable results. You can proceed with the skin condition analysis.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module to be executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

100: server
110: communication module
120: memory
130: processor
140: database
150: input module
200: user terminal

Claims (13)

In the user facial skin analysis method performed by the server,
(a) generating a first three-dimensional face model and a second three-dimensional face model each including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal;
(b) specifying a second region of interest matching the first region of interest in a second three-dimensional face model based on three-dimensional coordinate data of a first region of interest designated by a user in the first three-dimensional face model;
(c) matching the first region of interest to a second region of interest based on three-dimensional coordinate data and individual feature data of the first region of interest of the first three-dimensional face model and the second region of interest of the second three-dimensional face model; step; and
(d) extracting change data from the matched first region of interest and second region of interest to generate comparative analysis data, and generating skin change analysis result data including the individual characteristic data and the comparative analysis data to create the user including; providing to the terminal;
The step (a) includes generating a three-dimensional face model using a facial skin analysis model using an artificial neural network as an input of a plurality of received facial images,
The three-dimensional coordinate data includes three-dimensional coordinates and depth information of the vertices of the three-dimensional face model generated through the facial skin analysis model,
In the step (b), in order to designate the first region of interest in the first three-dimensional face model, one or more features of acne, dark circles, pigmentation, redness, and pore regions are learned through the facial skin analysis model to obtain skin features. to extract the area,
In step (c), a vertex for registration between the matched first ROI and the second ROI is selected, and the matched second ROI is aligned based on the selected vertex to form a matched second ROI. It includes the step of transforming,
The individual feature data includes histogram analysis data, which is a two-dimensional graph representing a distribution of pixel values of the first region of interest and the second region of interest;
The step (d) includes comparing and analyzing the first region of interest and the second region of interest based on a difference from a preset normal skin histogram,
At least one histogram of correlation, intersection, and chi-square to obtain a distribution difference between the histogram graph of the region set as the normal skin and the histogram graph of the first and second regions of interest Using a comparison technique, it is determined that the skin condition is worse as the similarity is lower as a result of comparison, and the histogram analysis data is included in the individual feature data of the first and second regions of interest.
User facial skin analysis method. delete delete The method of claim 1,
The step (b) includes designating the skin feature region as a first region of interest through the facial skin analysis model, which is a deep learning model based on a convolutional neural network, based on a three-dimensional face model input in advance.
User facial skin analysis method. delete delete delete In the user facial skin analysis server,
a memory in which a user facial skin analysis program is stored;
A processor for executing the program stored in the memory;
The processor generates a first three-dimensional face model and a second three-dimensional face model each including three-dimensional coordinate data based on the first face image and the second face image received from the user terminal by executing the program,
specifying a second region of interest matching the first region of interest in a second three-dimensional face model based on the three-dimensional coordinate data of the first region of interest designated by the user in the first three-dimensional face model;
matching the first region of interest with a second region of interest based on three-dimensional coordinate data and individual feature data of the first region of interest of the first three-dimensional face model and the second region of interest of the second three-dimensional face model;
Comparison analysis data is generated by extracting change data from the matched first region of interest and second region of interest, and skin change analysis result data including the individual characteristic data and the comparative analysis data is generated and provided to the user terminal but,
When the processor generates the first three-dimensional face model and the second three-dimensional face model, the processor generates a three-dimensional face model using a facial skin analysis model using an artificial neural network as an input to the received plurality of face images,
The three-dimensional coordinate data includes three-dimensional coordinates and depth information of the vertices of the three-dimensional face model generated through the facial skin analysis model,
The processor extracts a skin feature region by learning one or more features of acne, dark circles, pigmentation, redness, and pore regions through the facial skin analysis model to designate the first region of interest in the first three-dimensional face model. to do,
After matching the first region of interest with the second region of interest, the processor selects a vertex for matching between the matched first region of interest and the second region of interest, and selects a vertex for matching the second region of interest based on the selected vertex aligning the region of interest and transforming it into a matched second region of interest;
The individual feature data includes histogram analysis data, which is a two-dimensional graph representing a distribution of pixel values of the first region of interest and the second region of interest;
The processor compares and analyzes the first region of interest and the second region of interest based on a difference from a preset histogram of normal skin,
At least one histogram of correlation, intersection, and chi-square to obtain a distribution difference between the histogram graph of the region set as the normal skin and the histogram graph of the first and second regions of interest Using a comparison technique, it is determined that the skin condition is worse as the similarity is lower as a result of comparison, and the histogram analysis data is included in the individual feature data of the first and second regions of interest.
User facial skin analysis server. delete delete 9. The method of claim 8,
The processor designates the skin feature region as a first region of interest through the facial skin analysis model, which is a deep learning model based on a convolutional neural network, based on a three-dimensional face model input in advance.
User facial skin analysis server. delete A computer program for performing the user's facial skin analysis method according to claim 1 is recorded on a non-transitory computer-readable recording medium.

Images