KR20220128015A - System for analyzing skin - Google Patents

Abstract

According to an embodiment of the present invention, provided is a method performed on one or more processors of a skin analysis device. The method may include the steps of: extracting a facial region of interest from an obtained user image; dividing the facial region of interest into one or more patches and determining a skin condition for each of the one or more patches; and predicting a skin risk for the user based on the skin condition.

Description

Skin analysis system {SYSTEM FOR ANALYZING SKIN}

The present invention relates to a skin analysis system, and to a skin analysis system for analyzing a user's skin condition and providing information on skin risks expected for the user's skin condition.

As modern people's interest in appearance increases, a lot of money and time are invested in skin care. In particular, people visit hospitals to check skin conditions such as aging, wrinkles, blemishes, and whitening, which increases the time and economic burden of consumers.

To solve this problem, various systems for analyzing skin conditions have emerged. However, in the case of the conventional skin analysis system, it is difficult for the consumer to specify the area to be analyzed, and the data on the skin risk prediction model is insufficient, so that high performance cannot be achieved.

Therefore, a skin analysis system capable of solving such a problem is required.

An object of the present invention is to solve the above problems, and an object of the present invention is to provide a skin analysis system for analyzing a user's skin condition and providing information on a skin risk expected for the user's skin condition.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, there is provided a method performed on one or more processors of a skin analysis device. The method includes: extracting a facial ROI from the acquired user image; segmenting the facial region of interest into one or more patches, and determining a skin condition for each of the one or more patches; and predicting a skin risk for the user based on the skin condition.

Also, the user image may be an image captured and transmitted from a camera built into the smart mirror.

Also, the determining of the skin condition may be performed by processing the patch with a skin condition determination model including a learned artificial neural network.

In addition, the skin condition determination model consists of a first sub-model trained to extract an edge pattern or a color pattern from the patch and a second sub-model trained to determine a skin condition based on the output of the first sub model. can be

In addition, the step of determining the skin condition includes searching for skin wrinkles existing in the patch through edge detection in each patch, and determining the wrinkle state for each patch based on the ratio of the skin wrinkles to the normal skin of the patch. It may include a step of judging.

In addition, the determining of the skin condition may include determining the blemish state in each patch based at least in part on a color distribution for each pixel included in each patch.

In addition, the determining of the skin condition may include determining the whitening state of each patch based on color information of each patch.

In addition, the predicting of the skin risk to the user based on the skin condition may include predicting a skin risk learned to predict at least one of a type of a skin risk factor, an expected occurrence time, or an expected occurrence level based on the skin condition. It may include a step of predicting the skin risk for the user using the model.

In addition, the step of predicting the skin risk for the user based on the skin condition may include first learning data based on the skin condition of two or more points of view for one user, or the skin condition of two or more different points of view of the user. The method may include predicting a skin risk for the user by using a skin risk prediction model learned based on at least one of the second training data.

The method may further include transmitting at least one of the skin condition and skin risk prediction information to at least one of a user terminal and a smart mirror.

In addition, extracting a user emotion from the obtained user image; and analyzing the user's voice data to generate a dialogue response for the user.

According to an embodiment of the present invention, a computer program stored in a medium for performing a method performed by one or more processors of a skin analysis device is provided.

According to an embodiment of the present invention, a skin analysis device is provided. The apparatus comprises: a memory for storing computer executable instructions; and a processor for executing instructions stored in the memory, wherein the processor extracts a facial region of interest from the obtained user image, divides the facial region of interest into one or more patches, and for each of the one or more patches The skin condition may be determined, and a skin risk to the user may be predicted based on the skin condition.

According to the present invention, it is possible to provide information on skin risks expected for skin conditions by detecting skin wrinkles existing in the patch through edge detection and analyzing skin conditions such as the presence or absence of skin blemishes and whitening conditions. .

Also, according to the present invention, it is possible to extract a face ROI by learning about a general position occupied by the face region in the user image.

In addition, according to the present invention, by dividing the facial ROI into a plurality of patches having a small size and processing it using the skin condition determination model, it is possible to reduce the amount of computation, increase the training data, and reduce the increase of unnecessary pixels.

In addition, according to the present invention, it is possible to create a predictive model that solves a data shortage through the first and second training data and exhibits high performance even with a small number of data.

In order to more fully understand the drawings recited in the Detailed Description of the Invention, a brief description of each drawing is provided.
1 shows a skin analysis system according to an embodiment of the present invention.
2 shows a skin analysis device constituting a skin analysis system according to an embodiment of the present invention.
3 shows a dialogue system according to an embodiment of the present invention.
4 is a diagram for explaining an artificial neural network according to an embodiment of the present invention.

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the contents described in the accompanying drawings. In addition, a method of constructing and using an embodiment of the present invention will be described in detail with reference to the contents described in the accompanying drawings. The same reference numbers or reference numerals in each figure indicate parts or components that perform substantially the same functions. For convenience described below, the up, down, left and right directions are based on the drawings, and the scope of the present invention is not necessarily limited to that direction.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related items or any one of a plurality of related items.

The terms used herein are used to describe the embodiments, and are not intended to limit and/or limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the terms include or have is intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features, number, or step , it should be understood that it does not preclude in advance the possibility of the presence or addition of an operation, component, part, or combination thereof.

Throughout the specification, when it is said that a part is connected to another part, it includes not only a case in which it is directly connected, but also a case in which it is indirectly connected with another element interposed therebetween. Also, when it is said that a part includes a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

In the present invention, a network function, an artificial neural network, and a neural network may be used interchangeably.

1 shows a skin analysis system according to an embodiment of the present invention.

Referring to FIG. 1 , the skin analysis system may include a smart mirror 100 , a user terminal 300 , and a skin analysis device 200 .

The smart mirror 100 reflects the user's appearance and may include any device capable of performing additional functions.

In an embodiment, the smart mirror 100 may include a camera 110 for taking an image. The captured image of the user may be reproduced through the mirror unit 120 .

In an embodiment, the mirror unit 120 of the smart mirror 100 may be configured as a display, and is composed of a transparent display and a physical mirror to physically reflect the user's image and to display additional information on the transparent display. it might be However, the present invention is not limited thereto.

In an embodiment, the smart mirror 100 may include a lighting (not shown) to better display the user's appearance.

In an embodiment, the smart mirror 100 may include a sound output unit and a voice input unit to perform a voice conversation with a user using the mirror.

In an embodiment, the smart mirror 100 may include a collection transmission module for communication, and may communicate with the skin analysis device 200 to transmit/receive data.

In an embodiment, the smart mirror 100 may transmit a user image, a user voice, a user input, etc. to the skin analysis apparatus 200 .

In an embodiment, a plurality of smart mirrors 100 may be configured to obtain image information from different users and transmit the image information to the skin analysis apparatus 200 .

The user terminal 300 may include a general user handheld device, for example, a smart phone or the like. The user terminal 300 directly communicates with the smart mirror 100 and transmits the image obtained from the smart mirror 100 to the skin analysis device 200 , or transmits the image of the user terminal 300 to the skin analysis device 200 . can be sent to Also, the user terminal 300 may receive information such as a skin condition and a skin risk from the skin analysis apparatus 200 and transmit it to the user.

The skin analysis apparatus 200 may obtain the corresponding information from the smart mirror 100 or the user terminal 300, process the information, and provide it to the user. As described above, the skin analysis apparatus 200 may generate user skin condition and user skin risk prediction information from the user image and transmit it to the smart mirror 100 or the user terminal 300 . Also, the skin analysis apparatus 200 may analyze a user's emotion from the user's image and transmit a conversational response based on the user's emotion to the smart mirror 100 or the user terminal 300 .

According to an embodiment, the user terminal 300 may perform the function of the smart mirror 100 . The user terminal 300 may transmit the user image to the skin analysis apparatus 200 . The user terminal 300 may transmit a user input to the skin analysis apparatus 200 .

The skin analysis system shown in FIG. 1 is exemplary, and various configurations may be applied according to an embodiment to which the present invention is applied.

2 shows a skin analysis device constituting a skin analysis system according to an embodiment of the present invention.

The skin analysis apparatus 200 extracts a facial ROI from the acquired user image, divides the facial ROI into one or more patches, determines a skin condition for each of the one or more patches, and provides the user with a skin condition based on the skin condition. skin risk can be predicted.

In an embodiment, the user information stored in the skin analysis apparatus 200 may be information related to a user of a user image. For example, the user information may include user identification information. The face photo stored in the skin analysis apparatus 200 may be an image in which a face part is extracted from a user image.

In an embodiment, the skin analysis apparatus 200 may extract a facial ROI from the acquired user image. The user image may be an image including the user's face. For example, the user image may be an image captured and transmitted by the camera 110 built in the user terminal 300 or the smart mirror 100 .

In an embodiment, the skin analysis apparatus 200 may generate metadata, face data, and skin data from a user image.

In an embodiment, the metadata may include information related to the user of the image. For example, the metadata may include user identification information, user gender information, user age information, user income information, user occupation information, user lifestyle information, user body basic information, etc. is not limited thereto.

In an embodiment, the face data may be data for a partial region (ie, region of interest) extracted from the user image. The skin analysis apparatus 200 may extract a facial ROI from the user image. The facial interest region may include a partial region necessary to determine a skin condition in the user image. The skin analysis apparatus 200 may extract only the face part necessary for the analysis of the skin condition from the user image. For example, the facial ROI may be a portion of the user's face excluding hair, eyebrows, nose, and mouth, but the present invention is not limited thereto. may be a face region of

In an embodiment, the skin analysis apparatus 200 may normalize a face region based on user images for a plurality of users, and may learn about a general position occupied by the face region in the user image. The skin analysis apparatus 200 may extract a facial ROI from the user image based on the general location of the learned facial region. The skin analysis apparatus 200 may exclude a side portion in which it is difficult to determine the skin condition from the portion estimated as the face region. Since the side area of the face is not photographed in front of the camera 110 when the user image is acquired, it may not be easy to determine the skin condition of this portion or the accuracy may be low. Accordingly, the skin analysis apparatus 200 may finally extract the facial ROI from the part estimated to be the face region except for the side part.

In an embodiment, the skin data may include skin condition information and skin condition prediction information obtained by the skin analysis apparatus 200 analyzing a facial ROI. The skin data is obtained from the skin analysis apparatus 200 through user image analysis, and may include information related to the user's current skin condition and information related to the user's future skin risk.

In an embodiment, the skin analysis apparatus 200 may determine the skin state by analyzing the facial ROI. The skin analysis apparatus 200 may divide the facial ROI into one or more patches and determine a skin state for each hatch. The skin analysis apparatus 200 may divide the facial ROI into a plurality of smaller patches. The size of the patch may be a size suitable for input of the skin condition determination model. When the entire face region of interest is input to the skin condition determination model, the size of the skin condition determination model increases, which may cause difficulties in calculation. In addition, since the facial ROI may be different for each person, a rectangular shape must be created to input the entire ROI to the skin condition determination model, but unnecessary pixels may need to be added to the facial ROI for this purpose. Accordingly, the skin analysis apparatus 200 divides the facial region of interest into a plurality of patches composed of small sizes and processes them using the skin condition determination model, thereby reducing the amount of computation and increasing the learning data. , it is possible to reduce the increase of unnecessary pixels for processing using the skin condition determination model.

In embodiments, the size, placement, and total number of patches may be determined based on a predetermined algorithm. The skin analysis apparatus 200 may determine the size of the patch as a size that can be easily processed by the skin condition determination model. The skin analysis apparatus 200 may determine the arrangement of the patches in the facial ROI so that the number of additional pixels to be added in order for the patches to have a rectangular shape is minimal.

In an embodiment, the skin condition determination model may be composed of an artificial neural network capable of processing an image, for example, the skin condition determination model may be composed of a CNN (convolutional neural network), but the present invention is not limited thereto.

In an embodiment, the skin condition determination model may be trained to extract at least one of an edge pattern or a color pattern from an image. The skin condition determination model may be trained to determine the user's skin condition based on at least one of an edge pattern and a color pattern. The skin condition determination model may be composed of a sub-model for finding a feature on an image in an image patch, and a sub-model for finding a meaning related to a skin condition from a feature on the image. The skin condition determination model may be composed of a first sub-model trained to extract at least one of an edge pattern or a color pattern from a patch, and a second sub-model trained to determine a skin condition based on the output of the first sub model. .

In an embodiment, the skin analysis apparatus 200 may extract at least one of an edge pattern and a color pattern from a patch using the skin condition determination model. The skin analysis apparatus 200 may search for skin wrinkles existing in each patch through edge detection. The skin analysis apparatus 200 may determine a wrinkle state for each patch based on a ratio of skin wrinkles to normal skin in each patch. For example, when the pixel of the wrinkle portion in the 30x30 sized patch is 3x30, the skin analysis apparatus 200 may determine the wrinkle degree of the corresponding patch based on the 10% of the wrinkle portion in the corresponding patch. The skin analysis apparatus 200 may determine that the degree of wrinkles of the patch is severe as the proportion of pixels of the wrinkle portion in the patch increases. The skin analysis apparatus 200 may identify wrinkles and determine whether wrinkles continue in adjacent patches. The skin analysis apparatus 200 may determine whether wrinkles existing in the patch lead to the adjacent patch based on the edge detection results of the patch and the adjacent patch. When a wrinkle leads to an adjacent patch, the skin analysis apparatus 200 may connect it to display it to the user.

In an embodiment, the skin analysis apparatus 200 may perform a plurality of edge detection. For example, the skin analysis apparatus 200 may perform primary edge detection and secondary edge detection. The primary edge detection may be used to extract a region of interest from the user image, and the second edge detection may be performed to distinguish at least one patch from the region of interest. That is, by extracting a portion to be effectively analyzed from the user image through the extraction of the ROI through the primary edge detection, the amount of computation for analyzing the user image can be reduced. In addition, for the secondary edge detection, a more effective algorithm for distinguishing patches may be applied. For example, for the secondary edge detection, an algorithm effective for reading a texture may be applied. For example, the first edge detection may be based on the Kenny edge detection algorithm, and the second edge detection may be based on the LBP.

In an embodiment, the skin analysis apparatus 200 may perform a plurality of edge detection. For example, the skin analysis apparatus 200 may perform primary edge detection and secondary edge detection. The primary edge detection may be used to extract a region of interest from a user image and/or to distinguish at least one patch from the region of interest. In addition, the secondary edge detection may be performed for object detection in each patch. That is, by extracting the portion to be effectively analyzed from the user image through the extraction of the region of interest and/or the patch through the primary edge detection, the amount of calculation for analyzing the user image can be reduced. In addition, for the secondary edge detection, a more effective algorithm can be applied to skin analysis such as acne. For example, for the secondary edge detection, an algorithm effective for reading a texture may be applied. For example, the first edge detection may be based on the Kenny edge detection algorithm, and the second edge detection may be based on the LBP.

In an embodiment, the skin analysis apparatus 200 may perform partially different edge detection on each patch. That is, a separate algorithm effective for analyzing blemishes, for example, acne, wrinkles, etc., in each patch may be applied. This may be performed when a more detailed analysis is required based on the results of the primary analysis for each patch. For example, when a large number of acne is found in an arbitrary patch, by performing a secondary analysis after applying an edge detection algorithm effective for acne analysis, it is possible to provide analysis results with higher reliability.

In an embodiment, the skin analysis apparatus 200 may detect an object in each patch and read the presence or absence of skin blemishes. The skin analysis apparatus 200 may read objects located on the skin, such as melasma, freckles, acne, and pores, in each patch. The skin analysis apparatus 200 may determine whether a skin blemish exists in each patch by analyzing whether an object is present on the skin in each patch. The skin analysis apparatus 200 may determine a patch in which blemishes exist among each patch based on this.

In an embodiment, the skin analysis apparatus 200 may analyze a color pattern of each patch. The skin analysis apparatus 200 may determine a blemish state in each patch based on a color distribution for each pixel included in each patch. For example, the skin analysis apparatus 200 may classify each patch as a color difference in HSV three-dimensional space to determine the blemish state of the patch. For example, the skin analysis apparatus 200 may distinguish a normal part and a blemish part existing in the patch by a color difference. In this case, the skin analysis apparatus 200 may determine the degree of the blemish for each patch based on the ratio of the pixel having the blemish color to the pixel having the normal color. That is, the skin analysis apparatus 200 may determine the degree of the blemish based on the color of the blemish portion and the color distribution of the portion excluding the blemish with respect to the patch in which the blemish is determined to exist.

In an embodiment, the skin analysis apparatus 200 may determine the whitening state of the patch based on color information of the patch. The skin analysis apparatus 200 may determine the whitening state based on the overall color pattern of the patch.

In an embodiment, the patch information stored in the skin analysis apparatus 200 may include information related to each patch obtained by dividing the analysis ROI. The acne patch, the wrinkle patch, the blemish patch, etc. stored in the skin analysis apparatus 200 may be information about the patch including acne, wrinkles, and blemishes, respectively. The skin analysis apparatus 200 may separately store a patch including a skin abnormality.

In an embodiment, the skin analysis apparatus 200 may predict a skin risk to the user based on the skin condition. The skin analysis apparatus 200 may predict a skin abnormality that may occur for the user based on the skin condition. The skin analysis apparatus 200 may predict a skin risk to a user using a skin risk prediction model trained to predict at least one of a type of a skin risk factor, an expected occurrence time, or an expected occurrence degree based on the skin condition. .

In an embodiment, the skin risk prediction model is trained based on at least one of first learning data based on skin conditions at two or more time points of one user or second learning data based on skin conditions at two or more time points of different users. A skin risk prediction model may be used to predict skin risk for a user.

In an embodiment, the first learning data may be learning data for a skin condition of the same user at different points in time. The first learning data may be, for example, data on a skin condition at a previous point in time among different points of view for the same user is input data, and data on a skin condition at a later point in time may be training data composed of labels for the input data.

In an embodiment, as for the second learning data, among the data on the skin condition for the first user and the data on the skin condition for the second user, data at a previous time point on the basis of the user's life is input data, and data at a later time point is a label It may be training data composed of That is, it may be difficult to collect sufficient data for learning the skin condition data configured before and after the time point for the same user. Therefore, the second learning data may be data in which the data for learning is increased by configuring the input data and the label when there is a correlation between the user and the related information, even if the second learning data is data for a different user. That is, when the first user and the second user are different users or data that are related, the skin condition data for each user can be configured as a relationship between the input data and the label, and the learning data constructed in this way is 2 may be training data. Here, the correlation between users may be a correlation related to user information and the user's skin condition. For example, the first user and the second user are related by blood, the first user and the second user have similar body information such as gender and age, or the lifestyle information of the first user and the second user is similar. When there is a relationship, the skin analysis apparatus 200 may determine the first user and the second user as the related users. Before and after the time of determining the input data and the label in the second learning data can be determined based on the user's life, the development pattern of skin risk, and the like. That is, if the first user is 35 years old and the second user is 33 years old, even if the skin condition data of the first user is as of January 1, 2020 and the skin condition data of the second user is as of February 1, 2020, The skin condition data of the second user may be input data, and the skin condition data of the first user may be the label. In addition, when the first user is in the initial stage of acne, and the second user has the acne developed and has a larger shape, the skin condition data of the first user may be input data and the skin condition data of the second user may be the label. . Through this, the data shortage can be eliminated, and a predictive model with high performance can be created even with a small number of data.

In an embodiment, the skin risk prediction model may be an artificial neural network algorithm capable of processing data that changes over time. The skin risk prediction model may be, for example, an LSTM model, but the present invention is not limited thereto.

In an embodiment, the skin analysis apparatus 200 may transmit at least one of skin condition and skin risk prediction information to at least one of the user terminal 300 and the smart mirror 100 . The skin analysis device 200 transmits at least one of skin condition or skin risk prediction information to at least one of the user terminal 300 or the smart mirror 100 to share the prediction result to the user, provide information to the user, and provide the user with the prediction result. can urge action.

In an embodiment, the skin analysis apparatus 200 may interact with a user. The skin analysis apparatus 200 may extract a user's emotion from the user's image and generate a dialogue response to the user's voice input or text input. The skin analysis apparatus 200 may communicate with the user by analyzing the user's emotions. The skin analysis apparatus 200 may allow the smart mirror 100 or the user terminal 300 to communicate with the user. The skin analysis device 200 may allow the user of the smart mirror 100 used in daily life to obtain emotional comfort by analyzing the user's emotions and generating a dialogue response to the user, measuring their skin condition, and , the user of the smart mirror 100 that has obtained risk information on the skin condition may propose a method for solving problems related to his or her skin. The skin analysis apparatus 200 may allow the user of the smart mirror 100 to take care of emotions. Since the skin analysis apparatus 200 may analyze the emotion through the user's facial expression, it is possible to determine the development of the dialogue according to the user's emotion, not the dialogue through simple natural language processing.

In an embodiment, the learning data for conversation with the user may be configured based on data on SNS, phone data, voice data collected from a video platform, and the like. The skin analysis apparatus 200 may learn a conversation with the user based on this.

As shown in FIG. 2 , the skin analysis apparatus 200 may include a processor 210 , a memory 230 , and a network unit 250 .

The processor 210 may control the overall operation of the skin analysis apparatus 200 . For example, by executing a program stored in the skin analyzing apparatus 200 , the processor 210 may control overall components in the skin analyzing apparatus 200 . For example, the processor 210 may be configured to process at least one instruction by performing basic arithmetic, logic, and input/output operations for the operation of the skin analysis apparatus 200 . The above-described instructions may be provided to the processor 210 from the memory 230 . That is, the processor 210 may be configured to execute an instruction according to a program code stored in a recording device such as the memory 230 . Alternatively, the instruction may be received by the skin analysis apparatus 200 through the network unit 250 and provided to the processor 210 . In addition, the processor 210 may include one or more cores, a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). ) may include a processor for data analysis, deep learning, etc. The processor 210 may read a computer program stored in the memory 230 and perform data processing for machine learning according to an embodiment. According to an embodiment, the processor 210 may perform an operation for learning the neural network. The processor 210 for learning of the neural network, such as processing input data for learning in deep learning (DL, deep learning), extracting features from the input data, calculating an error, updating the weight of the neural network using backpropagation calculations can be performed. At least one of a CPU, a GPGPU, and a TPU of the processor 210 may process learning of a network function. For example, the CPU and the GPGPU can process learning of a network function and data classification using the network function. Also, in an embodiment, learning of a network function and data classification using the network function may be processed by using the processors 210 of the plurality of skin analysis apparatuses 200 together. In addition, the computer program executed in the skin analysis apparatus 200 according to the embodiment may be a CPU, GPGPU, or TPU executable program.

In an embodiment, the memory 230 may store any type of information generated or determined by the processor 210 and any type of information received by the network unit 250 .

In an embodiment, the memory 230 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory). etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), It may include at least one type of storage medium among a magnetic memory, a magnetic disk, and an optical disk. The skin analysis apparatus 200 may operate in relation to a web storage that performs a storage function of the memory 230 on the Internet. The description of the above-described memory 230 is only an example, and the present embodiment is not limited thereto.

In the embodiment, the network unit 250 according to the public switched telephone network (PSTN, Public Switched Telephone Network), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed) Various wired communication systems such as DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In the embodiment, the network unit 250 is CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-) A variety of wireless communication systems may be used, such as FDMA) and other systems.

In the embodiment, the network unit 250 may be configured regardless of its communication mode, such as wired and wireless, and is composed of various communication networks, such as a short-range network (PAN, Personal Area Network), and a local area network (WAN, Wide Area Network). can be In addition, the network may be a well-known World Wide Web (WWW), and may use a wireless transmission technology used for short-range communication, such as infrared (IrDA) or Bluetooth (Bluetooth).

The configuration of the skin analysis apparatus 200 shown in FIG. 2 is only a simplified example. In an embodiment, the skin analysis apparatus 200 may include other components for performing the computing environment of the skin analysis apparatus 200 , and only some of the disclosed components may constitute the skin analysis apparatus 200 .

3 shows a dialogue system according to an embodiment of the present invention.

The dialog system is implemented in the skin analysis system to communicate with the user, and specifically, it may be implemented in at least one of the smart mirror 100 , the user terminal 300 , and the skin analysis device 200 , but for convenience of description It is described as being implemented in the skin analysis device 200 for this purpose.

The dialogue system may be composed of an emotional dialogue scenario collection tool, an emotional dialogue scenario refining tool, an emotional dialogue scenario intention automatic creation tool, an emotional dialogue intention relationship network, a dialogue management tool, and a Samjeong dialogue scenario management tool.

In addition, the conversation system may collect emotional conversation scenarios based on call center call records, video and voice information collected from the user terminal 300 , text information collected from SNS, and the like.

The skin analysis apparatus 200 may collect emotional dialogue scenarios from the emotional dialogue scenario refinement tool, refine them, and extract intentions. The emotional conversation scenario refinement tool may perform pre-processing on the collected emotional conversation data. Pre-processing may include correction of errors, correction of typos, and transformation of words.

The skin analysis apparatus 200 may manage the emotional conversation scenario based on the collected conversation scenario, the user's reaction to the conversation generated by the computer, and the like.

The skin analysis apparatus 200 may generate a response to the user's conversation by using the emotional conversation intention relationship network.

4 is a diagram for explaining an artificial neural network according to an embodiment of the present invention.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is configured to include at least one or more nodes. Nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

In the neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may be in an input node relationship in a relationship with another node, and vice versa. As described above, an input node-to-output node relationship may be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the data of the output node may be determined based on data input to the input node. Here, a link interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by the user or algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and the links, and the value of a weight assigned to each of the links. For example, when the same number of nodes and links exist and there are two neural networks having different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may consist of a set of one or more nodes. A subset of nodes constituting the neural network may constitute a layer. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance n from the initial input node may constitute n layers. The distance from the initial input node may be defined by the minimum number of links that must be traversed to reach the corresponding node from the initial input node. However, the definition of such a layer is arbitrary for description, and the order of the layer in the neural network may be defined in a different way from the above. For example, a layer of nodes may be defined by a distance from the final output node.

The initial input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. In addition, the hidden node may mean nodes constituting the neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present invention may have the same number of nodes in the input layer as the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present invention may be a neural network in which the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes decreases as the number of nodes progresses from the input layer to the hidden layer. . In addition, in the neural network according to another embodiment of the present invention, the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes may increase as the number of nodes progresses from the input layer to the hidden layer. have. The neural network according to another embodiment of the present invention may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify the latent structures of data. In other words, it can identify the potential structure of photos, texts, videos, voices, and music (e.g., what objects are in the photos, what the text and emotions are, what the texts and emotions are, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, and a Generative Adversarial Network (GAN). The description of the deep neural network described above is only an example, and the present invention is not limited thereto.

In an embodiment of the present invention, the network function may include an autoencoder. The auto-encoder may be a kind of artificial neural network for outputting output data similar to input data. The auto encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input/output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to a dimension after preprocessing the input data. In the auto-encoder structure, the number of nodes of the hidden layer included in the encoder may have a structure that decreases as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so a certain number or more (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the training of a neural network, iteratively input the training data into the neural network, calculate the output of the neural network and the target error for the training data, and calculate the error of the neural network from the output layer of the neural network to the input layer in the direction to reduce the error. It is the process of updating the weight of each node of the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of comparative learning, the correct answer may not be labeled in each learning data. That is, for example, the learning data in the case of teacher learning regarding data classification may be data in which categories are labeled for each of the learning data. Labeled training data is input to the neural network, and an error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of comparison learning about data classification, an error may be calculated by comparing the input training data with the output of the neural network. The calculated error is back propagated in the reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back propagation. A change amount of the connection weight of each node to be updated may be determined according to a learning rate. The computation of the neural network on the input data and the backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, in the early stage of learning of a neural network, a high learning rate can be used to increase the efficiency by allowing the neural network to quickly obtain a certain level of performance, and in the late learning period, a low learning rate can be used to increase accuracy.

In training of a neural network, in general, the training data may be a subset of real data (that is, data to be processed using the trained neural network), and thus the error on the training data is reduced, but the error on the real data is reduced. There may be increasing learning cycles. Overfitting is a phenomenon in which errors on actual data increase by over-learning on training data as described above. For example, a phenomenon in which a neural network that has learned a cat by seeing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. In order to prevent such overfitting, various optimization methods can be used. In order to prevent overfitting, methods such as increasing the training data, regularization, and dropout that deactivate some of the nodes of the network in the process of learning, and the use of a batch normalization layer are applied. can

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. In addition, each embodiment may be operated in combination with each other as needed. For example, the operation of the device may be operated by combining parts of one embodiment and another embodiment with each other. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

A method performed on one or more processors of a skin analysis device, comprising:
extracting a facial ROI from the acquired user image;
segmenting the facial region of interest into one or more patches, and determining a skin condition for each of the one or more patches; and
predicting a skin risk to the user based on the skin condition. The method of claim 1,
The method of claim 1, wherein the user image is an image captured and transmitted from a camera built into the smart mirror. The method of claim 1,
The step of determining the skin condition is performed by processing the patch with a skin condition determination model including a learned artificial neural network. 4. The method of claim 3,
The skin condition determination model is composed of a first sub-model trained to extract an edge pattern or color pattern from the patch and a second sub-model trained to determine a skin condition based on an output of the first sub-model, Way. The method of claim 1,
The step of determining the skin condition includes searching for skin wrinkles existing in the patch through edge detection in each patch, and determining the wrinkle condition for each patch based on the ratio of the skin wrinkles to the normal skin of the patch. A method comprising steps. The method of claim 1,
The determining of the skin condition includes determining the blemish state in each patch based at least in part on a color distribution for each pixel included in each patch. The method of claim 1,
The determining of the skin condition includes determining a whitening state for each patch based on color information of each patch. The method of claim 1,
The step of predicting the skin risk to the user based on the skin condition includes using a skin risk prediction model trained to predict at least one of a type of a skin risk factor, an expected occurrence time, or an expected occurrence level based on the skin condition. predicting skin risk to the user using The method of claim 1,
Predicting the skin risk for the user based on the skin condition may include: first learning data based on skin conditions at two or more time points for one user, or second learning data based on skin conditions at two or more time points of different users. 2 Using a skin risk prediction model trained based on at least one of the training data to predict a skin risk for the user. The method of claim 1,
The method further comprising the step of transmitting at least one of the skin condition or skin risk prediction information to at least one of a user terminal or a smart mirror. The method of claim 1,
extracting user emotion from the obtained user image; and
and analyzing the user voice data to generate a dialog response to the user. A computer program stored on a medium for performing the method according to any one of claims 1 to 11. A skin analysis device comprising:
a memory for storing computer executable instructions; and
a processor for executing instructions stored in the memory;
The processor extracts a facial ROI from the acquired user image, divides the facial ROI into one or more patches, determines a skin condition for each of the one or more patches, and provides For predicting skin risk, the device.

Images