JPWO2020113326A5 - - Google Patents

Description

This application claims, with respect to the United States, Paris Convention priority of US Provisional Patent Application No. 62/775,117, filed December 4, 2018, the contents of which are incorporated herein by reference where permitted. incorporated into the book.

TECHNICAL FIELD This specification relates to skin diagnostics, such as dermatology, and skin treatment monitoring, and more particularly to systems and methods for automatic image-based skin diagnostics using deep learning.

Accurate skin analysis is an important area in both the medical and cosmetic fields. An image of the skin may be generated and analyzed to determine one or more skin conditions. It would be desirable to solve the problem of skin analysis using computer technology simply by observing the skin through images (the task of visual skin diagnostics). Successful resolution of this problem would make skin analysis faster and cheaper, since dermatologists would no longer need to see people in person.

An image, such as a face, has one or more skin conditions encoded within the pixels of the image. A computer-implemented method for performing or enabling to perform automatic image-based skin diagnostics using deep learning to decode one or more skin conditions from an image; It would be desirable to provide a computing device and other aspects.

Disclosed and described are deep learning-based systems and methods for skin diagnostics, and test metrics showing that such deep learning-based systems outperform human experts on apparent skin diagnostic tasks. be done. Also disclosed and described are systems and methods for monitoring skin treatment regimens using deep learning based systems and methods for skin diagnostics.

A skin diagnostic device is provided comprising a storage unit and a processing unit coupled to the storage unit, wherein the storage unit stores N skin symptom diagnoses for each of a plurality of N skin symptoms. storing and providing a CNN, a convolutional neural network that classifies pixels of an image for determination, said CNN being a deep layer for image classification configured to generate a diagnosis of each of said N skin signs; a neural network, wherein the CNN is trained using skin manifestation data for each of the N skin manifestations, and the processor receives the images and uses the CNN to process the images; and generate a skin symptom diagnosis for each of the N skin signs.

The CNN generates an encoder stage defined from a trained network for image classification and configured to encode features into a final encoder stage feature net and each of the N skin symptom diagnoses. a decoder stage configured to receive the feature net of said final encoder stage for decoding by means of a plurality of N parallel cutaneous manifestation branches. The decoder stage comprises a global pooling process that processes the feature nets of the final encoder stage to provide each of the N parallel skin manifestation branches. The CNN may be configured to classify the pixels to determine an ethnicity vector, the CNN using each of the N skin signs and skin sign data for a plurality of ethnicities. be learned. The decoder stage may comprise parallel ethnic branches for generating the ethnicity vector.

Each branch of the N parallel skin manifestation branches comprises a first fully connected layer, followed by a first activated layer, a second fully connected layer, a second activated layer, and a final activation layers, and output a final value comprising one of each of said N skin symptom diagnoses and said ethnicity vector. The final activation layer may be defined according to the function of Equation 1 below on the input score x received from the second activation layer, where α is the slope, a is the lower bound, and b is each of the N The upper bound of the score range for each of the skin sign diagnoses.

The CNN may be trained using multiple samples of the form (x _i , y _i ), where x _i is the ith training image and y _i is the ground truth skin sign diagnosis correspondence A vector, the CNN may be trained to minimize a loss function for each branch of the N parallel skin manifestation branches and the parallel ethnic branches. The CNN provides a loss function L that is a weighted combination of a loss function L2 for each of the N parallel skin manifestation branches and a standard cross-entropy classification loss L _ethnicity for the parallel ethnicity branches, It may be learned to minimize according to Equation 3 below, where λ controls the balance between score regression and ethnicity classification loss.

The storage unit may store a face and landmark detector for pre-processing the image, and the processing unit stores normalized from the image using the face and landmark detector. An image may be generated and configured to use the normalized image when using the CNN.

The CNN may be pre-configured from a pre-trained network for image classification adapted to generate each of the N skin symptom diagnoses, the network removing all connected layers of the pre-trained network. and define a group of N layers to decode in parallel the same feature net for each of said N skin symptom diagnoses.

The skin diagnostic device may be configured as one of a personal computing device comprising a mobile terminal and a server providing skin diagnostic services via a communication network.

The storage unit is responsive to at least some of each of the N skin symptom diagnoses when executed by the processing unit to obtain recommendations for at least one of a product and a treatment regimen. A code that provides a product selector may be stored.

The storage unit may store code that, when executed by the processing unit, provides an image acquisition function for receiving the image.

The storage unit may store code that, when executed by the processing unit, provides a therapy monitor for monitoring therapy for at least one skin indication.

The processing unit may be configured to at least one of remind, instruct and/or record treatment activities associated with application of the product for each treatment session.

The processing unit may be configured to process a second image using the CNN to generate a subsequent skin diagnosis received after a treatment session. The storage unit may store code that, when executed by the processing unit, presents comparison results with the subsequent skin diagnosis.

A computer-implemented method of skin diagnostics, the convolutional neural network configured to classify pixels of an image to determine N skin symptom diagnoses for each of a plurality of N skin symptoms. , wherein the CNN is a deep neural network for image classification configured to generate each of the N skin symptom diagnoses, the CNN comprising the N receiving the images by a processing unit coupled to the storage unit trained using skin manifestation data for each of the skin manifestations; and processing the images using the CNN to produce the Generating each of the N skin symptom diagnoses is provided.

A second method is a convolutional neural network configured to classify pixels of an image to determine N individual skin sign diagnoses for each of a plurality of N individual skin signs. training a CNN, said CNN being a deep neural network for image classification configured to generate a diagnosis for each of said N skin signs, said training comprising: A method is provided that is performed using the skin manifestation data of .

These and other aspects can be applied to any of the computer apparatus implementation methods described herein when the (non-transitory) storage stores instructions and the instructions are executed by the processing unit. It will be apparent to those skilled in the art that it includes aspects of a computer program product that configures the processing of a computing device to perform any of the following.

This is a composite of photographs showing skin conditions. 1 is a schematic diagram of a deep learning system according to one embodiment or example herein; FIG. 3 is a more detailed schematic diagram of the deep learning system of FIG. 2; FIG. 1 is a diagram of a computer network providing an environment for various aspects according to one embodiment of the present disclosure; FIG. 5 is a block diagram of a computer device of the computer network of FIG. 4; FIG. 6(a) and 6(b) are flowcharts of the operation of a computing device according to one embodiment of the present disclosure. 7(a) and 7(b) are flowcharts of the operation of a computing device according to one embodiment of the present disclosure.

The concepts of the present invention are best illustrated through the specific embodiments described herein with reference to the accompanying drawings, wherein like numerals refer to like features throughout. It should be understood that the term "invention", as used herein, is intended to connote the inventive concept underlying the embodiments described below and not merely the embodiments themselves. . Furthermore, it should be understood that the general concept of the invention is not limited to the exemplary embodiments described below, and that the following description should be read in that light.

<Preface>
As used herein, the term "skin signs" or "signs" includes statutory lines, wrinkles in various locations, lower facial ptosis, sebaceous pores, hyperpigmentation across the face, and vascular disorders. non-limiting), referring to certain skin conditions. FIG. 1 is a photographic composite 100 showing skin manifestations such as forehead wrinkles 102, glabellar wrinkles 104, under-eye wrinkles 106, normal lines 108, lip corner wrinkles 110, and lower facial drooping 112. . The appearance of the human face undergoes structural changes due to various factors. These factors include chronological aging, photoaging, dietary habits (anorexia or obesity), lifestyle habits (sleep problems, smoking, alcoholism, etc.). These structural changes are most evident with respect to localized wrinkles (forehead, glabella, upper lip, etc.), but general drooping of the face (ptosis, drooping eyes, cervical ptosis, etc.) and cheeks. often accompanied by enlarged pores. These changes can be subtle over years and decades, but they are manifested differently by gender and ethnicity. These changes are also associated with well-known effects on skin pigmentation (moles, dark spots, skin darkening) and skin vascular network (redness, capillary dilatation, etc.), whether or not they are associated with air pollution. ), as well as accentuated by variable (multiple) exposures to sunlight (ultraviolet).

Grading the severity of some facial indications can be dermatological (dermabrasion, corrective surgery, etc.), cosmetic (skin care, anti-aging products), or possible consumer assistance/advice, etc. It is an important need for different purposes. Such a need not only meets the primary scientific objectivity, but can also help detect false product claims. The purpose of this grading is L'Oreal S.M. A multi-volume reference skin pictorial collection (R. Bazin, E. Doublet, in: PE Med'Com (Ed.), Skin Aging Atlas. Volume 1, Caucasian Type, 2007. R. Bazin, F. Flament, in: PE Med'Com (Ed.), Skin Aging Atlas.Volume 2, Asian Type, 2010. R. Bazin, F. Flament, F. Giron, in: PE Med'Com (Ed.), Skin Aging Atlas.Volume 3, Afro-American Type, 2012. R. Bazin, F. Flament, V. Rubert, in: PE Med'Com (Ed.), Skin Aging Atlas.Volume 4, Indian Type, 2015. And, F. Flament, R. Bazin, H. Qiu, in: PE Med'Com (Ed.), Skin Aging Atlas. Volume 5, Photo-aging Face & Body, 2017) . This skin pictorial provides a visual grading of over 20 facial signs (and each scale of severity increasing from 0 to 4, 5, 6 or 7) in both sexes of four ethnic groups with age. Standardized by professionally processed photographs. By zooming in on a particular symptom regardless of the overall facial appearance, a skin professional can blindly grade the severity of each facial symptom. These skin illustrations show that aging affects people differently depending on their gender, but that the effects are similar for people of the same sex. However, some changes in facial features were ethnically specific. Not only can this method accurately describe age-related changes in facial signs in both genders of the four ethnic groups, but in ethnic Caucasian or Chinese women, some facial signs are associated with fatigue from a day's work. resulted in a judgment that it was related to or associated with But one more difficult and important step still remains: it was filmed by a mobile phone under a variety of real-world lighting conditions and during human activity (work, sports, public transportation, etc.). to develop an automated process that does not rely on human evaluation to grade structural facial features either through photographs (e.g., "selfies" or selfie videos) or through standardized photographs. be. In short, obtaining quantified data from a "blind/neutral" automated system is desirable in many applications.

Thus, we describe a deep learning approach to skin diagnosis developed using data from women of various ages and ethnicities, including technical aspects of the approach and the results obtained. We also present comparisons with data obtained by expert grading (using skin pictorials).

The apparent skin diagnosis problem of evaluating skin signs from images alone is posed as a supervised regression problem for computer implementation using deep learning. As represented by the schematic diagram of FIG. 2 showing deep learning system 200, given an image x 202 of a face, neural network 204 of system 200 returns a vector of scores y 206, with y=f _θ (x) , and f _θ is the neural network 204 parameterized by θ. Each component of y206 corresponds to a different skin indication. Factors such as ethnicity and other skin ratings may also be determined as described further below.

Although it is possible to design a separate neural network for each skin sign, due to the similarity of the low-level features learned across signs, the implementation of the above approach is limited to handling all signs by a single network. can be jointly estimated. A side benefit is higher computational efficiency.

Rather than designing neural networks from scratch, we can adapt architectures that have been proven to work well for different tasks. In particular, ResNet50 (a 50-layer residual network by Microsoft Research Asia, K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, in: Proceedings of the IEEE conference conference and pattern recognition, 2016, pp. 770-778, which is incorporated herein in its entirety) and MobileNetV2 (a second version of Google Inc.'s depth-wise separable convolutional neural network; Ｓａｎｄｌｅｒ，Ａ．Ｈｏｗａｒｄ，Ｍ．Ｚｈｕ，Ａ．Ｚｈｍｏｇｉｎｏｖ，Ｌ．－Ｃ．Ｃｈｅｎ，Ｉｎｖｅｒｔｅｄ Ｒｅｓｉｄｕａｌｓ ａｎｄ Ｌｉｎｅａｒ Ｂｏｔｔｌｅｎｅｃｋｓ：Ｍｏｂｉｌｅ Ｎｅｔｗｏｒｋｓ ｆｏｒ Ｃｌａｓｓｉｆｉｃａｔｉｏｎ，Ｄｅｔｅｃｔｉｏｎ ａｎｄ Ｓｅｇｍｅｎｔａｔｉｏｎ，ａｒＸｉｖ ｐｒｅｐｒｉｎｔ ａｒＸｉｖ：１８０１．０４３８１，１３ Ｊａｎ．２０１８に記載and incorporated herein in its entirety) can be applied.

ResNet50 and MobileNetV2 are convolutional neural networks trained on ImageNet (an open-source database that collects image data for classification). ResNet50 is used as the backbone of many state-of-the-art systems, and MobileNetV2 is a more efficient network that can be used with modest precision reductions when execution time and storage space are concerns. When used for classification, each of these networks contains a large fully convolutional part (e.g., the encoder stage) that yields a low-resolution but powerful set of CNN features, followed by a global max A (global max) pooling layer or global average pooling layer, a fully connected layer and a final classification layer follow (decoder stage). Each is a good candidate for adaptation.

FIG. 3 shows a neural network comprising encoder components 302 and decoder components 304 defined by layers (e.g., components with respective processing) from a source network such as ResNet50 or MobileNetV2. 2 is a schematic diagram of deep learning system 200 showing 202 in more detail; FIG. The decoder section 304 includes a global maxpooling layer 306 and respective parallel branches (eg, 308 shown for simplicity) for decoding each of the N skin manifestations in the output vector 206. , 310, and 312, where it is understood that there are N+1 parallel branches for N skin manifestations), and an ethnicity factor (output 314).

Instead of replacing only the final classification layer, each of the source networks is cropped after the pooling layer to build the feature net (neural network 204). Specifically, ResNet 50 is cropped after its average pooling layer, which is replaced by a global maxpooling layer (eg, 306) to produce a 1×1×2048 feature vector. Similarly, for MobileNetV2, the fully connected layer is truncated, the average pooling layer is replaced with a global max pooling layer, and the new feature net outputs 1x1x1280 feature vectors. Each of parallel branches 308 , 310 , 312 receives output from global maxpooling layer 306 .

This initial branch selection is made because different skin manifestations depend on potentially different image features, and this selection is verified through experimentation. Each skin manifestation branch (one of each parallel branch 308, 310) consists of two fully connected layers following the activation layer. First, connect the feature nets (ResNet50 or MobileNet) with a fully connected layer where the input size is the feature size after pooling (for example, 1×1×2048 or 1×1×1280, respectively) and the output size is 50, Then connect the ReLU activation layer (eg, normalized linear activation unit). Then a second fully connected layer with an input size of 50 and an output size of 1 is followed by a customized activation layer that outputs the final score.

The system complies with the internationally accepted skin score pictorial compendium maintained by L'Oreal, as referenced herein above, and as a result, skin manifestations are It has individual scales according to the type of person, the ethnicity of the person, and the gender. This is because each skin manifestation has boundaries. The final layer uses a custom function, namely an activation function like Leaky ReLU (called LeakyClamp), rather than a purely linear regression layer or another activation function. Leaky ReLU is an A.I. L. Maas, A.; Y. Hannun, A.; Y. Ng, Rectifier Nonlinearities Improve Neural Network Acoustic Models, in: Proc. International Conference on Machine Learning, Vol. 30, 2013, p. 3, which is incorporated herein by reference. Leaky ReLU attempts to address the problem of "dying ReLU" when x<0. While the standard ReLU function goes to 0 when x<0, Leaky ReLU has a small negative slope (eg, a slope of about 0.01 close to 0).

LeakyClamp has a slope close to 0 below min-activation and above max-activation, where max-activation is determined by sign as in Equation 1 below. different. α is the slope, a is the lower bound, and b is the upper bound of the score range. For learning, α is chosen to be 0.01 and a, b to be the score range for each symptom.

To train a deep learning network, as further described in the evaluation section of this specification, multiple obtain and use a sample of A loss function is minimized to find the optimal set of parameters θ. Experiments were performed with several loss functions and no one was found to be superior to the other.

Therefore, the standard L2 loss (equation 2) is minimized and used in the data presented herein, where L2 is given in equation 2 below.

Furthermore, due to the skin score ethnicity dependence, a separate ethnicity prediction branch (one of each parallel branch 312) with its own component structure and standard cross-entropy classification loss L _ethnicity Defined. The ethnicity branch (312) has one fully connected layer with the input size as the feature size and the output size as the number of ethnicities. The additional loss, L _identity , helps steer learning in the right direction, but it also helps during testing, allowing correct interpretation of output scores by using a person's ethnic group. The L _thnicity of the L2 loss and the cross-entropy classification loss, together with the weight λ, is the loss L, as shown in Equation 4 below. λ controls the balance between score regression and ethnicity classification loss. λ=0.002 was used for learning.

Following a common transfer learning approach, the network is pre-trained on ImageNet and then fine-tuned on skin diagnostic data using (eg, minimizing) the above losses. Also, similar to ImageNet's pre-training procedure, we normalize the image with a center of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225] applies. The Adam optimizer performs first-order gradient-based optimization of a stochastic objective function, fine-tuning the learning process with a learning rate of 0.0001 and a batch size of 16. The Adam optimizer was developed by D. P. Kingma, J.; Ba, Adam: A method for stochastic optimization, CoRR abs/1412.6980. arXiv: 1412.6980 as early as 22 Dec. 2014, which is incorporated herein by reference.

Apparent skin diagnostics has many scientific, commercial and other uses, including consumer applications. Although it is possible to control the imaging conditions for some such applications by taking images in controlled lighting conditions and utilizing standardized poses, such is particularly It may not be feasible for consumer applications. Therefore, it is desirable that a deep learning system be able to handle various lighting conditions and facial poses. Still referring to FIG. 3, as an example to address the latter, a face landmark detector 316 (V. Kazemi, J. Sullivan, One million face alignment with an ensemble of regression trees, in : IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1867-1874, incorporated herein by reference) to normalize faces (output image 202) based on detected landmarks. The image may be pre-processed in order to Thus, the input face image x is always an upright frontal image of the face at a fixed scale.

During training, even after landmark-based cropping, we augment the training data with various scales of cropping (chosen randomly from 0.8 to 1.0) to accommodate arbitrary scale variations. good. After randomly cropping the images, each input image is resized to a resolution of 448 pixels by 334 pixels (eg, to match the expected input resolution of the source network). In addition, the selected images are randomly flipped horizontally during training with a probability of 0.5. Also, in order to cope with illumination variations, learning is performed using images under various illumination conditions, as described in the evaluation section of this specification.
<Evaluation>

This model has been trained on two datasets of female images according to the following nine skin signatures.
・Wrinkles between the eyebrows ・Wrinkles on the forehead ・Wrinkles under the eyes ・Wrinkles on the corners of the lips ・Ptosis of the lower part of the face

Note that the last two skin manifestations are defined only for Caucasian and Asian ethnicities. The first data set consisted of images of 5834 women captured under controlled laboratory conditions with ideal lighting and facial poses using a professional camera (hereinafter referred to as the "analysis data set"). ing. Note that not all images in this dataset contain ground truth for all nine symptoms. The second dataset consists of selfie images taken with a mobile phone under uncontrolled lighting conditions (hereinafter referred to as the "selfie dataset"). This dataset contains images of 380 women of three ethnicities (Caucasian, Asian, and African), each of whom is exposed to outdoor daylight, indoor daylight, indoor artificial diffuse light, and indoor artificial direct light in four different lighting conditions. As a result, a total of 4560 images were obtained. For both datasets, 90% of the data was used for training and 10% for testing. Similarly, in both cases, the same face normalization framework is applied, although it is not necessary for some images in the analysis dataset. With this configuration, faces and facial landmarks cannot be detected in some images, which slightly reduces the amount of data for training and testing.

Both datasets were manually annotated by expert dermatologists and each image was annotated by 10-12 experts. The ground truth is the average of the expert's predictions.

Training may be done on images of men. Image conditions such as no facial hair may be imposed in order to obtain a clear image. Not only does facial hair have a large impact on the score of symptoms in areas of skin covered with facial hair, but it also impacts overall learning, as features are learned together for all symptoms. Skin manifestations in men and women are the same.

Several measures are used to evaluate a trained deep learning system 200 consisting of neural networks 202 . For ethnic precognition, the correctly classified percentage is measured. The test accuracies for the analysis and selfie datasets are 99.7% and 98.2%, respectively. For skin scores, two measurements are used. The first is the Mean Absolute Error (MAE), which is the average of the absolute differences between the predicted and ground truth values over all samples. However, a more meaningful error measure is the percentage of samples whose absolute error is below some threshold (%(M AE<T)). Depending on the application, this threshold may be more or less stringent. Therefore, this error measure is reported for several different thresholds. Below are the results for both the analysis dataset and the selfie dataset.

Table 1 shows the results for the analysis dataset and Table 2 shows the results for the selfie dataset. For example, a typical range of scores is 0-5-10, but it is noted that deep learning system 200 can predict scores within an absolute error of 1 over 90% for any skin manifestation. (more accurate for some indications).

For the selfie dataset (Table 2), the results are mostly even better, despite the uncontrolled lighting conditions. However, it is also observed that there is a very large variability in scores between the experts themselves and even between different lighting conditions of the same expert. Therefore, it is likely that the ground truth is biased and the system 200 is internally learning to predict proof conditions to better predict scores. Collecting more consistent ground truth across different lighting conditions can help.

Currently, however, scoring skin manifestations based on "in-the-wild" images is a difficult task, even for professional dermatologists, in which system 200 can help. It shows that it has a performance exceeding that of This is shown in Table 3, where the absolute error for each image is calculated by comparing each expert's prediction with the average expert's prediction for this image, where each image is scored by 12 experts on average. It was calculated by comparing with By comparing Tables 2 and 3, it can be seen that the system 100 is more accurate than the expert for all indications except pigmentation across the face.

In addition to validation of the model for image-based scores of selfie data, validation was also performed on a subset of subjects for whom dermatologists were able to score signs of their skin condition. Dermatologists were visited by 68 individuals (approximately 12 experts per subject) and evaluated live, regardless of subject's image-based score. Similar to the image-based analysis, for each skin condition indication, 1) for the system 200 model, compare the predictions from the model to the expert's average score for the particular subject's indication; For face-to-face expert assessments, the mean absolute error was calculated by comparing each expert's score vector to the expert's mean score vector for this subject. Two tables for model performance (Table 4) and expert performance (Table 5) are shown below. As with image-based scoring, even with face-to-face scoring by professionals, automatic score prediction from system 200 yields higher accuracy than predictions by professional dermatologists, which , for all indications.

In order to gain a deeper understanding of the results in Tables 4 and 5, validation analyzes similar to those in Tables 2 and 3 were performed, but only a subset of the 68 subjects actually evaluated was used. The results are shown in Tables 6 and 7 below. Again, prediction of model scores from system 200 provides significantly higher accuracy than expert scoring.

FIG. 4 is a block diagram of an exemplary computer network 400 in which a personal-use computing device 402 operated by a user 404 communicates, via a communications network 404, with remotely located server computing devices, namely server 406 and server 406 . Communicating with server 408 . . User 404 may be a dermatologist's patient and/or a consumer. Also shown is a second computing device 412 configured to communicate with a second user 410 via communications network 404 . A second user 410 may be a dermatologist. The computing device 402 is for the personal use of the user and is not available to the public like services from a server. Here, the public includes registered users and/or customers.

Briefly, computing device 402 is configured to perform skin diagnostics as described herein. Neural network 200 may be stored and utilized on board computing device 402 or may be provided from server 406 via cloud services, web services, etc. from image(s) received from computing device 402 . can be

The computing device 402 may, for example, provide skin diagnostic information and receive products/recommended treatments depending on the skin diagnostic and/or other information about the user (eg, age, gender, etc.). It is configured to communicate with server 408 . Computing device 402 is configured to communicate skin diagnostic information (which may include image data) to either or both of servers 406 and 408 for storage, for example, in a data store (not shown). obtain. Server 408 (or another service provider not shown) may provide an e-commerce service for selling the recommended product(s).

Computing device 402 is depicted as a portable mobile terminal (eg, smart phone or tablet). However, it may be another computing device such as a laptop, desktop, workstation, or the like. The skin diagnostics described herein may be implemented in other computing devices. Computing device 402 may be configured using, for example, one or more native or browser-based applications.

Computing device 402 may comprise, for example, a user device that acquires one or more images, such as photographs of skin, particularly a face, and processes the images to provide a skin diagnosis. Skin diagnostics may be performed in conjunction with a skin treatment plan in which images are periodically acquired and analyzed to determine a skin score for one or more skin manifestations. Scores may be stored (locally, remotely, or both) and compared between sessions to show trends, improvements, and the like. The skin score and/or skin image may be accessible to a user 404 of the computing device 402 and available to another user of the computer system 400 (eg, a second user 410), such as a dermatologist. (eg, communicated via server 406 or otherwise (electronically) via communication network 404). The second computing device 412 may also perform skin diagnostics as described above. It may receive images from a remote source (eg, computing device 402, server 406, server 408, etc.) and/or via an optical sensor (eg, camera) coupled thereto, or any other You can capture the image in any way. Neural network 200 may be stored and used from second computing device 412 or from server 406 as described.

Provides an application that performs skin diagnostics, recommends one or more products, and monitors skin changes over time after applying one or more products (which may define treatment sessions in a treatment plan). You can A computer application uses a series of instructional graphical user interfaces (GUIs) and/or other user interfaces (usually interactive and accepting user input) to perform any of the following activities: receive) or other workflows may be provided.
-Recommendation of products such as skin diagnostics -Treatment plans -Purchasing or otherwise obtaining products -Reminding, directing and/or recording (e.g., logging) the use of products in each treatment session
Subsequent (e.g. one or more follow-ups) skin diagnosis Presentation of results (e.g. comparison results) Data can also be generated according to the schedule of the treatment plan, e.g. to monitor the progress of the skin treatment plan . Any of these activities can be performed remotely, e.g., for review by user 410, for review by another individual, aggregated with other users' data to measure treatment plan effectiveness, etc. It can generate data to be stored.

Comparison results (eg, before and after results) may be presented via computing device 402, such as during and/or at the completion of the treatment plan. As noted above, aspects of skin diagnostics may be performed on computing device 400 or by a remotely coupled device (eg, a server in the cloud or another configuration).

FIG. 5 is a block diagram of computing device 402 in accordance with one or more aspects of the present invention. Computing device 402 includes one or more processors 502 , one or more input devices 504 , gesture-based input/output devices 506 , one or more communication units 508 , and one or more output devices 510 . Computing device 402 also includes one or more storage devices 512 that store one or more modules and/or data. The modules include a deep neural network model 514, an application 516 with components for a graphical user interface (GUI 518), and/or a workflow for therapy monitoring (e.g., therapy monitor 520), image acquisition 522 ( interface), and a therapy/product selector 530 (eg, interface). The data may include one or more images for processing (e.g., image 524), skin diagnostic data (e.g., respective scores, ethnicity, or other user data), log data related to specific treatments, and for reminders. It may also include treatment data such as treatment plans with schedules such as.

Application 516 provides functionality for acquiring one or more images, such as videos, and processing the images to determine a deep neural network skin diagnosis provided by neural network model 514 . The network model may be configured as the models shown in FIGS. 2 and 3. FIG. In another embodiment, the network model is remotely located and the computing device 402 can communicate images for processing and return of skin diagnostic data via the application 516 . Application 516 may also be configured to perform the aforementioned activities.

The storage device(s) 512 may include a processing system 532, a communication module, an image processing module (eg, for the GPU of processor 502), a map module, a contact module, a calendar module, a photo/gallery module, a photo (image/media ) may store additional modules such as other modules (not shown), including editors, media players and/or streaming modules, social media applications, browser modules, and the like. A storage device may be referred to herein as a storage unit.

Communication channel 538 may communicatively, physically and/or operationally connect each of components 502, 504, 506, 508, 510, 512 and any modules 514, 516, and 532 for inter-component communication. can be combined. In some examples, communication channel 538 may include a system bus, network connection, interprocess communication data structures, or any other method for communicating data.

One or more processors 502 may implement functionality and/or execute instructions within computing device 402 . For example, processor 502 may be configured to receive instructions and/or data from storage device 512 to perform the functionality of the modules (eg, processing system, application, etc.) shown in FIG. good. Computing device 402 may store data/information in storage device 512 . Some of the functions are further described herein below. It should be appreciated that the processing may not be exactly contained within modules 514, 516, and 532 of FIG. 5, and that one module may support the functionality of another module.

The computer program code for carrying out the processes may be written in one or more programming languages, e.g., object oriented programming languages such as Java, Smalltalk, C++, or conventional object oriented programming, such as the "C" programming language or similar programming languages. It can be written in any combination of languages.

Computing device 402 may generate output for display on the screen of gesture-based input/output device 506, and in some examples output for display by a projector, monitor, or other display device. may be generated. Gesture-based input/output devices 506 may be implemented using a variety of technologies (e.g., resistive touch screens, surface acoustic wave touch screens, capacitive touch screens, projected capacitive touch screens, pressure sensitive screens, acoustic pulse recognition touch screens, etc.). screen, or another touch-sensitive screen technology and output capability: liquid crystal display (LCD), light emitting diode (LED) display, organic light emitting diode (OLED) display, dot matrix display, e-ink, or similar monochrome or color display) It will be appreciated that it can be configured using

In the examples described herein, the gesture-based input/output device 506 includes a touch screen device capable of receiving tactile interactions or gestures as input from a user interacting with the touch screen. Such gestures may include tap gestures, drag or swipe gestures, flick gestures, pause gestures in which the user touches or points to one or more locations on the gesture-based input/output device 506 (e.g., the user touches or points at least a threshold amount of time). (when touching the same location on the screen across the screen). The gesture-based input/output device 506 can also include non-tap gestures. Gesture-based input/output device 506 can output or display information to a user, such as a graphical user interface. The gesture-based input/output device 506 can present various applications, functions, and capabilities of the computing device 402, including, for example, acquiring images, viewing images, processing images, creating new Applications 516 for displaying images, messaging applications, telephony, contacts and calendar applications, web browsing applications, gaming applications, e-book applications and finance, payments, and other applications or functions.

Although the present invention primarily describes gesture-based input/output devices 506 in the form of display screen devices (e.g., touch screens) that have input/output capabilities, other gestures that can detect motion and do not involve the screen itself. Base I/O devices are available. In such cases, computing device 402 includes a display screen or is coupled to a display device for presenting new images and GUIs of application 516 . Computing device 402 can receive gesture-based input from a trackpad/touchpad, one or more cameras, or another presence or gesture-sensing input device, where presence is, for example, movement of all or part of the user. It means the user's presence mode including.

One or more communication units 508 may transmit and/or receive network signals over one or more networks for the purposes described and/or other purposes (e.g., printing) over communication network 404 or the like. For example, it may communicate with an external device (eg, server 406, server 408, second computing device 412). The communication unit may include various antennas and/or network interface cards, chips (eg, global positioning satellites (GPS)), etc. for wireless and/or wired communication.

Input device 504 and output device 510 may be one or more buttons, switches, pointing devices, cameras, keyboards, microphones, one or more sensors (eg, biometrics, etc.), speakers, bells, one or more lights, tactile (vibration) devices, etc. may be included. One or more of them can be coupled via a Universal Serial Bus (USB) or other communication channel (eg, 538). The camera (input device 804) is positioned forward to allow the user to capture image(s) using the camera while looking at the gesture-based input/output device 506 to take a "selfie". (ie, to the same side).

The one or more storage devices 512 may take different forms and/or configurations, eg, as short-term memory or long-term memory. Storage device 512 may be configured for short-term storage of information as volatile memory, which does not retain stored contents when power is removed. Volatile memory includes random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and the like. Storage device 512, in some examples, may be a single memory device, for example, to store a larger amount of information than volatile memory and/or to retain information even when power is removed for long-term storage of that information. or multiple computer readable storage media. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory.

Although not shown, a computing device may be configured as a learning environment for training the neural network model 514 using, for example, a network such as that shown in FIG. 3 with appropriate training and/or test data. .

Deep neural networks are computers that are mobile terminals (e.g., smartphones or tablets) that have fewer processing resources than "larger" devices such as laptops, desktops, workstations, servers, or other comparable generation computing devices. It can be adapted to the light architecture for the device.

In one aspect, the deep neural network model is depthwise segregated, consisting of convolutions where each standard convolution is factored into a depthwise convolution and a pointwise convolution. It can be configured as a possible convolutional neural network. Depth-wise convolution is restricted to applying a single filter to each input channel, and point-wise convolution is restricted to combining the outputs of depth-wise convolutions.

It is understood that the second computing device 412 may be configured similarly to computing device 402 . The second computing device 412 may have a GUI, such as for requesting and displaying images and skin symptom diagnoses from data stored on the server 406 for various users.

6 and 7 are flow charts of respective processes 600, 610, 620, and 630 of computing device 402 (or 410), for example. Process 600 involves a user of computing device 402 using an application, such as application 516, to take a selfie containing an image of the user's face and perform a skin diagnosis for each of a plurality (N) of skin manifestations. . At 601, an image is received at the processor via a camera or other method (eg, from a message attachment).

At 602, the images are preprocessed to define normalized images to present to the CNN. Images may be centered and cropped to a predetermined size (resolution) in order to present similarly sized images to the CNN as the CNN learns. At 603, the normalized image is processed using a CNN (neural network model 514) to generate N skin symptom diagnosis results. An ethnicity vector is also generated. The N skin sign diagnoses and ethnicity vectors (or single values thereof) are presented at 604 via a GUI or the like that can also present images and/or normalized images. Presenting the images includes displaying an image (or a normalized image) for each (or at least one) of the N skin features that indicate which facial regions are associated with which skin features. Segmenting may be included. Extraction from the image may, for example, perform skin signatures using bounding boxes and/or masks to isolate regions prepared for presentation in the GUI. The CNN may be configured to output segmentation-related data that constitute bounding boxes and/or masks for each (or at least one) particular region. The image may be annotated using techniques such as augmented reality or virtual reality to highlight regions. For example, relevant pixels of regions within an image can be highlighted. A GUI may be provided that shows the image (or normalized image). Input, such as a pointing device or gesture, may be received to indicate or select one or more pixels in the region for which a skin symptom diagnosis was generated by the CNN. Pixels outside the indicated region may be blurred using a mask and/or bounding box for that region to enhance pixels in the selected region. Also, instead of blurring, pixels outside the region, eg pixels within the border (eg, 1-X pixels), may be colored with a highlight color to surround the region to create a halo effect. Pixels adjacent to the region may be darker (higher in color) and pixels further away from the region (within the border) may be lighter in color. Different signs may have different colored borders. A skin symptom diagnosis for the area may be displayed. Color may be used to indicate severity proportional to skin symptom diagnostic results, such as using a scaling factor. A single color may be used for a particular skin symptom diagnosis and its color depth (eg, light to dark) may be scaled proportionally to the scale of the skin symptom diagnosis. In another example, different colors may be used for each level of the skin symptom diagnosis scale. Whether the GUI shows a single color that varies with depth or uses different colors, it may provide a color legend that shows the relationship to scale. A user toggle control may be provided to turn on/off the augmented reality or virtual reality applied to the image, eg, to turn highlighting on/off. Analysis example images (or extracts of specific affected areas) showing representative images of others showing each of the skin sign diagnoses (e.g., one for each severity and each skin sign) for comparison. may be presented, and such examples may be presented in a manner that respects the privacy of others. Product and/or treatment recommendations may be provided, as described further below. As discussed further below, pre- and post-treatment images (e.g., post-images represent subsequent images taken after one or more treatments, with subsequent skin symptom diagnoses provided for comparison). Also, while it has been described that a gesture input via an image selects or indicates an area, for example, a GUI that automatically selects an area by receiving input of a given skin indicia may be used. For example, a GUI may present each skin symptom and/or skin symptom diagnosis in a tabular or other form of output Selecting a particular item from the tabular or other form may A GUI may be invoked to display an image (or normalized image) in which regions relevant to skin symptom diagnosis are highlighted, gesture-activated GUIs (and/or other input-activated GUIs ( For example, it is understood that a voice-activated GUI may be used in any of the examples herein in addition to or instead of text commands)).

FIG. 6B shows process 610 . At 611, a GUI is presented (note that a GUI may be presented for any of the processes 600, 610, 620, and 630) to initiate product and/or therapy recommendations. . Input may be received to initiate a performance. At 612, the recommendations are received and the operation communicates skin diagnostic information (eg, scores, ethnicity vectors, images, user information, etc.) to a remote server, such as server 408, to receive the recommendations. may include. Recommendations include those relating to a treatment regimen having one or more products, a method of application to an area of skin, and a schedule. At 613, recommendations are presented, eg, via a GUI. Multiple recommendations may be received and presented. At 614, a selection is made to indicate acceptance of the recommendation. This may be stored (logged) and may, for example, initiate a therapy monitoring function of computing device 402 . At 615, product purchases may be facilitated, such as through server 408 or other servers.

FIG. 7(a) shows a process 620, such as for monitoring. Monitoring may be responsive to treatment plans (eg, described in data) received by computing device 402 or accessible via a browser or the like. The treatment regimen may have a schedule such as application of the second product once a week (eg application of morning and evening products). The schedule may be announced (eg, at 621), for example, via notification by the native application, or via another means such as a calendar application. At 622, a GUI is provided to facilitate therapeutic activities, eg, documenting their occurrence and/or providing support for performing the activities. At 623, input is received, such as confirmation that the activity was performed. Images may be included to record activity. Data may be logged. Monitoring may measure how well the treatment regimen is being followed. At 624, repurchasing of the product may be facilitated, for example, in response to therapy monitoring, it may be determined that the quantity of product on hand may be insufficient.

FIG. 7(b) shows a process 630 for performing a comparison, for example, which may be performed as a monitoring activity. At 631, a comparison GUI is provided to guide the user and the like. At 632, the new image (eg, compared to the first image received at 601) is (optionally) stored. At 633, subsequent skin symptom diagnosis is performed using a CNN on the new image (eg, normalized, etc., similar to process 600). At 634, the GUI presents a comparison using the first skin symptom diagnosis and the subsequent skin symptom diagnosis, optionally using the first image and the new new image.

Although not shown in FIGS. 6 and 7, any data received or generated may be communicated to a remote storage device such as server 406 .

Diagnosis of skin manifestations, subsequent diagnosis of skin manifestations (optionally with other monitoring), and provision of data for aggregation may be used to investigate product or treatment efficacy and/or fraudulent claims of products and treatments. can make it possible. Data may be collected, analyzed and presented to dermatologists and/or other professionals and/or users. Thus, the systems and methods herein can facilitate a distributed research model for skin treatments.

The teachings herein provide local-to-global linking capabilities (e.g., specific conditions within facial regions while processing the entire face) and all critical regions (e.g., each layer of the face from forehead to mouth). It includes a function to comprehensively map the face targeting wrinkles present in the face.

A combination of local skin signs may be used to predict (classify) global appearance (eg, apparent age, radiance, fatigue, etc.). Appearance may also be determined and compared by performing a skin analysis in the presence of makeup. The cutaneous diagnostics herein are sufficiently comprehensive in terms of the nature and location of facial signs to be able to explain the perception of other humans when they are looking at the subject. Skin sign diagnostics can be used to draw further conclusions about apparent age, such as based on perceptions of 95% or more from others. In addition, if you wear makeup, skin diagnosis and general Further predictions/categorizations regarding personal appearance and attractiveness can be used.

The skin diagnostic methods and techniques of the present invention measure five analytical clusters of the face (wrinkles/texture, sagging, pigmentation disorders, vascular disorders, cheek pores), aging process, environmental conditions (sun exposure, , exposure to chronic urban pollution), or lifestyle (stress, fatigue, sleep quality, smoking, alcohol consumption, etc.). By measuring these over time, during exercise, and by comparing them with consumer age, this technique, and computer equipment, etc., can be used to predict accelerated aging, distinct environmental influences (some signs, some clusters and not other clusters).
・Recommendations for cosmetics and/or therapeutic or prophylactic agents (e.g. what types of filters, antioxidants, desquamating agents, etc. are needed geographically for sun exposure)
• Recommendations in terms of diet, lifestyle, sports/exercise, etc. may have a positive impact on the lesions or idiosyncrasies of facial signs. For example, facial signs are known to be affected by daily activities, and we suggest several strategies based on that.

The skin diagnostic methods and techniques thus described can be used to dynamically track consumers/patients in all dimensions in a highly accurate manner. Assessments may be taken at different times and/or in different areas to assess daily/seasonal/hormonal/resting effects and therapeutic/cosmetic/health benefits. Such an assessment provides a more accurate diagnosis and allows better solution recommendations.

The skin diagnostic methods and techniques thus described may be used to perform assessments on images of the user in motion, such as from selfies or other videos. The method and computing device may be configured to evaluate each frame or selected frames of the video and record the score(s) of the face when the face is moving. Dynamic curves such as wrinkles or sagging may be defined. The video may capture specific facial positions and transitions that induce stress on the face to aid in analysis of specific symptoms.

Instructions may be provided that force the user to perform specific gestures, poses, etc. that emphasize features and stress the face. As an example, the instructions (eg, via a graphical or other user interface) may ask the user to perform a particular gesture, eg, pinch the cheeks. Such an assessment provides a more accurate diagnosis and allows better solution recommendations.

Other stresses may be indicated, such as functional stress due to the body being in an upright or supine position. Functional stress is very important to young consumers and can register wrinkles not seen in the very neutral standard ID image. Small crow's feet wrinkles are seen when the user smiles or has certain emotions.

Thus, skin diagnostics and techniques can receive a video of the face in motion and then allow multiple images to be evaluated, eg, the video can be frames 1, 2, . . N, each frame can generate 20 scores for 20 symptoms. The system instructs the user to gesture (face pinch) and records the result. Images before and after pinching the face can be analyzed to conclude the condition of the skin before and after stress, the amount of water movement, etc. (Publication DermoTrace: Flament F, Bazin R. Influences of age, ethnic group, and skin sites. on a provisionary skin marking, experimentally induced, in vivo. Skin Res Technol 24, 180-186 (2018)). Exactly two frames should be used.

The skin diagnostic methods and techniques thus described may be used to further improve the performance of sentiment analysis through the development of analytical features. Performing a global assessment of the face may allow the assessment (eg, classification) of emotions by correlating combinations of skin signs with specific visual signs such as joy, fear, disgust, and the like.

The skin diagnostic methods and techniques thus described use the classification of symptoms as indicative of particular emotions, such as those who cannot communicate verbally or in the same language, to facilitate the practice of health care. May be used. Patients in pain may exhibit emotions associated with it, which can be analyzed and used to administer medications, etc. For example, the combination of glabellar symptoms and their severity can be important clues in the health field, especially in hospitals, for patients with difficult-to-communicate pain. By accurately reading your face, you can administer medicines or design specific treatments.

The skin diagnostic methods and techniques thus described may be used to further improve the performance of characterizing (eg, classifying) environmental or lifestyle influences. By comparing with databases, we can define extrinsic and intrinsic aging, and based on our knowledge database, environmental conditions (UV, pollution, etc.) or lifestyle (stress, diet, alcohol, smoking, sports, etc.) ) in terms of quantification (percentage of severity) and eligibility (the nature and location of facial signs). The skin diagnostic assessments described herein can be enhanced with information from databases to provide consumers with more accurate and personalized feedback on leading cosmetic products for urban aging. .

The skin diagnostic methods and techniques thus described may be used to enhance other medical diagnostics for other conditions. Combinations of cutaneous manifestations can be associated with specific medical conditions based on studies correlating specific facial manifestations with specific conditions or diseases. For example, forehead wrinkles are associated with heart disease.

Skin diagnostic methods and techniques, including product and/or application recommendations so described, are naturally and normally occurring and typically not disease-related skin manifestations (e.g., aging and/or environmental It will be understood that it may also be practiced in relation to skin manifestations not indicative of a disease state, such as those associated with exposure. However, the development and/or progression of such non-diseased cutaneous manifestations may be responsive to the respective pharmaceutical product and the respective regimen of application (which is not a medical treatment per se, but treatment in a broader sense). . Accordingly, provided herein are skin diagnostic devices and methods for non-diseased skin indications. Apparatus and methods are provided for recommending products for non-diseased skin indications. The apparatus comprises a storage unit and a processing unit coupled to the storage unit, the storage unit determining a respective N skin symptom diagnosis for each of a plurality of N non-diseased skin symptoms. storing and providing a CNN, a convolutional neural network that classifies pixels of an image for image classification, said CNN configured to generate a diagnosis of each of said N non-diseased skin manifestations for image classification; A deep neural network, wherein the CNN is trained using non-diseased skin manifestation data for each of each of the N non-diseased skin manifestations, the processing unit receiving the images; A CNN is used to process the image to generate each of the N non-diseased skin symptom diagnoses. The processing unit may include a product recommendation component, such as a rule-based system or other system that selects one or more products and, optionally, an application regimen for each product, for each product associated with each non-diseased skin indication. ) to generate product recommendations for at least one of each of the N non-disease skin symptom diagnoses. Product recommendation components, and even product recommendations, may be responsive to other factors such as gender, ethnicity, and the like. Learning methods and systems related to training a CNN and defining a system with a CNN to generate each of the N skin symptom diagnoses will be apparent.

In addition to computer device aspects, instructions for configuring the computer device to perform any of the method aspects described herein may be stored in non-transitory storage (e.g., memory, CD-ROM, One of ordinary skill in the art will appreciate that aspects of a computer program product stored on a DVD-ROM, disc, etc.) are disclosed.

Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may refer to methods, apparatus, systems, means for performing functions, program products, and otherwise combining the features described herein. can be While a number of embodiments have been described, it will be appreciated that various modifications can be made without departing from the spirit and scope of the processes and techniques described herein. Additionally, other steps may be provided or steps may be omitted from the described methods, and other components may be added or removed from the described systems. Accordingly, other aspects are within the scope of the claims.

Throughout the description and claims of this specification, the words "including" and "comprising" and variations thereof mean "including but not limited to" and to the exclusion of other elements, integers or steps. does not intend (do not exclude) Throughout this specification, the singular encompasses the plural unless the context otherwise requires. In other words, it is to be understood that this specification intends the plural as well as the singular, unless the context requires otherwise.

技術的思想８の皮膚の診断装置は、技術的思想４から７のいずれかに記載の皮膚の診断装置において、記ＣＮＮは、（ｘ _ｉ 、ｙ _ｉ ）形式の複数のサンプルを用いて学習され、ｘ _ｉ は、ｉ番目の学習画像であり、ｙ _ｉ は、グランドトゥルースの皮膚の徴候診断に対応するベクトルであり、前記ＣＮＮは、前記Ｎ個の並列の皮膚徴候の分岐、及び前記民族性に関する並列の分岐の各分岐に対する損失関数を最小化するように学習される。
技術的思想９の皮膚の診断装置は、技術的思想８記載の皮膚の診断装置において、前記ＣＮＮは、前記Ｎ個の並列の皮膚徴候の分岐のそれぞれについての損失関数Ｌ２に、前記民族性に関する並列の分岐についての標準交差エントロピー分類損失Ｌ _{ｅｔｈｎｉｃｉｔｙ} を重み付けして組み合わせた損失関数Ｌを、以下の数式３に従って最小化するように学習され、λは、スコア回帰と民族性の分類損失の間のバランスを制御する。

技術的思想１０の皮膚の診断装置は、技術的思想１から９のいずれかに記載の皮膚の診断装置において、前記記憶部は、前記画像を前処理するための顔およびランドマークの検出器を記憶し、前記処理部は、前記顔およびランドマークの検出器を用いて前記画像から正規化された画像を生成し、前記ＣＮＮを使用する際に前記正規化された画像を使用するように構成される。
技術的思想１１の皮膚の診断装置は、技術的思想１から１０のいずれかに記載の皮膚の診断装置において、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候診断を生成するように適合された画像分類のための学習済ネットワークを備え、前記学習済ネットワークの全結合層が省略され、Ｎ個の各層のグループは、前記Ｎ個の各皮膚の徴候診断のそれぞれについて同じ特徴ネットを並列に復号するように定義される。
技術的思想１２の皮膚の診断装置は、技術的思想１から１１のいずれかに記載の皮膚の診断装置において、モバイル端末からなる個人用のコンピュータ装置と、通信ネットワークを介して皮膚の診断サービスを提供するサーバと、のいずれかから構成される。
技術的思想１３の皮膚の診断装置は、技術的思想１から１２のいずれかに記載の皮膚の診断装置において、前記記憶部は、前記処理部によって実行されると、前記Ｎ個の各皮膚の徴候診断の少なくともいくつかに対応して、製品および治療計画のうちの少なくとも１つに関する推奨事項を取得するための治療製品セレクタを提供するコードを記憶する。
技術的思想１４の皮膚の診断装置は、技術的思想１から１３のいずれかに記載の皮膚の診断装置において、前記記憶部は、前記処理部によって実行されると、前記画像を受信するための画像取得機能を提供するコードを記憶する。
技術的思想１５の皮膚の診断装置は、技術的思想１から１４のいずれかに記載の皮膚の診断装置において、前記記憶部は、前記処理部によって実行されると、少なくとも１つの皮膚の徴候に対する治療をモニタするための治療モニタを提供するコードを記憶する。
技術的思想１６の皮膚の診断装置は、技術的思想１５記載の皮膚の診断装置において、前記処理部は、各治療セッションに対する製品の適用に関連する治療活動を、思い出させる、指示する、及び／又は記録する、のうちの少なくとも１つを行うように構成される。
技術的思想１７の皮膚の診断装置は、技術的思想１から１６のいずれかに記載の皮膚の診断装置において、前記処理部は、前記ＣＮＮを用いて第２の画像を処理し、治療セッション後に受信した後続の皮膚の診断を生成するように構成される。
技術的思想１８の皮膚の診断装置は、技術的思想１７記載の皮膚の診断装置において、前記記憶部は、前記処理部によって実行されると、前記後続の皮膚の診断を用いた比較結果の提示を行うコードを記憶する。
技術的思想１９のコンピュータ実装方法は、皮膚診断のコンピュータ実装方法であって、画像のピクセルを分類して、複数であるＮ個の各皮膚の徴候の各々についてＮ個の各皮膚の徴候診断を判定するように構成された畳み込みニューラルネットワークであるＣＮＮを記憶する記憶部を提供し、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候診断を生成するように構成された画像分類のための深層ニューラルネットワークであり、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候についての皮膚の徴候データを使用して学習され、前記記憶部に結合された処理部によって、前記画像を受信することと、前記ＣＮＮを用いて前記画像を処理して前記Ｎ個の各皮膚の徴候診断を生成することと、を実行する。
技術的思想２０のコンピュータ実装方法は、技術的思想１９記載のコンピュータ実装方法において、前記ＣＮＮは、画像分類のための学習済ネットワークから定義され、最終のエンコーダ段階の特徴ネットに特徴を符号化するように構成されたエンコーダ段階と、前記Ｎ個の各皮膚の徴候診断を生成するために、複数であるＮ個の並列の皮膚徴候の分岐によって復号化するための前記最終のエンコーダ段階の特徴ネットを受信するように構成されたデコーダ段階と、を備える。
技術的思想２１のコンピュータ実装方法は、技術的思想２０に記載のコンピュータ実装方法において、前記デコーダ段階は、前記最終のエンコーダ段階の特徴ネットを処理して前記Ｎ個の並列の皮膚徴候の分岐の各々に提供するグローバルプーリング処理を備える。
技術的思想２２のコンピュータ実装方法は、技術的思想２０又は２１に記載のコンピュータ実装方法において、前記ＣＮＮは、前記ピクセルを分類して民族性ベクトルを判定するように構成され、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候および複数の民族性に関する皮膚の徴候データを使用して学習され、前記ＣＮＮによる前記画像の処理は、前記民族性ベクトルを生成する。
技術的思想２３のコンピュータ実装方法は、技術的思想２２記載のコンピュータ実装方法において、前記デコーダ段階は、前記民族性ベクトルを生成するための民族性に関する並列の分岐を備える。
技術的思想２４のコンピュータ実装方法は、技術的思想２０から２３のいずれかに記載のコンピュータ実装方法において、前記Ｎ個の並列の皮膚徴候の分岐の各分岐は、第１の全結合層と、それに続く第１の活性化層と、第２の全結合層と、第２の活性化層と、最終の活性化層と、を連続して備え、前記Ｎ個の各皮膚の徴候診断および前記民族性ベクトルのうちの１つを含む最終値を出力する。
技術的思想２５のコンピュータ実装方法は、技術的思想２４記載のコンピュータ実装方法において、前記最終の活性化層は、前記第２の活性化層から受信した入力スコアｘに関する以下の数式１の関数に従って定義され、αは傾き、ａは下限、ｂは前記Ｎ個の各皮膚の徴候診断の各々のスコア範囲の上限である。

技術的思想２６のコンピュータ実装方法は、技術的思想２２から２５のいずれかに記載のコンピュータ実装方法において、前記ＣＮＮは、（ｘ _ｉ 、ｙ _ｉ ）形式の複数のサンプルを用いて学習され、ｘ _ｉ は、ｉ番目の学習画像であり、ｙ _ｉ は、グランドトゥルースの皮膚の徴候診断に対応するベクトルであり、前記ＣＮＮは、前記Ｎ個の並列の皮膚徴候の分岐、及び前記民族性に関する並列の分岐の各分岐に対する損失関数を最小化するように学習される。
技術的思想２７のコンピュータ実装方法は、技術的思想２６記載のコンピュータ実装方法において、前記ＣＮＮは、前記Ｎ個の並列の皮膚徴候の分岐のそれぞれについての損失関数Ｌ２に、前記民族性に関する並列の分岐についての標準交差エントロピー分類損失Ｌ _{ｅｔｈｎｉｃｉｔｙ} を重み付けして組み合わせた損失関数Ｌを、以下の数式３に従って最小化するように学習され、λは、スコア回帰と民族性の分類損失の間のバランスを制御する。

技術的思想２８のコンピュータ実装方法は、技術的思想１９から２７のいずれかに記載のコンピュータ実装方法において、前記記憶部は、前記画像を前処理するための顔およびランドマークの検出器を記憶し、前記方法は、前記処理部が前記顔およびランドマークの検出器を用いて記画像を前処理して前記画像から正規化された画像を生成し、前記ＣＮＮを使用する際に前記正規化された画像を使用する。
技術的思想２９のコンピュータ実装方法は、技術的思想１９から２８のいずれかに記載のコンピュータ実装方法において、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候診断を生成するように適合された画像分類のための学習済ネットワークを備え、前記学習済ネットワークの全結合層が省略され、Ｎ個の各層のグループは、前記Ｎ個の各皮膚の徴候診断のそれぞれについて同じ特徴ネットを並列に復号するように定義される
技術的思想３０のコンピュータ実装方法は、技術的思想１９から２９のいずれかに記載のコンピュータ実装方法において、前記記憶部および前記処理部は、モバイル端末からなる個人用のコンピュータ装置と、通信ネットワークを介して皮膚の診断サービスを提供するサーバと、のいずれかの構成要素である。
技術的思想３１のコンピュータ実装方法は、技術的思想１９から３０のいずれかに記載のコンピュータ実装方法において、前記記憶部は、前記処理部によって実行されると、前記Ｎ個の各皮膚の徴候診断の少なくともいくつかに対応して、製品および治療計画のうちの少なくとも１つに関する推奨事項を取得するための治療製品セレクタを提供するコードを記憶し、前記方法は、前記処理部によって前記治療製品セレクタの前記コードを実行して、製品および治療計画のうちの少なくとも１つに関する推奨事項を取得する
技術的思想３２のコンピュータ実装方法は、技術的思想１９から３１のいずれかに記載のコンピュータ実装方法において、前記記憶部は、前記処理部によって実行されると、前記画像を受信するための画像取得機能を提供するコードを記憶し、前記方法は、前記画像を受信するために、前記処理部によって前記画像取得機能の前記コードを実行する。
技術的思想３３のコンピュータ実装方法は、技術的思想１９から３２のいずれかに記載のコンピュータ実装方法において、前記記憶部は、前記処理部によって実行されると、少なくとも１つの皮膚の徴候に対する治療をモニタするための治療モニタを提供するコードを記憶し、前記方法は、少なくとも１つの皮膚の徴候に対する治療をモニタするために、前記処理部よって前記治療モニタのコードを実行する。
技術的思想３４のコンピュータ実装方法は、技術的思想３３記載のコンピュータ実装方法において、前記方法は、前記処理部によって、各治療セッションに対する製品の適用に関連する治療活動を、思い出させる、指示する、及び／又は記録する、のうちの少なくとも１つを行う。
技術的思想３５のコンピュータ実装方法は、技術的思想１９から３４のいずれかに記載のコンピュータ実装方法において、前記処理部によって、前記ＣＮＮを用いて第２の画像を処理し、治療セッション後に受信した後続の皮膚の診断を生成する。
技術的思想３６のコンピュータ実装方法は、技術的思想３５記載のコンピュータ実装方法において、処理部によって、前記後続の皮膚の診断を用いて比較結果の提示を提供する。
技術的思想３７の方法は、画像のピクセルを分類して、複数であるＮ個の各皮膚の徴候の各々についてＮ個の各皮膚の徴候診断を判定するように構成された畳み込みニューラルネットワークであるＣＮＮを学習させ、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候診断を生成するように構成された画像分類のための深層ニューラルネットワークであり、前記学習は、前記Ｎ個の各皮膚の徴候についての皮膚の徴候データを用いて実行される、方法。
技術的思想３８の方法は、技術的思想３７記載の方法において、前記ＣＮＮは、画像分類のための学習済ネットワークから定義され、最終のエンコーダ段階の特徴ネットに特徴を符号化するように構成されたエンコーダ段階と、前記Ｎ個の各皮膚の徴候診断を生成するために、複数であるＮ個の並列の皮膚徴候の分岐によって復号化するための前記最終のエンコーダ段階の特徴ネットを受信するように構成されたデコーダ段階と、を備える。
技術的思想３９の方法は、技術的思想３８記載の方法において、前記デコーダ段階は、前記最終のエンコーダ段階の特徴ネットを処理して前記Ｎ個の並列の皮膚徴候の分岐の各々に提供するグローバルプーリング処理を備える。
技術的思想４０の方法は、技術的思想３８又は３９に記載の方法において、前記ＣＮＮは、前記ピクセルを分類して民族性ベクトルを判定するように構成され、前記方法は、前記Ｎ個の各皮膚の徴候および複数の民族性に関する皮膚の徴候データを使用して前記ＣＮＮを学習させる。
技術的思想４１の方法は、技術的思想４０記載の方法において、前記デコーダ段階は、前記民族性ベクトルを生成するための民族性に関する並列の分岐を備える。
技術的思想４２の方法は、技術的思想３８から４１のいずれかに記載の方法において、前記Ｎ個の並列の皮膚徴候の分岐の各分岐は、第１の全結合層と、それに続く第１の活性化層と、第２の全結合層と、第２の活性化層と、最終の活性化層と、を連続して備え、前記Ｎ個の各皮膚の徴候診断および前記民族性ベクトルのうちの１つを含む最終値を出力する。
技術的思想４３の方法は、技術的思想４２記載の方法において、前記最終の活性化層は、前記第２の活性化層から受信した入力スコアｘに関する以下の数式１の関数に従って定義され、αは傾き、ａは下限、ｂは前記Ｎ個の各皮膚の徴候診断の各々のスコア範囲の上限である。

技術的思想４４の方法は、技術的思想４０から４４のいずれかに記載の方法において、前記学習は、（ｘ _ｉ 、ｙ _ｉ ）形式の複数のサンプルを用いて前記ＣＮＮを学習させ、ｘ _ｉ は、ｉ番目の学習画像であり、ｙ _ｉ は、グランドトゥルースの皮膚の徴候診断に対応するベクトルであり、前記学習は、前記Ｎ個の並列の皮膚徴候の分岐、及び前記民族性に関する並列の分岐の各分岐に対する損失関数を最小化するように前記ＣＮＮを学習させる。
技術的思想４５の方法は、技術的思想４４記載の方法において、前記学習は、前記Ｎ個の並列の皮膚徴候の分岐のそれぞれについての損失関数Ｌ２に、前記民族性に関する並列の分岐についての標準交差エントロピー分類損失Ｌ _{ｅｔｈｎｉｃｉｔｙ} を重み付けして組み合わせた損失関数Ｌを、以下の数式３に従って最小化するように前記ＣＮＮを学習させ、λは、スコア回帰と民族性の分類損失の間のバランスを制御する。

技術的思想４６のコンピュータ実装方法は、技術的思想１９から２７のいずれかに記載のコンピュータ実装方法において、前記ＣＮＮは、顔およびランドマークの検出によって前処理された正規化された画像を受信するように構成される。
技術的思想４７のコンピュータ実装方法は、技術的思想１９から２８のいずれかに記載のコンピュータ実装方法において、前記ＣＮＮは、前記Ｎ個の各皮膚の徴候診断を生成するように適合された画像分類のための学習済ネットワークを予め備え、前記学習済ネットワークの全結合層が省略され、Ｎ個の各層のグループは、前記Ｎ個の各皮膚の徴候診断のそれぞれについて同じ特徴ネットを並列に復号するように定義される。 Features, integral properties, compounds, chemical moieties or groups described in connection with a particular aspect, embodiment or example of the invention apply to any other aspect, embodiment or example unless incompatible with them. It should be understood that it is possible. Any and all features disclosed in this specification (including any appended claims, abstract, and drawings) or any method or process step so disclosed may be referred to as such features. Alternatively, they may be combined in any combination, except combinations where at least some of the steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments. The present invention resides in any novel invention or any novel combination of features disclosed in this specification (including the appended claims, abstract and drawings) or any technique disclosed. or extend to any novel one or any novel combination of process steps.
<Others>
<Means>
Technical idea 1 is a skin diagnostic device comprising a storage unit and a processing unit coupled to the storage unit, wherein the storage unit stores a plurality of N skin symptoms. , a CNN that is a convolutional neural network that classifies pixels of an image to determine N skin symptom diagnoses, said CNN generating said N skin symptom diagnoses. A deep neural network for image classification, wherein the CNN is trained using skin manifestation data for each of the N skin manifestations, and the processing unit processes the images as Receive and process the image using the CNN to generate a diagnosis of each of the N skin signs.
The skin diagnostic apparatus of technical idea 2 is the skin diagnostic apparatus according to technical idea 1, wherein the CNN is defined from a trained network for image classification, and encodes features into feature nets in the final encoder stage. and said final encoder stage for decoding with a plurality of N parallel skin manifestation branches to generate said N respective skin manifestation diagnoses. a decoder stage configured to receive the feature net.
Technical idea 3 is the skin diagnostic apparatus according to technical idea 2, wherein the decoder stage processes the feature net of the final encoder stage to branch the N parallel skin signs. , with global pooling processing provided to each of the .
Technical idea 4 is the skin diagnostic apparatus according to technical idea 2 or 3, wherein the CNN is configured to classify the pixels to determine an ethnicity vector, and the CNN comprises , is learned using each of the N skin signs and skin sign data for multiple ethnicities.
The skin diagnostic apparatus according to technical idea 5 is the skin diagnostic apparatus according to technical idea 4, wherein said decoder stage comprises parallel ethnic branches for generating said ethnicity vector.
Technical idea 6 is the skin diagnostic apparatus according to any one of technical ideas 2 to 5, wherein each branch of the N parallel skin symptom branches comprises a first fully connected layer followed by a first activated layer, a second fully connected layer, a second activated layer, and a final activated layer, wherein each of the N skin symptom diagnoses and output a final value containing one of said ethnicity vectors.
The skin diagnostic device according to technical idea 7 is the skin diagnostic device according to technical idea 6, wherein the final activation layer is the following formula 1 regarding the input score x received from the second activation layer is the slope, a is the lower bound, and b is the upper bound of the score range for each of the N skin sign diagnoses, defined according to a function.

The skin diagnostic device of technical idea 8 is the skin diagnostic device according to any one of technical ideas 4 to 7, wherein the CNN is learned using a plurality of samples of (x _i , y _i ) format. , x _i is the ith training image, y _i is the vector corresponding to the ground truth skin sign diagnosis, the CNN is the N parallel skin sign branches, and the ethnicity is learned to minimize the loss function for each branch of parallel branches with respect to .
The skin diagnostic apparatus according to technical idea 9 is the skin diagnostic apparatus according to technical idea 8, wherein the CNN is a loss function L2 for each of the N parallel skin sign branches, A loss function L that is a weighted combination of the standard cross-entropy classification loss L _ethnicity for parallel branches is learned to minimize according to Equation 3 below, where λ is the difference between score regression and ethnicity classification loss. Control your balance.

Technical idea 10 is the skin diagnostic apparatus according to any one of technical ideas 1 to 9, wherein the storage unit stores a face and landmark detector for preprocessing the image. and the processing unit is configured to generate a normalized image from the image using the face and landmark detector, and to use the normalized image when using the CNN. be done.
Technical idea 11 is the skin diagnostic apparatus according to any one of technical ideas 1 to 10, wherein the CNN is adapted to generate each of the N skin symptom diagnoses. A trained network for image classification, wherein a fully connected layer of said trained network is omitted, and each group of N layers decodes in parallel the same feature net for each of said N skin symptom diagnoses. defined to be
Technical idea 12 is the skin diagnostic device according to any one of technical ideas 1 to 11, wherein skin diagnostic service is provided via a personal computer device composed of a mobile terminal and a communication network. It consists of either a server that provides
Technical idea 13 is the skin diagnostic apparatus according to any one of technical ideas 1 to 12, wherein the storage unit, when executed by the processing unit, Code is stored that provides a therapeutic product selector for obtaining recommendations for at least one of a product and a therapeutic regimen, corresponding to at least some of the symptom diagnoses.
Technical idea 14 is the skin diagnostic apparatus according to any one of technical ideas 1 to 13, wherein the storage unit receives the image when executed by the processing unit. Stores code that provides image acquisition functionality.
Technical idea 15 is the skin diagnostic apparatus according to any one of technical ideas 1 to 14, wherein the storage unit stores information for at least one skin symptom when executed by the processing unit. Store code to provide a therapy monitor for monitoring therapy.
Technical idea 16. The skin diagnostic apparatus according to technical idea 15, wherein the processing unit reminds, instructs, and/or treats activities associated with application of products for each treatment session. or recording.
Technical idea 17 is the skin diagnostic apparatus according to any one of technical ideas 1 to 16, wherein the processing unit processes a second image using the CNN, and after a treatment session configured to generate a subsequent skin diagnosis received;
Technical idea 18 is the skin diagnostic apparatus according to technical idea 17, wherein the storage unit presents comparison results using the subsequent skin diagnosis when executed by the processing unit. memorize the code to do
The computer-implemented method of Technical Thought 19 is a computer-implemented method of skin diagnosis, classifying pixels of an image to generate N skin symptom diagnoses for each of a plurality of N skin symptoms. providing a storage unit for storing a CNN, a convolutional neural network configured to determine, said CNN being a deep neural network for image classification configured to generate each of said N skin symptom diagnoses; a network, the CNN being trained using skin manifestation data for each of the N skin manifestations, and receiving the images by a processing unit coupled to the storage unit; to generate each of the N skin symptom diagnoses.
The computer-implemented method of Technical Thought 20 is the computer-implemented method of Technical Thought 19, wherein the CNN is defined from a trained network for image classification and encodes features into a feature net in the final encoder stage. and a feature net of said final encoder stage for decoding by means of a plurality of N parallel cutaneous manifestation branches to generate said N respective cutaneous manifestation diagnoses. and a decoder stage configured to receive the
The computer-implemented method of Technical Thought 21 is the computer-implemented method of Technical Thought 20, wherein the decoder stage processes the feature net of the final encoder stage to generate the N parallel cutaneous manifestation branches. It has a global pooling process to serve each.
The computer-implemented method of Technical Thought 22 is the computer-implemented method of Technical Thought 20 or 21, wherein the CNN is configured to classify the pixels to determine an ethnicity vector, wherein the CNN comprises the The processing of the images by the CNN, learned using each of the N skin signs and skin sign data for multiple ethnicities, produces the ethnicity vector.
The computer-implemented method of technical idea 23 is the computer-implemented method of technical idea 22, wherein the decoder stage comprises parallel ethnicity branches for generating the ethnicity vector.
Technical idea 24 is the computer-implemented method of any one of technical ideas 20 to 23, wherein each branch of the N parallel skin manifestation branches comprises a first fully connected layer; successively comprising a first activation layer, a second all-connected layer, a second activation layer, and a final activation layer, wherein each of said N skin symptom diagnoses and said Output the final value containing one of the ethnicity vectors.
The computer-implemented method of Technical Thought 25 is the computer-implemented method of Technical Thought 24, wherein the final activation layer calculates the input score x received from the second activation layer according to the function of Equation 1 below: is the slope, a is the lower bound, and b is the upper bound of the score range for each of the N skin sign diagnoses.

The computer-implemented method of technical idea 26 is the computer-implemented method according to any one of technical ideas 22 to 25, wherein the CNN is trained using a plurality of samples of the form (x _i , y _i ), and x _i is the ith training image, _yi is the vector corresponding to the ground truth skin sign diagnosis, the CNN is the N parallel skin sign branches, and the parallel is learned to minimize the loss function for each of the branches of .
The computer-implemented method of technical idea 27 is the computer-implemented method according to technical idea 26, wherein the CNN includes a loss function L2 for each of the N parallel skin sign branches, a parallel A loss function L that is a weighted combination of the standard cross-entropy classification loss L _ethnicity for bifurcations is learned to minimize according to Equation 3 below, where λ is the balance between score regression and ethnicity classification loss. Control.

Technical idea 28 is the computer-implemented method according to any one of technical ideas 19 to 27, wherein the storage unit stores a face and landmark detector for preprocessing the image. , the method comprises: the processing unit preprocessing the image with the face and landmark detector to generate a normalized image from the image; and the normalized image in using the CNN. Use an image that is
The computer-implemented method of Technical Thought 29 is the computer-implemented method of any one of Technical Thoughts 19-28, wherein the CNN comprises image classification adapted to generate each of the N skin symptom diagnoses. , wherein the fully connected layers of said trained network are omitted, and each group of N layers decodes in parallel the same feature net for each of said N skin symptom diagnoses. defined as
The computer-implemented method of technical idea 30 is the computer-implemented method according to any one of technical ideas 19 to 29, wherein the storage unit and the processing unit comprise a personal computer device consisting of a mobile terminal and a communication network. a server that provides skin diagnostic services via;
Technical idea 31 is the computer-implemented method according to any one of technical ideas 19 to 30, wherein the storage unit, when executed by the processing unit, stores each of the N skin symptom diagnoses. and storing code for providing a therapeutic product selector for obtaining recommendations for at least one of a product and a treatment plan, corresponding to at least some of the methods comprising: to obtain recommendations for at least one of products and treatment plans
The computer-implemented method of technical idea 32 is the computer-implemented method according to any one of technical ideas 19 to 31, wherein, when executed by the processing unit, the storage unit performs image acquisition for receiving the image. Code that provides a function is stored, and the method executes the code of the image acquisition function by the processing unit to receive the image.
The computer-implemented method of Technical Thought 33 is the computer-implemented method of any of Technical Thoughts 19-32, wherein the storage unit, when executed by the processing unit, provides a treatment for at least one skin indication. Code for providing a therapy monitor for monitoring is stored, and the method executes the therapy monitor code by the processing unit to monitor therapy for at least one skin manifestation.
The computer-implemented method of Technical Thought 34 is the computer-implemented method of Technical Thought 33, wherein said method reminds, instructs, by said processor, a treatment activity associated with application of a product for each treatment session; and/or record.
Technical idea 35 is the computer-implemented method of any one of technical ideas 19 to 34, wherein the processing unit processes a second image using the CNN, received after a treatment session Generate a subsequent skin diagnosis.
The computer-implemented method of idea 36, wherein the computer-implemented method of idea 35, by the processing unit, provides presentation of comparison results using said subsequent skin diagnosis.
The method of idea 37 is a convolutional neural network configured to classify pixels of an image to determine N skin symptom diagnoses for each of a plurality of N skin symptoms. training a CNN, said CNN being a deep neural network for image classification configured to generate a diagnosis for each of said N skin signs, said training comprising: skin manifestation data.
The method of Technical Thought 38 is the method of Technical Thought 37, wherein said CNN is defined from a trained network for image classification and configured to encode features into a feature net in a final encoder stage. and said final encoder stage feature net for decoding by a plurality of N parallel skin symptom branches to generate said N respective skin symptom diagnoses. and a decoder stage configured to:
The method of Technical Thought 39 is the method of Technical Thought 38, wherein the decoder stage processes the feature net of the final encoder stage to provide each of the N parallel cutaneous manifestation branches. It has a pooling process.
Technical idea 40 is the method according to technical idea 38 or 39, wherein the CNN is configured to classify the pixels to determine an ethnicity vector, the method comprising: The CNN is trained using skin signs and skin sign data for multiple ethnicities.
The method of idea 41 is the method of idea 40, wherein said decoder stage comprises parallel branches on ethnicity to generate said ethnicity vector.
Technical idea 42 is the method according to any one of technical ideas 38 to 41, wherein each branch of the N parallel skin manifestation branches comprises a first fully connected layer followed by a first , a second fully connected layer, a second activation layer, and a final activation layer, wherein each of the N skin symptom diagnoses and the ethnicity vector Output the final value containing one of
Technical idea 43 is the method according to technical idea 42, wherein the final activation layer is defined according to the function of Equation 1 below with respect to the input score x received from the second activation layer, and α is the slope, a is the lower bound, and b is the upper bound of the score range for each of the N skin sign diagnoses.

The method of technical idea 44 is the method according to any one of technical ideas 40 to 44, wherein the training includes training the CNN using a plurality of samples of (x _i , y _i ) format, and x _i is the i-th training image, y _i is a vector corresponding to the ground truth skin sign diagnosis, and the learning consists of the N parallel skin sign branches and the parallel ethnicity The CNN is trained to minimize the loss function for each branch of branches.
The method of technical idea 45 is the method according to technical idea 44, wherein the learning is a loss function L2 for each of the N parallel skin sign branches, and a standard for the parallel branch for the ethnicity Train the CNN to minimize a loss function L that is a weighted combination of the cross-entropy classification loss L _ethnicity according to Equation 3 below, where λ controls the balance between score regression and ethnicity classification loss do.

The computer-implemented method of idea 46 is the computer-implemented method of any of ideas 19-27, wherein the CNN receives normalized images preprocessed by face and landmark detection. configured as
Technical idea 47 is the computer-implemented method of any one of technical ideas 19 to 28, wherein the CNN comprises an image classification adapted to generate each of the N skin symptom diagnoses. , wherein the fully connected layers of said trained network are omitted, and each group of N layers decodes in parallel the same feature net for each of said N skin symptom diagnoses is defined as

Claims (20)

1. A skin diagnostic device comprising a storage unit and a processing unit coupled to the storage unit, comprising:
the storage unit stores and provides a CNN that is a convolutional neural network that classifies pixels of an image for each of the plurality of N skin signs to determine each of the N skin sign diagnoses;
said CNN is a deep neural network for image classification configured to generate each of said N skin sign diagnoses;
the CNN is trained using skin manifestation data for each of the N skin manifestations;
The skin diagnosis apparatus, wherein the processing unit receives the image and processes the image using the CNN to generate each of the N skin symptom diagnoses. The CNN generates an encoder stage defined from a trained network for image classification and configured to encode features into a final encoder stage feature net and each of the N skin symptom diagnoses. a decoder stage configured to receive the final encoder stage feature net for decoding by a plurality of N parallel skin manifestation branches for diagnostic equipment. 3. The skin diagnostic apparatus of claim 2, wherein the decoder stage comprises a global pooling process for processing the feature nets of the final encoder stage to provide each of the N parallel skin manifestation branches. the CNN is configured to classify the pixels to determine an ethnicity vector;
4. The skin diagnostic apparatus of claim 2 or 3, wherein the CNN is trained using skin manifestation data for each of the N skin manifestations and multiple ethnicities. 5. The skin diagnostic apparatus of claim 4, wherein the decoder stage comprises parallel branches for ethnicity to generate the ethnicity vector. Each branch of the N parallel skin manifestation branches comprises a first fully connected layer, followed by a first activated layer, a second fully connected layer, a second activated layer, and a final activation layers, and outputting a final value comprising one of each of the N skin symptom diagnoses and the ethnicity vector. Skin diagnostic equipment. The final activation layer is defined according to the function of Equation 1 below for input scores x received from the second activation layer,
7. The skin diagnostic apparatus of claim 6, wherein [alpha] is the slope, a is the lower bound, and b is the upper bound of the score range for each of said N skin symptom diagnoses.

The CNN is trained using a plurality of samples of the form (x _i , y _i ),
x _i is the i-th training image,
y _i is a vector corresponding to the ground truth skin sign diagnosis,
8. The CNN of any of claims 4-7, wherein the CNN is trained to minimize a loss function for each branch of the N parallel skin manifestation branches and the parallel ethnicity branches. Skin diagnostic equipment. The CNN provides a loss function L that is a weighted combination of a loss function L2 for each of the N parallel skin manifestation branches and a standard cross-entropy classification loss L _ethnicity for the parallel ethnicity branches, learned to minimize according to Equation 3 below,
9. The skin diagnostic device of claim 8, wherein [lambda] controls the balance between score regression and ethnic classification loss.

the storage unit stores face and landmark detectors for preprocessing the image;
the processing unit is configured to generate a normalized image from the image using the face and landmark detector and to use the normalized image when using the CNN; The skin diagnostic device according to any one of claims 1 to 9. said CNN comprising a trained network for image classification adapted to generate each of said N skin sign diagnoses;
fully connected layers of the trained network are omitted,
11. A skin diagnostic apparatus according to any preceding claim, wherein each group of N layers is defined to decode in parallel the same feature net for each of said N skin symptom diagnoses. the storage unit for, when executed by the processing unit, for obtaining recommendations for at least one of products and treatment regimens corresponding to at least some of each of the N skin symptom diagnoses; 12. A skin diagnostic device according to any preceding claim, storing code for providing a therapeutic product selector. 13. The skin diagnostic apparatus of any of claims 1-12, wherein the storage unit stores code that, when executed by the processing unit, provides an image acquisition function for receiving the image. 4. The skin of any of claims 1-13 , wherein the storage unit stores code that, when executed by the processing unit, provides a therapy monitor for monitoring therapy for at least one skin indication. diagnostic equipment. 15. The method of claim 14 , wherein the processing unit is configured to at least one of remind, instruct and record therapeutic activities associated with application of the product for each treatment session. skin diagnostic equipment. 16. The skin of any of claims 1-15 , wherein the processing unit is configured to process a second image using the CNN to generate a subsequent skin diagnosis received after a treatment session. diagnostic equipment. 17. The skin diagnosis apparatus of claim 16 , wherein the storage unit stores code that, when executed by the processing unit, presents comparison results using the subsequent skin diagnosis. 18. A skin diagnostic device according to any of the preceding claims, wherein the CNN is configured to receive normalized images preprocessed by face and landmark detection. A skin diagnostic method using a storage unit and a processing unit coupled to the storage unit, comprising:
storing and providing in the storage unit a CNN, a convolutional neural network for classifying pixels of an image for each of a plurality of N skin signs to determine each of the N skin sign diagnoses;
said CNN is a deep neural network for image classification configured to generate each of said N skin sign diagnoses;
the CNN is trained using skin manifestation data for each of the N skin manifestations;
A method of diagnosing skin, wherein each of the N skin symptom diagnoses is generated by causing the processing unit to process the received image using the CNN . A skin diagnostic program for causing a computer comprising a storage unit and a processing unit coupled to the storage unit to perform skin diagnostic processing,
storing in the storage a CNN, a convolutional neural network that classifies pixels of an image for each of a plurality of N skin signs to determine each of the N skin sign diagnoses;
and causing the computer to perform the steps of: causing the processing unit to process the received images using the CNN to generate each of the N skin symptom diagnoses;
said CNN is a deep neural network for image classification configured to generate each of said N skin sign diagnoses;
A skin diagnostic program, wherein said CNN is trained using skin symptom data for each of said N skin symptoms.