CN115035068A

CN115035068A - Cross-domain self-photographing face pockmark grading image classification method capable of self-adapting skin color

Info

Publication number: CN115035068A
Application number: CN202210680710.8A
Authority: CN
Inventors: 谭敏; 王瑞瑞; 俞俊
Original assignee: Hangzhou Dianzi University
Current assignee: Hangzhou Dianzi University
Priority date: 2022-06-15
Filing date: 2022-06-15
Publication date: 2022-09-09

Abstract

The invention provides a cross-domain self-portrait face pockmark grading image classification method adaptive to skin color. The method comprises the following steps: 1. between the source domain and the target domain, the domain offset is narrowed down with cross-domain data enhancement against the generative network model. 2. And constructing two gating network self-adaptive learning optimal sample weights. An expert gating network self-adaptive learning optimal characteristic weight and a skin color gating network self-adaptive learning optimal skin color weight are constructed. 3. And aligning the sample characteristics between the source domain and the target domain by utilizing a multi-core maximum mean difference method so as to reduce the domain deviation between the source domain and the target domain. 4. And establishing a multi-task end-to-end deep learning model according to the steps, training the whole network on a specific data set, and testing the performance of the final model on a test set. The method can adaptively learn the most appropriate sample weight distribution aiming at a specific data set, and has strong practicability and universality.

Description

Cross-domain self-photographing face pockmark grading image classification method capable of self-adapting skin color

Technical Field

The invention relates to the field of image classification, in particular to a cross-domain selfie face pockmark classification image classification method adaptive to skin color.

Background

With the progress of the times and the rapid development of medical cosmetology, people begin to pay more and more attention to the health condition of their skin, and skin care gradually becomes a hot topic. Acne, commonly known as comedo, is the most common skin disorder. Acne develops in about 80% of the adolescents, with acne symptoms lasting to the end of adulthood in about 3% of men and 12% of women. Due to the influence of the new coronavirus epidemic situation, people have to wear a mask to protect. This makes acne related diseases more severe and occur at more different age stages. However, acne also leaves scars and pigmentation, often leading to inferior and depressed mood. Therefore, a large number of acne patients are in urgent need of more accurate treatment. The severity of acne is critical to the dermatologist making an accurate, normative treatment regimen.

However, when a dermatologist ranks the severity of acne on the skin of a patient, misdiagnosis may occur due to factors such as subjectivity and lack of experience, which may lead to the condition of the patient to be worsened. With the development of deep learning technology, various intelligent auxiliary diagnosis algorithms based on deep learning break through and innovate continuously, and obtain considerable scores in various traditional diagnosis tasks and emerging tasks of medical image analysis, such as medical image classification, detection, segmentation, registration, image-based guided therapy and intervention and the like, so that remarkable potential is shown.

Different from the natural image field, a series of public complete large-scale labeled data sets, such as MNIST, CIFAR, ImageNet and the like, are provided. The labeling process of the acne image data set is high in specificity, the labeled data set is relatively scarce, and due to the existence of the face privacy problem, large-scale complete available data resources are few and few. Meanwhile, the change of the image background, the illumination and the skin color of the patient can bring the change of the data field. Under the condition, the method utilizes a domain self-adaptive method to dig out the information with identifiability from small sample data or a small amount of marked data, and is an effective way for conforming to the characteristics of the neighborhood resources of the current acne image.

Domain adaptation, which essentially fits the distribution differences between different data domains, generally speaking, assumes that the task between different domains is the same. In machine learning problems, it is generally assumed that the test data and the training data have the same distribution. But if this assumption is not verified, the performance of the model on the test set may be significantly degraded. In computer vision applications, such distribution differences (domain shifts) are common in real life and may be the result of changes in conditions (e.g., changes in background, light intensity, etc.). In the classification task, the method guides a classifier which learns the label-free data in the target domain by using the learned knowledge of the label data existing in one or more related source domains. In real life, most of data in a target domain is label-free or the data volume needing to be labeled is too large, and a large amount of manual work is involved. Thus, in this case, a domain-adaptive technique may be selected to build the target model.

Disclosure of Invention

The invention provides a cross-domain selfie face pockmark grading image classification method adaptive to skin color. The traditional pox classification method detects the existing pox firstly, then calculates the number of the detected pox, and classifies the image according to the number. However, the method is different from the traditional variola classifying method, the cross-domain self-adaptation and the image classification are fused in a uniform deep neural network, an end-to-end deep learning model is completed, and the model can directly complete the variola rating and classifying task of the self-shot image and simultaneously complete the cross-domain self-adaptation task of a source domain and a target domain. In the aspect of cross-domain self-adaption tasks of a source domain and a target domain, a confrontation generation network model is utilized to realize style conversion between a source domain sample and a target domain sample, so that domain offset between the source domain sample and the target domain sample is reduced, and cross-domain self-adaption of the source domain and the target domain is realized; in the aspect of image classification of self-photographing image pox classification, a cross-domain self-adaptive module and a self-adaptive skin color sensitive module are utilized to enable the model to obtain a better effect on a related data set.

A self-adaptive skin color cross-domain self-photographing face pockmark grading image classification method comprises the following steps:

step (1): image data pre-processing

Because the sizes of the images in the data set are different, in order to adapt to a deep learning framework, the sizes of the images need to be changed before the model begins to train, and the sizes of the images in the data set are unified.

Step (2): cross-domain data enhancement

The invention introduces a publicly available data set as a source domain to help the learning of the classifier in the target domain. Due to the fact that the samples of the source domain data and the target domain data have different sample illumination, dissimilar sample background and the like, domain deviation exists between the source domain data and the target domain data. Thus, in both the source domain and the target domain, the domain offset is narrowed down with cross-domain data enhancement against the generative network model.

And (3): construction of self-adaptive skin color gating classification model

The invention considers that the skin color change has certain influence on the classification of the pox, for example, under the condition of fair skin, similar phenomena such as acne, allergy and the like occur, and the classifier can identify the acne better so as to provide a more accurate result of skin quality evaluation; in contrast, in the case of black skin, acne is not easily identifiable and it is more difficult for the classifier to make a correct assessment. Therefore, an adaptive skin color gating classification model is constructed, and the skin color gating classification model can adaptively learn a weight vector related to the skin color of the sample according to the sample, so that the classification model has better performance. In the source domain data set and the target domain data set, skin colors are divided into five categories, namely white, neutral, tan, brown and black according to an Individual Type Angle (ITA) index.

And (4): feature space alignment

Because of the domain offset between the source domain data and the target domain data, the respective real images and generated images in the source domain and target domain will also have domain offsets, for which reason a feature-based domain alignment penalty is designed to make up for the domain gap. A multi-core maximum mean difference method is used in the present method.

And (5): model training and testing

And establishing a multitask end-to-end deep learning model according to the steps. And training the network parameters on a specific data set through a back propagation algorithm until the whole network model converges, and then testing the performance of the final model on a test set.

The image data preprocessing in the step (1) comprises the following specific steps:

because the sizes of all images in the data set are different, all the images are uniformly adjusted to a certain fixed size by a bilinear interpolation method. And then randomly cutting the image after the size adjustment to obtain image data with the size of 256 × 256. And finally, carrying out normalization processing on the image.

The cross-domain data enhancement in the step (2) comprises the following specific processes:

2-1. the source field is a collection comprising n images, each image having a corresponding skin quality class label, denoted I ^s ＝{(x _i ,y _i ) I is more than or equal to 1 and less than or equal to n; the target domain is a collection of m images, each image having a corresponding skin quality class label, denoted as

Wherein y is _i E.g. Y and

are respectively an image x _i And

class label of (1), (2), (…), N _c Denotes the category label space, N _c Is the total number of categories.

Is an image generator that converts samples in the source domain to a sample style with the target domain, the collection of generated images in the target domain being denoted as

Is an image generator that converts samples in the target domain into a sample style having a source domain, the collection of generated images in the source domain being denoted as

The specific formula is defined as follows:

wherein, the first and the second end of the pipe are connected with each other,

refers to a loss function; | | non-woven hair ₁ The representation takes a 1 norm.

2-2, respectively constructing a pair of image discriminators in the image space and the feature space and recording the pair of image discriminators

And

and a pair of feature discriminators

And

for discriminating whether a sample in the source domain that has passed through the image discriminator network is a real image or a generated image;

for discriminating whether a sample in the target domain that has passed through the image discriminator network is a real image or a generated image.

The system is used for judging whether the features extracted by the classification network in the source domain come from real image samples or generated image samples;

for determining whether the features extracted via the classification network in the target domain are from real image samples or from generated image samples.

The discrimination loss in the image space and the feature space is specifically shown in formula 3:

wherein

And representing a classification network, s represents a source domain, t represents a target domain, and d takes the values of s and t.

Wherein l _i Real label representing ith sample image. When used to calculate the discriminant loss of image space

When the temperature of the water is higher than the set temperature,

when used to compute the discriminant loss of the feature space

When the temperature of the water is higher than the set temperature,

ψ (x) is an intermediate parameter variable, see equation 2.

Constructing the self-adaptive gating classification model in the step (3), wherein the specific process is as follows:

the invention designs an expert gating network and a skin color gating network to adaptively learn the optimal sample weights of the expert network and the sub-network respectively so as to solve the skin color label noise. This approach also guarantees a reasonable contribution of each sample to all sub-networks. In particular, the number of sub-networks is set differently than the number of skin tone categories to break the one-to-one mapping between the number of sub-networks and the number of skin tone categories.

Firstly, a sample obtains corresponding eigenvectors through a plurality of parallel expert networks, meanwhile, the sample can self-adaptively learn a gating weight vector through the expert gating networks, and the gating weight vector is multiplied by the eigenvectors to obtain the final weight combined eigenvector. Then the feature vector of the weight combination passes through a plurality of sub-networks simultaneously, and correspondingly outputs features extracted by the sub-networks; and multiplying the features extracted by the sub-network with the skin color ratio weight vector obtained by the sample through the skin color gating network in a self-adaptive manner to obtain the final combined feature.

In the method, a source domain and a target domain both have a classification network, so that the adaptive gating network has three structures, namely a classification network only in the source domain, a classification network only in the target domain and a classification network combined in the source domain and the target domain. The specific definition is as follows:

wherein, N' _t Indicating the number of sub-networks, N _e Which represents the number of the expert networks,

denotes the ith subnet full connection layer, M denotes the aggregation layer, E _j Denotes the jth expert network, G _j Represents N _e The jth element, W, of the dimensional weight vector _i Is N' _t The ith element of the weight vector is dimensioned.

Where τ (x) represents the skin tone class label of sample x, e _τ(x) Is N' _t The τ (x) -th element of the dimensional weight vector is 1;

a skin tone category label representing a real image to which the image sample is generated,

is N' _t The second of the dimensional weight vector

Each element is 1; γ and γ' represent the weighting parameters of the real image and the generated image, respectively.

The feature space alignment in the step (4) comprises the following specific processes:

in the method, a multi-kernel maximum mean difference penalty is used in the source domain and the target domain for the real image samples and the generated image samples in the respective domains, respectively. Thus, a cross-domain generated image from another domain is more compatible with a real image of a particular domain, ensuring the plausibility of sharing the same classification model on the real image and the generated image. The maximum mean difference loss for polynuclear is specifically defined as follows:

wherein

Is a weight parameter that controls the source domain,

is a weight parameter of the control target domain; phi is a unit of ^s Is a mapping function in the source domain, phi ^t Is a mapping function in the target domain;

2 norm is taken; e denotes expectation. The maximum mean difference penalty for multiple cores may be applied to a single domain or to two domains with shared or unshared mapping functions.

The step (5) of constructing the multitask deep learning model specifically refers to optimizing classification loss, image generation loss and domain alignment loss on a specific data set after an end-to-end frame is established according to the steps (2), (3) and (4). And in the training process, parameters of the confrontation generation network model and the classification model are updated simultaneously to obtain a final model, and the training effect is tested on the test set.

The invention has the beneficial effects that:

based on the thought of cross-domain image generation and gated network adaptive learning, a classification model for classifying the severity of face pox of a self-timer is provided. By introducing a publicly available source domain data set as an auxiliary data set, a deep migration learning-based vaccinia severity classification framework is proposed. An expert gating network and a skin color gating network model are designed to adaptively learn the correlation between the label and the skin color of the feature space.

Drawings

FIG. 1 is a schematic flow diagram of the process of the present invention.

Fig. 2 is a schematic diagram of a network framework constructed in the method of the present invention.

Figure 3 experiment of the combination of the number of expert networks and the number of subnetworks.

Details of the embodiments

The present invention will be further described with reference to FIGS. 1 and 2. The invention provides a cross-domain selfie face pockmark grading method adaptive to skin color. The method comprises the following steps: 1. between the source domain and the target domain, the domain offset is narrowed down with cross-domain data enhancement against the generative network model. 2. And constructing two gating network self-adaptive learning optimal sample weights. An expert gating network self-adaptive learning optimal characteristic weight is constructed, and a skin color gating network self-adaptive learning optimal skin color weight is constructed. 3. And aligning the sample characteristics between the source domain and the target domain by utilizing a multi-core maximum mean difference method so as to reduce the domain deviation between the source domain and the target domain. 4. And establishing a multi-task end-to-end deep learning model according to the steps, training the whole network on a specific data set, and testing the performance of the final model on a test set. The method can adaptively learn the most appropriate sample weight distribution aiming at a specific data set, and has strong practicability and universality.

The invention specifically realizes the following steps:

the first step is as follows:

we used the Acne04 dataset as the source domain and the three datasets Acnehdu, Acnehdu P and AcnePGP as the target domains, respectively, for verifying our classification model for self-portrait face pox severity rating. When the model is trained, firstly, the size of the images in the four data sets is adjusted to 286 × 286 by a bilinear interpolation method, then each image is randomly cut into 256 × 256, and finally, the pixel values of the images are normalized. When the model is tested, the data processing procedure is the same as that of training.

The second step is that:

in the method, an Acne04 data set is a source domain data set introduced by us, and three data sets of Acnehdu, Acnehdu P and AcnePGP are respectively used as target domain data sets. The following procedures are described with the Acne04 and Acnehdu datasets as examples.

First, a sample image in the source domain data set Acne04, denoted as x _s Through a generator network

To obtain a reaction product of _s Corresponding generated image, denoted x _s2t (ii) a Second, sample image x in the target domain dataset Acnehdu _t Through a generator network

To obtain a reaction product of _t Corresponding generated image, denoted x _t2s . In the source domain x _s And x _t2s All participate in the training of classification models in the source domain; in the target domain x _t And x _s2t Will participate in the training of the classification model in the target domain.

Then, x in the source domain _s And x _t2s Pass through discriminator D _s In the target domain x _t And x _s2t Pass through discriminator D _t 。D _s And D _t It is discriminated whether the sample is a real image or a generated image. Meanwhile, a pair of discriminators with the same function is also arranged in the feature space extracted by the classification network.

The third step:

we first tested different structures in three data sets and the results are shown in table 1 below. We have found that the optimal gating network structure is different in different target domain datasets due to the presence of domain offsets. At the same time, we evaluated the number of two important parametric expert networks (N) in the gated classification network _e ) And number of subnetworks (N' _t ) The results are shown in Table 2. We found 1) optimal N' _t May be different from the number of skin tone categories and the number of sub-networks (N' _t ) The larger the more readily better performance is obtained; 2) optimal number of expert networks (N) _e ) Tendency toFrom less than the number of optimal sub-networks (N' _t ) This means that the relevance and effectiveness of the underlying feature space is shared across multiple skin tones. The optimal number of subnetworks and expert networks for a particular data set is adaptively learned using a gating network in our method.

TABLE 1 ablation experiments on gated classification network modules of different structures

The fourth step:

we impose a multinuclear maximum mean difference (MK-MMD) penalty on the generated and real image datasets to achieve cross-domain alignment. On acneggp, we first tested different numbers of nuclei, K, and alignment structures using gaussian nuclei under a model with cross-domain data enhancement. The results are shown in table 2:

TABLE 2 Multi-core maximum mean Difference loss model experiment

Type (B)	Only in the source domain	Only in the target domain	Federated sharing	Federated unshared
					K＝3	0.872	0.849	0.847	0.857
K＝5	0.857	0.766	0.857	0.852
					K＝8	0.864	0.766	0.766	0.849

In particular, we build MK-MMD penalties on only the source domain, only the target domain, the dual domain sharing the mapping, and the dual domain not sharing the mapping, respectively. We observed that the alignment structure had a significant impact on the performance results, and to achieve better performance we applied the best alignment structure for each dataset under the three gaussian kernels.

Claims

1. A self-adaptive skin color cross-domain self-photographing face pockmark grading image classification method is characterized by comprising the following steps:

step (1): preprocessing image data, and unifying the sizes of images in a data set;

step (2): enhancing cross-domain data, introducing a publicly available data set as a source domain to help the learning of a classifier in a target domain;

and (3): constructing a self-adaptive skin color gating classification model;

and (4): aligning the feature space, and designing domain alignment loss based on features to make up domain difference;

and (5): training and testing a model; establishing a multi-task end-to-end deep learning model, and training network parameters on a specific data set through a back propagation algorithm until the whole network model converges.

2. The method for classifying facial pox-classified images through self-adaptive skin color cross-domain self-timer face according to claim 1, wherein said cross-domain data enhancement of step (2) is performed as follows:

Wherein y is _i E.g. Y and

are respectively an image x _i And

class label of (1), (2), (…), N _c Denotes the category label space, N _c Is the total number of categories;

The specific formula is defined as follows:

wherein x is _i ∈I ^s ，

Refers to a loss function; | | non-woven hair ₁ The expression is taken as 1 norm;

And

and a pair of feature discriminators

And

for discriminating whether a sample in the source domain that has passed through the image discriminator network is a real image or a generated image; for discriminating whether a sample in the target domain that has passed through the image discriminator network is a real image or a generated image;

for judging extraction through classified network in target domainWhether the feature is from a real image sample or a generated image sample;

the discrimination loss in the image space and the feature space is specifically shown as the following formula:

wherein

Representing a classification network, wherein s represents a source domain, t represents a target domain, and d takes the values of s and t;

wherein l _i A real label representing the ith sample image; when used to calculate the discriminant loss of image space

When the temperature of the water is higher than the set temperature,

when used to compute the discriminant loss of the feature space

When the temperature of the water is higher than the set temperature,

ψ (x) is an intermediate parameter variable.

3. The method for classifying facial pox-classified images of self-adaptive skin color cross-domain self-timer face according to claim 2, wherein said step (3) of constructing an adaptive gating classification model comprises the following steps:

designing an expert gating network and a skin color gating network to respectively and adaptively learn the optimal sample weights of the expert network and the sub-network so as to solve skin color label noise; and the number of sub-networks set is different from the number of skin color categories, thereby destroying the one-to-one mapping between the number of sub-networks and the number of skin color categories;

firstly, a sample obtains corresponding eigenvectors through a plurality of parallel expert networks, meanwhile, the sample can self-adaptively learn a gating weight vector through the expert gating networks, and the gating weight vector is multiplied by the eigenvectors to obtain a final weight combined eigenvector;

then, the feature vector of the weight combination passes through a plurality of sub-networks simultaneously, and correspondingly outputs features extracted by the sub-networks; and multiplying the features extracted by the sub-network with the skin color ratio weight vector obtained by the sample through the skin color gating network in a self-adaptive manner to obtain the final combined feature.

4. The method for classifying facial pox-classified images of self-adaptive skin color cross-domain self-timer face according to claim 3, characterized by adding the following technical characteristics:

there is a classification network in both the source domain and the target domain, so the adaptive skin color gating network has three structures, namely a classification network only in the source domain, a classification network only in the target domain, and a classification network combined in the source domain and the target domain, which are specifically defined as follows:

wherein, N' _t Indicating the number of subnetworks, N _e Which is indicative of the number of expert networks,

represents the ith sub-network full connection layer, M represents the aggregation layer，E _j Representing the jth expert network, G _j Represents N _e The jth element, W, of the dimensional weight vector _i Is N' _t The ith element of the weight vector is dimensioned.

Where τ (x) represents the skin tone category label of sample x, e _τ(x) Is N' _t The τ (x) -th element of the dimensional weight vector is 1;

is N' _t The second of the dimensional weight vector

Each element is 1; γ and γ' represent weight parameters of the real image and the generated image, respectively.

5. The method for classifying facial pox images of self-portrait across domains according to claim 3 or 4, wherein said step (4) of said image feature space alignment comprises the following steps:

using multi-kernel maximum mean difference loss for real image samples and generated image samples in respective domains in a source domain and a target domain respectively; thus, a cross-domain generated image from another domain is more compatible with a real image of a particular domain, ensuring the reasonableness of sharing the same classification model on the real image and the generated image; the maximum mean difference loss for polynuclear is specifically defined as follows:

wherein

Is a weight parameter that controls the source domain,

2 norm is taken; e represents expectation; the maximum mean difference penalty for multiple cores may be applied to a single domain or to two domains with shared or unshared mapping functions.