CN117292217A - Skin typing data augmentation method and system based on countermeasure generation network - Google Patents

Abstract

The invention discloses a skin classification data augmentation method and system based on adversarial generation network, which is used to accurately classify skin pathology types. Due to the small number of skin pathology images that can be collected, over-fitting problems will occur when directly used for training classifier models, resulting in unsatisfactory classification results. It is necessary to use adversarial generation networks to generate realistic skin pathology images to alleviate this shortcoming. . This method performs image segmentation on the skin pathology data set, then performs data augmentation through an adversarial generation network, and uses multi-level feature fusion technology to classify skin pathology. The main steps of the present invention include: first, collecting skin pathology image data, performing image segmentation and manually completing label operations; then, constructing an adversarial generation network to generate high-quality skin pathology images to expand the available data set; finally, using multi-branch feature extraction The network extracts features and models the target image, and uses support vector machines to achieve accurate skin classification.

Description

A skin typing data augmentation method and system based on adversarial generative network

Technical field

The invention relates to the fields of data analysis and medical technology, and in particular to a method and system for augmenting skin classification data based on adversarial generation networks.

Background technique

Skin pathology classification is one of the important research contents in the field of clinical dermatology. Classifying and describing skin lesions can provide an important reference for the diagnosis and treatment of skin diseases. The type of skin pathology can be assessed by observing multi-colored patches and surface ulcers on the skin surface, and then the type of skin pathology can be initially determined. Under normal circumstances, image recognition technology can be used to classify four common types of pathological skin, namely squamous cell skin cancer, basal cell skin cancer, malignant melanoma, and Merkel cell skin cancer.

Currently, skin typing is mainly achieved through visual inspection. However, due to the specificity and diversity of skin types, traditional methods have exposed many flaws, and there are also certain differences in the diagnostic results of different doctors. This subjectivity and uncertainty have led to the reliability and uncertainty of artificial skin typing. Less accurate.

In order to solve the above problems, more and more researchers have begun to explore how to use computer vision technology to achieve skin classification. Compared with human eyes observing skin images, computer vision technology can automatically extract features from a large number of skin images and classify and predict them through machine learning models. This method can reduce the negative impact of subjectivity and improve the accuracy and reliability of skin typing. However, in the field of skin typing, the application of data augmentation technology is greatly limited due to the lack of large-scale annotated data sets. In deep learning, the larger the amount of data, the better the performance of the trained model. Therefore, in the absence of large-scale data sets, how to improve the accuracy and reliability of skin classification through data augmentation technology is an urgent problem that needs to be solved.

Data augmentation technology based on Generative Adversarial Networks (GAN) can improve the classification performance and generalization ability of the model by generating new synthetic data samples. GAN consists of two parts: a generator and a discriminator. The generator generates new synthetic data samples by learning the distribution of the training data set, and the discriminator is used to determine whether the generated data samples are real. By training the generator and discriminator, GAN can generate high-quality synthetic data samples, thereby expanding the size of the original data set and improving the robustness and generalization ability of the model.

Therefore, using data augmentation technology based on adversarial generative networks to achieve skin classification has very important application value. Using GAN to generate more synthetic data can expand the size of the data set and improve the accuracy and reliability of skin classification without increasing the cost of data collection. By analyzing and processing skin images using computer vision technology, efficient classification of skin pathology types can be achieved, providing an important reference for the diagnosis and treatment of skin pathology in clinical medicine.

Contents of the invention

In view of this, one of the purposes of the present invention is to provide a skin typing data augmentation method based on an adversarial generative network. The adversarial generative network model is used to achieve skin typing data augmentation, which can be used when the original image data is small or the skin is small. Provides diverse samples when images are complex to assist medical judgment and improve skin diagnosis efficiency.

One of the purposes of the present invention is achieved through the following technical solutions:

This method of skin typing data augmentation based on adversarial generation network includes the following steps:

Step S1: Collect and preprocess the original skin image data;

Step S2: Construct an adversarial generative network model to generate high-quality synthetic skin pathology image samples;

Step S3: Segment the generated skin pathology image and synthesize it with the healthy skin image to achieve data augmentation;

Step S4: Train an adversarial generative network model to expand the size of the original skin pathology image data set;

Step S5: Construct a multi-branch feature extraction network to extract multi-level structural features of skin pathology images and achieve skin classification.

Further, the original skin image data is collected and preprocessed, including:

Step S101: Obtain multiple types of skin images from medical image databases, online skin image databases, large medical research institutions and other channels. The main information of the skin images includes: skin lesion appearance, skin lesion shape, and skin lesion size. , skin lesion types, skin lesion grading;

Step S102: Map the skin image into an undirected graph, treat each pixel as a node, and then connect each pixel node to form an edge. The weight of each edge represents the texture difference between pixels to build grayscale symbiosis. The matrix P calculates the texture differences that exist between different pixels and is constructed as follows:

In the above formula, i represents the current pixel, j represents the adjacent pixel of i pixel, (x, y) represents the coordinate of the current pixel i, αx and βy represent the coordinate offset between the two pixels, and θ represents the pixel pair ( The frequency of occurrence of i, j) in the image, W represents the width of the image, H represents the length of the image, G_value(·) is the grayscale calculation function, which represents the grayscale value at the coordinates (x, y) in the image;

Use the texture feature function to calculate the sum of squares of gray value differences between pixels, which is used to reflect the degree of texture difference between pixels. The calculation method is as follows:

D(i,j)＝∑ _i,j [G_value(x,y)-G_value(x+αx,y+βy)] ² ×I(G_value(x,y)＝i)×I(G_value(x+ αx,y+βy)＝j)

In the above formula, D(·) represents the difference degree calculation function, and I(·) represents the indicator function. When I(·) is true, the value is 1, otherwise the value is 0;

According to the pixel texture difference measurement formula, the weight of each edge in the undirected graph is calculated as follows:

Step S103: Add two additional node sets in the constructed undirected graph, namely the source node set S and the sink node set T, which are connected to each vertex in the undirected graph to form an edge. Calculate and divide the nodes in the graph into two The minimum cost of a set is calculated as follows:

Cost＝min∑ _{(u,v)∈I,s∈S,t∈T} c(u,v)

In the above formula, min represents the minimum value, u and v represent any two edges in the undirected graph, c(·) represents the path flow calculation function, I represents the skin image, S represents the source node set, and T represents the sink node Set, by finding the augmented path that exists between the source node and the sink node, the weight of each edge on the path is included in the flow of the augmented path. When the augmented path cannot be found, it represents the flow of the current path. When the maximum value is reached, the image is segmented according to the maximum flow path to retain the pathological areas in the image;

Step S104: Use linear interpolation method to complete the image into a distorted image block of 512×512 fixed size to make up for the defect of inconsistent size of the segmented image. The completion calculation formula is as follows:

Pixel'(x _m ,y _m )=(1-U)*(1-V)*Pixel(x _m ,y _m )+U*(1-V)*Pixel(x _m ,y _m )+(1 -U)*V*Pixel(x _m ,y _m )

Pixel(x _m ,y _m )＝w ₁ *R+w ₂ *G+w ₃ *B

In the above formula, (x _m , y _m ) represents the coordinate value of the missing pixel, (x ₀ , y ₀ ) represents the coordinate origin, (x _r , y _r ) represents the coordinate value of the complete pixel near the randomly selected missing pixel, U represents the horizontal offset of the missing pixel, V represents the vertical offset of the missing pixel, Pixel'(·) represents the image grayscale estimation function, Pixel(·) is the image grayscale calculation function, and uses the weighted average method to calculate the target pixel The gray value of w ₁ , w ₂ , and w ₃ represents the weight of the red component, green component, and blue component respectively. It should be noted that the weights of the three color components can be adjusted according to the actual situation and the sum of the weights is 1;

Step S105: Divide the completed image into blocks, use a sliding window to traverse the image and crop the image into a 256×256 fixed-size image G _path , and add corresponding labels to the cropped image. If the original image contains The label information adds corresponding labels based on the original information. If the original image lacks or omits label information, it will be manually identified and specific label information will be added.

Furthermore, an adversarial generative network model is constructed to generate high-quality synthetic skin pathology image samples, including:

Step S201: Construct a generator network with a symmetric structure, and learn the latent features between the input image and the corresponding reference image. The constructed generator network consists of 3 convolutional layers. The first convolutional layer contains 32 pixels of size The convolution kernel is 7×7 and the convolution step is set to 1. The second convolution layer contains 64 convolution kernels of size 5×5 and the convolution step is set to 1. The third convolution layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; then, set up 3 residual network units composed of 1 layer of convolution layer and 1 layer of ReLU activation function, and the residual network unit is convolved. The convolutional layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; finally, 3 transposed convolution layers are set up to achieve feature sampling and image recovery. The first transposed convolution layer contains 128 convolution kernels of size 3 × 3 and the convolution step size is set to 1. The second transposed convolution layer contains 64 convolution kernels of size 5 × 5 and the convolution step size is set to 1. The three transposed convolutional layers contain 32 convolution kernels with a size of 7×7 and the convolutional layer stride is set to 1. The output features of the last transposed convolutional layer are fused with the input image to obtain the final output. Image G _gen , the expression of the convolution operation is as follows:

In the above formula, Represents the feature map output by the current channel, K represents the size of the convolution kernel, W represents the width of the image, H represents the height of the image, /> Represents the feature map of the current channel input, x, y represents the coordinate position of the output feature map in channel n, w, h represents the element coordinates of the convolution kernel weight matrix in channel n;

Step S202: Construct a discriminator network composed of 5 convolutional layers, which can extract different features in the input image and identify different types of skin. The convolutional layers in the discriminator network all use convolutions with a size of 3×3. Kernel, each convolutional layer is added with a batch normalization layer. In order, the number of convolutional layers included in the convolutional layer are 32, 64, 128, 256, 1 respectively. The first four convolutional layers use ReLU activation. function, the last convolutional layer uses the Sigmoid activation function to output the probability value of whether the corresponding input image is the generated image.

Furthermore, the generated skin pathology images are segmented and synthesized with healthy skin images to achieve data augmentation, including:

Step S301: Traverse each pixel of the image G _gen and obtain the corresponding gray value, and count the pixel frequency of each gray value to construct a gray histogram corresponding to the image G _gen . The expression form of the gray histogram is as follows:

k(l)=f(l)×m ^-1

In the above formula, k(l) represents the frequency of occurrence of pixels with gray level l in the image, f(l) represents the frequency of occurrence of pixels with gray level l in the image, and m represents the number of pixels in the image;

Step S302: Calculate the inter-class variance corresponding to different gray level thresholds, obtain the threshold δ to maximize the difference between the disease texture and normal skin, and convert the image G _gen into the binary image G _bin according to the threshold δ, calculation method as follows:

δ=max{δ ₁ , δ ₂ ,..., δ _i }, i=h(l)

In the above formula, δ _i represents the threshold for constructing a binary image based on the i-th gray level, Represents the inter-class variance of the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 0 in the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 255 in the image when the threshold value is δ _i , f(l) indicates the frequency of occurrence of pixels with a gray level of l in the image, m indicates the number of pixels in the image, h(l) indicates the image The number of medium gray levels, max{·} is the maximum operator, and δ represents the gray level threshold corresponding to the maximum inter-class variance;

Step S303: There are obvious pixel differences between the skin part and the pathological part in the image G _gen obtained by the generator. Use the binary image G _bin to segment the image G _gen to obtain a more refined pathological image G _acc , and then input it to the discriminator. middle;

Step S304: Use the pathological image generated by the generator network to synthesize the healthy skin. The edge area of the pathological image is significantly different from the pixels of the healthy skin. Use mean filtering to smooth the edges of the pathological image. The image edge mean filter smoothing formula is as follows :

G _i '=λ×G _i +(1-λ)×G _j

In the above formula, G _i ' represents the filtered pixel value, G _i represents the pixel value to be filtered, G _j represents the pixel value of the pixel adjacent to G _i , and λ represents the weight controlling image filtering;

Step S305: Use the bilinear interpolation method to perform multi-scale processing on the synthesized image to construct an image pyramid structure to ensure that the synthesized image can retain the main pathological features. The position coordinates of the pixels and the pixel value are calculated as follows:

In the above formula, l _i represents the coordinate value of the target point to be calculated in the image, x _src represents the abscissa of the image after scaling, y _src represents the abscissa of the image after scaling, x _dst represents the abscissa of the image before scaling, y _dst represents the ordinate of the image before scaling, W _src represents the width of the image after scaling, W _dst represents the width of the image before scaling, H _src represents the width of the image after scaling, H _dst represents the width of the image before scaling, G _i represents the target to be calculated The pixel value of the point, G _j represents the pixel value of the adjacent pixel point of the target point to be calculated, and l _j represents the coordinate value of the adjacent pixel point of the target point to be calculated in the image.

Furthermore, an adversarial generative network model is trained to expand the size of the original skin pathology image data set, including:

Step S401: Use the residual loss function to measure the dissimilarity between the generated image G _acc and the original image G _path . Ideally, the number of pixel patches between the generated image and the original image is the same, the residual loss value is 0, and the residual The formula of the loss function is as follows:

In the above formula, Loss _gen (·) represents the residual loss function, P _G∈Data (G _path ) represents the probability that the input image belongs to real image data, s represents the number of images, and log(·) represents the logarithmic function;

Step S402: Use the feature loss function to identify the feature difference between the generated image G _acc and the original image G _path . The formula of the feature loss function is as follows:

Loss _dis (G _acc ,G _path )＝-[log(P _G∈Data (G _path ))+log(1-P _G∈Gen (G _acc ))]

In the above formula, Loss _dis (·) represents the feature loss function, and P _G∈Gen (G _acc ) represents the probability that the input image belongs to the sample created by the generator;

Step S403: Weighted splicing of the two loss functions, mapping the target skin image to the latent space, and using the Adam optimizer to update the parameters in the generator network and discriminator network to help the two networks reach a convergence state faster during alternate training. To improve the stability and reliability of the model and finally achieve convergence, the loss function splicing formula is as follows:

Loss(G _acc ,G _path )＝α ₁ *Loss _gen (G _acc ,G _path )+β ₁ *Loss _dis (G _acc ,G _path )

In the above formula, α ₁ represents the parameter that controls the weight of the loss function corresponding to the generator network, β ₁ represents the parameter that controls the weight of the loss function corresponding to the discriminator network, and the sum of α ₁ and β ₁ is 1.

Furthermore, a multi-branch feature extraction network is constructed to extract multi-level structural features of skin pathology images and achieve skin classification, including:

Step S501: Construct a multi-branch feature extraction network containing 4 residual blocks. Each residual block includes a layer of atrous convolution layer, a layer of batch normalization layer and a layer of adaptive average pooling layer. The atrous convolution layer is calculated as follows:

In the above formula, q _i represents the feature tensor of the output image of the i-th residual block, q _i-1 represents the feature tensor of the output image of the i-1 residual block, and z represents the convolution kernel in the dilated convolution layer. The size of , p represents the filling parameter, b represents the moving step size, and eta represents the expansion rate of atrous convolution;

Step S502: Perform feature fusion on image features at different levels to avoid losing important features of skin pathology images and improve the accuracy of skin classification. The calculation method of feature fusion is as follows:

In the above formula, Q represents the feature tensor output after feature fusion, q _i represents the feature tensor output by the i-th residual block, and σ _i represents the feature weight corresponding to the i-th residual block;

Step S503: The fused feature tensor Q is normalized and encoded according to the minimum-maximization principle, and the element value range of the feature tensor is mapped to the interval of [0,1]. The normalization calculation method is as follows:

In the above formula, Q' represents the normalized feature tensor, min(Q) represents the minimum value of all elements in tensor Q, max(Q) represents the maximum value of all elements in tensor Q;

Step S504: Select the kernel function to train the support vector machine model to classify the feature tensor Q', obtain the category label of the target skin image, and complete the skin classification.

The second object of the present invention is to provide a system for skin typing data augmentation based on adversarial generation network, including a memory, a processor and a computer program stored in the memory and capable of running on the processor. The processor executes The computer program implements the method as described above.

A third object of the present invention is to provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method as described above is implemented.

The beneficial effects of the present invention are:

(1) The method of the present invention uses a generative adversarial network model to achieve augmentation of skin classification data, which can provide diverse samples when the amount of original image data is small or the skin image is complex, without affecting the quality of the data. Under certain conditions, it can greatly reduce the over-fitting of the prediction model and provide effective clinical decision-making support for medical staff to diagnose patients' skin conditions;

(2) The present invention uses image binarization method to solve the problem of large scale of generated images when segmenting and generating pathological images. Binarizing image pixels can accurately and intuitively separate pathological skin areas from healthy areas, making it more convenient Extract salient features of skin pathological areas. In addition, the binary method has low time complexity, which is beneficial to reducing the time overhead required for adversarial generation network training and ensuring the timeliness of skin classification results;

(3) The present invention uses the mean filtering method to solve the problem of image non-smoothness when synthesizing pathological images. It achieves the purpose of smoothing the image by calculating the average value of neighborhood pixels around the pixel, effectively reducing noise pixels and making the overall image presentation more smooth. The smooth appearance is beneficial to the adversarial generation network to generate more realistic sample images, ensuring the accuracy of skin classification in the next step;

(4) The present invention adds additional synthetic samples when training the adversarial generation network, and synthesizes additional new samples through pathological skin images and healthy skin images generated by the generator, increasing the diversity of training data and providing specific categories of skin pathologies. The data set provides more sufficient example samples, which is beneficial to improving the generalization ability of the adversarial generation network, thereby improving the generation ability of the adversarial generation network on this type of skin pathology;

(5) The present invention uses a multi-branch network structure when extracting features at different levels of the image. Each branch focuses on a different receptive field size, which can effectively capture local and global feature information in the image, and adopt feasible methods during the feature fusion process. Compared with fixed feature weights, adjusted feature weights can better improve the expressive ability of images and effectively improve the accuracy of skin classification tasks.

Other advantages, objects, and features of the present invention will, to the extent that they are set forth in the description that follows, and to the extent that they will become apparent to those skilled in the art upon examination of the following, or may be derived from This invention is taught by practicing it. The objectives and other advantages of the invention may be realized and attained by the following description and the preceding claims.

Description of drawings

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the accompanying drawings, in which:

Figure 1 is a schematic flow diagram of a skin typing data augmentation method and system flow based on an adversarial generation network according to the present invention;

Figure 2 is a schematic diagram of raw skin image data implemented in the present invention;

Figure 3 is a schematic diagram of the maximum flow path image segmentation implemented in the present invention;

Figure 4 is a schematic diagram of the implementation of the present invention to build a generator network;

Figure 5 is a schematic diagram of the discriminator network constructed by the implementation of the present invention;

Figure 6 is a schematic diagram of the implementation of the present invention to construct a binary image;

Figure 7 is a schematic diagram of the implementation of binary image segmentation according to the present invention;

Figure 8 is a schematic diagram of the mean filtering and smoothing synthetic image implemented in the present invention;

Figure 9 is a schematic diagram of scaling image features using the image pyramid strategy according to the present invention.

Detailed ways

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention and are not intended to limit the scope of the present invention.

The present invention is a skin typing data augmentation method and system based on adversarial generation network, as shown in Figure 1, including the following steps:

Step S1: Collect and preprocess the original skin image data;

Step S2: Construct an adversarial generative network model to generate high-quality synthetic skin pathology image samples;

Step S3: Segment the generated skin pathology image and synthesize it with the healthy skin image to achieve data augmentation;

Step S4: Train an adversarial generative network model to expand the size of the original skin pathology image data set;

Step S5: Construct a multi-branch feature extraction network to extract multi-level structural features of skin pathology images and achieve skin classification.

The specific steps of the above method will be further explained below through a specific embodiment.

In this embodiment, step S1 specifically includes the following steps:

Step S101: Collect various types of skin images from medical image databases, online skin image databases, large medical research institutions and other channels and save them to the MongoDB database to establish a skin image database. The main information of the skin images includes: skin lesions Appearance, skin lesion shape, skin lesion size, skin lesion type, and skin lesion grade, as shown in Figure 2. The data acquisition channels include but are not limited to medical image databases, online skin image databases, large medical research institutions, etc.;

Step S102: Map the skin image into an undirected graph, treat each pixel as a node, and then connect each pixel node to form an edge. The weight of each edge represents the texture difference between pixels to build grayscale symbiosis. The matrix P calculates the texture differences that exist between different pixels and is constructed as follows:

In the above formula, i represents the current pixel, j represents the adjacent pixel of i pixel, (x, y) represents the coordinate of the current pixel i, αx and βy represent the coordinate offset between the two pixels, and θ represents the pixel pair ( The frequency of occurrence of i, j) in the image, W represents the width of the image, H represents the length of the image, G_value(·) is the grayscale calculation function, which represents the grayscale value at the coordinates (x, y) in the image;

Use the texture feature function to calculate the sum of squares of gray value differences between pixels, which is used to reflect the degree of texture difference between pixels. The calculation method is as follows:

D(i,j)＝∑ _i,j [G_value(x,y)-G_value(x+αx,y+βy)] ² ×I(G_value(x,y)＝i)×I(G_value(x+ αx,y+βy)＝j)

In the above formula, D(·) represents the difference degree calculation function, and I(·) represents the indicator function. When I(·) is true, the value is 1, otherwise the value is 0;

According to the pixel texture difference measurement formula, the weight of each edge in the undirected graph is calculated as follows:

Step S103: Add two additional node sets in the constructed undirected graph, namely the source node set S and the sink node set T, which are connected to each vertex in the undirected graph to form an edge. Calculate and divide the nodes in the graph into two The minimum cost of a set is calculated as follows:

Cost＝min∑ _{(u,v)∈I,s∈S,t∈T} c(u,v)

In the above formula, min represents the minimum value, u and v represent any two edges in the undirected graph, c(·) represents the path flow calculation function, I represents the skin image, S represents the source node set, and T represents the sink node Set, by finding the augmented path that exists between the source node and the sink node, the weight of each edge on the path is included in the flow of the augmented path. When the augmented path cannot be found, it represents the flow of the current path. Reaching the maximum value, the image is segmented according to the maximum flow path to retain the pathological area in the image, as shown in Figure 3;

Step S104: Use linear interpolation method to complete the image into a distorted image block of 512×512 fixed size to make up for the defect of inconsistent size of the segmented image. The completion calculation formula is as follows:

Pixel'(x _m ,y _m )=(1-U)*(1-V)*Pixel(x _m ,y _m )+U*(1-V)*Pixel(x _m ,y _m )+(1 -U)*V*Pixel(x _m ,y _m )

Pixel(x _m ,y _m )＝0.299R+0.587G+0.114B

In the above formula, (x _m , y _m ) represents the coordinate value of the missing pixel, (x ₀ , y ₀ ) represents the coordinate origin, (x _r , y _r ) represents the coordinate value of the pixel that is randomly selected near the missing pixel, and U represents The horizontal offset of the missing pixel, V represents the vertical offset of the missing pixel, Pixel'(·) represents the image grayscale estimation function, Pixel(·) is the image grayscale calculation function, and the weighted average method is used to calculate the target pixel For grayscale values, the weight of the red component R is set to 0.299, the weight of the green component G is set to 0.587, and the weight of the blue component B is set to 0.114. It should be noted that the weights of the three color components can be adjusted according to the actual situation and the weights The sum is 1;

Step S105: Divide the completed image into blocks, use a sliding window to traverse the image and crop the image into a 256×256 fixed-size image G _path , and add corresponding labels to the cropped image. If the original image contains The label information adds corresponding labels based on the original information. If the original image lacks or omits label information, manual identification will add specific label information. Specifically, if the original image is manually identified as squamous cell skin cancer, the image will be manually identified. Add a skin pathology type tag with the value "Squamous Cell Skin Cancer".

In this embodiment, step S2 specifically includes the following steps:

Step S201: Construct a generator network with a symmetric structure, and learn the latent features between the input image and the corresponding reference image. The constructed generator network consists of 3 convolutional layers. The first convolutional layer contains 32 pixels of size The convolution kernel is 7×7 and the convolution step is set to 1. The second convolution layer contains 64 convolution kernels of size 5×5 and the convolution step is set to 1. The third convolution layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; then, set up 3 residual network units composed of 1 layer of convolution layer and 1 layer of ReLU activation function, and the residual network unit is convolved. The convolutional layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; finally, 3 transposed convolution layers are set up to achieve feature sampling and image recovery. The first transposed convolution layer contains 128 convolution kernels of size 3 × 3 and the convolution step size is set to 1. The second transposed convolution layer contains 64 convolution kernels of size 5 × 5 and the convolution step size is set to 1. The three transposed convolutional layers contain 32 convolution kernels with a size of 7×7 and the convolutional layer stride is set to 1. The output features of the last transposed convolutional layer are fused with the input image to obtain the final output. Image G _gen , as shown in Figure 4, the expression of the convolution operation is as follows:

In the above formula, Represents the feature map output by the current channel, K represents the size of the convolution kernel, W represents the width of the image, H represents the height of the image, /> Represents the feature map of the current channel input, x, y represents the coordinate position of the output feature map in channel n, w, h represents the element coordinates of the convolution kernel weight matrix in channel n;

Step S202: Construct a discriminator network composed of 5 convolutional layers, extract different features in the input image, and identify different types of skin. The convolutional layers in the discriminator network all use convolution kernels with a size of 3×3. , each convolutional layer is added with a batch normalization layer. In order, the number of convolutional layers included in the convolutional layer are 32, 64, 128, 256, and 1 respectively. The first four convolutional layers use ReLU activation. function, the last convolutional layer uses the Sigmoid activation function to output the probability value of whether the input image is a generated image, as shown in Figure 5.

Step S3 specifically includes the following steps:

Step S301: Traverse each pixel of the image G _gen and obtain the corresponding gray value, and count the pixel frequency of each gray value to construct a gray histogram corresponding to the image G _gen . The expression form of the gray histogram is as follows:

k(l)=f(l)×m ^-1

In the above formula, k(l) represents the frequency of occurrence of pixels with gray level l in the image, f(l) represents the frequency of occurrence of pixels with gray level l in the image, m represents the number of pixels in the image, For example, the frequency f(0) of pixels with gray level 0 in the image is 28, and the number of pixels m in the image is 625, then the frequency k(0) of pixels with gray level 0 in the image is 0.0448;

Step S302: Calculate the inter-class variance corresponding to different gray level thresholds, obtain the threshold δ to maximize the difference between the disease texture and normal skin, and convert the image G _gen into the binary image G _bin according to the threshold δ, calculation method as follows:

δ=max{δ ₁ , δ ₂ ,..., δ _i }, i=h(l)

In the above formula, δ _i represents the threshold for constructing a binary image based on the i-th gray level, Represents the inter-class variance of the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 0 in the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 255 in the image when the threshold value is δ _i , f(l) indicates the frequency of occurrence of pixels with a gray level of l in the image, m indicates the number of pixels in the image, h(l) indicates the image The number of medium gray levels, max{·} is the maximum operator, and δ represents the gray level threshold corresponding to the maximum inter-class variance, as shown in Figure 6;

Step S303: There are obvious pixel differences between the skin part and the pathological part in the image G _gen obtained by the generator. Use the binary image G _bin to segment the image G _gen to obtain a more refined pathological image G _acc and then input it into the discriminator. , as shown in Figure 7;

Step S304: Use the pathological image generated by the generator network to synthesize the healthy skin. The pixel difference between the edge area of the pathological image and the healthy skin is obvious. Use mean filtering to smooth the edges of the pathological image. The image edge mean filter smoothing formula as follows:

G _i '=λ×G _i +(1-λ)×G _j

In the above formula, G _i ' represents the filtered pixel value, G _i represents the pixel value to be filtered, G _j represents the pixel value of the pixel adjacent to G _i , and λ represents the weight that controls image filtering, with a value of 0.625. Specifically, if the pixel value of G _i to be filtered is 220 and the pixel value of its adjacent pixel point G _j is 176, then the pixel value of G _i ' after filtering is 203.5, as shown in Figure 8;

Step S305: Use the bilinear interpolation method to perform multi-scale processing on the synthesized image to construct an image pyramid structure to ensure that the synthesized image can retain the main pathological features. The position coordinates of the pixels and the pixel value are calculated as follows:

In the above formula, l _i represents the coordinate value of the target point to be calculated in the image, x _src represents the abscissa of the image after scaling, y _src represents the abscissa of the image after scaling, x _dst represents the abscissa of the image before scaling, y _dst represents the ordinate of the image before scaling, W _src represents the width of the image after scaling, W _dst represents the width of the image before scaling, H _src represents the width of the image after scaling, H _dst represents the width of the image before scaling, G _i represents the target to be calculated The pixel value of the point, G _j represents the pixel value of the adjacent pixel point of the target point to be calculated, and l _j represents the coordinate value of the adjacent pixel point of the target point to be calculated in the image, as shown in Figure 9.

Step S4 specifically includes the following steps:

Step S401: Use the residual loss function to measure the dissimilarity between the generated image G _acc and the original image G _path . Ideally, the number of patches between the generated image and the original image is the same, the residual loss value is 0, and the residual loss The formula of the function is as follows:

In the above formula, Loss _gen (·) represents the residual loss function, P _G∈Data (G _path ) represents the probability that the input image belongs to real image data, s represents the number of images, and log(·) represents the logarithmic function;

Step S402: Use the feature loss function to identify the feature difference between the generated image G _acc and the original image G _path . The formula of the feature loss function is as follows:

Loss _dis (G _acc ,G _path )＝-[log(P _G∈Data (G _path ))+log(1-P _G∈Gen (G _acc ))]

In the above formula, Loss _dis (·) represents the feature loss function, and P _G∈Gen (G _acc ) represents the probability that the input image belongs to the sample created by the generator;

Step S403: Weighted splicing of the two loss functions, mapping the target skin image to the latent space, and using the Adam optimizer to update the parameters in the generator network and discriminator network to help the two networks reach a convergence state faster during alternate training. To improve the stability and reliability of the model and finally achieve convergence, the loss function splicing formula is as follows:

Loss(G _acc ,G _path )＝α ₁ *Loss _gen (G _acc ,G _path )+β ₁ *Loss _dis (G _acc ,G _path )

In the above formula, α ₁ represents the parameter that controls the weight of the loss function corresponding to the generator network, and the value is 0.3. β ₁ represents the parameter that controls the weight of the loss function corresponding to the discriminator network. The value is 0.7. The sum of α ₁ and β ₁ is 1; Specifically, the value of Loss _gen (G _acc ,G _path ) is calculated in step S401 to be 26.523; the value of Loss _dis (G _acc ,G _path ) is calculated in step S402 to be 1.046; finally, the value of Loss (G _acc ,G path ) is calculated. _path ) value is 8.689.

Step S5 specifically includes the following steps:

Step S501: Build a multi-branch feature extraction network containing 4 residual blocks. Each residual block includes a dilated convolution layer, a batch normalization layer and an adaptive average pooling layer. The dilated convolution layer is calculated as follows:

In the above formula, q _i represents the feature tensor of the output image of the i-th residual block, q _i-1 represents the feature tensor of the output image of the i-1 residual block, and z represents the convolution kernel in the dilated convolution layer. size, p represents the filling parameter, b represents the moving step size, eta represents the expansion rate of dilated convolution, for example, select a convolution kernel with a size of 3×3 to extract smaller local pathological features, and select "same" for the filling parameter. is 1, the moving step size is set to 1 to ensure that the size of the output image remains unchanged, and the expansion rates of the four residual blocks take values of 1, 4, 9, and 16 respectively;

Step S502: Perform feature fusion on image features at different levels to avoid losing important features of skin pathology images and improve the accuracy of skin classification. The calculation method of feature fusion is as follows:

In the above formula, Q represents the feature tensor output after feature fusion, q _i represents the feature tensor output by the i-th residual block, and σ _i represents the feature weight corresponding to the i-th residual block;

Step S503: The fused feature tensor Q is normalized and encoded according to the minimum-maximization principle, and the element value range of the feature tensor is mapped to the interval of [0,1]. The normalization calculation method is as follows:

In the above formula, Q' represents the normalized feature tensor, min(Q) represents the minimum value of all elements in tensor Q, max(Q) represents the maximum value of all elements in tensor Q;

Step S504: Select the Gaussian kernel as the kernel function of the support vector machine, learn an appropriate decision boundary according to the characteristics and labels of the tensor, maximize the interval between different categories, and use the trained support vector machine model to compare the feature tensor Q' Classify, obtain the category label of the target skin image, and complete the skin classification.

It should be noted that the present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, based on this The idea of the invention will be subject to change in the specific implementation and scope of application. In summary, the contents of this specification should not be understood as limiting the invention.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified. Modifications or equivalent substitutions without departing from the purpose and scope of the technical solution shall be included in the scope of the claims of the present invention.

Claims (8)

1. A method of skin classification data augmentation based on adversarial generation network, characterized in that: the method includes the following steps: Step S1: Collect and preprocess the original skin image data; Step S2: Construct an adversarial generative network model to generate high-quality synthetic skin pathology image samples; Step S3: Segment the generated skin pathology image and synthesize it with the healthy skin image to achieve data augmentation; Step S4: Train an adversarial generative network model to expand the size of the original skin pathology image data set; Step S5: Construct a multi-branch feature extraction network to extract multi-level structural features of skin pathology images and achieve skin classification. 2. A method of skin typing data augmentation based on adversarial generation network according to claim 1, characterized in that: the step S1 specifically includes: Step S101: Obtain multiple types of skin images from medical image databases, online skin image databases, large medical research institutions and other channels. The main information of the skin images includes: skin lesion appearance, skin lesion shape, and skin lesion size. , skin lesion types, skin lesion grading; Step S102: Map the skin image into an undirected graph, treat each pixel as a node, and then connect each pixel node to form an edge. The weight of each edge represents the texture difference between pixels to build grayscale symbiosis. The matrix P calculates the texture differences that exist between different pixels and is constructed as follows: In the above formula, i represents the current pixel, j represents the adjacent pixel of i pixel, (x, y) represents the coordinate of the current pixel i, αx and βy represent the coordinate offset between the two pixels, and θ represents the pixel pair ( The frequency of occurrence of i, j) in the image, W represents the width of the image, H represents the length of the image, G_value(·) is the grayscale calculation function, which represents the grayscale value at the coordinates (x, y) in the image; Use the texture feature function to calculate the sum of squares of gray value differences between pixels, which is used to reflect the degree of texture difference between pixels. The calculation method is as follows: D(i,j)＝∑ _i,j [G_value(x,y)-G_value(x+αx,y+βy)] ² ×I(G_value(x,y)＝i)×I(G_value(x+ αx,y+βy)＝j) In the above formula, D(·) represents the difference degree calculation function, and I(·) represents the indicator function. When I(·) is true, the value is 1, otherwise the value is 0; According to the pixel texture difference measurement formula, the weight of each edge in the undirected graph is calculated as follows: Step S103: Add two additional node sets in the constructed undirected graph, namely the source node set S and the sink node set T, which are connected to each vertex in the undirected graph to form an edge. Calculate and divide the nodes in the graph into two The minimum cost of a set is calculated as follows: Cost＝min∑ _{(u,v)∈I,s∈S,t∈T} c(u,v) In the above formula, min represents the minimum value, u and v represent any two edges in the undirected graph, c(·) represents the path flow calculation function, I represents the skin image, S represents the source node set, and T represents the sink node Set, by finding the augmented path that exists between the source node and the sink node, the weight of each edge on the path is included in the flow of the augmented path. When the augmented path cannot be found, it represents the flow of the current path. When the maximum value is reached, the image is segmented according to the maximum flow path to retain the pathological areas in the image; Step S104: Use linear interpolation method to complete the image into a distorted image block of 512×512 fixed size to make up for the defect of inconsistent size of the segmented image. The completion calculation formula is as follows: Pixel'(x _m ,y _m )=(1-U)*(1-V)*Pixel(x _m ,y _m )+U*(1-V)*Pixel(x _m ,y _m )+(1 -U)*V*Pixel(x _m ,y _m ) Pixel(x _m ,y _m )＝w ₁ *R+w ₂ *G+w ₃ *B In the above formula, (x _m , y _m ) represents the coordinate value of the missing pixel, (x ₀ , y ₀ ) represents the coordinate origin, (x _r , y _r ) represents the coordinate value of the complete pixel near the randomly selected missing pixel, U represents the horizontal offset of the missing pixel, V represents the vertical offset of the missing pixel, Pixel'(·) represents the image grayscale estimation function, Pixel(·) is the image grayscale calculation function, and uses the weighted average method to calculate the target pixel The gray value of w ₁ , w ₂ , and w ₃ represents the weight of the red component, green component, and blue component respectively. It should be noted that the weights of the three color components can be adjusted according to the actual situation and the sum of the weights is 1; Step S105: Divide the completed image into blocks, use a sliding window to traverse the image and crop the image into a 256×256 fixed-size image G _path , and add corresponding labels to the cropped image. If the original image contains The label information adds corresponding labels based on the original information. If the original image lacks or omits label information, it will be manually identified and specific label information will be added. 3. A method of skin typing data augmentation based on adversarial generation network according to claim 1, characterized in that: the step S2 specifically includes: Step S201: Construct a generator network with a symmetric structure, and learn the latent features between the input image and the corresponding reference image. The constructed generator network consists of 3 convolutional layers. The first convolutional layer contains 32 pixels of size The convolution kernel is 7×7 and the convolution step is set to 1. The second convolution layer contains 64 convolution kernels of size 5×5 and the convolution step is set to 1. The third convolution layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; then, set up 3 residual network units composed of 1 layer of convolution layer and 1 layer of ReLU activation function, and the residual network unit is convolved. The convolutional layer contains 128 convolution kernels with a size of 3×3 and the convolution step size is set to 1; finally, 3 transposed convolution layers are set up to achieve feature sampling and image recovery. The first transposed convolution layer contains 128 convolution kernels of size 3 × 3 and the convolution step size is set to 1. The second transposed convolution layer contains 64 convolution kernels of size 5 × 5 and the convolution step size is set to 1. The three transposed convolutional layers contain 32 convolution kernels with a size of 7×7 and the convolutional layer stride is set to 1. The output features of the last transposed convolutional layer are fused with the input image to obtain the final output. Image G _gen , the expression of the convolution operation is as follows: In the above formula, Represents the feature map output by the current channel, K represents the size of the convolution kernel, W represents the width of the image, H represents the height of the image, /> Represents the feature map of the current channel input, x, y represents the coordinate position of the output feature map in channel n, w, h represents the element coordinates of the convolution kernel weight matrix in channel n; Step S202: Construct a discriminator network composed of 5 convolutional layers, extract different features in the input image, and identify different types of skin. The convolutional layers in the discriminator network all use convolution kernels with a size of 3×3. , each convolutional layer is added with a batch normalization layer. In order, the number of convolutional layers included in the convolutional layer are 32, 64, 128, 256, 1 respectively. The first four convolutional layers use the ReLU activation function. , the last convolutional layer uses the Sigmoid activation function to output the probability value of whether the corresponding input image is a generated image. 4. A method of skin typing data augmentation based on adversarial generation network according to claim 1, characterized in that: the step S3 specifically includes: Step S301: Traverse each pixel of the image G _gen and obtain the corresponding gray value, and count the pixel frequency of each gray value to construct a gray histogram corresponding to the image G _gen . The expression form of the gray histogram is as follows: k(l)=f(l)×m ^-1 In the above formula, k(l) represents the frequency of occurrence of pixels with gray level l in the image, f(l) represents the frequency of occurrence of pixels with gray level l in the image, and m represents the number of pixels in the image; Step S302: Calculate the inter-class variance corresponding to different gray level thresholds, obtain the threshold δ to maximize the difference between the disease texture and normal skin, and convert the image G _gen into the binary image G _bin according to the threshold δ, calculation method as follows: δ=max{δ ₁ , δ ₂ ,..., δ _i }, i=h(l) In the above formula, δ _i represents the threshold for constructing a binary image based on the i-th gray level, Represents the inter-class variance of the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 0 in the image when the threshold value is δ _i ,/> Indicates the number of pixels with a gray level of 255 in the image when the threshold value is δ _i , f(l) indicates the frequency of occurrence of pixels with a gray level of l in the image, m indicates the number of pixels in the image, h(l) indicates the image The number of medium gray levels, max{·} is the maximum operator, and δ represents the gray level threshold corresponding to the maximum inter-class variance; Step S303: There are obvious pixel differences between the skin part and the pathological part in the image G _gen obtained by the generator. Use the binary image G _bin to segment the image G _gen to obtain a more refined pathological image G _acc , and then input it to the discriminator. middle; Step S304: Use the pathological image generated by the generator network to synthesize the healthy skin. The edge area of the pathological image is significantly different from the pixels of the healthy skin. Use mean filtering to smooth the edges of the pathological image. The image edge mean filter smoothing formula is as follows : G _i '=λ×G _i +(1-λ)×G _j In the above formula, G _i ' represents the filtered pixel value, G _i represents the pixel value to be filtered, G _j represents the pixel value of the pixel adjacent to G _i , and λ represents the weight controlling image filtering; Step S305: Use the bilinear interpolation method to perform multi-scale processing on the synthesized image to construct an image pyramid structure to ensure that the synthesized image can retain the main pathological features. The position coordinates of the pixels and the pixel value are calculated as follows: In the above formula, l _i represents the coordinate value of the target point to be calculated in the image, x _src represents the abscissa of the image after scaling, y _src represents the abscissa of the image after scaling, x _dst represents the abscissa of the image before scaling, y _dst represents the ordinate of the image before scaling, W _src represents the width of the image after scaling, W _dst represents the width of the image before scaling, H _src represents the width of the image after scaling, H _dst represents the width of the image before scaling, G _i represents the target to be calculated The pixel value of the point, G _j represents the pixel value of the adjacent pixel point of the target point to be calculated, and l _j represents the coordinate value of the adjacent pixel point of the target point to be calculated in the image. 5. A method of skin typing data augmentation based on adversarial generation network according to claim 1, characterized in that: the step S4 specifically includes: Step S401: Use the residual loss function to measure the dissimilarity between the generated image G _acc and the original image G _path . Ideally, the number of pixel patches between the generated image and the original image is the same, the residual loss value is 0, and the residual The formula of the loss function is as follows: In the above formula, Loss _gen (·) represents the residual loss function, P _G∈Data (G _path ) represents the probability that the input image belongs to real image data, s represents the number of images, and log(·) represents the logarithmic function; Step S402: Use the feature loss function to identify the feature difference between the generated image G _acc and the original image G _path . The formula of the feature loss function is as follows: Loss _dis (G _acc ,G _path )＝-[log(P _G∈Data (G _path ))+log(1-P _G∈Gen (G _acc ))] In the above formula, Loss _dis (·) represents the feature loss function, and P _G∈Gen (G _acc ) represents the probability that the input image belongs to the sample created by the generator; Step S403: Weighted splicing of the two loss functions, mapping the target skin image to the latent space, and using the Adam optimizer to update the parameters in the generator network and discriminator network to help the two networks reach a convergence state faster during alternate training. To improve the stability and reliability of the model and finally achieve convergence, the loss function splicing formula is as follows: Loss(G _acc ,G _path )＝α ₁ *Loss _gen (G _acc ,G _path )+β ₁ *Loss _dis (G _acc ,G _path ) In the above formula, α ₁ represents the parameter that controls the weight of the loss function corresponding to the generator network, β ₁ represents the parameter that controls the weight of the loss function corresponding to the discriminator network, and the sum of α ₁ and β ₁ is 1. 6. A method of skin typing data augmentation based on adversarial generation network according to claim 1, characterized in that: the step S5 specifically includes: Step S501: Construct a multi-branch feature extraction network containing 4 residual blocks. Each residual block includes a layer of atrous convolution layer, a layer of batch normalization layer and a layer of adaptive average pooling layer. The atrous convolution layer is calculated as follows: In the above formula, q _i represents the feature tensor of the output image of the i-th residual block, q _i-1 represents the feature tensor of the output image of the i-1 residual block, and z represents the convolution kernel in the dilated convolution layer. The size of , p represents the filling parameter, b represents the moving step size, and eta represents the expansion rate of atrous convolution; Step S502: Perform feature fusion on image features at different levels to avoid losing important features of skin pathology images and improve the accuracy of skin classification. The calculation method of feature fusion is as follows: In the above formula, Q represents the feature tensor output after feature fusion, q _i represents the feature tensor output by the i-th residual block, and σ _i represents the feature weight corresponding to the i-th residual block; Step S503: The fused feature tensor Q is normalized and encoded according to the minimum-maximization principle, and the element value range of the feature tensor is mapped to the interval of [0,1]. The normalization calculation method is as follows: In the above formula, Q' represents the normalized feature tensor, min(Q) represents the minimum value of all elements in tensor Q, max(Q) represents the maximum value of all elements in tensor Q; Step S504: Select the kernel function to train the support vector machine model to classify the feature tensor Q', obtain the category label of the target skin image, and complete the skin classification. 7. A system for skin typing data augmentation based on adversarial generation network, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: the processor executes the A computer program implements the method according to any one of claims 1-6. 8. A computer-readable storage medium on which a computer program is stored, characterized in that: when the computer program is executed by a processor, the method according to any one of claims 1-6 is implemented.