CN109685801B - Skin mirror image processing method combining texture features and deep neural network information - Google Patents
Skin mirror image processing method combining texture features and deep neural network information Download PDFInfo
- Publication number
- CN109685801B CN109685801B CN201811502946.2A CN201811502946A CN109685801B CN 109685801 B CN109685801 B CN 109685801B CN 201811502946 A CN201811502946 A CN 201811502946A CN 109685801 B CN109685801 B CN 109685801B
- Authority
- CN
- China
- Prior art keywords
- texture
- neural network
- map
- layers
- pooling layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 42
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000011176 pooling Methods 0.000 claims description 55
- 238000013527 convolutional neural network Methods 0.000 claims description 45
- 238000000034 method Methods 0.000 claims description 41
- 230000004044 response Effects 0.000 claims description 21
- 230000006870 function Effects 0.000 claims description 20
- 238000013507 mapping Methods 0.000 claims description 16
- 238000005096 rolling process Methods 0.000 claims description 16
- 230000003902 lesion Effects 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 230000000295 complement effect Effects 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 6
- 239000002131 composite material Substances 0.000 abstract description 3
- 230000011218 segmentation Effects 0.000 description 28
- 230000008569 process Effects 0.000 description 10
- 206010040882 skin lesion Diseases 0.000 description 10
- 231100000444 skin lesion Toxicity 0.000 description 10
- 238000013135 deep learning Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 230000010354 integration Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 210000000977 primary visual cortex Anatomy 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241001390738 Belemia Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 210000004209 hair Anatomy 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30088—Skin; Dermal
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a dermatoscope image processing method combining textural features and deep neural network information, which comprises the following steps: processing the input image through a deep neural network to generate a feature map; processing the input image through a shallow network having a texture to generate a texture map; and fusing the feature map and the texture map by a convolution layer to generate a composite score map.
Description
Technical Field
The invention relates to the technical field of image processing. In particular, the invention relates to a dermatoscope image processing method combining texture features and deep neural network information.
Background
Existing methods for skin mirror image classification can be broadly classified into traditional histogram-based thresholding, clustering, edge-based detection, region-based detection, morphological detection, model-based, active contour (snap method and its variants), and supervised learning-based methods. For example, mm.e.celebi, q.wen, s.hwang, h.iyatomi and g.schaefer segment specific Skin features in Skin mirror Images Using segmentation Methods Using Thresholding Methods in the title "cause Border Detection in reduction Images Using segmentation of segmentation Methods", Skin Research and technology, vol.19, pp.e252-E258, Feb 2013. The result of threshold fusion shows a better skin lesion segmentation effect. However, this method may not process the image well when a large number of artifacts (such as hairs or bubbles) are present, as these factors can significantly change the histogram, resulting in biased threshold calculations. C.a.z.barcelos and v.b.pires in the title "An automatic based non-linear dispersion equalization for skin segmentation," Applied Mathematics and Computation, vol.215, pp.251-261, Sep 12009, An anisotropic diffusion filter is used to segment lesion edges before the edge detector. The results show that most of the unwanted edges are removed, however some skin lesion areas are missed.
Recent advances in machine learning, particularly in the field of deep learning such as Convolutional Neural Networks (CNNs), have greatly improved the latest techniques for identifying, classifying, and quantifying underlying patterns in medical images. In particular, it is at the heart of the progress to utilize hidden hierarchical features that are learned only from the data. The application progress of deep learning in medical image registration, segmentation, computer-aided disease diagnosis and prognosis diagnosis is introduced. In recent years, methods based on deep learning have also made great progress in the automatic segmentation of skin lesions.
The skin mirror image can be automatically segmented by utilizing the complete convolution neural network. To address the limitation of the coarse segmentation boundary generated by the original fully convolutional neural network due to lack of label refinement and consistency (especially for skin lesions with blurred boundaries and/or low variation of texture between foreground and background), the multi-level fully convolutional neural network was investigated to learn the complementary visual features of different skin lesions. The former stages of the fully convolutional neural network learn the coarse appearance and localization information, while the latter stages of the fully convolutional neural network learn the subtle features of the lesion boundary. The supplementary information from the various segmentation stages is then combined to obtain the final segmentation result. This approach has shown very promising results on the ISBI 2016 skin lesion challenge match data set.
Disclosure of Invention
In the existing skin feature segmentation method based on deep learning, much effort is made to design a new complete convolution neural network architecture with a specific loss function to realize better pre-generation. However, few attempts have been made to encode clinically valuable a priori knowledge into a deep learning regime to achieve accurate segmentation of skin features. To achieve this goal, the present invention aims to develop a generic framework that allows modeling and integration of background information into a deep fully convolutional neural network to achieve fully automatic skin feature segmentation, where the fully convolutional neural network aims to fuse information from intermediate convolutional layers to be output through the skip-join and deconvolution layers, so that low-level appearance information and high-level semantic information can be considered. In addition to data-driven features from a fully convolutional neural network, clinical prior knowledge of skin lesions of low-level edges and textural features is also considered. These low-level edge and texture features are derived from a predefined skin feature-specific filtering kernel, using a new shallow convolutional network, and then built into a full convolutional neural network workflow using convolution operations. The proposed new network architecture is trained in an end-to-end manner. In this way, the domain-specific texture features can complement other hierarchical and semantic features learned from the fully convolutional neural network to enhance the fine details of the skin lesion for more accurate segmentation.
According to one aspect of the invention, a dermatoscope image processing method combining texture features and deep neural network information is provided, which comprises the following steps:
processing the input image through a deep neural network to generate a feature map;
processing the input image through a shallow network having a texture to generate a texture map; and
the feature map and texture map are fused by the convolutional layer to generate a synthetic score map.
In one embodiment of the present invention, the method for processing a dermatoscope image combining texture features and deep neural network information further comprises: and inputting the comprehensive score map into a softmax function with a loss layer to form an end-to-end trainable network.
In one embodiment of the invention, the network is trained using a small batch of random gradient descent SGDs with momentum.
In one embodiment of the invention, the deep neural network comprises:
a plurality of stacks, each stack comprising one or more convolutional layers and pooling layers; and
and a plurality of transform convolution layers and upsampling layers provided after the plurality of stacks, and performing convolution operation and first upsampling operation on predictions output by the plurality of stacks to generate an activation map.
In one embodiment of the invention, the plurality of stacks is 5 stacks, the first stack comprises one or more first rollup layers and a first pooling layer, and the input image is processed by the one or more first rollup layers and the first pooling layer to generate a first pooling layer prediction; the second stack includes one or more second rolling layers and second pooling layers, the first pooling layer prediction generates a second pooling layer prediction after being processed by the one or more second rolling layers and the second pooling layers; the third stack includes one or more third rolling layers and a third pooling layer, the second pooling layer prediction generates a third pooling layer prediction after being processed by the one or more third rolling layers and the third pooling layer; the fourth stack includes one or more fourth rolling layers and fourth pooling layers, the third pooling layer prediction generates a fourth pooling layer prediction after being processed by the one or more fourth rolling layers and the fourth pooling layers; the fifth stack includes one or more fifth rolling layers and a fifth pooling layer, and the fourth pooling layer prediction is processed by the one or more fifth rolling layers and the fifth pooling layer to generate a fifth pooling layer prediction.
In one embodiment of the present invention, the method for processing a dermatoscope image combining texture features and deep neural network information further comprises:
fusing the fourth pooling layer prediction with the activation map output by the upsampling layer, and then performing a second upsampling operation to generate a second upsampling prediction;
and fusing the third pooling layer prediction with the second upsampling prediction, and then performing a third upsampling operation to generate a third upsampling prediction.
In one embodiment of the invention, processing the input image through the shallow network having texture comprises:
generating a texture dictionary;
convolving the input image with filters in a filter bank to produce a filter response;
assigning each pixel of the input image to a texture tag in a texture dictionary (D) based on a minimum between texture and filter response at each pixel, generating a texture tag map;
converting the texture label map into an intensity map;
and replacing the label indexes in the texture label graph with corresponding average intensity to form texture mapping.
In one embodiment of the present invention, the generating the texture dictionary includes:
preparing two types of patches by using a truth value mark;
a two-dimensional second-order partial derivative of a gaussian with 12 directions is applied to each patch in each set;
clustering filter responses generated from all patches in the same group to generate a class of textures; and
all the trained textures are stored in a texture dictionary.
In one embodiment of the invention, the feature map and texture map are fused in a complementary manner by a two-block convolutional layer.
In one embodiment of the invention, during the training process, more detail from the texture feature map can be supplemented to the feature map of the fully convolved neural network, while suppressing the effect of non-lesion edges in the texture feature map by the fully convolved neural network feature map.
The invention provides a method for combining the prior knowledge of shallow network coding with a deep complete convolution neural network for segmenting skin features in a skin mirror image. The existing convolutional neural network needs a deep multi-layer neural network to ensure the effectiveness of the model. We propose a method that combines a priori knowledge of shallow network coding with a deep fully convolutional neural network. First, a priori knowledge uses extraction of textural features, where a spatial filter model simplifies the function of the cellular receptive domain on the primary visual cortex (V1). A priori knowledge encoded by the shallow network is then coupled with the hierarchical data-driven features learned from the fully convolutional neural network (FCN) for detailed segmentation of skin features using an efficient fusion strategy of the skip-join and convolution operators. The invention applies this new neural network for the first time to a detailed segmentation of skin features. Compared with the existing deep learning model, the method provided by the invention has better stability, meanwhile, the method does not need an ultra-deep neural network architecture, and does not need data expansion or comprehensive parameter adjustment, and experimental results show that compared with other advanced methods, the method has effective model popularization.
Drawings
To further clarify the above and other advantages and features of embodiments of the present invention, a more particular description of embodiments of the invention will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. In the drawings, the same or corresponding parts will be denoted by the same or similar reference numerals for clarity.
FIG. 1 shows a flow diagram of a dermoscopic image processing method incorporating texture features and deep neural network information, according to one embodiment of the present invention.
FIG. 2 illustrates a fully convolutional neural network architecture with VGG16 (fully convolutional neural network classification net) according to one embodiment of the present invention.
FIG. 3 illustrates a framework for a shallow network in which the texture dictionary generation process is shown in the bottom flowchart and the top flowchart shows the process of generating a texture map using a learned dictionary, according to one embodiment of the present invention.
FIG. 4 illustrates the effect of fusing feature maps from two networks according to one embodiment of the invention.
Detailed Description
In the following description, the invention is described with reference to various embodiments. One skilled in the relevant art will recognize, however, that the embodiments may be practiced without one or more of the specific details, or with other alternative and/or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of embodiments of the invention. Similarly, for purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the embodiments of the invention. However, the invention may be practiced without specific details. Further, it should be understood that the embodiments shown in the figures are illustrative representations and are not necessarily drawn to scale.
Reference in the specification to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.
The architecture of the disclosed dermatoscope image processing method of the present invention comprises two networks: fully convolutional neural networks and shallow texture-based networks. The two networks are complementary to achieve a more accurate skin feature segmentation. FIG. 1 shows a flow diagram of a dermoscopic image processing method incorporating texture features and deep neural network information, according to one embodiment of the present invention. Both data-driven and hand-made features are considered for segmentation. More specifically, hierarchical semantic features are learned by a fully convolutional neural network, while contextual information related to skin feature boundaries is modeled by texture features derived from a shallow network. The two networks are then integrated by fusing the feature maps generated from each network using the convolution operator. The two networks interact in a learning phase to achieve a more detailed segmentation.
In our method, the basis of the feature learning process for both networks is derived from convolution operations. For deep networks, the convolution filter kernel is represented by weights that are automatically learned from the raw image data, while the texton-based shallow networks learn the raw elements by manually designing the filter kernel based on domain-specific knowledge (such as skin lesion boundaries and textures). The combination of these two networks has two benefits: on one hand, some important clues obtained from clinical priori knowledge or context information are emphasized in the learning process, and on the other hand, an automatic weight learning scheme designed in a complete convolutional neural network is helpful for optimizing a manually-made element diagram.
First, at step 110, the input image is processed by a deep neural network to generate a feature map.
The fully convolutional neural network architecture is the latest technology of semantic segmentation, where segmentation is obtained by pixelized prediction. A fully convolutional neural network is trained by mapping inputs to their labeled ground truth in an end-to-end supervised learning manner. In the present invention, the complete convolutional neural network can be trained to learn the hierarchical features using the existing architecture, and the training part will not be described in detail in order to simplify the present specification.
FIG. 2 illustrates a fully convolutional neural network architecture with VGG16 (fully convolutional neural network classification net) according to one embodiment of the present invention. The architecture of VGG16 can be used as the basic network in the complete convolution neural network, and the advantages are as follows: 1) a more representative feature can be learned by stacking two or three convolutional layers with small filters (3 x 3) because it increases the complexity of the non-linear function; 2) the problem of limited training data quantity can be solved by transfer learning, namely a pre-training model which is learned from abundant natural images can be used for training a complete convolution neural network to carry out skin lesion segmentation; 3) a deep 16-level architecture with a large amount of weights can encode more complex high-level semantic features.
The VGG16 network includes 5 stacks and 3 transform convolutional layers, each block containing several convolutional and pooling layers. The convolutional layer can be viewed as a filter-based feature extractor. The filter response is generated by the discrete convolution operation of the local acceptance domain of the feature map of the previous layer, and is defined as:
wherein
Is a filter containing weights in the current layer lA wave core. Input mapping
Is the feature map in layer (l-1). b is the deviation in the l-th layer. The result of the local weighted sum is passed to the activation function. Compared to Sigmoid methods, Linear rectification functions (relus) can achieve better performance in terms of learning efficiency, resulting in faster deep neural network training, using relus as activation functions in each convolutional layer. The pooling layer provides a mechanism to preserve feature space invariance. However, it reduces the resolution of the element map. Typical pooling operations include sub-sampling and maximum pooling. In the invention, maximum cell operation can be adopted, and the maximum cell operation is obviously superior to secondary sampling operation through research.
Referring to the specific example shown in fig. 2, the first stack includes convolutional layer 1, convolutional layer 2, and pooling layer 1. The convolutional layer 1 performs feature extraction on an input image using 64 filters to obtain 64 feature maps, and the output feature map of the convolutional layer 1 is obtained by stacking the 64 feature maps. The operation of convolutional layer 2 is similar to convolutional layer 1, except that convolutional layer 1 performs a convolution operation on the original image input, and convolutional layer 2 performs a convolution operation on the output result of convolutional layer 1 to generate a feature map of convolutional layer 2. The feature map of convolutional layer 2 is predicted by pooling layer 1 to generate pooling layer 1. Pooling layer 1 predicts entry into the second stack. The processing of the second stack through the fifth stack is similar to the first stack, except that the input to the first stack is the original image, and the input to the subsequent stack is the prediction of the previous stack.
Conventional fully convolutional neural networks include: convolution-activation-pooling-fully-connected-classification or regression. In the present invention, a full convolutional neural network is implemented by converting the last full connection layer of a conventional full convolutional neural network into a convolutional layer, and then adding an upsampling and deconvolution layer to the converted full convolutional neural network. Replacing fully connected layers enables the network to accept image input of any size. The upsampling and deconvolution layers generate an output activation map and make the size of the feature map consistent with the size of the input image, thereby realizing pixel prediction. As mentioned above, merging layers reduce the resolution of the feature map that may lead to coarse prediction, in which case skip-joins are introduced to combine coarse prediction at deep levels and fine-scale prediction at shallow levels to improve segmentation detail.
More specifically, the skip join fuses the 2 up-sampled prediction computed on the last layer of stride 32 with the prediction from pooled layer 4 of stride 16. I.e. the 2-fold upsampled prediction is superimposed with the prediction of the pooling layer 4, the sum of which then upsamples the prediction back to the picture by a step 16. Also, a relatively better prediction (stride 8 prediction) is achieved by fusing the shallow prediction (pooling layer 3) with the sum of the two predictions from pooling layer 4 and the last layer of the 2-fold upsampling.
Various terms are used in this specification to refer to feature images generated by subjecting an input image to different stages of operations, such as feature maps, activation maps, filter responses, feature maps. These terms are sometimes used interchangeably.
Although the coarse segmentation problem can be mitigated using skip connections, some details are still missing in feature mapping that is recovered from the shallow layer by the deconvolution layer. Furthermore, the lack of spatial regularization of a fully convolutional neural network may result in a reduction in spatial consistency of the segmentation. Local dependencies are not adequately accounted for in a fully convolutional neural network, and some prediction inconsistencies between pixels within the same structure may also result. The present invention will hereinafter address this problem by introducing a shallow network by integrating texture-based spatial information into a fully convolutional neural network architecture.
Returning to FIG. 1, at step 120, the input image is processed through the shallow network with textures to generate a texture map. Texture is one of the representative spatial information that provides identifying features for the pattern recognition task. Texture has shown its advantages in encoding texture information and is represented by the response to a set of filter kernels (W)1,W2,…Wn)：
R＝[W1*I(x，y)，W2*I(x，y)...Wn*I(x，y)] [2]
Where denotes the convolution operation, n is the number of filter kernels (W), and the texture is defined as a set of eigenvectors generated by clustering of the filter responses in R.
Designing a suitable filter bank is a key step in extracting specific texture features. Based on the edge information, we use a second derivative gaussian filter to provide clinical prior knowledge of important cues for skin segmentation. The two-dimensional gaussian function is defined as:
the filter implements an anisotropic filter by introducing directional parameters, since edges can be present in arbitrary directions. The second partial derivative of equation (3) in the y-axis direction is:
x′＝x cosθ-y sinθ
y′＝x sinθ+y cosθ [4]
the filter is designed as a special edge detector and σ is 1.5, θ is [0 °, 15 °, 30 °, 45 °, 60 °, 75 °, 90 °, 105 °, 120 °, 135 °, 150 °, 165 °. Furthermore, in the present invention, a standard gaussian scale σ of 1 is employed to extract non-edge structures, which also imposes a simple smoothing constraint on the feature representation. To reduce computational redundancy and increase the feature representation, for anisotropic filters, the maximum response to the filter kernel across all directions is considered, while the response to the isotropic filter is recorded directly.
Texture is computed using the second order partial derivatives from the gaussian and k-means (k-means) clustering of the filter response of the standard gaussian. Pixels with similar filter responses are grouped into the same group. Each group is assigned a tag ID, i.e., a texture tag. To create the texture (primitive elements) of both lesions and non-lesions, two types of patches (lesions and non-lesions) are prepared using ground truth labeling. In the training phase, a two-dimensional second-order partial derivative of a gaussian with 12 directions (equation 4) is applied to each patch in each group, and the filter responses generated from all patches in the same group are clustered to generate a class of textures. As a result, there are k × c textures, where k (e.g., k ═ 8) is the number of centroids in the k-means and c is the number of classes (e.g., c ═ 2, i.e., lesions and non-lesions). All the trained textures are stored in a dictionary (D) which will be used to compute the texture atlas. This process is illustrated in the bottom flow chart in fig. 3.
Once the texture dictionary is generated from the training phase, each input image is mapped to a texture map using a similar implementation. More specifically, given a skin image, it is first convolved with the filters in the filter bank to produce a filter response, and then each pixel in the image is assigned to a texture label in the texture dictionary (D) based on the minimum between the texture and the filter response at each pixel
Through this process, a texture label map is generated, which is further converted to an intensity map. That is, for pixels having the same texture label, the average intensity of those corresponding pixels in the input image is calculated. The tag indices in the texture tag map are then replaced with the corresponding average intensities. We refer to this map as texture mapping, a process which is shown in the top flow chart of fig. 3.
This shallow network encodes global and local spatial information using convolution from a manually designed filter of domain specific knowledge that can decompose higher order visual features or structures into some original elements (here edges, dots and blobs). Each image may be represented by a different distribution of these elements, depending on the designed filter bank. Herein, each skin image is represented by the second-order partial derivative using gaussian and the edges extracted from the spots using standard gaussian extraction. Furthermore, we can distinguish edges with different gradient magnitudes depending on the number of k. Thus, a strong boundary of the lesion and a weak edge within the lesion may be distinguished. In addition, instead of using a pool-like operation that may reduce resolution in a fully convolutional neural network, clustering is implemented on filter response images having the same input image size in a shallow network. Therefore, our shallow network may retain more edge details. Notably, some edges in non-diseased regions may also be present due to the shallow nature of the network (e.g., no non-linear transformation). However, these non-lesion edges can be suppressed by fusing the fully convolved neural network feature maps.
Returning to FIG. 1, at step 130, the feature map and texture map are fused by the convolutional layer to generate a synthetic score map.
The feature map derived from the fully convolutional neural network and the texture map derived from the shallow network are fused in a single network by an integration block. Formally let the required mapping function be M(x)Which represents the slave input x1To the output xl+1Is performed by a non-linear transformation. It is assumed that this functionality can be learned more efficiently by introducing a priori knowledge models in the deep network. We do not fit M directly through the deep neural network(x)Instead, the fully convolutional neural network mapping function F: (x) Is set to F(x)＝M(x)-T(x). Thus, the original mapping function can be expressed as:
M(x)＝F(x)+T(x) [5]
wherein, T(x)Is a mapping function. With respect to the mapping function T(x)The output of the transform is constant, since the weights of the filter bank are predefined and fixed. In this case, T is added(x)Can affect forward propagation without affecting forward direction F(x)Backward propagation of (a). This is because, in the backward propagation calculation, the gradients are calculated by (local gradient x upstream gradient), and since the local gradients are all local gradients
The upstream gradient derived from the loss can be transferred directly to F(x)The weights are updated and the weights of the filter bank (by T)(x)Coding a priori knowledge) remains unchanged.
To fuse the two mappings in a complementary way, we increase the functional complexity by introducing a two-block convolutional layer. Equation (5) can be expressed as:
M(x)＝C(F(x)，{Wi})+C(T(x)，{Wi})
C＝W2λ(μ，W1) [6]
wherein { WiDenotes a set of weights (i is the number of layers) in the convolution block C, where the inputs μ and λ are the activation functions ReLU.
Each map (e.g., the feature map and texture map of the complete convolutional neural network output) is input into a separate block that contains two convolutional layers. Trainable weights in the convolutional layer are considered as filters that can learn functions that allow the two feature maps to fuse properly. That is, during the training process, more detail from the texture feature map can be supplemented to the feature map of the fully convolved neural network, while the effect of non-lesion edges in the texture feature map is suppressed by the fully convolved neural network feature map. In our experiments, the filter number and kernel size are empirically defined as follows: for each block, there are two inputs and four outputs in the first convolution layer, the kernel size being 5 by 5 pixels. In the second convolutional layer, there are four inputs and two outputs with the same kernel size. Finally, the sum of the responses of the filters from each block is fed to the whole network. That is, the filter responses for each block are added together, and then the network is trained by minimizing softmax cross-entropy loss.
FIG. 4 illustrates the effect of fusing feature maps from two networks according to one embodiment of the invention. As shown in fig. 4, (a) is a very low contrast sample image, particularly a local area displayed in a magnified box, top (b) is a composite score map generated from the network of the present invention, and (c) is a score map using only a fully convolutional neural network.
The improvement of the network of the present invention can be visually observed in fig. 4, where (a) is the input image (top) and the red boxed area is the magnified image (bottom), and we can clearly observe the original details. Sub-graph (b) shows the composite score graph (upper left corner of b) derived from the network according to the invention and its surface (lower right corner of b). Sub graph (c) is a score graph using only a fully convolutional neural network. As we see, the graph in (b) has finer detail than the graph in (c), and the region with red boxes in (a) is predicted to have a high probability of lesion in (b) using our proposed network, but misses it in (c). To see in further detail, we can see more local detail on the surface of (b) than the surface shown in (c).
Returning to FIG. 1, at step 140, the network is trained. Once the texture dictionary is generated, a texture map for each image may be calculated based on the minimum distance between the texture in the dictionary and the filter response for each pixel in the image. This in turn enables the network of the present invention to be trained in an end-to-end fashion. More specifically, the input image is passed in parallel through a fully convolved neural network and a shallow network to produce two feature maps, which are further fused using two block convolution layers (as shown in block 210 in FIG. 2). The final score map is input to softmax with the lossy layer, forming a new complete trainable network.
We train our network with small batches of random gradient descent (SGD) with momentum. In our experiment, we set the batch size to 20. The tilt rate of the fully convolutional neural network was set to 0.0001 and the momentum to 0.9. The learning rate of the last integration layer is set to 0.001. We initialize the network weights and initialize the integration layer using a pre-trained VGG16 model for a fully convolutional neural network. We used a 0.5 rate dip tube layer after the sixth and seventh convolutional layers in fig. 2 to reduce overfitting.
The invention provides a method for combining the prior knowledge of shallow network coding with a deep complete convolution neural network for segmenting skin features in a skin mirror image. The existing convolutional neural network needs a deep multi-layer neural network to ensure the effectiveness of the model. We propose a method that combines a priori knowledge of shallow network coding with a deep fully convolutional neural network. First, a priori knowledge uses extraction of textural features, where a spatial filter model simplifies the function of the cellular receptive domain on the primary visual cortex (V1). A priori knowledge encoded by the shallow network is then coupled with the hierarchical data-driven features learned from the fully convolutional neural network (FCN) for detailed segmentation of skin features using an efficient fusion strategy of the skip-join and convolution operators. The invention applies this new neural network for the first time to a detailed segmentation of skin features. Compared with the existing deep learning model, the method provided by the invention has better stability, meanwhile, the method does not need an ultra-deep neural network architecture, and does not need data expansion or comprehensive parameter adjustment, and experimental results show that compared with other advanced methods, the method has effective model popularization.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various combinations, modifications, and changes can be made thereto without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention disclosed herein should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
Claims (7)
1. A dermatoscope image processing method combining texture features and deep neural network information comprises the following steps:
processing the input image through a deep neural network to generate a feature map;
processing the input image through a shallow network with texture to generate a texture map, the shallow network with texture employing a second derivative Gaussian filter, wherein processing the input image through the shallow network with texture comprises: generating a texture dictionary; convolving the input image with filters in a filter bank to produce a filter response; assigning each pixel of the input image to a texture tag in a texture dictionary based on a minimum between a texture and a filter response at each pixel, generating a texture tag map; converting the texture label map into an intensity map; replacing the tag indices in the texture tag map with corresponding average intensities to form a texture map, wherein generating the texture dictionary comprises: preparing two types of patches by using a truth value mark; a two-dimensional second-order partial derivative of a gaussian with 12 directions is applied to each patch in each set; clustering filter responses generated from all patches in the same group to generate a class of textures; storing all the trained textures in a texture dictionary; and
fusing the feature map and texture map by a convolutional layer to generate a synthetic score map,
wherein the feature mapping and texture mapping are fused in a complementary manner by means of a two-block convolutional layer, the mapping function M (x) used representing the input x1To the output x1+1The formula is:
M(x)＝C(F(x)，{Wi})+C(T(x)，{Wi})
C＝W2λ(μ，W1)
f (x) is the complete convolutional neural network mapping function, T(x)Is a mapping function, { WiDenotes a set of weights in the convolution block C, i is the number of layers, and the inputs μ and λ are the activation functions ReLU.
2. The dermatoscopic image processing method combining textural features and deep neural network information of claim 1, further comprising:
and inputting the comprehensive score map into a softmax function with a loss layer to form an end-to-end trainable network.
3. The method of dermoscopic image processing in conjunction with texture feature and deep neural network information of claim 2 wherein the network is trained using a small batch of random gradient descent (SGD) with momentum.
4. The method of dermoscopic image processing in conjunction with textural features and deep neural network information of claim 1 wherein the deep neural network comprises:
a plurality of stacks, each stack comprising one or more convolutional layers and pooling layers; and
and a plurality of transform convolution layers and upsampling layers provided after the plurality of stacks, and performing convolution operation and first upsampling operation on predictions output by the plurality of stacks to generate an activation map.
5. The method of dermoscopic image processing in conjunction with textural features and deep neural network information of claim 4 wherein the plurality of stacks is 5 stacks, a first stack includes one or more first convolution layers and a first pooling layer, and the input image is processed by the one or more first convolution layers and the first pooling layer to generate a first pooling layer prediction; the second stack includes one or more second rolling layers and second pooling layers, the first pooling layer prediction generates a second pooling layer prediction after being processed by the one or more second rolling layers and the second pooling layers; the third stack includes one or more third rolling layers and a third pooling layer, the second pooling layer prediction generates a third pooling layer prediction after being processed by the one or more third rolling layers and the third pooling layer; the fourth stack includes one or more fourth rolling layers and fourth pooling layers, the third pooling layer prediction generates a fourth pooling layer prediction after being processed by the one or more fourth rolling layers and the fourth pooling layers; the fifth stack includes one or more fifth rolling layers and a fifth pooling layer, and the fourth pooling layer prediction is processed by the one or more fifth rolling layers and the fifth pooling layer to generate a fifth pooling layer prediction.
6. The dermatoscopic image processing method combining textural features and deep neural network information of claim 5, further comprising:
fusing the fourth pooling layer prediction with the activation map output by the upsampling layer, and then performing a second upsampling operation to generate a second upsampling prediction;
and fusing the third pooling layer prediction with the second upsampling prediction, and then performing a third upsampling operation to generate a third upsampling prediction.
7. The method of dermoscopic image processing in combination with texture features and deep neural network information of claim 1 wherein during training, more details from the texture map are supplemented to the feature map of the fully convolutional neural network while suppressing the effects of non-lesion edges in the texture map by the fully convolutional neural network feature map.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811502946.2A CN109685801B (en) | 2018-12-10 | 2018-12-10 | Skin mirror image processing method combining texture features and deep neural network information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811502946.2A CN109685801B (en) | 2018-12-10 | 2018-12-10 | Skin mirror image processing method combining texture features and deep neural network information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109685801A CN109685801A (en) | 2019-04-26 |
| CN109685801B true CN109685801B (en) | 2021-03-26 |
Family
ID=66187420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811502946.2A Active CN109685801B (en) | 2018-12-10 | 2018-12-10 | Skin mirror image processing method combining texture features and deep neural network information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109685801B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110363720A (en) * | 2019-07-04 | 2019-10-22 | 北京奇艺世纪科技有限公司 | A kind of color enhancement method, apparatus, equipment and the storage medium of image |
| CN110633662B (en) * | 2019-09-03 | 2022-03-25 | 云南白药集团健康产品有限公司 | Image processing method, device and system |
| CN110619598B (en) * | 2019-09-03 | 2023-04-18 | 云南白药集团健康产品有限公司 | Image processing method, device and system |
| CN111062296B (en) * | 2019-12-11 | 2023-07-18 | 武汉兰丁智能医学股份有限公司 | Automatic white blood cell identification and classification method based on computer |
| CN111583256B (en) * | 2020-05-21 | 2022-11-04 | 北京航空航天大学 | Dermatoscope image classification method based on rotating mean value operation |
| KR20220013071A (en) * | 2020-07-24 | 2022-02-04 | 에스케이하이닉스 주식회사 | Device for generating a depth map |
| CN112784806A (en) * | 2021-02-04 | 2021-05-11 | 中国地质科学院矿产资源研究所 | Lithium-containing pegmatite vein extraction method based on full convolution neural network |
| KR102495889B1 (en) * | 2022-07-13 | 2023-02-06 | 주식회사 룰루랩 | Method for detecting facial wrinkles using deep learning-based wrinkle detection model trained according to semi-automatic labeling and apparatus for the same |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9881234B2 (en) * | 2015-11-25 | 2018-01-30 | Baidu Usa Llc. | Systems and methods for end-to-end object detection |
| CN108629326A (en) * | 2018-05-14 | 2018-10-09 | 中国科学院自动化研究所 | The action behavior recognition methods of objective body and device |
| CN108664922A (en) * | 2018-05-10 | 2018-10-16 | 东华大学 | A kind of infrared video Human bodys' response method based on personal safety |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5426684A (en) * | 1993-11-15 | 1995-06-20 | Eastman Kodak Company | Technique for finding the histogram region of interest for improved tone scale reproduction of digital radiographic images |
| US10753881B2 (en) * | 2016-05-27 | 2020-08-25 | Purdue Research Foundation | Methods and systems for crack detection |
| CN106650830A (en) * | 2017-01-06 | 2017-05-10 | 西北工业大学 | Deep model and shallow model decision fusion-based pulmonary nodule CT image automatic classification method |
| CN108734719A (en) * | 2017-04-14 | 2018-11-02 | 浙江工商大学 | Background automatic division method before a kind of lepidopterous insects image based on full convolutional neural networks |
| CN107203999B (en) * | 2017-04-28 | 2020-01-24 | 北京航空航天大学 | Dermatoscope image automatic segmentation method based on full convolution neural network |
| CN107341265B (en) * | 2017-07-20 | 2020-08-14 | 东北大学 | Mammary gland image retrieval system and method fusing depth features |
| CN108010049A (en) * | 2017-11-09 | 2018-05-08 | 华南理工大学 | Split the method in human hand region in stop-motion animation using full convolutional neural networks |
| CN108510504B (en) * | 2018-03-22 | 2020-09-22 | 北京航空航天大学 | Image segmentation method and device |
-
2018
- 2018-12-10 CN CN201811502946.2A patent/CN109685801B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9881234B2 (en) * | 2015-11-25 | 2018-01-30 | Baidu Usa Llc. | Systems and methods for end-to-end object detection |
| CN108664922A (en) * | 2018-05-10 | 2018-10-16 | 东华大学 | A kind of infrared video Human bodys' response method based on personal safety |
| CN108629326A (en) * | 2018-05-14 | 2018-10-09 | 中国科学院自动化研究所 | The action behavior recognition methods of objective body and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109685801A (en) | 2019-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109685801B (en) | Skin mirror image processing method combining texture features and deep neural network information | |
| Adegun et al. | Deep learning techniques for skin lesion analysis and melanoma cancer detection: a survey of state-of-the-art | |
| Nandhini Abirami et al. | Deep CNN and deep GAN in computational visual perception-driven image analysis | |
| JP6395158B2 (en) | How to semantically label acquired images of a scene | |
| Lin et al. | Network in network | |
| Huang et al. | Deep hyperspectral image fusion network with iterative spatio-spectral regularization | |
| Liu et al. | Watershed merge tree classification for electron microscopy image segmentation | |
| Elmannai et al. | Deep learning models combining for breast cancer histopathology image classification | |
| Guan et al. | Multistage dual-attention guided fusion network for hyperspectral pansharpening | |
| Kar et al. | A review on progress in semantic image segmentation and its application to medical images | |
| CN109615614B (en) | Method for extracting blood vessels in fundus image based on multi-feature fusion and electronic equipment | |
| CN112560831A (en) | Pedestrian attribute identification method based on multi-scale space correction | |
| Banerjee et al. | Novel volumetric sub-region segmentation in brain tumors | |
| CN115546466A (en) | Weak supervision image target positioning method based on multi-scale significant feature fusion | |
| DE102023104829A1 (en) | Object class inpainting - or manipulation in digital images using class-specific inpainting neural networks | |
| Aalan Babu et al. | Water‐body segmentation from satellite images using Kapur's entropy‐based thresholding method | |
| Wang et al. | Multi-focus image fusion framework based on transformer and feedback mechanism | |
| Pham et al. | Toward Deep-Learning-Based Methods in Image Forgery Detection: A Survey | |
| Wu | An iterative convolutional neural network algorithm improves electron microscopy image segmentation | |
| Choudhary et al. | Mathematical modeling and simulation of multi-focus image fusion techniques using the effect of image enhancement criteria: A systematic review and performance evaluation | |
| Lu et al. | Efficient object detection for high resolution images | |
| Gupta et al. | A robust and efficient image de-fencing approach using conditional generative adversarial networks | |
| CN116030308A (en) | Multi-mode medical image classification method and system based on graph convolution neural network | |
| Chiu et al. | Real-time monocular depth estimation with extremely light-weight neural network | |
| Riad et al. | An industrial portrait background removal solution based on knowledge infusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200714 Address after: Room 1001-1008, 15 Block 57, Baiyang Street Science Park Road, Hangzhou Economic and Technological Development Zone, Zhejiang Province Applicant after: HANGZHOU DISHI TECHNOLOGY Co.,Ltd. Applicant after: DIGONG (HANGZHOU) SCIENCE AND TECHNOLOGY INDUSTRY Co.,Ltd. Address before: Room 1001-1008, 15 Block 57, Baiyang Street Science Park Road, Hangzhou Economic and Technological Development Zone, Zhejiang Province Applicant before: HANGZHOU DISHI TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |