DE202022106317U1 - A system for classifying melanoma images - Google Patents

Abstract

A system (100) for classifying melanoma images, the system (100) comprising:a dataset collection module (102) for collecting a plurality of images for training and testing, the plurality of images for training being comparatively greater than that of the plurality of images for testing; anda classification module (104) coupled to the data set collection module (102) for classifying the plurality of images to detect melanoma, the classification module (104) being formed by generating and stacking a plurality of trained networks (106), loading weights for each of the plurality of trained networks (106), freezing their layers to avoid changing the weights during adaptation of an ensemble module to the trained network (106), concatenating the frozen layers, and finally adding a dense one Single-output layer with a sigmoid activation function for binary classification of the melanoma images into malignant and benign.

Description

FIELD OF THE INVENTION

The present invention relates to the field of convolutional neural networks. In particular, the present invention relates to a system for classifying melanoma images for diagnostic assessments by convolutional neural networks.

BACKGROUND OF THE INVENTION

One of the most common human diseases is skin cancer. The number of skin cancer cases has increased enormously in the last decades. It is the leading cause of death in both men and women. It occurs when the skin cells responsible for pigment production, called melanocytes, which are responsible for melanin, get out of control. The reason for the mutation in the melanocytes that leads to skin cancer is unknown.

Knowing how to recognize melanoma is important because early-stage melanoma is highly treatable. To recognize the warning signs that a spot on the skin could be melanoma, the American Academy of Dermatology has proposed an "ABCDE" memory facility or rule. Medical image-guided diagnostic systems for skin lesions can be used as a suitable tool to support dermatologists in classifying skin cancer. This study specifically addresses binary classification issues associated with melanoma and benign skin cancers. Deep learning methods like CNN are capable of powerful classification and detection. Deep learning classification techniques provide a highly effective way to tackle complex problems and detect various diseases.

Various pre-trained models like ResNet50, InceptionV3, MobileNetV2 can achieve up to 90-95% accuracy. Since it is about melanoma detection, it is important to have a model that is highly accurate.

Automatic detection of melanoma has become an important public health issue due to the poor knowledge people have about certain diseases. Recently, deep neural networks are widely used for segmentation, classification, object recognition, etc.

According to a researcher, deep learning methods are applied with augmentation to avoid complex pre-processing steps and he examines the performance of different neural network architectures like ResNet50, Xception, VGG16 and InceptionV3 with accuracy of 0.94, 0.95, 0.93 and 0.94 and for lesions BCDU -Net, SegNet and U-Net. The deep learning methods performed well with no complex preprocessing techniques except for normalization and augmentation.

According to another research, a Keras Image Data Generator, images of skin lesions were pre-processed. The MobileNet model provided accurate and stable performance with an additional faster and lightweight architecture. The overall accuracy of the model used was 83%.

Another study discusses a domain-specific AI for skin image analysis. The ABCD rule for understanding melanoma using locally interpretable model-agnostic explanations (LIME) has been implemented and the results are presented for classification of skin images using deep neural networks (DNN).

US8543519B2 describes a remote melanoma diagnosis system and method for diagnosing a disease or condition from digital images captured by a remote user and transmitted to an image analysis server connected to a distributed network. The image analysis server contains a trained learning engine for classifying the images. The classification result is post-processed to create a risk score that is sent to the remote user. A database connected to the server may contain referral information for geographic mapping of the remote user to a local doctor. An optional process involves the collection of financial information to ensure payment for analytics services.

US10176574B2 discloses a structure-preserving composite model for skin lesion segmentation that includes partitioning a dermoscopic image into superpixels at a first scale. Each super pixel is a vertex on a graph defined by color coordinates and space coordinates and represents a number of pixels of the dermoscopic image according to the first scale. Furthermore, a multiplicity of k background templates are created by k-means clustering of selected super-pixels in space and color. In addition, generating sparse representations of the multitude of super pixel based on the variety of background templates. Also, calculate a reconstruction error for each superpixel by comparing its sparse representation with its original color and space coordinates. Also output a confidence map identifying each pixel of the dermoscopic image as belonging or not belonging to a skin lesion based on the reconstruction errors of the representative superpixels.

Most of the work presented is based on the ABCD rule, i. H. based on asymmetry, borders, colors and diameter, used by dermatologists to determine melanoma by visual inspection. An irregular thickness can also be a sign of melanoma. Existing melanoma classification models do not recognize such differences in skin thickness. The dataset used for these studies consisted primarily of Caucasian subjects. In individuals with highly pigmented or dark skin, it is difficult to distinguish the boundaries of the mole, so in such cases the classifier cannot detect abnormalities in the skin lesion or mole.

Therefore, there is a need to develop an automatic system for classifying images of skin lesions that is more accurate and allows reliable detection of the melanoma, solves the problem of overfitting and less compared to existing machine learning techniques such as neural networks and support vector machines is complex in terms of computational effort.

The technical advance disclosed by the present invention overcomes the limitations and disadvantages of existing and conventional systems and methods.

SUMMARY OF THE INVENTION

The present invention relates generally to a system for classifying melanoma images for diagnostic evaluation.

The aim of the present invention is to develop a system for the detection and classification of melanoma images.

Another object of the present invention is to train and store the MobileNetV2, InceptionV3 and ResNet50 models individually;

Another object of the present invention is to build an ensemble model using MobileNetV2, InceptionV3 and ResNet50 to classify melanoma images into benign and malignant using CNN; and

Another aim of the present invention is to implement LIME to understand the decision behind the diagnostic judgments of the classifier.

• ein Datensatzsammelmodul zum Sammeln einer Vielzahl von Bildern zum Trainieren und Testen, wobei die Vielzahl von Bildern zum Trainieren vergleichsweise höher ist als die der Vielzahl von Bildern zum Testen; und
• ein Klassifizierungsmodul, das mit dem Datensatzsammelmodul verbunden ist, um die Vielzahl von Bildern zu klassifizieren, um ein Melanom zu erkennen, wobei das Klassifizierungsmodul gebildet wird, indem eine Vielzahl von trainierten Netzwerken erzeugt und gestapelt wird, wobei Gewichte für jedes der Vielzahl von trainierten Netzwerken geladen werden, ihre Schichten eingefroren werden, um eine Änderung der Gewichte während der Anpassung eines Ensemblemoduls an das trainierte Netzwerk zu vermeiden, die eingefrorenen Schichten verkettet werden und schließlich mit einer dichten Schicht mit einem einzelnen Ausgang und einer Sigmoid-Aktivierungsfunktion zur binären Klassifizierung der Melanombilder in bösartig und gutartig hinzugefügt werden.

In one embodiment, a system for classifying melanoma images, the system comprising:

• a dataset collection module for collecting a plurality of images for training and testing, wherein the plurality of images for training is comparatively higher than that of the plurality of images for testing; and
• a classification module coupled to the dataset collection module to classify the plurality of images to detect melanoma, the classification module formed by generating and stacking a plurality of trained networks, weights for each of the plurality of trained networks loaded, their slices frozen to avoid changing the weights during fitting of an ensemble module to the trained network, the frozen slices concatenated, and finally with a single-output dense slice and a sigmoid activation function for binary classification of the melanoma images be added into malignant and benign.

In one embodiment, the plurality of images are read and stored in a NumPy array of dimensions (number of images × image shape × channels), where the corresponding designations of the plurality of images are also stored in the 1-D NumPy array.

In one embodiment, the data set was divided into a training and a test set containing 80% and 20% of the skin cancer image, respectively.

In one embodiment, the majority of the trained networks are built using ResNet, Inception, and Mobile Net.

In one embodiment, the ResNet has a plurality of convolution layers, an average pool and a max pool layer, and a floating point operation of 3.8x109, where the ResNet has a plurality of different kernels with size 2 strides, each with a kernel size of 7x7, with one tier having a max pool with a stride of size 2.

In one embodiment, the concept is an image recognition model that consists of multiple layers and has lower error rates to reduce computational effort when the number of parameters in a network is reduced.

In one embodiment, the inception has different sizes of convolution kernels, allowing it to possess a receptive field of different spatial frequencies, before finally combining to condense the design space of the network, allowing for the integration of features at different scales, where the inception inserts a stack normalization layer as a regularizer between an auxiliary classifier and the fully connected layers.

In one embodiment, the grid size of the inception is reduced to increase the activation dimension of the line filters by concatenating the pooling layer and two parallel convolution blocks.

In one embodiment, the Mobile Net is a convolutional neural network for feature extraction, object detection and segmentation with a variety of deep layers for different visual recognition tasks with two different types of blocks, a reduction block of size stride 2 and a remainder block of strides1, where the Mobile Net has several has layers for both types of blocks.

In one embodiment, the convolutional network has a first layer with a rectified linear activation function (ReLU) in 1x1 convolution, a second layer with a depthwise convolution, and a third layer with a 1x1 convolution without the non-linearity.

In order to further clarify the advantages and features of the present invention, a more detailed description of the invention will be made by reference to specific embodiments thereof illustrated in the accompanying figures. It is understood that these figures show only typical embodiments of the invention and therefore should not be considered as limiting its scope. The invention will be described and illustrated with additional specificity and detail with the accompanying figures.

character list

• 1 ein Blockdiagramm eines Systems zur Klassifizierung von Melanombildern zeigt,
• 2 eine architektonische Darstellung des Klassifizierungsmoduls zeigt, und
• 3 ein Blockdiagramm der Kalkauswertung des Klassifizierungsmoduls zeigt.

These and other features, aspects and advantages of the present invention will be better understood when the following detailed description is read with reference to the accompanying figures, in which like characters represent like parts throughout the figures, wherein:

• 1 shows a block diagram of a system for classifying melanoma images,
• 2 shows an architectural representation of the classification module, and
• 3 shows a block diagram of the lime evaluation of the classification module.

Those skilled in the art will understand that the elements in the figures are presented for simplicity and are not necessarily drawn to scale. For example, the flowcharts illustrate the method in terms of key steps that may aid in understanding aspects of the present disclosure. In addition, one or more components of the device may be represented in the figures by conventional symbols, and the figures only show the specific details relevant to understanding the embodiments of the present disclosure, not to encircle the figures with details to overload, which are easily recognizable to those skilled in the art familiar with the present description.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the invention, reference will now be made to the embodiment illustrated in the figures and specific language will be used to describe the same. It should be understood, however, that no limitation on the scope of the invention is intended, and such alterations and further modifications to the illustrated system and such further applications of the principles of the invention set forth therein are contemplated as would occur to those skilled in the art invention would normally come to mind.

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be limiting.

When this specification refers to "an aspect,""anotheraspect," or the like, it means that a particular feature, structure, or characteristic described in connection with the embodiment is present in at least one embodiment of the present invention. As such, the phrases "in one embodiment,""in another embodiment," and similar phrases may be used in this description all refer to the same embodiment, but do not have to.

The terms "comprises," "including," or other variations thereof are intended to cover non-exclusive inclusion such that a method or method that includes a list of steps includes not only those steps, but may also include other steps that are not expressly stated or pertaining to any such process or method. Likewise, any device or subsystem or element or structure or component preceded by "comprises...a" does not, without further limitation, exclude the existence of other devices or other subsystem or other element or other structure or other component or additional device or additional subsystems or additional elements or additional structures or additional components.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The system, methods, and examples provided herein are for purposes of illustration only and are not intended to be limiting.

Embodiments of the present invention are described in detail below with reference to the attached figures.

1 shows a block diagram of a system (100) for classification of melanoma images, the system (100) comprising: a dataset collection module (102), a classification module (104), a plurality of trained networks (106), ResNet (106a), Inception ( 106b) and Mobile Net (106c).

The data set collection module (102) collects a plurality of images for training and testing, the number of images for training being comparatively higher than the number of images for testing. The plurality of images are read and stored in a NumPy array of dimensions (number of images × image shape × channels), and the corresponding labels of the plurality of images are also stored in the 1-D NumPy array.

The classification module (104) is connected to the data set collection module (102) to classify the plurality of images to detect melanoma, the classification module (104) being formed by generating and stacking a plurality of trained networks (106) wherein weights are loaded for each of the plurality of trained networks (106), freezing their layers to avoid changing the weights during the fitting of an ensemble module to the trained network (106), concatenating the frozen layers and finally adding a dense layer with a single output and a sigmoid activation function for binary classification of the melanoma images into malignant and benign.

The majority of the trained networks (106) are created with ResNet (106a), Inception (106b) and Mobile Net (106c). The ResNet (106a) has a variety of convolution layers, an average pool and a max pool layer, and has a floating point operation of 3.8x109, where ResNet (106a) has a variety of different kernels with size 2 strides, each with a kernel size of 7x7, with one tier having a max pool with a stride of size 2. Inception (106b) is an image recognition engine that consists of multiple layers and has lower error rates to reduce computational power when the number of parameters in a network is reduced. Inception (106b) has convolution kernels of different sizes, allowing it to possess a receptive field of different spatial frequencies, before finally combining to condense the design space of the network, allowing for the integration of features at different scales, where Inception (106b) inserts a stack normalization layer as a regularization layer between an auxiliary classifier and the fully connected layers. The grid size of the Inception is reduced to increase the activation dimension of the line filters by concatenating a pooling layer and two parallel convolution blocks. The Mobile Net (106c) is a convolutional neural network for feature extraction, object detection and segmentation with multiple deep layers for various visual recognition tasks with two different types of blocks, a stride 2 size minification block and a strides1 residual block, where the Mobile Net has multiple layers for has both types of blocks. The convolutional network has a first layer with a rectified linear activation function (ReLU) in 1x1 convolution, a second layer with a depthwise convolution and a third layer with a 1x1 convolution without the non-linearity.

2 shows an architectural representation of the classification module.

The classification engine is created by built-in stacked ensemble models composed of networks such as ResNet50, InceptionV3, and MobileNetV2. These models will initially individually trained and stored, then their outputs are concatenated together with an additional dense layer. It does this by loading the modules in list form and using a separate input header for each of the loaded models to feed data into the larger stacked ensemble model. The input layer of each partial model serves as its own input header for the model. This means that the model receives k copies of the input data, where k is the number of input models, in this case 3.

The ResNet-50 deep neural network has 48 convolution layers and 1 average pool and 1 max pool layer and has a floating point operation of 3.8x109. In this model, the vanishing/exploding gradient problems were solved by using a shallower and deeper model and incorporating an identity layer. They use a jump connection or shortcut between every two shifts. In this way, activation can be transferred from one layer to another layer, so that the parameters of a network are preserved in lower layers. ResNet50 has 64 different kernels with size 2 strides, each with a kernel size of 7x7, making one tier. It has a maximum pool with an increment of 2. Identity mapping by skipping a connection is given by: $$\text{Y}=\text{f}\left(\text{X}{\text{,W}}_{\text{i}}\right)+{\text{W}}_{\text{s}}\text{X}$$

The input and output vectors of the layers are represented here as X and Y and F(X,Wi) denotes the residual map.

The ResNet-50 model consists of five stages, each with an identity block and a convolution block. Each convolution block contains three layers of convolution and each identity block has three layers of convolution. The ResNet50 has over 23 million trainable parameters.

$$\left\{\text{Y}={\text{BN}}_{\mathrm{\alpha }\mathrm{,\beta }}\left(\text{Xi}\right)\right\}$$

Inception V3 consists of forty-two layers and has lower error rates than its previous versions. Inception V3 features factored convolutions that help reduce computational overhead by reducing the number of parameters within a network. The large folds within a model have been factored into smaller folds. The BN formulas are expressed as follows: $$\text{B}=\left\{\text{X1}...\text{m}\right\},\mathrm{a}\mathrm{,\beta }$$

$$\left\{\text{Y}={\text{B.N}}_{\mathrm{a}\mathrm{,\beta }}\left(\text{Xi}\right)\right\}$$

$${\sigma }^2\text{B}={\text{1/m}}_{i=1}^2{\left({\text{X}}_i-{\mu }_b\right)}^2$$

Here, x designates the minimum activation value of batch B, m the number of activation values and the learnable parameters are given by α and β. $$\mathrm{\mu }={\text{1/m}}_{i=1}^m{X}_i$$

$${\sigma }^2\text{B}={\text{1/m}}_{i=1}^2{\left({\text{X}}_i-{\mu }_b\right)}^2$$

The mean value in one dimension is given by µ, the standard deviation in each dimension of the feature map is given by σ ² and ∈ is a constant. $${\text{Y}}_{\text{i}}={\text{X}}_{\text{i}}+\mathrm{\beta =}\text{B.N}{\text{.}}_{\mathrm{\beta }}\left(\text{Xi}\right)$$

The Inception V1 model has a 5x5 convolution layer and is very computationally intensive. To reduce computational costs, the convolution layer was replaced with a 3x3 convolution layer. The 5x5 convolution layer of the original Inception module was replaced with a 3x3 convolution module, which also reduced the number of parameters and the computational cost.

Asymmetric folding can make the model even more efficient, so the 3x3 fold was replaced with a 1x3 fold, followed by a 3x1 fold. Inception also uses auxiliary classifiers to improve deep network convergence. In order to reduce the grid sizes in Inception V3, the activation dimension of the network filters is increased. This is done by concatenating a pooling layer and two parallel convolution blocks.

MobileNetV2 has 53 deep layers for various visual recognition tasks. There are two different types of blocks in Mobile Net, a downsizing block of size stride 2 and a remainder block of strides1. It has three layers for both types of blocks. ReLU6 is used as the first layer in the 1x1 convolution. The second layer consists of a deep-acting fold. The depth-wise separable convolution has been integrated into the MobileNetV2 layer. Two operations, i. H. the depth-wise convolution and the point-wise convolution are combined to produce a depth-wise separable convolution.

The concatenation layer takes the inputs and concatenates them along certain specific dimensions. The results of the individual models are combined in the concatenation layer. Simple concatenation is used here, with each of the three models combining two class probabilities into a single six-element vector.

The dense layer is basically a layer closely connected to the previous layer is. The neurons of this layer are connected to all neurons of the previous layers. The outputs of all models are given as input to the concatenation layer, which merges the inputs. Then a dense layer with some units is added, followed by a dense layer with a singular output and an activation function of Sigmoid since the task is a binary classification.

3 shows a block diagram of the lime evaluation of the classification module.

LIME is essentially model-independent and serves as a decision-making aid for the diagnosis of melanoma. The proposed classification engine is trained on data, and the trained model is then sent along with the data to the LIME explainer to provide insights into specific predictions made by the classification engine. The LIME takes the data and prediction values from the classification engine as input and creates an explanation based on the inputs. The LIME provides an explanation for skin cancer classification by providing green and red super pixels on the image. The green super pixels are mapped to the area where there is a skin lesion or mole. Based on the association pattern, the classification model predicts an image as benign or malignant. The mapping on the benign image is typically evenly distributed, while the green superpixel mapping on the malicious images is typically scattered and unevenly distributed.

By developing a localized explainable model of g ∈ G, where G represents a category of understandable models that includes decision trees and linear classifiers, LIME can understand the results of any model classifier f. It measures the complexity, represented as Ω(g) of the model, which indicates the importance of a factor in how easily explanations are regenerated. Additionally, an error of g in approximating f is computed using a distance or loss function represented as L(f,g), where the final explanation ξ(g) is: $$\mathrm{\xi }\left(\text{G}\right)=\text{argmin}\left(\text{f}\text{,G}\right)+\mathrm{\Omega }\left(\text{G}\right)$$

LIME performs the actions listed below to understand or interpret a black box model.

Input data perturbation: When the input data is an image, LIME creates multiple samples by turning on and off some of the image's super-pixels corresponding to the input image.

Predicting the class of an artificial data point: LIME determines the class of each artificial data point generated by a trained model. In the case of image data, the prediction for each changed or disturbed image is created in this phase.

Calculating the Weight of the Artificial Data Points: The weights for each artificial data point are then calculated; the cosine distance metric is used to determine how far each generated data point is from the actual input data. The distance is then mapped to a value between 0 and 1, and in the following phase a kernel function is applied. If the difference in distance between the artificial data point and the original data points is smaller, the weights become larger. Larger weights mean that an artificial data point is more important. In the case of the entered image data, the cosine distance to the original image is calculated and compared for each disturbed image. The weight and importance of a disturbed image increases when there is a greater resemblance between it and the original image.

Fitting a linear regression model: Fitting a linear regression model using weighted artificial data points is the last step in this process. After that, the fitted coefficient for each feature is determined, similar to the usual linear regression analysis. The coefficients are then sorted, and the features with the highest coefficients play an important role in determining the classification model's predictions.

The figures and the preceding description give examples of embodiments. Those skilled in the art will understand that one or more of the elements described may well be combined into a single functional element. Alternatively, certain elements can be broken down into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of the processes described herein may be changed and is not limited to the manner described herein. Additionally, the actions of a flowchart need not be performed in the order shown; Also, not all actions have to be carried out. Also, the actions that are not dependent on other actions can be performed in parallel with the other actions. The scope of the embodiments is limited by this spe in no way limited to specific examples. Numerous variations are possible, regardless of whether they are explicitly mentioned in the description or not, e.g. B. Differences in structure, dimensions and use of materials. The scope of the embodiments is at least as broad as indicated in the following claims.

Advantages, other benefits, and solutions to problems have been described above with respect to particular embodiments. However, the benefits, advantages, problem solutions, and components that can cause an advantage, benefit, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all claims.

reference list

100

A system (100) for classifying melanoma images.

102

Record collection module

104

classification module

106

trained network

106a

residual network

106b

Beginning

106c

Mobile network

202

Entry layer for Resnet50

204

Resnet50 architecture

206

Output layer for Resnet50

208

Input layer for InceptionV3

210

Inception V3 architecture

212

Output layer for InceptionV3

214

Input layer for Mobile NetV2

216

Mobile NetV2 architecture

218

Output layer for MobileNetV2

220

concatenation layer

222

Dense layer (ensemble model output)

302

Data

304

classification module

306

forecast

308

explanation

310

LIME

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US8543519B2 [0009]
• US 10176574 B2 [0010]

Claims (10)

A system (100) for classifying melanoma images, the system (100) comprising: a data set collection module (102) for collecting a plurality of images for training and testing, the plurality of images for training being comparatively higher than that of the plurality of images for testing; and a classification module (104) coupled to the dataset collection module (102) for classifying the plurality of images to detect melanoma, the classification module (104) being formed by generating and stacking a plurality of trained networks (106), loading weights for each of the plurality of trained networks (106), freezing their layers to avoid changing the weights during adaptation of an ensemble module to the trained network (106), concatenating the frozen layers, and finally adding a dense one Single-output layer with a sigmoid activation function for binary classification of the melanoma images into malignant and benign. system after claim 1 , the plurality of images being read and stored in a NumPy array of dimensions (number of images x image shape x channels), the corresponding labels of the plurality of images also being stored in the 1-D NumPy array. system after claim 1 , where the dataset is divided into a training set and a test set, containing 80% and 20% of the skin cancer image, respectively. system after claim 1 , the plurality of trained networks (106) being created with residual networks (ResNet) (106a), inception (106b) and mobile net (106c). system after claim 4 , where the ResNet (106a) has a variety of convolutional layers, an average pool and a max pool layer, and has a floating point operation of 3.8x109, where the ResNet (106a) has a variety of different kernels with strides of size 2 each with a kernel size of 7x7, with one tier having a max pool with a stride of size 2. system after claim 4 , where the inception (106b) is an image recognition engine composed of multiple layers and has lower error rates to reduce computational power when the number of parameters in a network is reduced. system after claim 6 , where the Inception (106b) has different sizes of convolution kernels that allow it to possess a receptive field of different spatial frequencies, before finally combining to condense the design space of the network, allowing the integration of features at different scales where the inception (106b) inserts a stack normalization layer as a regularizer between an auxiliary classifier and the fully connected layers. system after claim 6 , where the grid size of the inception is reduced to increase the activation dimension of the network filters by concatenating the pooling layer and two parallel convolution blocks. system after claim 4 , wherein the Mobile Network (106c) is a convolutional neural network for feature extraction, object recognition and segmentation, having several deep layers for different visual recognition tasks with two different types of blocks, a reduction block of size stride 2 and a remainder block of strides1, where the Mobile network has multiple layers for both types of blocks. system after claim 9 , wherein the convolutional network comprises a first layer of a rectified linear activation function (ReLU) in 1x1 convolution, a second layer of depth-wise convolution, and a third layer of 1x1 convolution without the non-linearity.

Images