CN110503654A

CN110503654A - A kind of medical image cutting method, system and electronic equipment based on generation confrontation network

Info

Publication number: CN110503654A
Application number: CN201910707712.XA
Authority: CN
Inventors: 王书强; 吴昆�; 陈卓
Original assignee: Shenzhen Institute of Advanced Technology of CAS
Current assignee: Shenzhen Institute of Advanced Technology of CAS
Priority date: 2019-08-01
Filing date: 2019-08-01
Publication date: 2019-11-26
Anticipated expiration: 2039-08-01
Also published as: CN110503654B; WO2021017372A1

Abstract

This application involves a kind of based on the medical image cutting method, system and the electronic equipment that generate confrontation network.Firstly, how research generator extracts the Pixel-level feature of different classes of high quality graphic, and carries out structured features expression using capsule model, and then realize the generation of Pixel-level mark sample；Secondly the suitable arbiter of building, for differentiating the true and false for generating Pixel-level mark sample, and design suitable error majorized function, it will differentiate that result is fed back respectively in the model of generator and arbiter, pass through continuous dual training, the sample generative capacity and discriminating power of generator and arbiter is respectively increased, finally generates Pixel-level using trained generator and marks sample, realizes the Pixel-level segmentation of image level mark medical image.The application is effectively reduced dependence of the parted pattern to Pixel-level labeled data, can improve the efficiency for generating sample and authentic specimen dual training and effectively realize that high-precision pixel-level image is divided.

Description

Medical image segmentation method and system based on generation countermeasure network and electronic equipment

Technical Field

The present application relates to the field of medical image processing technologies, and in particular, to a medical image segmentation method and system based on a generative countermeasure network, and an electronic device.

Background

With the rapid development of medical imaging technology, medical images have been widely and deeply applied in clinical medicine. According to statistics, tens of millions of cases are diagnosed and treated by medical images in every year around the world. In the conventional method of diagnosis and treatment based on medical image, a physician reads and recognizes medical image data and makes a judgment on diagnosis and treatment of a disease. The diagnosis and treatment mode is very inefficient, the individual difference is large, doctors are easy to miss diagnosis and misdiagnose according to personal experience, the doctor is tired when the doctor reads the film for a long time, and the film reading accuracy rate is reduced. With the rise of artificial intelligence, the machine is used for screening and judging image data in advance, key suspicious regions are marked, and then the images are sent to doctors for diagnosis and treatment, so that the workload of the doctors can be greatly reduced, and the results are comprehensive, stable and efficient. Therefore, artificial intelligence has an important application prospect in the field of medical imaging.

The medical imaging technology comprises two parts of medical imaging and medical image processing. Common medical Imaging techniques are mainly MRI (Magnetic Resonance Imaging), computed tomography Imaging (CT), Positron Emission Tomography (PET), ultrasound Imaging (US) and X-ray Imaging. Different imaging technologies have advantages in different applications of disease diagnosis and treatment, and selection of a corresponding imaging technology for diagnosis and treatment purposes of a specific disease is gradually formed in specific clinical applications. For example, magnetic resonance imaging can have excellent resolution for soft tissue imaging, has no harm of ionizing radiation and the like, and has wide application in diagnosis and treatment of parts such as the brain, the uterus and the like. The basic flow of the application of deep learning to medical image processing is shown in fig. 1.

In the conventional medical image segmentation task based on generation of the countermeasure network, in order to be able to sufficiently train the neural network and achieve a high accuracy result, a large amount of relevant medical image data needs to be prepared, and the medical image data needs to be labeled manually pixel by pixel. For example, to study the segmentation of tumor regions in the human brain, it is necessary to manually label the corresponding brain tumor images. Medical diseases are various, corresponding medical images are also various, deep learning is utilized to carry out segmentation on medical images, the medical images corresponding to each disease need manual labeling, and a large amount of manpower and material resources are consumed. Even the largest common data set can only provide pixel-level annotation samples of limited semantic categories. High quality data is scarce in medical image data sets, severely limiting the accuracy of semantic segmentation models.

A Generative Adaptive Networks (GAN) is a deep learning model, and is one of the most promising methods for unsupervised learning in complex distribution in recent years. The model passes through (at least) two modules in the framework: the mutual game learning of the Generative Model (Generative Model) and the Discriminative Model (Discriminative Model) yields a reasonably good output. The existing image segmentation model based on the generation countermeasure network can be applied to the segmentation of object images across categories, but in the field of medical images, the model has the problems of insufficient feature extraction, large calculation amount of countermeasure training and the like.

Disclosure of Invention

The application provides a medical image segmentation method, a medical image segmentation system and electronic equipment based on a generation countermeasure network, and aims to solve at least one of the technical problems in the prior art to a certain extent.

In order to solve the above problems, the present application provides the following technical solutions:

a medical image segmentation method based on generation of a countermeasure network comprises the following steps:

step a: respectively acquiring pixel-level labeling samples of other medical images and image-level labeling samples of medical images to be segmented;

step b: training a generation countermeasure network based on a capsule network through the pixel-level labeling samples of the other medical images and the image-level labeling samples of the medical images to be segmented, wherein the generation countermeasure network comprises a generator and a discriminator;

step c: the generator extracts pixel-level features of pixel-level annotation samples of other medical images, processes the image-level annotation samples of the medical images to be segmented through the pixel-level features to generate the pixel-level annotation samples of the medical images to be segmented, and generates segmentation prediction samples of the medical images to be segmented based on the pixel-level annotation samples;

step d: inputting the segmentation prediction sample generated by the generator and the real labeling sample of the image to be segmented into a discriminator together for generation-confrontation training, judging the authenticity of the segmentation prediction sample, and optimizing the generator and the discriminator according to an error function to obtain a trained generation confrontation network;

step e: inputting the medical image to be segmented marked at the image level into a trained generation countermeasure network, and outputting the pixel-level segmented image of the medical image to be segmented through the generation countermeasure network.

The technical scheme adopted by the embodiment of the application further comprises the following steps: in step c, the generator includes a capsule network module and a region positioning network, and the generating of the segmentation prediction samples of the medical image to be segmented specifically includes:

step b 1: pre-training a capsule network module through pixel-level annotation samples of other medical images to obtain a semantic label-free sample, and processing the image-level annotation sample of the image to be segmented through the semantic label-free sample to distinguish the background and the effective segmentation area of the image-level annotation sample of the image to be segmented;

step b 2: inputting the image-level labeling sample of the image to be segmented into a capsule network module after pre-training, and outputting a reconstructed image of the image-level labeling sample of the image to be segmented through the capsule network module;

step b 3: the region positioning network utilizes the characteristic extraction of the convolution layer to generate a characteristic graph containing position information of an image-level labeling sample of an image to be segmented, and adopts a global average pooling layer to apply weight (w)₁,w₂…,w_n) Carrying out weighted average on the feature map to obtain an area positioning feature map of an image-level labeling sample of the image to be segmented;

step b 4: and executing a self-diffusion algorithm according to the reconstructed image and the region positioning characteristic graph, determining a region pixel point segmentation line, and obtaining a segmentation prediction sample of the image-level annotation sample of the image to be segmented.

The technical scheme adopted by the embodiment of the application further comprises the following steps: in step b2, the capsule network module includes a convolution layer, a PrimaryCaps layer, a DigitCaps layer, and a decoding layer, and the capsule network module records the direction and position information of the edge pixels of the image-level labeled sample segmentation region of the image to be segmented by using the output vector of a single capsule neuron, extracts the classified probability value by using the nonlinear activation function of the vector, determines the segmentation region and the background of the image-level labeled sample of the image to be segmented, calculates the edge loss, and outputs the reconstructed image of the image-level labeled sample of the image to be segmented.

The technical scheme adopted by the embodiment of the application further comprises the following steps: in step b4, the performing a self-diffusion algorithm according to the reconstructed image and the region localization feature map specifically includes: and diffusing pixel points in the area with the larger activation value in the area positioning feature map by using a random walk self-diffusion algorithm, calculating the Gaussian distance from each pixel to the input point on the image by using the input point of the area positioning feature map, selecting an optimal path from the Gaussian distances, obtaining a segmentation line of the pixel points in the area, and finally generating a segmentation prediction sample.

The technical scheme adopted by the embodiment of the application further comprises the following steps: in the step d, the discriminator comprises a Cascade Cascade module, a Capsule network module and a parameter optimization module, and the training of the discriminator for 'generation-confrontation' specifically comprises:

step d 1: extracting pixels with wrong labels, key pixels with confidence degrees lower than a set threshold value and corresponding ground struts in the segmentation prediction sample through a Cascade Cascade module, and filtering the pixels with correct labels and confidence degrees higher than the set threshold value;

step d 2: processing the extracted key pixels and the corresponding grountruth through a Capsule network module, and generating errors;

step d 3: the parameter optimization module optimizes the network parameters of the generator and the discriminator by using errors generated by the Capsule network module; wherein for a given partitioned prediction sample { I }^f,L^f*And the corresponding true labeled samples { I }^f,L^fIntegral of the networkThe error function is:

in the above formula, θ_SAnd theta_pParameters representing generator and arbiter, respectively, J_bRepresenting a binary cross-entropy loss function, O_sAnd O_pRepresenting the output of the generator and the arbiter, respectively, when the input comes from the true annotated sample { I^f,L^fAnd partition prediction samples { I }^f,L^f*And when the pixel point type is judged to be true, judging whether the pixel point type is true or false by outputting 1 and 0.

Another technical scheme adopted by the embodiment of the application is as follows: a medical image segmentation system based on a generation countermeasure network comprises a sample acquisition module and the generation countermeasure network,

a sample collection module: the image segmentation method comprises the steps of acquiring pixel-level annotation samples of other medical images and image-level annotation samples of medical images to be segmented respectively;

training a generation countermeasure network based on a capsule network through the pixel-level labeling samples of the other medical images and the image-level labeling samples of the medical images to be segmented;

the generation countermeasure network comprises a generator and a discriminator, wherein the generator carries out pixel-level feature extraction on pixel-level labeling samples of other medical images, processes image-level labeling samples of medical images to be segmented through the pixel-level features to generate pixel-level labeling samples of the medical images to be segmented, and generates segmentation prediction samples of the medical images to be segmented based on the pixel-level labeling samples;

inputting the segmentation prediction sample generated by the generator and the real labeling sample of the image to be segmented into a discriminator together for generation-confrontation training, judging the authenticity of the segmentation prediction sample, and optimizing the generator and the discriminator according to an error function to obtain a trained generation confrontation network;

inputting the medical image to be segmented marked at the image level into a trained generation countermeasure network, and outputting the pixel-level segmented image of the medical image to be segmented through the generation countermeasure network.

The technical scheme adopted by the embodiment of the application further comprises the following steps: the generator comprises a pre-training module, a capsule network module, a region positioning network module and a sample generating module:

a pre-training module: the capsule network module is pre-trained through pixel-level annotation samples of other medical images to obtain a semantic label-free sample, and the image-level annotation sample of the image to be segmented is processed through the semantic label-free sample to distinguish the background and the effective segmentation area of the image-level annotation sample of the image to be segmented;

a capsule network module: the capsule network module is used for inputting the image-level labeling sample of the image to be segmented into the capsule network module after pre-training is finished, and outputting a reconstructed image of the image-level labeling sample of the image to be segmented through the capsule network module;

area positioning network: feature map containing position information for generating image-level annotation samples of an image to be segmented by using feature extraction of convolutional layers, and weighting (w) by using global average pooling layer₁,w₂…,w_n) Carrying out weighted average on the feature map to obtain an area positioning feature map of an image-level labeling sample of the image to be segmented;

a sample generation module: and the segmentation prediction module is used for executing a self-diffusion algorithm according to the reconstructed image and the region positioning characteristic graph, determining a segmentation line of a region pixel point, and obtaining a segmentation prediction sample of an image-level annotation sample of the image to be segmented.

The technical scheme adopted by the embodiment of the application further comprises the following steps: the capsule network module comprises a convolution layer, a Primarycaps layer, a Digitcaps layer and a decoding layer, the capsule network module adopts the output vector of a single capsule neuron, records the direction and the position information of the edge pixel of the image-level labeling sample segmentation area of the image to be segmented, extracts the classification probability value by adopting the nonlinear activation function of the vector, determines the segmentation area and the background of the image-level labeling sample of the image to be segmented, calculates the edge loss and outputs the reconstructed image of the image-level labeling sample of the image to be segmented.

The technical scheme adopted by the embodiment of the application further comprises the following steps: the sample generation module executes a self-diffusion algorithm according to the reconstructed image and the region positioning characteristic map, and specifically comprises the following steps: and diffusing pixel points in the area with the larger activation value in the area positioning feature map by using a random walk self-diffusion algorithm, calculating the Gaussian distance from each pixel to the input point on the image by using the input point of the area positioning feature map, selecting an optimal path from the Gaussian distances, obtaining a segmentation line of the pixel points in the area, and finally generating a segmentation prediction sample.

The technical scheme adopted by the embodiment of the application further comprises the following steps: the discriminator comprises a Cascade Cascade module, a Capsule network module and a parameter optimization module:

cascade of Cascade modules: the method comprises the steps of extracting pixels with wrong labels, key pixels with confidence degrees lower than a set threshold value and corresponding ground route in the segmentation prediction sample, and filtering the pixels with correct labels and confidence degrees higher than the set threshold value;

a Capsule network module: the method is used for processing the extracted key pixels and the corresponding ground route and generating errors;

a parameter optimization module: the device is used for optimizing the network parameters of the generator and the discriminator by utilizing the error generated by the Capsule network module; wherein for a given partitioned prediction sample { I }^f,L^f*And the corresponding true labeled samples { I }^f,L^fThe overall error function of the network is:

The embodiment of the application adopts another technical scheme that: an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the one processor to cause the at least one processor to perform the following operations of the above-described medical image segmentation method based on generation of a countermeasure network:

Compared with the prior art, the embodiment of the application has the advantages that: the medical image segmentation method, the system and the electronic equipment based on the generation of the countermeasure network optimize the deep convolutional neural network through the fusion Capsule mechanism, fuse the ideas of the Capsule network and the cascade waterfall, generate a new training image sample under the condition that the quantity of the medical image sample is small, realize the semantic segmentation of the medical image data with low quality and only image level labels, transfer the learned segmentation knowledge from the full labeling data of pixel level labeling to the weak labeling data of image level, thereby improving the model characteristic expression capability, expanding the applicability of the medical image labeling sample, effectively reducing the dependence of the segmentation model on the pixel level labeling data, having the characteristics of less network information redundancy and sufficient feature extraction, and under the premise of a small quantity of pixel level labeling samples, improving the efficiency of the countertraining of the generated sample and a real sample, and can effectively realize high-precision pixel-level image segmentation.

Drawings

FIG. 1 is a basic flow diagram of the application of deep learning to medical image processing;

FIG. 2 is a flowchart of a medical image segmentation method based on generation of a countermeasure network according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a structure of a generation countermeasure network according to an embodiment of the present application;

FIG. 4 is a schematic diagram of the structure of a capsule network module;

FIG. 5 is a schematic network structure diagram of a regional positioning network;

FIG. 6 is a schematic structural diagram of a medical image segmentation system based on generation of a countermeasure network according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a hardware device of a medical image segmentation method based on generation of a countermeasure network according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

In order to solve the defects of the prior art, the method for segmenting the medical image based on the generation countermeasure network improves the generation countermeasure network by fusing a capsule mechanism, firstly, the generator is researched to extract pixel-level characteristics of high-quality images of different categories, and a capsule model is utilized to carry out structural characteristic representation, so that the generation of a pixel-level labeled sample is realized; and secondly, constructing a proper discriminator for discriminating the authenticity of the generated pixel-level labeling sample, designing a proper error optimization function, feeding discrimination results back to models of the generator and the discriminator respectively, improving the sample generation capacity and discrimination capacity of the generator and the discriminator respectively through continuous confrontation training, and finally generating the pixel-level labeling sample by utilizing the trained generator to realize the pixel-level segmentation of the image-level labeling medical image. In the following embodiments, the present application is described in detail by taking only the medical image segmentation of Cervical Spondylotic Myelopathy (CSM) as an example, and the type of the disease source is not limited to a single case, and can be extended to the image segmentation scenes of multiple cases, such as brain MRI image segmentation, etc. Aiming at image segmentation of different cases, only training samples of corresponding cases need to be collected in a data collection stage, and the training samples are replaced in a generator of the model.

Please refer to fig. 2, which is a flowchart of a medical image segmentation method based on generation of a countermeasure network according to an embodiment of the present application. The medical image segmentation method based on the generation countermeasure network comprises the following steps:

step 100: respectively acquiring pixel level annotation samples and CSM image level annotation samples of other medical images;

in step 100, the other medical images include medical images of the lung and other parts, and the acquisition mode of the labeled sample specifically includes: collection of a few fully labeled CSM image samples { I }^f,L^f,T^fA total of 500 DTI samples (diffusion tensor imaging) with an image size of 28x28, 60 CSM groups of 60 cases, wherein 27 men and 33 women are 33, the ages of 20-71, and the average age of 45, comprise pixel-level annotation samples and image-level annotation samples { L }^f,T^f}; pixel-level labeling samples { I } of other medical images (e.g., human lungs)^O,L^O8000 DTI samples of (Dacron-Dacro,the image size was 28x 28. The method comprises the steps of respectively obtaining the CSM image and DTI samples of other medical images after correcting deformation and selecting a threshold value, determining a region of interest (ROI) on the DTI samples, and placing the ROI in the center of a spinal cord lesion area, so that the size of the DTI samples is uniform, and the influence of spinal fluid and artifacts is avoided.

Please refer to fig. 3, which is a schematic structural diagram of a generation countermeasure network according to an embodiment of the present application. The generation countermeasure network of the embodiment of the application comprises a generator and a discriminator, wherein the generator is responsible for generating pixel-level labeling data of the medical image, and the discriminator is responsible for refining the generated labels. The generator comprises a capsule network module and a regional positioning network, and is used for labeling samples { I ] of other medical images at pixel level^O,L^OThe CSM image level labeling samples are used as pre-training samples of the capsule network module, and the CSM image level labeling samples are marked with samples (L)^f,T^fAnd the training samples are used as the training samples of the area positioning network.

Step 200: pre-training a capsule network module through pixel-level labeled samples of other medical images to obtain a semantic label-free sample, and processing a CSM image-level labeled sample through the semantic label-free sample to distinguish the background and effective segmentation areas of the CSM image-level labeled sample;

in step 200, the capsule network module employs a migratable semantic segmentation model that can transfer learned segmentation knowledge from pixel-level labeling of fully labeled data to image-level labeling of weakly labeled data. In practical application, high-quality medical images are difficult to acquire, so that the method and the device enable new data to be matched with data of a target domain through a trained model, acquire the characteristics of pixel-level segmentation, and can also realize high-precision image segmentation under the condition of a small sample size. The pre-training process of the capsule network module specifically comprises the following steps:

step 201: processing pixel-level labeling samples of other medical images to convert the pixel-level labeling samples into semanteme-free segmentation image labels;

in step 201, the semantic-free segmentation image labeling is to label only the effective segmentation region of the sample and the background, and not to label the shape of the sample. The network trained by the training data learns that the knowledge for distinguishing the object from the background belongs to wide high-order features, and the image is distinguished by the object from the background, so the wide high-order features have strong universality and can be easily migrated to other different tasks, the knowledge learned from the medical image labeled at a high-quality pixel level can be migrated to the medical image labeled at a low-quality image level, and the high-quality image and the low-quality image do not need to have direct relevance.

Step 202: processing the CSM image-level labeled sample according to the semanteme-free segmentation image label of other medical images to generate a semanteme-free label sample { I) of the CSM image-level labeled sample^O,L^OAnd obtaining pixel-level semantic-free labels L by filtering semantic information^O；

In step 202, the purpose of obtaining pixel-level semantic label-free data is to distinguish the effective segmentation region and background of the data, so that the learned knowledge can be more easily migrated in the strongly and weakly labeled data.

Step 300: annotating CSM image level with samples { L^f,T^fInputting the capsule network module after the pre-training is finished, and outputting a reconstructed image of the CSM image-level labeling sample by the capsule network module;

in step 300, the capsule network module records the direction and position information of the edge pixels of the CSM image-level labeling sample segmentation region by using the output vector of a single capsule neuron, extracts the classification probability value by using the nonlinear activation function of the vector, determines the segmentation region and the background of the CSM image-level labeling sample, calculates the edge loss and outputs the reconstructed image of the CSM image-level labeling sample. According to the method and the device, parameter information of instantiated pixels of the capsule network module is utilized, the position and angle information of the segmented region is recorded by the activated vector, and the sharpening degree of the boundary region of the segmented region can be effectively improved.

The structure of the capsule network module is shown in fig. 4. The model comprises a convolution layer, a Primarycaps layer, a Digitcaps layer and a decoding layer, wherein each capsule represents a function, activated vectors are output, and the length of the vectors represents the correct probability of a region dividing line searched by the capsule. The functions of each layer of the capsule network module are respectively as follows:

and (3) rolling layers: by labeling CSM image level with samples { L^f,T^fConvolution operation is carried out to obtain primary characteristics of the spinal disc morphology, the pressed position and the like. Taking the CSM image with the input size of 28x28 as an example, the convolutional layer has 256 convolutional kernels of 9x9x1 with the step size of 1, and a feature tensor of 20x20x256 is output through feature extraction of the convolutional layer by using the Relu activation function.

PrimaryCaps layer: the capsule comprises 32 main capsules, receives primary characteristics obtained by the convolutional layer, generates a combination of the characteristics for each capsule and expresses the combination in a vectorization mode, and each dimension of a vector represents information such as the direction, the position and the like of the characteristics. Each primary capsule applies 8 convolution kernels of 9x9x256 to the input tensor of 20x20x256, which is a feature tensor of 6x6x8x32, due to the 32 primary capsules.

DigitCaps layer: each digital capsule corresponds to a vector output by a Primarycaps layer, each digital capsule receives a tensor of 6x6x8x32 as input, output variables of each main capsule are mapped into the multi-layer digital capsules in a nested mode by adopting a dynamic route, key features of the vectors are activated, and 8-dimensional input space is mapped into 16-dimensional capsule output space through a weight matrix of 8x 16.

A decoding layer: the decoding layer is the final full-connection layer of the network, comprises a Relu function and a Sigmoid function, receives a correct 16-dimensional vector output by the Digitcaps layer, is used as multiple characteristics expressed by an input learning capsule output vector, calculates edge loss, and learns to reconstruct an image with the size of 28x28 with the same pixels as an input image.

In the embodiment of the present application, the loss function of the capsule network module is as follows:

for a given CSM image level annotation sample { I^f,T^f} the corresponding loss function is:

in the formula (1), O_L((I^f,T^f)；θ_L) Representing the output of the capsule network module, theta_LWeights and parameters representing network training, J_bThe binary cross entropy loss function of the element in brackets is shown.

Application of Capsule network Module to CSM image-level labeled samples { I^f,T^fAnd outputting a semantic-free rough segmentation graph M ═ O_L((I^f,T^f)；θ_L)。

S400: tagging CSM image level with samples { I over a regional positioning network^f,T^fPredicting pixel level labels and outputting an area positioning feature map;

in step 400, the network structure of the area location network is shown in fig. 5. The regional positioning network generates a feature map containing position information by using the feature extraction of the convolutional layer, and adopts a global average pooling layer to apply weight (w)₁,w₂…,w_n) And carrying out weighted average on the feature map to obtain a region positioning feature map. The region with larger activation value in the region positioning characteristic map is most probably the damaged region segmentation position of the cervical vertebra of the spinal cord. The area positioning network fully utilizes the hot spot area of the primary characteristic positioning characteristic diagram obtained after the convolutional layer operation of the training sample, and the parameter quantity of the network can be reduced by recycling the characteristic diagram due to the fact that the convolutional neural network has a plurality of parameters needing to be trained, so that the training efficiency of the model is higher.

Step 500: the sample generation module executes a self-diffusion algorithm according to the reconstructed image output by the capsule network and the area positioning characteristic image output by the area positioning network, determines the partition line of the area pixel point, and obtains a rough partition prediction sample { I }^f,L^f*}；

In step 500, in order to output the segmentation map M containing the semantic information as the segmentation sample labeled at the pixel level, a Random Walk (Random Walk) idea is applied to the region with the larger activation value, the pixel points are diffused through a self-diffusion algorithm of the Random Walk, the input points of the region positioning feature map are utilized, the gaussian distance from each pixel to the input point on the image is calculated, the optimal path is selected from the input points, the segmentation line of the pixel points of the region is obtained, and the classification with the larger activation value is performed from each classificationPoint diffusion to finally generate a coarser partitioned prediction sample { I }^f,l^f*}。

Given a CSM image-level annotation sample { I^f,T^f-converting it into superpixels p ═ p1, p 2., pN }, the images being described by an undirected graph model G, where each node corresponds to a particular superpixel, and then performing a self-diffusion algorithm on the undirected graph model G. Defining an objective function of the self-diffusion process of the category on the basis of the rough segmentation graph M:

in equation (2), q ═ q1, q 2.., qN ] denotes a label vector of all superpixels p, and if pi ∈ a, qi is fixed to 1, otherwise its initial value is 0.

Z_i,j＝exp(-||F(p_i)-F(p_j)||/2σ²) (3)

In the formula (3), Z_i,jRepresenting the gaussian distance between two adjacent superpixels.

Through the above operations, semantic segmentation of the CSM image with only image-level annotations can be realized by using a small amount of high-quality image with pixel-level annotations.

Step 600: partitioned prediction samples { I } to be output by the generator^f,L^f*And true labeled samples { I }^f,L^fInputting the data into a discriminator to carry out 'generation-confrontation' training, and optimizing the discriminator;

in step 600, the discriminator uses the capsule network module to record the direction and position information of the segmentation region, improve the sharpening degree of the boundary region of the segmentation region, extract key region pixels which are difficult to correctly classify in the image by using the cascade mode, filter out simple and clear flat region pixels, and perform 'generation-confrontation' training by using the processed image until nash equilibrium is formed between the generator and the discriminator, and the discriminator cannot distinguish the segmentation prediction sample { I) generated by the generator from the image^f,L^f*Whether it is the true annotated sample { I }^f,L^fAnd finishing the training of generating the confrontation network.

As shown in fig. 3, in the embodiment of the present application, the discriminator includes a Cascade module, a Capsule network module, and a parameter optimization module; the specific functions of each module are as follows:

cascade of Cascade modules: a key pixel for taking charge of extracting a segmentation prediction sample; in the image segmentation process, the labeling difficulty of each pixel is different, a flat background area can be easily distinguished, but the boundary pixels of an object and the background area are difficult to distinguish. The prior network structure puts the pixels into the network for processing, which causes unnecessary redundancy of the network. The method adopts the concept of Cascade Cascade, and treats the pixels differently, so as to focus on key pixel regions which are difficult to classify. Extracting the key pixels with the labeling errors and the confidence coefficient lower than a certain threshold value and the corresponding ground route in the segmentation prediction samples generated by the generator, and filtering the pixels with the correct labeling and high confidence coefficient. Therefore, the pixels input to the next stage of training are only key pixels which are not easy to distinguish, redundant information in the network can be reduced, and the working efficiency of the network is improved.

The Capsule network module is responsible for processing the extracted key pixels and the corresponding ground route and generating errors; specifically, the functions of the Capsule network module include:

step 610: extracting local features; respectively inputting the key pixels and the corresponding ground truth into the corresponding convolution layers as input; then, a plurality of convolution layers are utilized to respectively convolve the input key pixels and the corresponding ground pixels, and a segmentation prediction sample { I } is extracted^f,L^f*Low-level features in (j); wherein the activation function of the convolutional layer is a ReLU function.

Step 611: extracting high-dimensional features; by constructing a PrimaryCaps layer, inputting the extracted low-level features into the PrimaryCaps layer to obtain high-dimensional feature vectors containing spatial position information; constructing Digitcaps layers, nesting and mapping output variables in the Primarycaps layers into the Digitcaps layers by adopting dynamic routing, constructing high-level features which can represent all input features most currently, and inputting the high-level features into the next layer;

the calculation between the PrimaryCaps layer and the DigitCaps layer involved in step 611 is as follows:

let u be the feature vector extracted after convolution of the key pixel and the corresponding group channel_iLow-level feature vector u_iAs input to the Primary caps layer, with a weight matrix W_ijMultiplying to obtain a prediction vectorWherein:

and weighted sum S can be obtained by linear combination between prediction vectors_jWeight coefficient of c_ijWherein:

obtaining a weighted sum S_jThen, S is compressed by a compression function_jLimiting the vector length to obtain an output vector V_jWherein:

in equation (6), the first half is the input vector S_jThe second half of the scaling scale of (1) is S_jThe unit vector of (2). While calculating S_jIn the course of (1), coefficient c_ijIs a constant value c_ijThe calculation formula of (2) is as follows:

in the formula (7), b_ijIs a constant value, b_ijB of the last iteration_ijValue of and V_jAndis obtained by summing the products of, i.e., b_ijThe updating method comprises the following steps:

step 612: the discriminator puts the high-level feature vector V output by the Digitcaps layer into the decoding layer, and finally outputs the discrimination result of the authenticity of the image through a plurality of full connection layers; the method specifically comprises the following steps: if the output result is 0, judging the input image to be false, and indicating that the input image is judged to be a fake image; if the output result is 1, the judgment is true, which indicates that the input image is mixed with the judger successfully.

A parameter optimization module: and optimizing the network parameters of the generator and the discriminator by using the error generated by the Capsule network module, so that the generator can output a more optimized segmentation result.

Prediction samples for a given partition { I^f,L^f*And the corresponding true labeled samples { I }^f,L^fThe overall error function of the network is as follows:

in formula (9), θ_SAnd theta_pParameters representing generator and arbiter, respectively, J_bRepresenting a binary cross-entropy loss function, O_sAnd O_pRespectively representing the output of the generator and the arbiter. When the input comes from a true annotated sample { I^f,L^fAnd partition prediction samples { I }^f,L^f*And when the pixel points are true, outputting 1 and 0 to mark the truth of the pixel point types.

In the embodiment of the present application, the process of parameter optimization includes two parts:

step 620: fixed generator parameter θ_SOptimizing the discriminator parameter theta_p(ii) a In the countermeasure training process, the generator parameter θ is first fixed_SThe divided prediction samples generated by the generator are sent to a discriminator,judging authenticity by a discriminator, and adjusting a discriminator parameter theta by a back propagation algorithm by using a discriminator error function_pAnd self-discrimination capability is improved. The error function corresponding to the discriminator is:

in the training process, the parameters of the discriminator are continuously optimized, the discrimination capability is continuously enhanced, and the generated image of the generator is more and more easily distinguished, so that the next stage is entered.

Step 621: fixed discriminator parameter theta_pOptimization of the Generator parameter θ_S(ii) a The network brings the discrimination result of the discriminator into the error function of the generator, and adjusts the parameter theta of the generator through a back propagation algorithm_SThe generator is caused to generate higher quality segmentation results, and thus the generator generates more accurate results to confuse the discriminator. And the error function for the generator is:

repeating the two optimization steps, and finally forming Nash balance between the generator and the discriminator, wherein the discriminator cannot distinguish the image from the segmentation prediction sample { I) output by the generator^f,L^f*Whether it is the true annotated sample { I }^f,L^fAnd fifthly, generating the confrontation network to finish training.

Step 700: and inputting the CSM image labeled at the image level into a trained generated countermeasure network, and outputting a pixel-level segmentation image of the CSM image through the generated countermeasure network.

Please refer to fig. 6, which is a schematic structural diagram of a medical image segmentation system based on generation of a countermeasure network according to an embodiment of the present application. The medical image segmentation system based on the generation countermeasure network comprises a sample acquisition module and the generation countermeasure network, the image sample acquired by the sample acquisition module is trained to generate the countermeasure network, the generation countermeasure network comprises a generator and a discriminator, the generator utilizes a capsule model to carry out structural feature representation, and further generation of a pixel-level labeled sample is achieved, the discriminator is used for distinguishing authenticity of the generated pixel-level labeled sample, a proper error optimization function is designed, discrimination results are respectively fed back to models of the generator and the discriminator, sample generation capacity and discrimination capacity of the generator and the discriminator are respectively improved through continuous countermeasure training, finally the trained generator is utilized to generate the pixel-level labeled sample, and pixel-level segmentation of the image-level labeled medical image is achieved. Specifically, the method comprises the following steps:

a sample collection module: the system comprises a processor, a display and a display module, wherein the processor is used for respectively acquiring pixel level annotation samples and CSM image level annotation samples of other medical images; wherein, other medical images include the medical image of parts such as lung, and the acquisition mode of mark sample specifically is: collection of a few fully labeled CSM image samples { I }^f,L^f,T^fA total of 500 DTI samples (diffusion tensor imaging) with an image size of 28x28, 60 CSM groups of 60 cases, wherein 27 men and 33 women are 33, the ages of 20-71, and the average age of 45, comprise pixel-level annotation samples and image-level annotation samples { L }^f,T^f}; pixel-level labeling samples { I } of other medical images (e.g., human lungs)^O,L^O8000 DTI samples of image size 28x 28. The method comprises the steps of respectively obtaining the CSM image and DTI samples of other medical images after correcting deformation and selecting a threshold value, determining a region of interest (ROI) on the DTI samples, and placing the ROI in the center of a spinal cord lesion area, so that the size of the DTI samples is uniform, and the influence of spinal fluid and artifacts is avoided.

The generator comprises a pre-training module, a capsule network module, a region positioning network module and a sample generating module, wherein the functions of the modules are as follows:

a pre-training module: the method is used for pre-training a capsule network module through pixel-level labeled samples of other medical images to obtain semantic label-free samples, processing CSM image-level labeled samples through the semantic label-free samples, and distinguishing backgrounds and effective segmentation areas of the CSM image-level labeled samples; the capsule network module adopts a migratable semantic segmentation model, and the model can transfer the learned segmentation knowledge from the pixel-level annotation of the fully-annotated data to the image-level annotation of the weakly-annotated data. In practical application, high-quality medical images are difficult to acquire, so that the method and the device enable new data to be matched with data of a target domain through a trained model, acquire the characteristics of pixel-level segmentation, and can also realize high-precision image segmentation under the condition of a small sample size. The pre-training process of the capsule network module specifically comprises the following steps:

1. processing pixel-level labeling samples of other medical images to convert the pixel-level labeling samples into semanteme-free segmentation image labels; the semantic-free segmentation image labeling only distinguishes the effective segmentation area of the sample from the background, but does not distinguish the shape of the sample. The network trained by the training data learns that the knowledge for distinguishing the object from the background belongs to wide high-order features, and the image is distinguished by the object from the background, so the wide high-order features have strong universality and can be easily migrated to other different tasks, the knowledge learned from the medical image labeled at a high-quality pixel level can be migrated to the medical image labeled at a low-quality image level, and the high-quality image and the low-quality image do not need to have direct relevance.

2. Processing the CSM image-level labeled sample according to the semanteme-free segmentation image label of other medical images to generate a semanteme-free label sample { I) of the CSM image-level labeled sample^O,L^OAnd obtaining pixel-level semantic-free labels L by filtering semantic information^O(ii) a The purpose of obtaining the semantic-free label at the pixel level is to distinguish an effective segmentation area and a background of data, so that learned knowledge can be transferred in the strongly and weakly labeled data more easily.

A capsule network module: for labelling CSM image level with samples { L^f,T^fInputting the capsule network module after the pre-training is finished, and outputting a reconstructed image of the CSM image-level labeling sample by the capsule network module; wherein, the capsule network module adopts the output vector of a single capsule neuron and records the edge pixels of the CSM image-level labeling sample segmentation regionAnd direction and position information, extracting the classified probability value by adopting a vector nonlinear activation function, determining the segmentation region and the background of the CSM image-level labeling sample, calculating the edge loss and outputting a reconstructed image of the CSM image-level labeling sample. According to the method and the device, parameter information of instantiated pixels of the capsule network module is utilized, the position and angle information of the segmented region is recorded by the activated vector, and the sharpening degree of the boundary region of the segmented region can be effectively improved.

The capsule network module comprises a convolution layer, a Primarycaps layer, a Digitcaps layer and a decoding layer, wherein each capsule represents a function, activated vectors are output, and the length of the vectors represents the correct probability of the region dividing line searched by the capsule. The functions of each layer of the capsule network module are respectively as follows:

A regional positioning network module: for annotating CSM image level samples { I^f,T^fPredicting pixel level labels and outputting an area positioning feature map; the area positioning network utilizes the feature extraction of the convolutional layer to generate a feature map containing position information, and adopts a global average pooling layer to apply weight (w)₁,w₂…,w_n) And carrying out weighted average on the feature map to obtain a region positioning feature map. The region with larger activation value in the region positioning characteristic map is most probably the damaged region segmentation position of the cervical vertebra of the spinal cord. The area positioning network fully utilizes the hot spot area of the primary characteristic positioning characteristic diagram obtained after the convolutional layer operation of the training sample, and the parameter quantity of the network can be reduced by recycling the characteristic diagram due to the fact that the convolutional neural network has a plurality of parameters needing to be trained, so that the training efficiency of the model is higher.

A sample generation module: the method is used for executing a self-diffusion algorithm according to the reconstructed image output by the capsule network module and the area positioning characteristic image output by the area positioning network, determining the partition line of the area pixel point and obtaining a rough partition prediction sample { I }^f,L^f*}; in order to output a segmentation graph M containing semantic information as a segmentation sample labeled at a pixel level, a random walk (random walk) idea is applied to a region with a larger activation value, pixel points are diffused through a random walk self-diffusion algorithm, the input points of a region positioning feature graph are utilized, the Gaussian distance from each pixel to the input points on an image is calculated, an optimal path is selected from the Gaussian distances, segmentation lines of the pixel points in the region are obtained, diffusion is performed from each classification point with a larger activation value, and finally a rough segmentation prediction sample { I is generated^f,L^f*}。

Z_i,j＝exp(-||F(p_i)-F(p_j)||/2σ²) (3)

Segmented prediction samples { I } to be generated by a generator^f,L^f*And true labeled samples { I }^f,L^fInputting the image into a discriminator to carry out confrontation training, recording the direction and position information of the segmentation area by the discriminator by utilizing a capsule network, improving the sharpening degree of the boundary area of the segmentation area, and extracting the image which is difficult to be correctly classified by utilizing a cascade modeKey region pixels of the class are filtered, simple and clear flat region pixels are filtered, and processed images are used for performing 'generation-confrontation' training until Nash equilibrium is formed between a generator and a discriminator, and the discriminator cannot distinguish that the images are segmentation prediction samples { I ] generated by the generator^f,L^f*Whether it is the true annotated sample { I }^f,L^fAnd finishing the training of generating the confrontation network.

Specifically, the discriminator comprises a Cascade Cascade module, a Capsule network module and a parameter optimization module; the specific functions of each module are as follows:

cascade of Cascade modules: a key pixel for taking charge of extracting a segmentation prediction sample; in the image segmentation process, the labeling difficulty of each pixel is different, a flat background area can be easily distinguished, but the boundary pixels of an object and the background area are difficult to distinguish. The prior network structure puts the pixels into the network for processing, which causes unnecessary redundancy of the network. The method adopts the concept of Cascade Cascade, and treats the pixels differently, so as to focus on key pixel regions which are difficult to classify. Pixels with wrong labels and pixels with confidence degrees lower than a certain threshold value in the segmentation prediction samples generated by the generator are extracted, and pixels with correct labels and high confidence degrees are filtered out. Therefore, the pixels input to the next stage of training are only key pixels which are not easy to distinguish, redundant information in the network can be reduced, and the working efficiency of the network is improved.

The Capsule network module is responsible for processing the extracted key pixels and generating errors; specifically, the functions of the Capsule network module include:

1. extracting local features; partitioned prediction samples { I } to be output by the generator^f,L^f*Filtering by a Cascade Cascade module, extracting key pixels and corresponding ground pixel, and inputting the key pixels and the corresponding ground pixel as input into corresponding convolution layers respectively; then, a plurality of convolution layers are utilized to respectively convolve the input key pixels and the corresponding ground pixels, and a segmentation prediction sample { I } is extracted^f,L^f*Low-level features in (j); wherein the activation of the convolutional layerThe number is the ReLU function.

2. Extracting high-dimensional features; by constructing a PrimaryCaps layer, inputting the extracted low-level features into the PrimaryCaps layer to obtain high-dimensional feature vectors containing spatial position information; constructing Digitcaps layers, nesting and mapping output variables in the Primarycaps layers into the Digitcaps layers by adopting dynamic routing, constructing high-level features which can represent all input features most currently, and inputting the high-level features into the next layer;

among the above, the calculation method between the PrimaryCaps layer and the DigitCaps layer is as follows:

3. the discriminator puts the high-level feature vector V output by the Digitcaps layer into the decoding layer, and finally outputs the discrimination result of the authenticity of the image through a plurality of full connection layers; the method specifically comprises the following steps: if the output result is 0, judging the input image to be false, and indicating that the input image is judged to be a fake image; if the output result is 1, the judgment is true, which indicates that the input image is mixed with the judger successfully.

A parameter optimization module: the method is used for optimizing the network parameters of the generator and the discriminator by utilizing the errors generated by the Capsule network module, so that the generator can output a more optimized segmentation result.

1. fixed generator parameter θ_SOptimizing the discriminator parameter theta_p(ii) a In the countermeasure training process, the generator parameter θ is first fixed_SThe divided prediction samples generated by the generator are sent to a discriminator, the discriminator judges the truth and the truth, and the discriminator parameter theta is adjusted by a back propagation algorithm by utilizing a discriminator error function_pAnd self-discrimination capability is improved. The error function corresponding to the discriminator is:

2. Fixed discriminator parameter theta_pOptimization of the Generator parameter θ_S(ii) a The network brings the discrimination result of the discriminator into the error function of the generator, and adjusts the parameter theta of the generator through a back propagation algorithm_SThe generator is caused to generate higher quality segmentation results, and thus the generator generates more accurate results to confuse the discriminator. And the error function for the generator is:

And inputting the CSM image to be segmented marked at the image level into a trained generated countermeasure network, and outputting the pixel-level segmented image of the CSM image to be segmented through the generated countermeasure network.

Fig. 7 is a schematic structural diagram of a hardware device of a medical image segmentation method based on generation of a countermeasure network according to an embodiment of the present application. As shown in fig. 3, the device includes one or more processors and memory. Taking a processor as an example, the apparatus may further include: an input system and an output system.

The processor, memory, input system, and output system may be connected by a bus or other means, as exemplified by the bus connection in fig. 3.

The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device, i.e., implements the processing method of the above-described method embodiment, by executing the non-transitory software program, instructions and modules stored in the memory.

The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processing system over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input system may receive input numeric or character information and generate a signal input. The output system may include a display device such as a display screen.

The one or more modules are stored in the memory and, when executed by the one or more processors, perform the following for any of the above method embodiments:

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the methods provided in the embodiments of the present application.

Embodiments of the present application provide a non-transitory (non-volatile) computer storage medium having stored thereon computer-executable instructions that may perform the following operations:

Embodiments of the present application provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the following:

The medical image segmentation method, the system and the electronic equipment based on the generation of the countermeasure network optimize the deep convolutional neural network through the fusion Capsule mechanism, fuse the ideas of the Capsule network and the cascade waterfall, generate a new training image sample under the condition that the quantity of the medical image sample is small, realize the semantic segmentation of the medical image data with low quality and only image level labels, transfer the learned segmentation knowledge from the full labeling data of pixel level labeling to the weak labeling data of image level, thereby improving the model characteristic expression capability, expanding the applicability of the medical image labeling sample, effectively reducing the dependence of the segmentation model on the pixel level labeling data, having the characteristics of less network information redundancy and sufficient feature extraction, and under the premise of a small quantity of pixel level labeling samples, improving the efficiency of the countertraining of the generated sample and a real sample, and can effectively realize high-precision pixel-level image segmentation.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical image segmentation method based on a generation countermeasure network is characterized by comprising the following steps:

2. The method for medical image segmentation based on generation of confrontation network according to claim 1, wherein in the step c, the generator comprises a capsule network module and a region localization network, and the generation of the segmentation prediction samples of the medical image to be segmented by the generator comprises in particular:

3. The medical image segmentation method based on generation countermeasure network of claim 2, wherein in the step b2, the capsule network module comprises a convolution layer, a PrimaryCaps layer, a DigitCaps layer and a decoding layer, the capsule network module records the direction and position information of the edge pixel of the image-level labeled sample segmentation region of the image to be segmented by using the output vector of a single capsule neuron, extracts the classified probability value by using the non-linear activation function of the vector, determines the segmentation region and the background of the image-level labeled sample of the image to be segmented, calculates the edge loss and outputs the reconstructed image of the image-level labeled sample of the image to be segmented.

4. The method for medical image segmentation based on generation of confrontation networks according to claim 2, wherein in step b4, the performing of the self-diffusion algorithm according to the reconstructed image and the region localization feature map specifically comprises: and diffusing pixel points in the area with the larger activation value in the area positioning feature map by using a random walk self-diffusion algorithm, calculating the Gaussian distance from each pixel to the input point on the image by using the input point of the area positioning feature map, selecting an optimal path from the Gaussian distances, obtaining a segmentation line of the pixel points in the area, and finally generating a segmentation prediction sample.

5. The medical image segmentation method based on generation of confrontation network according to any one of claims 1 to 4, wherein in the step d, the discriminator comprises a Cascade Cascade module, a Capsule network module and a parameter optimization module, and the training of "generation-confrontation" performed by the discriminator specifically comprises:

step d 2: processing the extracted key pixels and the corresponding ground channel through a Capsule network module, and generating errors;

step d 3: the parameter optimization module optimizes the network parameters of the generator and the discriminator by using errors generated by the Capsule network module; wherein for a given partitioned prediction sample { I }^f,L^f*And the corresponding true labeled samples { I }^f,L^fThe overall error function of the network is:

6. A medical image segmentation system based on a generation countermeasure network is characterized by comprising a sample acquisition module and the generation countermeasure network,

7. The generation-confrontation-network-based medical image segmentation system of claim 6 wherein the generator comprises a pre-training module, a capsule network module, a region-locating network module, and a sample generation module:

8. The generation-based countermeasure network medical image segmentation system of claim 7, wherein the capsule network module comprises a convolution layer, a PrimaryCaps layer, a DigitCaps layer and a decoding layer, the capsule network module records direction and position information of edge pixels of image-level labeled sample segmentation regions of the image to be segmented by using output vectors of single capsule neurons, extracts classified probability values by using a non-linear activation function of the vectors, determines segmentation regions and backgrounds of the image-level labeled samples of the image to be segmented, calculates edge loss and outputs a reconstructed image of the image-level labeled samples of the image to be segmented.

9. The system for medical image segmentation based on generation of a countermeasure network according to claim 7, wherein the sample generation module performs a self-diffusion algorithm based on the reconstructed image and the region localization feature map, specifically comprising: and diffusing pixel points in the area with the larger activation value in the area positioning feature map by using a random walk self-diffusion algorithm, calculating the Gaussian distance from each pixel to the input point on the image by using the input point of the area positioning feature map, selecting an optimal path from the Gaussian distances, obtaining a segmentation line of the pixel points in the area, and finally generating a segmentation prediction sample.

10. The medical image segmentation system based on generation of confrontation networks according to any one of claims 6 to 9, characterized in that the discriminator comprises a Cascade Cascade module, a Capsule network module and a parameter optimization module:

11. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the following operations of any one of the above 1 to 5 of the medical image segmentation method based on generation of countermeasure networks: