CN114782760B - Stomach disease picture classification system based on multitask learning - Google Patents

Stomach disease picture classification system based on multitask learning Download PDF

Info

Publication number
CN114782760B
CN114782760B CN202210711643.1A CN202210711643A CN114782760B CN 114782760 B CN114782760 B CN 114782760B CN 202210711643 A CN202210711643 A CN 202210711643A CN 114782760 B CN114782760 B CN 114782760B
Authority
CN
China
Prior art keywords
picture
disease
stomach
classification
auxiliary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210711643.1A
Other languages
Chinese (zh)
Other versions
CN114782760A (en
Inventor
戴捷
赖春晓
张希钢
鹿伟民
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zidong Information Technology Suzhou Co ltd
Original Assignee
Zidong Information Technology Suzhou Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zidong Information Technology Suzhou Co ltd filed Critical Zidong Information Technology Suzhou Co ltd
Priority to CN202210711643.1A priority Critical patent/CN114782760B/en
Publication of CN114782760A publication Critical patent/CN114782760A/en
Application granted granted Critical
Publication of CN114782760B publication Critical patent/CN114782760B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2431—Multiple classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Molecular Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a stomach disease picture classification system based on multitask learning, which comprises the following steps: the data acquisition module is used for acquiring pictures and respectively making data sets; the first feature processing module is used for obtaining a first picture feature; the second characteristic processing module is used for obtaining a second picture characteristic; the auxiliary disease classification module is used for training an auxiliary disease classifier by utilizing the first picture characteristics and inputting the second picture characteristics into the auxiliary disease classifier to obtain the disease probability; the third characteristic processing module is used for fusing the morbidity probability and the second picture characteristic through element-by-element multiplication to obtain a third picture characteristic; and the stomach disease classification module is used for training a stomach disease classifier by using the third picture characteristic and inputting the stomach disease picture test set into the trained stomach disease classifier to obtain a stomach disease picture classification result. According to the method, the similarity among diseases is utilized, and the multi-label classification recognition rate of the stomach picture is improved through a multi-task artificial intelligence model.

Description

Stomach disease picture classification system based on multitask learning
Technical Field
The invention relates to the technical field of image recognition, in particular to a stomach disease image classification system based on multitask learning.
Background
Gastric diseases are organic or functional diseases occurring in the stomach, and the causes of the diseases are very complex, including physical and chemical stimulation, infection, toxin, heredity, mental factors, developmental disturbance, operation influence and the like. Gastritis, gastric polyp, gastric ulcer, gastric cancer, bile reflux and gastrorrhagia are common in clinic. Early stage stomach diseases have no obvious symptoms, so the early stage stomach diseases are easy to ignore by people. When the stomach is diseased, the secretion and motor functions of the stomach are disturbed, which may affect digestion and absorption and sometimes cause anemia, emaciation or even life-threatening problems. In clinic, digestive diseases are the most common, and stomach diseases are the most common. The diagnosis of gastric diseases by stomach picture analysis of doctors consumes time and effort of doctors. The stomach image classification method based on the artificial intelligence technology can assist doctors to make more accurate judgment and can reduce the probability of the doctors missing suspicious early gastric cancer.
In the field of artificial intelligence, single-picture-based classification methods are commonly used for the determination of whether a stomach picture contains a stomach disease. This artificial intelligence approach is a process of extracting meaning from a single image using computer vision and machine learning algorithms. The most widespread artificial intelligence method at present is to model the problem as inputting an image and adopting a conventional classification algorithm to output labels of a plurality of diseases.
However, the existing model often processes the situation that the model input is a single picture, and ignores the situation that a plurality of pictures are used as input; secondly, because the cost for labeling the pictures with the stomach disease labels by professionals is higher, the number of the conventional labeled pictures containing the stomach diseases is less; in addition, the existing model only utilizes the stomach disease picture, but cannot utilize other related resources. Therefore, the recognition rate of the existing multi-label classification technology based on stomach pictures is not high enough.
Disclosure of Invention
The invention aims to provide a stomach disease image classification system based on multitask learning, which utilizes the similarity between diseases and improves the multi-label classification recognition rate of stomach images through a multitask artificial intelligence model.
In order to solve the above technical problem, the present invention provides a stomach disease picture classification system based on multitask learning, comprising:
the data acquisition module is used for acquiring pictures and respectively making an esophageal tumor picture data set, a duodenal ulcer picture data set and a stomach disease picture data set; the stomach disease picture data set comprises a stomach disease picture training set and a stomach disease picture testing set, and the pictures in the esophagus tumor picture data set, the duodenal ulcer picture data set and the stomach disease picture training set are all labeled with corresponding disease categories;
the first feature processing module is used for extracting the image features in the esophageal tumor image data set and the duodenal ulcer image data set to obtain a first image feature;
the second feature processing module is used for extracting picture features in the stomach disease picture training set to obtain second picture features;
the auxiliary disease classification module is used for training an auxiliary disease classifier by utilizing the first picture characteristics and inputting the second picture characteristics into the trained auxiliary disease classifier to obtain the classification probability of the stomach disease picture about the auxiliary disease;
the third feature processing module is used for fusing the classification probability and the second picture feature through element-by-element multiplication to obtain a third picture feature;
and the stomach disease classification module is used for training a stomach disease classifier by using the third picture characteristic and inputting the stomach disease picture test set into the trained stomach disease classifier to obtain a stomach disease picture classification result.
As a further improvement of the invention, the esophageal tumor picture data set, the duodenal ulcer picture data set and the stomach disease picture data set comprise a plurality of samples, and each sample comprisesnAnd (5) opening a picture.
As a further improvement of the present invention, the first feature processing module includes a first picture preprocessing module, and the first picture preprocessing module includes:
a first feature extractor for extracting features of the images in any one of the esophageal tumor image data set and the duodenal ulcer image data set to obtain image features
Figure 214321DEST_PATH_IMAGE001
；
A first feature segmentation module for segmenting features of the picture
Figure 291999DEST_PATH_IMAGE001
Partitioning to obtain a feature-segmented picture region set
Figure 294852DEST_PATH_IMAGE002
；
A first feature serialization module to serialize a feature of the first set of features
Figure 39954DEST_PATH_IMAGE002
All the regions in (a) are spliced in sequence to form a sequence
Figure 601386DEST_PATH_IMAGE003
。
As a further improvement of the present invention, the second feature processing module includes a second picture preprocessing module, and the second picture preprocessing module includes:
a second feature extractor for extracting features of the pictures in any sample of the stomach disease picture training set to obtain picture features
Figure 759835DEST_PATH_IMAGE004
；
A second feature segmentation module for segmenting the picture features
Figure 268176DEST_PATH_IMAGE004
Partitioning to obtain a feature-segmented picture region set
Figure 398943DEST_PATH_IMAGE005
；
A second feature serialization module to serialize the second feature
Figure 836003DEST_PATH_IMAGE004
All the regions in (a) are spliced in sequence to form a sequence
Figure 685010DEST_PATH_IMAGE006
。
As a further improvement of the present invention, the auxiliary disease classifier comprises:
the first Encoder layer adopts a pre-trained Vision Transformer framework, the pixel value of each region plus the position of the region in the whole sequence is embedded through the Encoder layer, and the intermediate representation of each region is obtained through multi-head attention mechanism learning of the Vision Transformer framework
Figure 669147DEST_PATH_IMAGE007
；
A first characteristic pooling module for pooling characteristics output by the first Encoder layer to obtain final representation of the picture
Figure 982317DEST_PATH_IMAGE008
；
A first fully-connected layer of
Figure 88813DEST_PATH_IMAGE009
（softmax) And (4) classifying:
Figure 425116DEST_PATH_IMAGE010
wherein the content of the first and second substances,
Figure 681785DEST_PATH_IMAGE011
and
Figure 115041DEST_PATH_IMAGE012
representing learnable weights and biases in the first fully connected layer,
Figure 64542DEST_PATH_IMAGE013
representing the classification probability of the output of the assisted disease classifier.
As a further improvement of the present invention, the training of the auxiliary disease classifier optimizes the objectives:
Figure 888142DEST_PATH_IMAGE014
wherein the content of the first and second substances,
Figure 309021DEST_PATH_IMAGE015
representing cross entropy loss between real auxiliary disease labels of images in the esophageal tumor image dataset and the duodenal ulcer image dataset and output prediction results of an auxiliary disease classifier;
Figure 472149DEST_PATH_IMAGE016
the first in the label representing true auxiliary diseasejThe number of the elements is one,
Figure 982765DEST_PATH_IMAGE017
indicating the probability of the auxiliary disease classifier outputting the predicted outcomejThe number of the elements is one,
Figure 28081DEST_PATH_IMAGE018
indicating the number of total categories of auxiliary diseases.
As a further improvement of the invention, the auxiliary disease classification module classifies the sequences
Figure 626553DEST_PATH_IMAGE019
Inputting the trained auxiliary disease classifier to obtain the classification probability of the stomach disease picture about the auxiliary disease
Figure 34400DEST_PATH_IMAGE020
。
As a further improvement of the present invention, the third feature processing module includes:
a matrix conversion module for converting the classification probabilities
Figure 122442DEST_PATH_IMAGE020
Expansion to the diseased matrix:
Figure 858317DEST_PATH_IMAGE021
wherein the content of the first and second substances,
Figure 385113DEST_PATH_IMAGE022
representing the matrix of disease
Figure 522834DEST_PATH_IMAGE023
To middleiLine ofjThe elements of the column are,
Figure 611138DEST_PATH_IMAGE024
representing classification probabilities
Figure 631046DEST_PATH_IMAGE020
In the prediction of disease classcThe probability of (a) of (b) being,cbelong to
Figure 164796DEST_PATH_IMAGE025
，
Figure 953760DEST_PATH_IMAGE026
The number of all regions for a plurality of pictures,
Figure 852446DEST_PATH_IMAGE027
the dimension of the first Encoder layer, i.e.,
Figure 687547DEST_PATH_IMAGE023
has a row dimension of
Figure 431512DEST_PATH_IMAGE026
In the column dimension of
Figure 668458DEST_PATH_IMAGE027
；
Feature fusion module to fuse sequences
Figure 472466DEST_PATH_IMAGE019
Diseased matrix corresponding to picture
Figure 30749DEST_PATH_IMAGE023
Performing element-by-element multiplication while adding sequences
Figure 578405DEST_PATH_IMAGE019
Thereby obtaining the final pictureSign representation
Figure 341961DEST_PATH_IMAGE028
：
Figure 707084DEST_PATH_IMAGE029
Wherein the content of the first and second substances,
Figure 126564DEST_PATH_IMAGE030
the correlation coefficients representing the different disease classes,
Figure 336965DEST_PATH_IMAGE030
belong to
Figure 158291DEST_PATH_IMAGE031
And represents a general multiplication of the data of the original,
Figure 366418DEST_PATH_IMAGE032
representing the multiplication of corresponding elements of the two matrices.
As a further improvement of the present invention, the gastric disease classifier comprises:
the second Encoder layer adopts a pre-trained Vision Transformer framework, the pixel value of each region and the position of the region in the whole sequence are embedded through the Encoder layer, and the middle representation of each region is obtained through multi-head attention mechanism learning of the Vision Transformer framework
Figure 397828DEST_PATH_IMAGE033
；
A second feature pooling module for pooling features output by the second Encoder layer to obtain final representation of the picture
Figure 287287DEST_PATH_IMAGE034
；
A second fully-connected layer of
Figure 589217DEST_PATH_IMAGE035
（sigmoid) And (4) classifying:
Figure 171508DEST_PATH_IMAGE036
wherein the content of the first and second substances,
Figure 690214DEST_PATH_IMAGE037
and
Figure 383364DEST_PATH_IMAGE038
representing learnable weights and biases in the second fully connected layer;
Figure 710440DEST_PATH_IMAGE039
a set of classification probabilities for each label representing gastric disease.
As a further improvement of the present invention, the training of the gastric disease classifier optimizes the objectives:
Figure 588266DEST_PATH_IMAGE040
wherein the content of the first and second substances,
Figure 204055DEST_PATH_IMAGE041
representing a two-class cross entropy loss between a true gastric disease label for a picture in the gastric disease picture dataset and a prediction result output by the gastric disease classifier;
Figure 825529DEST_PATH_IMAGE042
the first of the labels representing real gastric diseasesjThe number of the elements is one,
Figure 272691DEST_PATH_IMAGE043
representing the probability of the classifier outputting the prediction result of the gastric diseasejAn element;
Figure 931206DEST_PATH_IMAGE044
indicating the number of gastric disease categories.
The invention has the beneficial effects that: the method solves the defects of the prior art, utilizes the correlation among diseases, uses the prediction result of the esophageal tumor and the prediction result of the duodenal ulcer to assist the prediction of the stomach diseases, and assists the prediction of the existing stomach diseases through the prediction results of related tasks (non-stomach diseases such as the esophageal diseases and the duodenal ulcer), thereby improving the identification accuracy of the multi-label classification technology based on stomach pictures; the invention can process the multi-picture input of a case; meanwhile, the invention adopts a multitask artificial intelligence model: the auxiliary disease classifier and the stomach disease classifier can actually predict the categories of auxiliary diseases, and play a role in win-win; the auxiliary diseases adopted by the invention are similar to the input pictures used by stomach diseases, the prediction is assisted by utilizing the correlation among the diseases, and the marked data can be expanded actually, so that the burden of data annotation is relieved.
Drawings
FIG. 1 is a schematic diagram of the system architecture of the present invention;
fig. 2 is a process schematic of an embodiment of the invention.
Detailed Description
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
As described in the background, the prior art is divided into the following steps: (1) a professional annotates a large number of pictures with gastric disease labels, each picture is used as a sample, and a plurality of annotation corpora with annotation samples are obtained; (2) training and labeling linguistic data based on a deep learning network (generally a multilayer convolutional neural network) to obtain a classification model; (3) and predicting the picture with an unknown label by using a classification model to obtain the stomach disease label of the picture. In the prediction process, each time the classification model is input, a single picture is input. In the classification of gastric diseases, the labels are 7 of gastritis, gastric polyps, gastric ulcers, gastric cancer, bile reflux, gastric bleeding and normal.
The deep learning network of the second step generally adopts a convolutional neural network and an attention-based Transformer network, and includes an Encoder layer (Encoder) and an FC layer (full connection layer). The Encoder layer is responsible for extracting the features of the image and generally comprises a series of convolutional layers, activation layers, pooling layers, self-attention layers and the like, and commonly used Encoder layers comprise transformers, VGG, Residual Net, Dense Net, efficiency Net and the like. The FC layer is responsible for mapping image features to classes of pictures. Inputting an image, and coding the image through a series of convolutional layers, activation layers, pooling layers, self-attention layers and the like in the model to obtain the characteristics of the image; and finally, classifying the images through full connection. It is worth noting that: (1) the input of the traditional disease diagnosis model is a single picture, and the input of a plurality of pictures cannot be processed simultaneously; (2) the traditional disease diagnosis model can only use limited stomach disease labeling samples, and can not use labeling samples of other diseases.
However, there are many similarities between esophageal and duodenal diseases and gastric diseases, for example, the shape of esophageal tumors is similar to the shape of gastric tumors, the pictures of duodenal ulcers and gastric ulcers are similar, and the existing data sets for esophageal tumors and duodenal ulcers are relatively numerous. In other words, if a model can effectively predict the symptoms presented to the duodenum, then the model can more likely accurately infer the disease state of the stomach. However, in the prior art, the similarity of diseases of different parts is neglected, and the similarity can assist the prediction of the existing stomach diseases through the prediction result of related tasks (non-stomach diseases such as esophageal diseases and duodenal ulcer) on the premise of not increasing the labor cost, so that the identification accuracy of the multi-label classification technology based on the stomach images is improved.
Thus, with reference to fig. 1, the present invention provides a picture classification system for gastric diseases based on multitask learning, comprising:
the data acquisition module is used for acquiring pictures and respectively making an esophageal tumor picture data set, a duodenal ulcer picture data set and a stomach disease picture data set; the stomach disease picture data set comprises a stomach disease picture training set and a stomach disease picture testing set, and pictures in the esophagus tumor picture data set, the duodenal ulcer picture data set and the stomach disease picture training set are all labeled with corresponding disease types;
the first feature processing module is used for extracting picture features in the esophageal tumor picture data set and the duodenal ulcer picture data set to obtain first picture features;
the second feature processing module is used for extracting picture features in the stomach disease picture training set to obtain second picture features;
the auxiliary disease classification module is used for training an auxiliary disease classifier by utilizing the first picture characteristics and inputting the second picture characteristics into the trained auxiliary disease classifier to obtain the classification probability of the stomach disease picture about the auxiliary disease;
the third feature processing module is used for fusing the classification probability and the second picture feature through element-by-element multiplication to obtain a third picture feature;
and the stomach disease classification module is used for training the stomach disease classifier by using the third picture characteristic and inputting the stomach disease picture test set into the trained stomach disease classifier to obtain a stomach disease picture classification result.
The deep learning model of the stomach disease image classification based on the multitask learning is shown in figure 1, firstly, a large number of marked esophageal tumor image data sets and duodenal ulcer image data sets are obtained, each sample in the data sets is composed of a plurality of images, an auxiliary disease classifier composed of Encoder1 and FC1 layers is constructed, and the auxiliary classifier is pre-trained (esophageal tumor and duodenal ulcer images are input, and disease categories are output); secondly, acquiring a small amount of stomach picture samples containing disease categories, and acquiring the probability that the stomach picture is possibly ill about the esophagus or the duodenum through an auxiliary disease classifier; and finally, taking the prevalence probability of the auxiliary diseases as a clue, fusing the prevalence probability with the stomach pictures, and training the stomach disease classifier through a newly constructed main task classifier (consisting of Encoder2 and FC2 layers). The auxiliary disease classifier was trained using all of the picture samples in the esophageal tumor picture dataset and the duodenal ulcer picture dataset (both of these tasks may be referred to as auxiliary tasks). Then, a plurality of pictures in the same case in the stomach disease data set (the task is a main task) are processed by an auxiliary disease classifier, and the presumed disease probability about the esophagus or the duodenum of the stomach pictures is obtained. And finally, fusing the stomach picture characteristics with the morbidity probability of the auxiliary diseases, and obtaining the type prediction of the stomach diseases through a stomach disease classifier formed by an Encoder2 layer and an FC2 layer. The invention can process the multi-picture input of a case, adopts a multi-task artificial intelligence model: the auxiliary disease classifier and the stomach disease classifier can actually predict the type of the auxiliary disease and play a role of win-win, the auxiliary disease adopted by the invention is similar to an input picture used by the stomach disease, the prediction is assisted by utilizing the correlation among the diseases, and the effect of expanding marked data can be actually played, so that the burden of data annotation is relieved.
The invention includes assisting in the training of disease classifiers and in the training of gastric disease classifiers. Specifically, the method comprises the following steps:
first, training phase of auxiliary disease classifier
1. Characteristic extraction: in one example of a given esophageal tumor image dataset and duodenal ulcer image datasetnPicture frame
Figure 660389DEST_PATH_IMAGE045
Sequentially transmitting them to a feature extractor of the picture to obtain picture features
Figure 695342DEST_PATH_IMAGE046
. The feature extractor adopts the current commonly used picture feature extractor Residual Network (2)ResNet）：
Figure 262589DEST_PATH_IMAGE047
Wherein the content of the first and second substances,
Figure 216639DEST_PATH_IMAGE048
is shown as
Figure 72599DEST_PATH_IMAGE048
And (5) opening a picture.
2. And (3) feature segmentation: characterizing a picture
Figure 770297DEST_PATH_IMAGE046
Partitioning to obtain a plurality of non-overlapping regular pixel blocks: a picture is divided into
Figure 926472DEST_PATH_IMAGE049
The number of the small areas is small,npicture is divided into
Figure 192368DEST_PATH_IMAGE050
And (4) small area:
Figure 394679DEST_PATH_IMAGE051
wherein the content of the first and second substances,
Figure 771434DEST_PATH_IMAGE052
for the set of picture regions after feature segmentation,
Figure 865337DEST_PATH_IMAGE053
is a blocking function.
3. Characteristic ordering: will be provided with
Figure 98872DEST_PATH_IMAGE052
All the small regions in the sequence are spliced in sequence to form a sequence
Figure 663845DEST_PATH_IMAGE054
。
4. Sequence coding: the Encoder layer adopts a pre-trained Vision transform framework (VT), pixel values of each small region are added with position embedding (positions in the whole sequence), the middle representation of each small region is obtained through multi-head attention mechanism learning of a multi-layer transform through the Encoder
Figure 703346DEST_PATH_IMAGE055
：
Figure 37375DEST_PATH_IMAGE056
4. Characteristic pooling: encoder output characteristics are fused (pooled) via featuresPoolingObtain the final representation of the picture:
Figure 769708DEST_PATH_IMAGE057
5. and (4) classification: through the use of a full connection layer
Figure 618715DEST_PATH_IMAGE058
（softmax) And (4) classifying:
Figure 337272DEST_PATH_IMAGE059
wherein the content of the first and second substances,
Figure 650442DEST_PATH_IMAGE011
and
Figure 429042DEST_PATH_IMAGE012
representing learnable weights and biases in the fully connected layer.
Figure 594706DEST_PATH_IMAGE060
Namely, it is
Figure 913692DEST_PATH_IMAGE008
，
Figure 956735DEST_PATH_IMAGE061
Representing the classification probability of the output of the assisted disease classifier.
6. Loss optimization: in summary, the following are optimization goals for the assisted disease classifier:
Figure 30870DEST_PATH_IMAGE062
wherein the content of the first and second substances,
Figure 588890DEST_PATH_IMAGE063
cross entropy loss between the true auxiliary disease label representing the picture in the esophageal tumor picture dataset, the duodenal ulcer picture dataset, and the auxiliary disease classifier output prediction results, which needs to be minimized;
Figure 242725DEST_PATH_IMAGE064
the first in the label (category) representing true auxiliary diseasesjThe number of the elements is one,
Figure 202591DEST_PATH_IMAGE065
indicating the probability of the auxiliary disease classifier outputting the predicted outcomejThe number of the elements is one,
Figure 322994DEST_PATH_IMAGE066
indicating the number of total categories of auxiliary diseases.
Second, training phase of stomach disease classifier
Picture preprocessing: extracting, segmenting and ordering the features of multiple stomach pictures in the same case in a given stomach disease picture data set to form a sequence, and obtaining the sequence and the auxiliary disease pictures
Figure 696206DEST_PATH_IMAGE054
The same procedure is used. However, to distinguish from the sequence of picture generation for the auxiliary disease, the sequence of picture acquisition for the gastric disease is represented as
Figure 356995DEST_PATH_IMAGE067
。
1. Auxiliary disease incidence: sequence of
Figure 374629DEST_PATH_IMAGE067
Is trainedLater assisted disease classifier to obtain classification probabilities
Figure 557611DEST_PATH_IMAGE020
。
2. Generating a disease matrix: probability of classification
Figure 27907DEST_PATH_IMAGE020
Expansion to the disease matrix:
Figure 820282DEST_PATH_IMAGE068
wherein the content of the first and second substances,
Figure 489161DEST_PATH_IMAGE022
representing the matrix of the disease
Figure 216946DEST_PATH_IMAGE023
To middleiLine ofjThe elements of the column are,
Figure 299171DEST_PATH_IMAGE024
indicating the outcome of the assisted disease classification
Figure 301762DEST_PATH_IMAGE020
In the prediction of disease classcProbability of belonging to
Figure 293989DEST_PATH_IMAGE025
，
Figure 317309DEST_PATH_IMAGE026
The number of all the small squares for a plurality of pictures,
Figure 496617DEST_PATH_IMAGE027
dimension of Encoder. Therefore, the temperature of the molten metal is controlled,
Figure 132260DEST_PATH_IMAGE023
has a row dimension of
Figure 41310DEST_PATH_IMAGE026
In the column dimension of
Figure 845318DEST_PATH_IMAGE027
。
3. Characteristic processing: characterizing a picture sequence
Figure 167715DEST_PATH_IMAGE067
Diseased matrix corresponding to picture
Figure 449792DEST_PATH_IMAGE023
Performing element-by-element multiplication while adding the original picture feature sequence
Figure 541245DEST_PATH_IMAGE019
To obtain the final picture feature representation
Figure 312892DEST_PATH_IMAGE028
：
Figure 997951DEST_PATH_IMAGE029
Wherein the content of the first and second substances,
Figure 473932DEST_PATH_IMAGE069
is set correlation coefficient of different disease categories, and the range belongs to
Figure 29678DEST_PATH_IMAGE031
Different from each othercCorresponding to
Figure 67166DEST_PATH_IMAGE069
The settings are different. Denotes a normal multiplication for multiplying the coefficients.
Figure 770680DEST_PATH_IMAGE032
Representing the multiplication of corresponding elements of two matrices. Thus the disease information of the auxiliary task is fused to the characteristic sequence of the main task.
4. Sequence coding: v with pretrained Encoder layerThe final feature representation of each small region plus position embedding (positions in the whole sequence) passes through an Encoder, and the intermediate representation of each small region is obtained through multi-head attention mechanism learning of a multi-layer Transformer
Figure 660139DEST_PATH_IMAGE033
：
Figure 460605DEST_PATH_IMAGE070
5. Characteristic pooling: and (3) obtaining final representation of the multivariate picture through feature fusion (pooling) of the Encoder output features:
Figure 777316DEST_PATH_IMAGE071
6. and (4) classification: through the use of a full connection layer
Figure 968126DEST_PATH_IMAGE072
（sigmoid) And (4) classifying:
Figure 785910DEST_PATH_IMAGE073
wherein the content of the first and second substances,
Figure 316248DEST_PATH_IMAGE037
and
Figure 194074DEST_PATH_IMAGE038
representing learnable weights and biases in the fully connected layer.
Figure 872180DEST_PATH_IMAGE039
A set of classification probabilities for each label representing gastric disease.
7. Loss optimization: in summary, the following are the optimization objectives:
Figure 932802DEST_PATH_IMAGE040
wherein the content of the first and second substances,
Figure 379964DEST_PATH_IMAGE041
representing a two-class cross entropy loss between a true gastric disease label for a picture in the gastric disease picture dataset and a prediction result output by the gastric disease classifier;
Figure 38479DEST_PATH_IMAGE042
the first in the label representing real stomach diseasesjThe number of the elements is one,
Figure 531777DEST_PATH_IMAGE043
representing the probability of the outcome of a prediction output by a gastric disease classifierjAn element;
Figure 566729DEST_PATH_IMAGE044
indicating the number of gastric disease categories.
Examples
As shown in FIG. 2, the samples in the stomach disease picture test set are input into the trained artificial intelligence model of the stomach disease picture classification system based on multi-task learning provided by the present invention to obtain the classification result of the stomach disease. Meanwhile, 2 experienced endoscopists are invited to interpret and diagnose the stomach pictures of the test set. The overall accuracy, sensitivity and positive predictive value of the model and 2 doctors in disease diagnosis are obtained.
The method for calculating the diagnostic effect evaluation index comprises the following steps: overall accuracy = number of cases identified correct/number of actual cases tested set disease x 100%; sensitivity = number of correctly identified cases of a certain category/number of actual cases of the category x 100%; positive predictive value = number of cases identified as correct for a certain category/number of cases identified as that category by the model or endoscopist x 100%.
Specifically, raw data is collected: the invention collects gastroscopic pictures of patients in endoscopic central gastroscopy (including painless gastroscopy, conscious sedation gastroscopy and ordinary gastroscopy). The picture taking devices are mainly endoscopes of Olympus 240, 260, 290 series and Fujinon 560, 580 series, japan. All pictures are taken in a white light non-amplification mode, and optical dyeing such as BLI, FICE, NBI and the like and chemical dyeing such as indigo carmine, acetic acid dyeing amplification and the like are not studied for the moment. Inclusion criteria were: the diagnosis is as follows: gastritis, gastric polyps, gastric ulcers, gastric cancer, bile reflux, gastrorrhagia and normal gastroscopic mucosa pictures. Exclusion criteria: patients are under 16 years of age or over 95 years of age; secondly, the observed pictures are influenced by abnormal blurring, artifacts, abnormal distortion and the like of the pictures; and thirdly, a large amount of foam, viscous lake or food and other pictures with serious interference exist.
Construction of a data set and picture preprocessing: according to different application models, the method is divided into an esophageal tumor image data set, a 12-finger intestinal ulcer image data set and a stomach disease image data set. The esophageal tumor and 12-finger intestinal ulcer picture data set contains 20,000 samples in total, and the stomach disease picture data set contains 3460 samples in total. In the data set of the esophageal tumor and 12-finger intestinal ulcer pictures, the number of cases of esophageal tumor, normal esophagus, 12-finger intestinal ulcer and normal 12-finger intestine is respectively as follows: 4872, 5675, 4325 and 5128. In the classification data set of stomach diseases, the number of cases of gastritis, gastric polyp, gastric ulcer, gastric cancer, bile reflux, gastrorrhagia and normal stomach are respectively: 650 cases, 610 cases, 400 cases, 500 cases, 200 cases, 450 cases, and 650 cases. The acquired case image data is processed through a series of image operations such as image format conversion, image size scaling, image enhancement, image normalization and the like so as to ensure the identification of the image by the artificial intelligence model. In order to ensure that no repeated part exists in the cases of the training set, the verification set and the test set, the original cases are divided into the training set (520 cases of gastritis, 488 cases of gastric polyp, 320 cases of gastric ulcer, 400 cases of gastric cancer, 160 cases of bile reflux, 360 cases of gastric bleeding and 520 cases of normal stomach) and the test set (65 cases of gastritis, 61 cases of gastric polyp, 40 cases of gastric ulcer, 50 cases of gastric cancer, 200 cases of bile reflux, 450 cases of gastric bleeding and 650 cases of normal stomach) and (65 cases of gastritis, 61 cases of gastric polyp, 40 cases of gastric ulcer, 50 cases of gastric cancer, 200 cases of bile reflux, 450 cases of gastric bleeding and 650 cases of normal stomach) according to the proportion of approximately 8:1:1 by using a random principle. Training the model parameters of the artificial intelligence through a training set, then verifying the effectiveness of the model by using a verification set, improving the generalization capability of the model, carrying out optimal adjustment on the parallel hyper-parameters to form a final artificial intelligence algorithm model, and finally evaluating and considering the performance of the artificial intelligence algorithm model through a test set.
The specific results are as follows: the overall accuracy of the doctor was 86%, and the overall accuracy of the invention was 91.5%. The sensitivity of the doctor was 87%, and the sensitivity of the present invention was 90%. The positive predictive value of the doctor is 89%, the positive predictive value of the invention is 92%, and the overall accuracy, sensitivity and positive predictive value of the stomach disease identification of the invention are obviously superior to those of an experienced endoscope doctor; compared with the method for training the stomach disease classifier (single task) independently, the auxiliary disease strategy (multi-task) adopted by the method can effectively reduce data annotation of the stomach disease, and as the single task method needs to label 2000 samples under the condition that the sample prediction accuracy is 85%, the multi-task method provided by the invention only needs to label 1500 samples.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention. The protection scope of the invention is subject to the claims.

Claims (10)

1. A stomach disease picture classification system based on multitask learning is characterized in that: the method comprises the following steps:
the data acquisition module is used for acquiring pictures and respectively making an esophageal tumor picture data set, a duodenal ulcer picture data set and a stomach disease picture data set; the stomach disease picture data set comprises a stomach disease picture training set and a stomach disease picture testing set, and the pictures in the esophagus tumor picture data set, the duodenal ulcer picture data set and the stomach disease picture training set are all labeled with corresponding disease categories;
the first feature processing module is used for extracting picture features in the esophageal tumor picture data set and the duodenal ulcer picture data set to obtain first picture features;
the second feature processing module is used for extracting picture features in the stomach disease picture training set to obtain second picture features;
the auxiliary disease classification module is used for training an auxiliary disease classifier by utilizing the first picture characteristics and inputting the second picture characteristics into the trained auxiliary disease classifier to obtain the classification probability of the stomach disease picture about the auxiliary disease;
the third feature processing module is used for fusing the classification probability and the second picture features through element-by-element multiplication to obtain third picture features;
and the stomach disease classification module is used for training the stomach disease classifier by using the third picture characteristic and inputting the stomach disease picture test set into the trained stomach disease classifier to obtain a stomach disease picture classification result.
2. The system of claim 1, wherein the image classification system comprises: the esophageal tumor image data set, the duodenal ulcer image data set and the stomach disease image data set comprise a plurality of samples, and each sample comprisesnAnd (5) opening a picture.
3. The system of claim 2, wherein the image classification system comprises: the first feature processing module comprises a first picture preprocessing module, and the first picture preprocessing module comprises:
a first feature extractor for extracting features of the images in any one of the esophageal tumor image data set and the duodenal ulcer image data set to obtain image features
Figure 474398DEST_PATH_IMAGE001
；
A first feature segmentation module for segmenting features of the picture
Figure 919286DEST_PATH_IMAGE001
Partitioning to obtain a feature-segmented picture region set
Figure 191480DEST_PATH_IMAGE002
；
A first feature serialization module for aggregating picture regions
Figure 303792DEST_PATH_IMAGE002
All the regions in (a) are spliced in sequence to form a sequence
Figure 311062DEST_PATH_IMAGE003
。
4. The system of claim 3, wherein the image classification system comprises: the second feature processing module comprises a second picture preprocessing module, and the second picture preprocessing module comprises:
a second feature extractor for extracting features of the pictures in any sample in the stomach disease picture training set to obtain picture features
Figure 508826DEST_PATH_IMAGE004
；
A second feature segmentation module for segmenting the picture features
Figure 56482DEST_PATH_IMAGE004
Partitioning to obtain a feature-segmented picture region set
Figure 288880DEST_PATH_IMAGE005
；
A second feature serialization module for aggregating picture regions
Figure 263789DEST_PATH_IMAGE004
All the regions in (a) are spliced in sequence to form a sequence
Figure 152111DEST_PATH_IMAGE006
。
5. The system of claim 4, wherein the image classification system comprises: the auxiliary disease classifier includes:
the first Encoder layer adopts a pre-trained Vision Transformer framework, the pixel value of each region and the position of the region in the whole sequence are embedded through the Encoder layer, and the intermediate representation of each region is obtained through multi-head attention mechanism learning of the Vision Transformer framework
Figure 503458DEST_PATH_IMAGE007
；
A first characteristic pooling module for pooling characteristics output by the first Encoder layer to obtain final representation of the picture
Figure 590362DEST_PATH_IMAGE008
；
A first fully-connected layer of
Figure 1752DEST_PATH_IMAGE009
The classification is carried out, and the classification is carried out,
Figure 908528DEST_PATH_IMAGE009
by usingsoftmaxFunction:
Figure 797987DEST_PATH_IMAGE010
wherein the content of the first and second substances,
Figure 739398DEST_PATH_IMAGE011
and
Figure 256442DEST_PATH_IMAGE012
representing learnable weights in a first fully connected layerAnd a bias means for biasing the movable member in a direction perpendicular to the axis,
Figure 650515DEST_PATH_IMAGE013
representing the classification probability of the output of the assisted disease classifier.
6. The system of claim 5, wherein the image classification system comprises: training optimization objectives of the assisted disease classifier are as follows:
Figure 343664DEST_PATH_IMAGE014
wherein the content of the first and second substances,
Figure 139582DEST_PATH_IMAGE015
representing cross entropy loss between real auxiliary disease labels of images in the esophageal tumor image dataset and the duodenal ulcer image dataset and output prediction results of an auxiliary disease classifier;
Figure 892774DEST_PATH_IMAGE016
the first in the label representing true auxiliary diseasejThe number of the elements is one,
Figure 39722DEST_PATH_IMAGE017
indicating the probability of the auxiliary disease classifier outputting the predicted outcomejThe number of the elements is one,
Figure 270983DEST_PATH_IMAGE018
indicating the number of total categories of auxiliary diseases.
7. The system of claim 6, wherein the image classification system comprises: the auxiliary disease classification module classifies the sequences
Figure 921407DEST_PATH_IMAGE019
Inputting a trained auxiliary disease classifierObtaining the classification probability of the stomach disease picture about the auxiliary disease
Figure 845501DEST_PATH_IMAGE020
。
8. The system of claim 7, wherein the image classification system comprises: the third feature processing module includes:
a matrix conversion module for converting the classification probabilities
Figure 214165DEST_PATH_IMAGE020
Expansion to the disease matrix:
Figure 249117DEST_PATH_IMAGE021
wherein the content of the first and second substances,
Figure 957310DEST_PATH_IMAGE022
representing the matrix of disease
Figure 52305DEST_PATH_IMAGE023
To middleiLine ofjThe elements of the column(s) are,
Figure 908266DEST_PATH_IMAGE024
representing classification probabilities
Figure 478400DEST_PATH_IMAGE020
In the prediction of disease classcThe probability of (a) of (b) being,cbelong to
Figure 103416DEST_PATH_IMAGE025
，
Figure 634892DEST_PATH_IMAGE026
The number of all regions for a plurality of pictures,
Figure 712569DEST_PATH_IMAGE027
is a dimension of the first Encoder layer, that is,
Figure 354903DEST_PATH_IMAGE023
has a row dimension of
Figure 834426DEST_PATH_IMAGE026
In the column dimension of
Figure 271224DEST_PATH_IMAGE027
；
Feature fusion module to fuse sequences
Figure 39460DEST_PATH_IMAGE019
Diseased matrix corresponding to picture
Figure 954326DEST_PATH_IMAGE023
Performing element-by-element multiplication while adding sequences
Figure 553935DEST_PATH_IMAGE019
To obtain the final picture feature representation
Figure 161633DEST_PATH_IMAGE028
：
Figure 213903DEST_PATH_IMAGE029
Wherein the content of the first and second substances,
Figure 932460DEST_PATH_IMAGE030
the correlation coefficients representing the different disease classes,
Figure 386575DEST_PATH_IMAGE030
belong to
Figure 430755DEST_PATH_IMAGE031
And denotes a number of the ordinary multiplications,
Figure 967391DEST_PATH_IMAGE032
representing the multiplication of corresponding elements of the two matrices.
9. The system of claim 8, wherein the image classification system comprises: the gastric disease classifier includes:
the second Encoder layer adopts a pre-trained Vision Transformer framework, the pixel value of each region and the position of the region in the whole sequence are embedded through the Encoder layer, and the middle representation of each region is obtained through multi-head attention mechanism learning of the Vision Transformer framework
Figure 692901DEST_PATH_IMAGE033
；
A second feature pooling module for pooling features output by the second Encoder layer to obtain final representation of the picture
Figure 1523DEST_PATH_IMAGE034
；
A second fully-connected layer of
Figure 951024DEST_PATH_IMAGE035
The classification is carried out, and the classification is carried out,
Figure 712307DEST_PATH_IMAGE035
by usingsigmoidFunction:
Figure 303825DEST_PATH_IMAGE036
wherein the content of the first and second substances,
Figure 466953DEST_PATH_IMAGE037
and
Figure 587356DEST_PATH_IMAGE038
representing learnable weights and biases in the second fully connected layer;
Figure 101514DEST_PATH_IMAGE039
a set of classification probabilities for each label representing gastric disease.
10. A gastric disease picture classification system based on multitasking learning according to any one of claims 1-9, characterized by that: the training of the gastric disease classifier optimizes a goal:
Figure 965565DEST_PATH_IMAGE040
wherein the content of the first and second substances,
Figure 186462DEST_PATH_IMAGE041
representing a two-class cross entropy loss between a true gastric disease label for a picture in the gastric disease picture dataset and a prediction result output by the gastric disease classifier;
Figure 743345DEST_PATH_IMAGE042
the first of the labels representing real gastric diseasesjThe number of the elements is one,
Figure 479220DEST_PATH_IMAGE043
representing the probability of the outcome of a prediction output by a gastric disease classifierjAn element;
Figure 881382DEST_PATH_IMAGE044
indicating the number of gastric disease categories.
CN202210711643.1A 2022-06-22 2022-06-22 Stomach disease picture classification system based on multitask learning Active CN114782760B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210711643.1A CN114782760B (en) 2022-06-22 2022-06-22 Stomach disease picture classification system based on multitask learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210711643.1A CN114782760B (en) 2022-06-22 2022-06-22 Stomach disease picture classification system based on multitask learning

Publications (2)

Publication Number Publication Date
CN114782760A CN114782760A (en) 2022-07-22
CN114782760B true CN114782760B (en) 2022-09-13

Family

ID=82422505

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210711643.1A Active CN114782760B (en) 2022-06-22 2022-06-22 Stomach disease picture classification system based on multitask learning

Country Status (1)

Country Link
CN (1) CN114782760B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115082747B (en) * 2022-08-23 2023-01-06 紫东信息科技(苏州)有限公司 Zero-sample gastric ulcer classification system based on block confrontation
CN115171889B (en) * 2022-09-09 2023-01-06 紫东信息科技(苏州)有限公司 Small sample gastric tumor diagnosis system
CN115240847B (en) * 2022-09-23 2023-01-06 紫东信息科技(苏州)有限公司 Gastric ulcer diagnosis device, gastric ulcer diagnosis equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017107905A1 (en) * 2014-12-26 2017-06-29 夏小叶 Tracking and early-warning method and system involving cooperation between multiple networks
CN111243042A (en) * 2020-02-28 2020-06-05 浙江德尚韵兴医疗科技有限公司 Ultrasonic thyroid nodule benign and malignant characteristic visualization method based on deep learning
CN114504298A (en) * 2022-01-21 2022-05-17 南京航空航天大学 Physiological feature distinguishing method and system based on multi-source health perception data fusion

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017107905A1 (en) * 2014-12-26 2017-06-29 夏小叶 Tracking and early-warning method and system involving cooperation between multiple networks
CN111243042A (en) * 2020-02-28 2020-06-05 浙江德尚韵兴医疗科技有限公司 Ultrasonic thyroid nodule benign and malignant characteristic visualization method based on deep learning
CN114504298A (en) * 2022-01-21 2022-05-17 南京航空航天大学 Physiological feature distinguishing method and system based on multi-source health perception data fusion

Also Published As

Publication number Publication date
CN114782760A (en) 2022-07-22

Similar Documents

Publication Publication Date Title
CN114782760B (en) Stomach disease picture classification system based on multitask learning
Igarashi et al. Anatomical classification of upper gastrointestinal organs under various image capture conditions using AlexNet
CN113496489B (en) Training method of endoscope image classification model, image classification method and device
CN114399465B (en) Benign and malignant ulcer identification method and system
CN115082747B (en) Zero-sample gastric ulcer classification system based on block confrontation
Hicks et al. Dissecting deep neural networks for better medical image classification and classification understanding
CN115115897B (en) Multi-modal pre-trained gastric tumor classification system
CN112466466B (en) Digestive tract auxiliary detection method and device based on deep learning and computing equipment
CN114882978A (en) Stomach image processing method and system introducing picture translation information
CN113781489B (en) Polyp image semantic segmentation method and device
CN113222957A (en) Multi-class focus high-speed detection method and system based on capsule lens image
Masmoudi et al. Optimal feature extraction and ulcer classification from WCE image data using deep learning
CN112651375A (en) Helicobacter pylori stomach image recognition and classification system based on deep learning model
CN116759068A (en) Medical visual question-answering method and system for digestive system
You et al. Vocal cord leukoplakia classification using deep learning models in white light and narrow band imaging endoscopy images
CN115171889B (en) Small sample gastric tumor diagnosis system
Ali et al. A shallow extraction of texture features for classification of abnormal video endoscopy frames
Yang et al. Identification of upper GI diseases during screening gastroscopy using a deep convolutional neural network algorithm
CN116612340B (en) Peptic ulcer part classification method, device and readable storage medium
Li Recognizing polyps in wireless endoscopy images using deep stacked auto encoder with constraint image model in flexible medical sensor platform
Malviya et al. Deep Learning Based Gastro Intestinal Disease Analysis Using Wireless Capsule Endoscopy Images
CN117671573B (en) Helicobacter pylori infection state identification method and device based on gastroscope image
Nguyen et al. Automatic classification of upper gastrointestinal tract diseases from endoscopic images
CN115240847B (en) Gastric ulcer diagnosis device, gastric ulcer diagnosis equipment and storage medium
Bravo et al. Automatic polyp detection and localization during colonoscopy using convolutional neural networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP02 Change in the address of a patent holder

Address after: 215000 e3-501, artificial intelligence Industrial Park, 88 Jinjihu Avenue, Suzhou Industrial Park, Suzhou area, China (Jiangsu) pilot Free Trade Zone, Suzhou City, Jiangsu Province

Patentee after: Zidong information technology (Suzhou) Co.,Ltd.

Address before: No. 188, Shihu West Road, Wuzhong District, Suzhou City, Jiangsu Province

Patentee before: Zidong information technology (Suzhou) Co.,Ltd.

CP02 Change in the address of a patent holder