CN116228759A - Computer-aided diagnosis system and apparatus for renal cell carcinoma type - Google Patents
Computer-aided diagnosis system and apparatus for renal cell carcinoma type Download PDFInfo
- Publication number
- CN116228759A CN116228759A CN202310504664.0A CN202310504664A CN116228759A CN 116228759 A CN116228759 A CN 116228759A CN 202310504664 A CN202310504664 A CN 202310504664A CN 116228759 A CN116228759 A CN 116228759A
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- kidney tissue
- kidney
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004195 computer-aided diagnosis Methods 0.000 title claims abstract description 23
- 208000006265 Renal cell carcinoma Diseases 0.000 title claims abstract description 20
- 210000005084 renal tissue Anatomy 0.000 claims abstract description 77
- 239000013598 vector Substances 0.000 claims abstract description 55
- 238000012549 training Methods 0.000 claims abstract description 22
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 11
- 239000010410 layer Substances 0.000 claims description 45
- 239000011159 matrix material Substances 0.000 claims description 37
- 206010028980 Neoplasm Diseases 0.000 claims description 29
- 201000011510 cancer Diseases 0.000 claims description 27
- 238000000034 method Methods 0.000 claims description 24
- 238000003745 diagnosis Methods 0.000 claims description 19
- 210000003734 kidney Anatomy 0.000 claims description 15
- 201000009030 Carcinoma Diseases 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 7
- 239000002344 surface layer Substances 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 201000010240 chromophobe renal cell carcinoma Diseases 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims 1
- 238000013145 classification model Methods 0.000 abstract description 3
- 208000008839 Kidney Neoplasms Diseases 0.000 description 15
- 210000004027 cell Anatomy 0.000 description 13
- 208000030808 Clear cell renal carcinoma Diseases 0.000 description 10
- 206010038389 Renal cancer Diseases 0.000 description 8
- 201000010982 kidney cancer Diseases 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 210000003292 kidney cell Anatomy 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007792 addition Methods 0.000 description 3
- 230000001575 pathological effect Effects 0.000 description 3
- 108090000623 proteins and genes Proteins 0.000 description 3
- 238000012163 sequencing technique Methods 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 210000001519 tissue Anatomy 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 206010073251 clear cell renal cell carcinoma Diseases 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000007170 pathology Effects 0.000 description 1
- 239000013047 polymeric layer Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000000391 smoking effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30024—Cell structures in vitro; Tissue sections in vitro
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30084—Kidney; Renal
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Public Health (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Quality & Reliability (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a computer-aided diagnosis system and equipment for renal cell carcinoma type, the system comprises: the data acquisition module is used for collecting kidney tissue full-slice images and constructing a training set; the data processing module is used for preprocessing the kidney tissue full-slice image; the feature learning module comprises a feature learner and performs training by adopting a training set; the image feature library is constructed by extracting feature vectors of training samples by a feature learner; the classification module is used for preprocessing the kidney tissue full-section image to be classified through the data processing module, extracting feature vectors from the feature learner after preprocessing, and judging the type of the kidney tissue full-section image to be classified according to the feature vectors of the kidney tissue full-section image to be classified and all the feature vectors in the image feature library. The invention solves the problem that the classification model with high accuracy cannot be trained due to insufficient rare class data and large characteristic difference.
Description
Technical Field
The invention relates to the technical field of computer-aided medical treatment, in particular to a computer-aided diagnosis system and equipment for renal cell carcinoma types.
Background
Renal cell carcinoma is one of the most common among malignant tumors of the kidney. There are three common types of renal cell carcinoma (about 90% of renal cell carcinoma): clear cell carcinoma of the kidney, papillary cell carcinoma of the kidney, and chromophobe cell carcinoma of the kidney. The current routine diagnostic methods in clinic are: the doctor of the intensive pathology department observes the processed kidney tissue section image under the microscope, searches the position of the cancer cell area, and analyzes which type according to the medical characteristics of the cancer cell structure and the like. For the three types that are common, doctors can diagnose more easily.
However, rare cases of kidney cell-like cancers (also including multiple sub-types) are rare, and some rare are similar in structure to common comparisons, so that pathologists are difficult to judge clinically and prone to misjudge (especially for inexperienced doctors).
The judgment of rare kidney cancers requires doctors to let patients fill in questionnaires, including information such as age, genetic history, smoking history or other disease history. Particularly for inherited rare kidney-like cancers, it is also desirable to combine expensive gene sequencing results to judge.
There are many vision-based methods for classifying kidney cancer. They have proposed various treatment methods and classification models to detect renal cell carcinoma (both carcinoma and carcinoma-free) or to classify subtypes (classification of the above-mentioned common three subtypes, no rare class), and to achieve good results. For example, chinese patent publication No. CN113222933a discloses an image recognition system applied to full-chain diagnosis of renal cell carcinoma, which includes an image segmentation module, wherein the image segmentation module segments an original pathological image including a cancer genome map TCGA and an LH data set provided by a local hospital after labeling of a cancer region, a cancer subtype and a cancer grade, then inputs the segmented image into an image cancer region detection module to train and predict the image, the image processed by the image cancer region detection module is subjected to accuracy improvement by an accuracy improvement module to obtain a more accurate cancer region prediction thermodynamic diagram, the region predicted as cancer is marked and sent to a cancer region typing module to be further typed to obtain a subdivided subtype of cancer, and the report output module outputs an image recognition result report after typing.
However, in the existing scheme, whether the method of the feature library or the classification model method is adopted, a feature library or a training model is established by using a plurality of case image data of each subtype, however, the data of the rare kidney cancer is very little and not easy to obtain, and large intra-class differences exist between the rare kidney cancers, so the rare kidney cell cancers cannot be accurately identified by using the existing scheme. There is no computer-aided method, system for diagnosis of the full class of renal cancers including rare classes (no cancer, common three classes, rare classes) to aid physicians in rapid diagnosis.
Disclosure of Invention
The invention provides a computer-aided diagnosis system and equipment for renal cell carcinoma type, which solve the problems of low accuracy of prediction classification and incomplete classification type of the traditional computer-aided diagnosis system.
The technical scheme of the invention is as follows:
a computer-aided diagnosis system of a renal cell carcinoma type, comprising:
the data acquisition module is used for collecting the marked kidney tissue full-section images and constructing a training set; the types of the kidney tissue whole-section images include a normal type, a kidney transparent cell carcinoma type, a kidney papillary cell carcinoma type and a kidney chromophobe cell carcinoma type;
the data processing module is used for preprocessing the kidney tissue full-slice image;
the feature learning module comprises a feature learner based on a kidney tissue full-slice image constructed by a deep learning network, and the feature learner is trained by adopting the training set;
the image feature library inputs samples of the training set into a trained feature learner, and feature vectors of the samples are extracted; collecting all feature vectors and types corresponding to samples with correct prediction and probability larger than a set value, and constructing an image feature library; calculating the in-class cosine similarity between feature vectors in an image feature library, and setting the minimum value of the in-class cosine similarity as a diagnosis threshold;
the classification module inputs the kidney tissue full-section image to be classified into the data processing module for preprocessing, inputs the preprocessed kidney tissue full-section image into the trained feature learner for extracting feature vectors, and calculates cosine similarity between the feature vectors of the kidney tissue full-section image to be classified and all the feature vectors in the image feature librarysimilarityThe method comprises the steps of carrying out a first treatment on the surface of the Calculated maximumsimilarityDifferences from diagnostic thresholdλThe method comprises the steps of carrying out a first treatment on the surface of the If it isλ>0, classifying the kidney tissue whole-section image to be classified into a rare kidney cell-like cancer type; if it isλ<0, classifying the kidney tissue full-section image to be classified assimilarityThe type to which the largest belongs.
Preferably, the preprocessing comprises the operations of denoising, sliding slicing, encoding image blocks and generating an image matrix in sequence.
Further, the preprocessing includes:
(1-1) removing a background noise area in a kidney tissue whole-section image by adopting a threshold method, and reserving a foreground cell area;
(1-2) slicing an image area with a sliding window under a certain multiplying power to obtain a plurality of image blocks and arranging the image blocks;
(1-3) encoding Image blocks using a pre-trained res Net50 on an Image-Net dataset, each Image block encoded as a feature vector;
(1-4) fusing the feature vectors of all image blocks in a first dimension to obtain an image matrix for characterizing the renal tissue whole slice image.
Preferably, the feature learner includes:
feature enhancement layer for kidney tissue whole section mapThe image features are enhanced to obtain a feature matrix of the kidney tissue full-section imageH i c ；
Comprising the following steps: the image matrix of the kidney tissue whole-section image is expressed as:P i c ={P i,1 ;…;P i,n }，P i,j represent the firstiImage of the kidney tissue in full sectionjThe feature vectors of the individual image blocks,j=1…n，ce { normal type; KIRC type; KIRP type; KICH type }; get greater than
Is expressed as the square of the smallest integer of (2)gFront of image matrixg-nFeature vector addition to individual image blocksP i c In (3) obtaining an enhanced feature matrix:H i c =Concat(P i,1 ;…;P i,n ;P i,1 ;…;P i,g-n )；
the surface layer feature modeling layer is constructed based on a structure of a transducer and uses the feature matrixH i c For input, extracting surface features of the kidney tissue full-section image to obtain a first feature mapF 1 ；
Comprising the following steps: matrix the featuresH i c Is embedded into a feature matrix to obtain multi-head self-attention input:I i =Concat(p i,class ;H i c )，p i,class representing a category vector;
based on the input sequence and the corresponding feature space, a query matrix is calculated
Key matrix->
Sum matrix->
, wherein W Q 、W K 、W V Is a linear transformation matrix obtained by mapping the input sequence in different feature spaces;
calculation of attention coefficients based on Nystrfoster algorithmA i ：
wherein ,N Q andN K representing from the originalQAndKis selected from the group consisting of the marker points,drepresents the dimension constant, [] + Representing the Moore-Penrose pseudo-inverse,softmax(. Cndot.) represents an activation function;
use value matrixVAttention coefficientA i Weighting to obtain
；
Using different sets of linear projectionsQ、K、VMapping by a learnable parameterW O Weighting the group output to obtain a first feature map:
, wherein Z i,j Represent the firstiImage No. 1jThe group-weighted output is used to determine,j∈[1,h]；
the deep feature modeling layer adopts a double-scale convolution structure to form a first feature mapF 1 For input, extracting depth features of the kidney tissue full-slice image, and fusing surface features and depth features to obtain a second feature mapF 2 ；
The deep feature modeling layer consists of two convolution layers with different scales, and the first feature mapF 1 After two-layer convolution transformation, obtaining a double-scale depth feature:f t ,t=1, 2; fusing depth features and surface features on depth channels to obtainSecond feature map to image:F 2 =Concat(F 1 ,f 1 ,f 2 )；
a feature aggregation layer having the same structure as the surface layer feature modeling layer and using the second feature mapF 2 For input, obtaining the characteristic expression of the kidney tissue full-section imageF O ；
Multilayer perceptron expressed by the characteristicsF O For input, fitting the kidney tissue full-section image, and predicting the type of the kidney tissue full-section image.
In step (3) or in step (4), extracting feature vectors of a sample or a whole slice image of kidney tissue to be classified, including: and extracting the output of the last hidden layer of the multi-layer perceptron to obtain the feature vector of the sample or the kidney tissue full-section image to be classified.
In the step (2), when the feature learner is trained, the loss function is
； wherein ,x i the characteristic vector output by the last hidden layer of the multi-layer perceptron is represented,c yi representing the center of the category of the whole slice image of kidney tissue,brepresenting the size of the mini-batch.
In the step (2), adam is used as an optimizer when training a feature learner; initial learning rate of 2e-4。
In the step (2), the accuracy, recall, precision, and AUC are used as the evaluation index of the feature learner when training the feature learner.
In the step (3), the set value is most preferably 0.85.
Image feature library collectionSExpressed as:S={a 1c c ,…,a kc c }, wherein a kc c Represent the firstcClass 1kAnd feature vectors.
The cosine similarity is the cosine value of the included angle of the two vectors, and the calculation formula is as follows:
wherein ,a kc c represent the firstcClass 1kThe number of feature vectors is chosen to be the same,i、j∈[1,k]。
the in-class cosine similarity refers to cosine similarity among all feature vectors of each type in the image feature library.
The diagnostic threshold is used to distinguish rare types from other types and is the minimum value of cosine similarity in the class in the image feature library. Most preferably, the diagnostic threshold is 0.26.
Preferably, the computer-aided diagnosis system further comprises a display module, and the display module displays the prediction type output by the classification module.
The invention also provides a computer-aided diagnosis device for renal cell carcinoma type, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory stores an image feature library and a trained feature learner;
the processor, when executing the computer program, performs the steps of:
preprocessing the kidney tissue full-section image to be classified, inputting the preprocessed kidney tissue full-section image into a trained feature learner to extract feature vectors, and calculating cosine similarity between the feature vectors of the kidney tissue full-section image to be classified and all feature vectors in an image feature librarysimilarity；
Calculated maximumsimilarityDifferences from diagnostic thresholdλThe method comprises the steps of carrying out a first treatment on the surface of the If it isλ>0, classifying the kidney tissue whole-section image to be classified into a rare kidney cell-like cancer type; if it isλ<0, classifying the kidney tissue full-section image to be classified assimilarityThe type to which the largest belongs.
The invention integrates a vision-based method into a computer system, automatically classifies and outputs the category of the kidney tissue slice image, and outputs the rare category if the kidney tissue slice image is rare renal cell carcinoma; if not rare, the system of the invention can also automatically output specific categories, namely one of normal categories, KIRC types, KIRP types and KICH types.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, the rare class training data is not used, the kidney tissue slices of the whole classes (normal, common three classes and rare classes) are accurately classified, and the problems of low classification prediction accuracy and incomplete classes (not containing rare classes or normal classes) in the existing auxiliary diagnosis method and system are solved.
The invention can excavate the depth characteristic of the bottom layer of the pathological image, is not only the structural distribution characteristic of the macroscopic surface layer, improves the diagnosis efficiency of the doctor of the pathological department on the rare kidney cancer, omits the processes of expensive gene sequencing and the like to a certain extent, and relieves the burden of the doctor and the patient.
Drawings
FIG. 1 is a schematic diagram of a computer-aided diagnosis system of renal cell carcinoma type;
FIG. 2 is a schematic workflow diagram of a computer-aided diagnosis system of renal cell carcinoma type;
fig. 3 is a schematic diagram of a network structure of the feature learner.
Detailed Description
The invention will be described in further detail below with reference to the drawings and examples, it being noted that the examples described below are intended to facilitate an understanding of the invention and are not intended to limit the invention in any way.
As shown in fig. 1, a computer-aided diagnosis system for rare and common renal cell carcinoma comprises a data acquisition unit, a data processing unit, a feature learner calculation unit, a feature processing and calculation unit, a diagnosis unit and a computer display unit. Wherein:
the data acquisition unit is used for acquiring the data of the kidney tissue slice images marked by the professional doctor;
and the data processing unit is used for preprocessing all the kidney tissue whole slice images. Removing background, sliding slice, coding image block and generating image matrix;
the feature learner computing unit is used for modeling the features of the image, including surface features and deep features, and enlarging the inter-class difference of the data;
the feature processing and calculating unit is used for screening features, establishing a kidney tissue image feature library and determining a diagnosis threshold;
a diagnostic unit for aiding in diagnosing a type of kidney tissue of the patient.
And the computer display unit is used for displaying the final diagnosis result.
As shown in fig. 2, the workflow of the computer-aided diagnosis system of renal cell carcinoma type includes the steps of:
s100, data collection: case data of kidney tissue whole-section images (WSI) are collected and processed and image-level annotated by a professional pathologist.
Case data for kidney tissue whole-section images (WSI) include normal section images, kidney clear cell carcinoma (KIRC) section images, kidney papillary cell carcinoma (KIRP) section images, kidney chromophobe cell carcinoma (KICH) section images, and rare kidney cell carcinoma-like section images. Wherein the data of normal type, KIRC type, KIRP type and KICH type are used for training and testing of the system, and the data of rare kidney cell carcinoma is only used for testing of the system.
S101, data processing: for preprocessing all the kidney tissue whole-section images in S100. Including denoising, sliding slicing, encoding image blocks, generating image matrices, in particular:
denoising: removing a background noise area in the kidney tissue image by adopting a threshold method, and reserving a foreground cell area;
sliding slice: at 20
Slicing the image area by using a sliding window with the size of 512 to obtain a plurality of image blocks and arranging the image blocks side by side;
encoding an image block: encoding all Image blocks using a pre-trained ResNet50 on an Image-Net dataset; fine tuning ResNet50, changing the number of the neurons of the last full-connection layer from 1000 to 512; resulting in each image block being encoded as 1
512 feature vectors;
generating an image matrix: fusing the feature vectors of all the image blocks in the first dimension to obtain n
512 and encoded as a binary file ending in the. Pt format for characterizing the renal tissue whole slice image. Where n represents the number of valid image blocks in the WSI.
S102, feature learning: a feature learner for a full slice image of kidney tissue is trained using an end-to-end deep learning network. Features for modeling images of kidney tissue sections, and increasing the inter-class distance of the images. Features of an image include surface features and deep features, and inter-class distances refer to differences between different types of images. Specifically:
according to 80%: the ratio of 20% divides the four other data files in S101 except rare renal cell carcinoma into training and test sets. Data enhancement method adopting random clipping with clipping size of 224 2 。
The feature learner is obtained through training by a deep learning method, as shown in fig. 3, the specific network structure of the feature learner is as follows: the feature enhancement layer is used for enriching slice-level features of the WSI; the surface feature modeling layer SFM adopts a structure based on a transducer and is used for extracting surface features of the image; the deep feature modeling layer DFM adopts a double-scale convolution structure and is used for extracting deep features of the image; the feature aggregation layer FA adopts a structure based on a transducer and is used for aggregating two features of WSI; the multi-layer perceptron MLP is used to fit labels of slice image data.
The feature enhancement layer is used for enriching slice-level features of WSI, and specifically comprises the following steps:
reading the image matrix file of the step S101, and expressing WSI as:P i c ={P i,1 ;…;P i,n }，get greater than
Is expressed as the square of the smallest integer of (2)gFront of image matrixg-nFeature vector addition to individual image blocksP i c Obtaining an aggregated slice-level feature matrix:H i c =Concat(P i,1 ;…;P i,n ;P i,1 ;…;P i,g-n ) As input to a feature learner. Wherein the method comprises the steps ofcE { normal type; KIRC type; KIRP type; KICH-type } represents the class of kidney tissue images,P i,j represent the firstiFirst of WSIjThe feature vectors of the individual image blocks,H i c represent the firstiFeature matrix of each WSI. />
The surface feature modeling layer SFM adopts a structure based on a transducer and is used for extracting image surface features to obtain a first feature map:
, wherein SMF(. Cndot.) represents the entire skin feature modeling layer. The specific extraction steps are as follows:
the input is a feature matrix derived from a feature enhancement layerH i c ；
Embedding the category vector of the feature matrix into the feature matrix to obtain an input expression of multiple self-attentiveness in the structure:I i =Concat(p i,class ;H i c )，p i,class representing a category vector. Based on the input sequence and the corresponding feature space, a "query" is generated using the following formula "Q：
And "keyK：
. wherein W Q AndW K is a linear transformation matrix obtained by mapping the input sequence in different feature spaces.
Calculation of attention coefficients using an Nystr-m based algorithmA i ：
wherein N Q AndN K representing from the originalQAndKis selected from the group consisting of the marker points,drepresents the dimension constant, [] + Representing the Moore-Penrose pseudo-inverse,softmax(. Cndot.) represents an activation function.
Then use the "value"V：
Weighting the features to obtain a weighted outputZ i ：
. wherein W V Is a linear transformation matrix obtained by mapping an input sequence in a feature space;
for the above using different sets of linear projectionsQ、K、VMapping is performed. By a parameter which can be learnedW O To weight the group output. Obtaining a first characteristic diagram:
, wherein Z i,j Represent the firstiImage No. 1jAnd (5) group weighted output.
The deep feature modeling layer adopts a double-scale convolution structure and is used for extracting deep features of the image to obtain a second feature map of the image:F 2 =Concat(F 1 ,f 1 ,f 2). wherein DFM(. Cndot.) represents the entire deep feature modeling layer. The specific extraction steps are as follows:
the deep feature modeling layer consists of two convolutional layers, the kernel sizes of which are 3 and 5, respectively, the padding (padding) is 1 and 2,after convolution transformation, obtaining a double-scale depth feature:f t ,t=1,2；
fusing the obtained depth features and surface features on the depth channel to obtain a second feature map of the imageF 2 ：F 2 =Concat(F 1 ,f 1 ,f 2 )；f 1 、f 2 Is a dual-scale depth feature.
The feature aggregation layer FA adopts a structure based on a transducer and is used for aggregating the surface layer features and the deep layer features of WSI, and specifically comprises the following steps:
the input is a second feature map derived from the deep feature modeling layerF 2 ；
Using the same structure as the surface layer feature modeling layer, a feature expression of WSI was obtainedF O ：
, wherein FA(. Cndot.) represents the entire feature polymeric layer.
The multi-layer perceptron MLP is used for fitting labels of normal type, KIRC type, KIRP type and KICH type WSI to obtain slice-level characteristics predicted by the characteristic learnerO：O=MLP[LN(F O )+F O ]. wherein LN(. Cndot.) represents the normalized layer,MLP(. Cndot.) represents a multi-layer perceptron.
The above model uses Center Loss as the Loss function, with the formula:
for maximizing inter-class variation and minimizing intra-class variation for multiple classes. Wherein the method comprises the steps ofx i The feature vector representing the last hidden layer of the multi-layer perceptron,c yi representing the center of the category of several slice images,brepresenting the size of the mini-batch.
Adam was used as an optimizer; initial learning rate of 2e-4；
The accuracy, recall, precision, and AUC are used as the evaluation index of the feature learner.
Training the feature learner, and visualizing the training result of each round in the two-dimensional plane until the model converges and a larger inter-class distance is seen in the two-dimensional plane;
the test set is used for testing the trained learner, and the learner can accurately learn the characteristics of each type of WSI.
S103, establishing a feature library: for creating a feature library of renal tissue image data. The feature library contains clusters of feature vectors other than rare class tissue images. The method comprises the following steps:
inputting training data except rare types in S100 into the feature learner trained in S102;
extracting the last hidden layer and the last output layer of the MLP in S102 to obtain the feature vector of the WSI and the probability of belonging to a certain type;
adding feature vectors corresponding to all data with correct prediction and probability more than 0.85 to an image feature library for diagnosis of rare renal cancer and other 4 tissue images; feature library collectionSExpressed as:S={a 1c c ,…,a kc c }, wherein a kc c Represent the firstcClass 1kA feature vector;
calculating the in-class cosine similarity between feature vectors in a feature library, establishing a histogram by using a calculation result, observing the histogram and setting a diagnosis threshold, specifically:
cosine similarity refers to the cosine value of the included angle of two vectors, and the formula is:
The in-class cosine similarity refers to cosine similarity among all feature vectors of each type in a feature library;
histogram, the horizontal axis is image similarity degree value distribution, and the vertical axis is image appearance frequency;
the diagnostic threshold is used to distinguish rare types from other types and is the minimum value of cosine similarity in the class in the feature library. In this embodiment, the diagnostic threshold is 0.26.
S104 type diagnosis: for diagnosing the type to which the input image belongs, in particular:
obtaining feature vectors of the new image generated by the feature learner pre-trained in S102, calculating cosine similarity between the feature vectors and all kidney tissue image features in the feature library in S103similarity；
Calculated maximumsimilarityDifferences from diagnostic thresholdλ. If it isλ>0, diagnosing the patient as rare kidney cell-like cancer; outputting a patient diagnosis result; if it isλ<0, diagnosing the patient assimilarityThe type of the largest category;
and outputting and displaying the final diagnosis result.
The existing scheme is characterized by classification, identification, feature library matching and other technologies, and the purposes of identification and classification can be achieved by training a model. However, rare cases of kidney cancer have few samples and low intra-class similarity. The prior art solutions in this case do not allow to detect well the full category of cases including rare kidney cancers.
In the existing scheme, the detection of normal/cancer of the kidney cancer and the detection of common types of the kidney cancer (KIRC type, KIRP type and KICH type) can be realized. In the clinic, doctors can diagnose both cases relatively easily, but rare kidney cancers are relatively difficult to diagnose. Diagnosis needs to be confirmed based on some other information or symptoms of the patient, even with expensive gene sequencing. There is currently no computer-aided system that includes diagnosis of the full class of rare types of kidney cancer (no cancer, clear cell cancer, papillary cell cancer, chromophobe cell cancer, rare cancer) to aid physicians in rapid diagnosis. The computer-aided diagnosis system integrates a vision-based method into a computer system, can automatically diagnose five kidney conditions (normal, common three types and rare types) including rare kidney cancers under the condition that the image training data of rare kidney cell cancer slices are not used, and outputs the rare types if a patient is the rare kidney cell cancer; if not rare, the system of the invention automatically outputs a specific category, namely one of normal category, KIRC type, KIRP type and kirh type.
Aiming at the characteristics of kidney cancer, the invention designs a special data processing method, a characteristic learner and a characteristic matching scheme, so that the method and the system of the invention have high accuracy and high speed for five types of diagnosis.
The foregoing embodiments have described the technical solutions and advantages of the present invention in detail, and it should be understood that the foregoing embodiments are merely illustrative of the present invention and are not intended to limit the invention, and any modifications, additions, substitutions and the like that fall within the principles of the present invention should be included in the scope of the invention.
Claims (9)
1. A computer-aided diagnosis system of a renal cell carcinoma type, comprising:
the data acquisition module is used for collecting the marked kidney tissue full-section images and constructing a training set; the types of the kidney tissue whole-section images include a normal type, a kidney transparent cell carcinoma type, a kidney papillary cell carcinoma type and a kidney chromophobe cell carcinoma type;
the data processing module is used for preprocessing the kidney tissue full-slice image;
the feature learning module comprises a feature learner based on a kidney tissue full-slice image constructed by a deep learning network, and the feature learner is trained by adopting the training set;
the image feature library inputs samples of the training set into a trained feature learner, and feature vectors of the samples are extracted; collecting all feature vectors and types corresponding to samples with correct prediction and probability larger than a set value, and constructing an image feature library; calculating the in-class cosine similarity between feature vectors in an image feature library, and setting the minimum value of the in-class cosine similarity as a diagnosis threshold;
the classification module inputs the kidney tissue full-section image to be classified into the data processing module for preprocessing, inputs the preprocessed kidney tissue full-section image into the trained feature learner for extracting feature vectors, and calculates the feature vectors and the images of the kidney tissue full-section image to be classifiedCosine similarity of all feature vectors in image feature librarysimilarityThe method comprises the steps of carrying out a first treatment on the surface of the Calculated maximumsimilarityDifferences from diagnostic thresholdλThe method comprises the steps of carrying out a first treatment on the surface of the If it isλ>0, classifying the kidney tissue whole-section image to be classified into a rare kidney cell-like cancer type; if it isλ<0, classifying the kidney tissue full-section image to be classified assimilarityThe type to which the largest belongs.
2. The computer aided diagnosis system of claim 1, wherein the preprocessing includes sequentially performing denoising, sliding slicing, encoding image blocks, generating image matrix operations.
3. The computer-aided diagnosis system of claim 2, wherein the preprocessing includes:
(1-1) removing a background noise area in a kidney tissue whole-section image by adopting a threshold method, and reserving a foreground cell area;
(1-2) slicing an image area with a sliding window under a certain multiplying power to obtain a plurality of image blocks and arranging the image blocks;
(1-3) encoding Image blocks using a pre-trained res Net50 on an Image-Net dataset, each Image block encoded as a feature vector;
(1-4) fusing the feature vectors of all image blocks in a first dimension to obtain an image matrix for characterizing the renal tissue whole slice image.
4. The computer-aided diagnosis system of claim 1, wherein the feature learner includes:
the characteristic enhancement layer is used for enhancing the characteristics of the kidney tissue full-section image to obtain a characteristic matrix of the kidney tissue full-section imageH i c ；
The surface layer feature modeling layer is constructed based on a structure of a transducer and uses the feature matrix H i c For input, extract surface features of whole slice image of kidney tissueCharacterization, obtain a first feature mapF 1 ；
The deep feature modeling layer adopts a double-scale convolution structure to form a first feature mapF 1 For input, extracting depth features of the kidney tissue full-slice image, and fusing surface features and depth features to obtain a second feature mapF 2 ；
A feature aggregation layer having the same structure as the surface layer feature modeling layer and using the second feature mapF 2 For input, obtaining the characteristic expression of the kidney tissue full-section imageF O ；
Multilayer perceptron expressed by the characteristicsF O For input, fitting the kidney tissue full-section image, and predicting the type of the kidney tissue full-section image.
5. The computer-aided diagnosis system of claim 4, wherein the feature enhancement layer enhances features of the renal tissue full slice image, comprising:
the image matrix of the kidney tissue whole-section image is expressed as:P i c ={P i,1 ;…;P i,n }，P i,j represent the firstiImage of the kidney tissue in full sectionjThe feature vectors of the individual image blocks,j=1…n， ce { normal type; KIRC type; KIRP type; KICH type };
get greater than
Is expressed as the square of the smallest integer of (2)gFront of image matrixg-nFeature vector addition to individual image blocksP i c In (1) obtaining an enhanced feature matrixH i c =Concat(P i,1 ;…;P i,n ; P i,1 ;…;P i,g-n )。
6. According toThe computer-aided diagnosis system of claim 5, wherein the surface feature modeling layer uses the feature matrixH i c For input, extracting surface features of the kidney tissue full-section image to obtain a first feature mapF 1 Comprising:
matrix the featuresH i c Is embedded into the feature matrix to obtain multi-head self-attention inputI i =Concat(p i,class ; H i c )；
Based on the input sequence and the corresponding feature space, a query matrix is calculated
Key matrix->
Sum matrix
, wherein W Q 、W K 、W V Is a linear transformation matrix obtained by mapping the input sequence in different feature spaces;
calculation of attention coefficients based on Nystrfoster algorithmA i ；
Use value matrixVAttention coefficientA i Weighting to obtain
；
Using different sets of linear projectionsQ、K、VMapping by a learnable parameterW O Weighting the group output to obtain a first feature map
, wherein Z i,j Represent the firstiImage No. 1jThe group-weighted output is used to determine,j∈[1,h]。
7. the computer-aided diagnosis system of claim 1, wherein the cosine similarity is calculated by the formula:
wherein ,a kc c represent the firstcClass 1kThe number of feature vectors is chosen to be the same,i、j∈[1,k]。
8. the computer-aided diagnosis system of claim 8, further comprising a display module, wherein the display module displays the prediction type output by the classification module.
9. A computer-aided diagnosis device of the renal cell carcinoma type, comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said memory has stored therein a library of image features and a trained feature learner;
the processor, when executing the computer program, performs the steps of:
preprocessing the kidney tissue full-section image to be classified, inputting the preprocessed kidney tissue full-section image into a trained feature learner to extract feature vectors, and calculating cosine similarity between the feature vectors of the kidney tissue full-section image to be classified and all feature vectors in an image feature librarysimilarity；
Calculated maximumsimilarityDifferences from diagnostic thresholdλThe method comprises the steps of carrying out a first treatment on the surface of the If it isλ>0, classifying the kidney tissue whole-section image to be classified into a rare kidney cell-like cancer type; if it isλ<0, classifying the kidney tissue full-section image to be classified assimilarityThe type to which the largest belongs.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310504664.0A CN116228759B (en) | 2023-05-08 | 2023-05-08 | Computer-aided diagnosis system and apparatus for renal cell carcinoma type |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310504664.0A CN116228759B (en) | 2023-05-08 | 2023-05-08 | Computer-aided diagnosis system and apparatus for renal cell carcinoma type |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116228759A true CN116228759A (en) | 2023-06-06 |
| CN116228759B CN116228759B (en) | 2023-08-01 |
Family
ID=86579093
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310504664.0A Active CN116228759B (en) | 2023-05-08 | 2023-05-08 | Computer-aided diagnosis system and apparatus for renal cell carcinoma type |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116228759B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117636064A (en) * | 2023-12-21 | 2024-03-01 | 浙江大学 | Intelligent neuroblastoma classification system based on pathological sections of children |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010231277A (en) * | 2009-03-25 | 2010-10-14 | Fuji Xerox Co Ltd | Image processing apparatus, memory controller, and program |
| CN105224960A (en) * | 2015-11-04 | 2016-01-06 | 江南大学 | Based on the corn seed classification hyperspectral imagery model of cognition update method of clustering algorithm |
| CN108960289A (en) * | 2018-06-08 | 2018-12-07 | 清华大学 | Medical imaging sorter and method |
| CN109598189A (en) * | 2018-10-17 | 2019-04-09 | 天津大学 | A kind of video classification methods based on Feature Dimension Reduction |
| CN112784822A (en) * | 2021-03-08 | 2021-05-11 | 口碑(上海)信息技术有限公司 | Object recognition method, object recognition device, electronic device, storage medium, and program product |
| CN112819042A (en) * | 2021-01-18 | 2021-05-18 | 首都医科大学附属北京朝阳医院 | Method, system and medium for processing esophageal squamous dysplasia image |
| CN114972341A (en) * | 2022-07-28 | 2022-08-30 | 华南理工大学 | WSI image classification method, system and medium based on Bayesian assisted learning |
| CN115641956A (en) * | 2022-10-26 | 2023-01-24 | 中科(厦门)数据智能研究院 | Phenotype analysis method for disease prediction |
-
2023
- 2023-05-08 CN CN202310504664.0A patent/CN116228759B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010231277A (en) * | 2009-03-25 | 2010-10-14 | Fuji Xerox Co Ltd | Image processing apparatus, memory controller, and program |
| CN105224960A (en) * | 2015-11-04 | 2016-01-06 | 江南大学 | Based on the corn seed classification hyperspectral imagery model of cognition update method of clustering algorithm |
| CN108960289A (en) * | 2018-06-08 | 2018-12-07 | 清华大学 | Medical imaging sorter and method |
| CN109598189A (en) * | 2018-10-17 | 2019-04-09 | 天津大学 | A kind of video classification methods based on Feature Dimension Reduction |
| CN112819042A (en) * | 2021-01-18 | 2021-05-18 | 首都医科大学附属北京朝阳医院 | Method, system and medium for processing esophageal squamous dysplasia image |
| CN112784822A (en) * | 2021-03-08 | 2021-05-11 | 口碑(上海)信息技术有限公司 | Object recognition method, object recognition device, electronic device, storage medium, and program product |
| CN114972341A (en) * | 2022-07-28 | 2022-08-30 | 华南理工大学 | WSI image classification method, system and medium based on Bayesian assisted learning |
| CN115641956A (en) * | 2022-10-26 | 2023-01-24 | 中科(厦门)数据智能研究院 | Phenotype analysis method for disease prediction |
Non-Patent Citations (3)
| Title |
|---|
| P. KANMANI，P. MARIKKANNU: "MRI Brain Images Classification: A Multi-Level Threshold Based Region Optimization Technique", 《JOURNAL OF MEDICAL SYSTEMS》, pages 1 - 12 * |
| 曹建敏 等: "烤烟致香物质GC/MS 指纹图谱在产区识别中的应用", 《中国烟草科学》, vol. 35, no. 6, pages 85 - 89 * |
| 陈镇香: "基于指静脉的生物识别算法研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 138 - 2768 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117636064A (en) * | 2023-12-21 | 2024-03-01 | 浙江大学 | Intelligent neuroblastoma classification system based on pathological sections of children |
| CN117636064B (en) * | 2023-12-21 | 2024-05-28 | 浙江大学 | Intelligent neuroblastoma classification system based on pathological sections of children |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116228759B (en) | 2023-08-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xue et al. | An application of transfer learning and ensemble learning techniques for cervical histopathology image classification | |
| CN110969626B (en) | Method for extracting hippocampus of human brain nuclear magnetic resonance image based on 3D neural network | |
| Lashari et al. | A framework for medical images classification using soft set | |
| CN110097921B (en) | Visualized quantitative method and system for glioma internal gene heterogeneity based on image omics | |
| Xu et al. | Using transfer learning on whole slide images to predict tumor mutational burden in bladder cancer patients | |
| CN111767952A (en) | Interpretable classification method for benign and malignant pulmonary nodules | |
| CN116228759B (en) | Computer-aided diagnosis system and apparatus for renal cell carcinoma type | |
| CN114399634B (en) | Three-dimensional image classification method, system, equipment and medium based on weak supervision learning | |
| CN117036288A (en) | Tumor subtype diagnosis method for full-slice pathological image | |
| CN114581698A (en) | Target classification method based on space cross attention mechanism feature fusion | |
| Meng et al. | Clinical applications of graph neural networks in computational histopathology: A review | |
| CN109740669B (en) | Breast cancer pathological image classification method based on depth feature aggregation | |
| Tian et al. | Radiomics and its clinical application: artificial intelligence and medical big data | |
| CN113707317B (en) | Disease risk factor importance analysis method based on mixed model | |
| Xu et al. | Identification of benign and malignant lung nodules in CT images based on ensemble learning method | |
| CN117994587A (en) | Pathological image classification method based on deep learning two-stage reasoning network | |
| CN114093507A (en) | Skin disease intelligent classification method based on contrast learning in edge computing network | |
| CN112861881A (en) | Honeycomb lung recognition method based on improved MobileNet model | |
| Ghazouani et al. | Towards non-data-hungry and fully-automated diagnosis of breast cancer from mammographic images | |
| Roselin et al. | Fuzzy-rough feature selection for mammogram classification | |
| CN116664932A (en) | Colorectal cancer pathological tissue image classification method based on active learning | |
| CN115409812A (en) | CT image automatic classification method based on fusion time attention mechanism | |
| CN115375632A (en) | Lung nodule intelligent detection system and method based on CenterNet model | |
| CN112151175B (en) | Computer-aided multi-person decision-making method, system and equipment based on imaging data | |
| CN113889235A (en) | Unsupervised feature extraction system for three-dimensional medical image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |