CN117557844B - Multi-model fusion tongue image intelligent classification method based on data enhancement - Google Patents
Multi-model fusion tongue image intelligent classification method based on data enhancement Download PDFInfo
- Publication number
- CN117557844B CN117557844B CN202311513218.2A CN202311513218A CN117557844B CN 117557844 B CN117557844 B CN 117557844B CN 202311513218 A CN202311513218 A CN 202311513218A CN 117557844 B CN117557844 B CN 117557844B
- Authority
- CN
- China
- Prior art keywords
- data set
- image classification
- tongue image
- training
- trainer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 76
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012549 training Methods 0.000 claims abstract description 120
- 238000002474 experimental method Methods 0.000 claims abstract description 109
- 230000002787 reinforcement Effects 0.000 claims abstract description 4
- 238000011156 evaluation Methods 0.000 claims description 47
- 239000011159 matrix material Substances 0.000 claims description 41
- 238000002679 ablation Methods 0.000 claims description 38
- 238000012360 testing method Methods 0.000 claims description 23
- 238000005457 optimization Methods 0.000 claims description 21
- 230000003068 static effect Effects 0.000 claims description 21
- 239000002775 capsule Substances 0.000 claims description 18
- 230000005540 biological transmission Effects 0.000 claims description 15
- 238000012163 sequencing technique Methods 0.000 claims description 9
- 238000004458 analytical method Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 230000009467 reduction Effects 0.000 claims description 5
- 230000000052 comparative effect Effects 0.000 claims description 4
- 238000012803 optimization experiment Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 3
- 230000006854 communication Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000000605 extraction Methods 0.000 description 4
- 238000007792 addition Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 241000208340 Araliaceae Species 0.000 description 2
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 2
- 235000003140 Panax quinquefolius Nutrition 0.000 description 2
- 230000003321 amplification Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 239000003814 drug Substances 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 235000008434 ginseng Nutrition 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/098—Distributed learning, e.g. federated learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/008—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols involving homomorphic encryption
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Biodiversity & Conservation Biology (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of tongue image classification, and particularly discloses a multi-model fusion tongue image intelligent classification method based on data enhancement, which comprises the following steps: constructing an original data set, expanding the original data set to obtain a training data set, and adding a prediction label and a real label to each training sample in the training data set; selecting a difficult-to-train sample, setting a difficult-to-train sample queue, and performing reinforcement training; extracting and fusing the characteristics of the training samples; and carrying out data exchange on each local trainer and the global trainer, training, classifying tongue images according to the data exchange, carrying out a plurality of training data set comparison experiments, and analyzing comparison experiment data. The method effectively solves the problem that the accuracy of the classification result is difficult to guarantee when the current data volume is insufficient, expands the use scene of tongue image classification, avoids the scene restriction of the current tongue image classification mode, and further improves the flexibility, applicability and reliability of tongue image classification.
Description
Technical Field
The invention belongs to the technical field of tongue image classification, and relates to a multi-model fusion tongue image intelligent classification method based on data enhancement.
Background
In recent years, with the continuous development of artificial intelligence technology, tongue image intelligent classification plays an increasingly important role in the field of traditional Chinese medicine diagnosis. Tongue images are one of important basis of traditional Chinese medicine diagnosis, and the type, degree and change trend of diseases can be judged through observation and analysis of tongue images.
In the tongue image intelligent classification task, a training set can be expanded by using a data enhancement method, so that the diversity of training data can be effectively increased, and the robustness of a model is improved. However, the current model has higher requirements on sample size, and parameters are difficult to be effectively trained when the sample size is insufficient, so that under fitting is often caused, so that the model is difficult to achieve a good effect, and therefore, the current tongue image classification has the following defects: 1. the accuracy of classification is difficult to guarantee when the data volume is insufficient: cases of medical related organizations are confidential and the structured data that can actually be used for machine learning differs significantly. Large medical organizations may have many case resources, but for small medical organizations, cases tend to be limited, with certain limitations.
2. Federal learning is difficult to develop under limited and medical history resources, and federal model retraining is performed without adding a pseudo tag to the difficult-to-train samples.
3. The feature extraction is incomplete: the residual network, the VGG16, the LSTM and other algorithms can well extract static features of images, texts and the like, but the spatial association relation of different parts of the images is difficult to effectively model.
4. The current data transmission amount is huge, so that a large amount of communication overhead is caused, communication links are easy to block, and tongue classification efficiency cannot be guaranteed.
Disclosure of Invention
In view of this, in order to solve the problems set forth in the above background art, a multi-model fusion tongue image intelligent classification method based on data enhancement is now proposed.
The aim of the invention can be achieved by the following technical scheme: the invention provides a multi-model fusion tongue image intelligent classification method based on data enhancement, which comprises the following steps: step 1, taking each tongue fur image as each training sample, forming an original data set, expanding the original data set to obtain a training data set, and adding a prediction label and a real label to each training sample in the training data set.
And 2, respectively extracting the characteristics of different dimensions of each training sample in the training data set by combining a plurality of network structures, and fusing to obtain fused characteristics.
And 3, selecting the difficult-to-train samples of each trainer, setting a difficult-to-train sample queue according to the difficult-to-train sample, and performing reinforcement training on the difficult-to-train sample queue.
And 4, extracting attribute labels of all trainers, respectively marking the trainers with the attribute labels of the overall and the body as the overall trainers and the local trainers, exchanging data between each local trainer and the overall trainers, retraining the exchanged data, and classifying tongue images according to training results.
And 5, performing a plurality of training data set comparison experiments to obtain comparison experiment data, analyzing the comparison experiment data to obtain comparison experiment results, and further outputting the comparison experiment results.
Preferably, the specific implementation process of the fusion in the step 2 includes the following steps: and selecting training samples in the training data set in a random voting mode, and marking each selected training sample as a target sample.
And extracting static characteristics of the target sample through a residual error network.
Spatial features between internal body structures of the target sample are extracted through the capsule network.
And reducing the dimension of the static features extracted from the residual network and the spatial features extracted from the capsule network through a plurality of layers of convolution.
And fusing the static characteristics and the space characteristics after dimension reduction through a fusion algorithm.
Preferably, the fusion algorithm is specifically expressed as: x is training sample data of a trainer i, i is a trainer number, i=1, 2, &..n, gi, α, β is a network structure representation of the ith trainer, α represents a sharing parameter between trainers, β represents a local private parameter of the trainer, fr, fc respectively represent a static feature and a spatial feature, δ and λ respectively represent classification contribution factors corresponding to the static feature and the spatial feature, and f merge represents a fusion feature.
Preferably, the difficult training sample selection is obtained through a training sample selection model, wherein the training sample selection model specifically represents: Wherein j represents the training sample number in the training data set, j=1, 2, &..u, M is a reference minimum cosine value for setting the features and fusion features of the difficult-to-train sample, M j represents the cosine distance between the predicted label and the real label corresponding to the jth training sample in the training data set, H j←x if mj is less than or equal to M, when the cosine distance between the predicted label and the real label corresponding to the jth training sample in the training data set is less than or equal to the reference minimum cosine value for the features and fusion features of the difficult-to-train sample, the training samples are collected into a difficult-to-train sample queue H j, L is the number of labels, and Avg j′ represents the average value of all the features of the real sample corresponding to the real label in the training samples of the trainer i.
Preferably, the data exchange between each local trainer and the global trainer includes: and taking the residual network and the capsule network as basic network architecture of each trainer under the federal learning model, and encrypting the fusion characteristics of each local trainer through a homomorphic encryption model based on the federal learning model, so as to obtain an encryption parameter matrix P g of each local trainer, wherein g represents the number of the local trainer, and g=1, 2.
A restoring parameter matrix W g of each ontology trainer is constructed based on the encryption parameter matrix,
Wherein, the sig function represents the sign of the element in the encryption matrix, a represents the dimension of the matrix to be encrypted, d represents the length of the vector in the dimension of the matrix to be encrypted, y is a set natural constant, x 'is a matrix element, and x' =f a,d.
Setting decryption parameter matrix The parameter Q is the decryption function of homomorphic encryption matrix,/>Representing the inner product of the matrix corresponding elements.
And according to the encryption parameter matrix and the decryption parameter matrix of the corresponding fusion characteristics of each local trainer, each local trainer and each global trainer exchange data in an asynchronous transmission mode.
Preferably, the homomorphic encryption model is specifically expressed as: In the/> Representing rounding up symbols.
Preferably, the performing a plurality of training dataset comparison experiments comprises the steps of: a1, taking the current multi-model fusion tongue image classification as a target tongue image classification rule, and extracting each tongue image classification rule which is currently accumulated and formulated from a tongue image classification database as each reference tongue image classification rule.
A3, setting each comparison data set, taking each comparison data set as each experimental data set, and simultaneously formulating each evaluation index.
A2, randomly extracting Y 0 local trainers as local trainers of each experiment, randomly extracting Y 1 global trainers as global trainers of each experiment, and formulating an experiment data set allocation strategy.
And A4, carrying out fusion comparison experiments on all experimental data sets based on the target tongue image classification rules and all reference tongue image classification rules, and recording fusion comparison experiment data.
And A5, updating the test data set, taking the updated test data set as an ablation experiment data set, carrying out an ablation comparison experiment on the ablation experiment data set based on the target tongue image classification rule and each reference tongue image classification rule, and recording ablation comparison experiment data.
Preferably, the formulating the experimental dataset distribution strategy includes: and integrating the number of the experimental global trainers and the number of the experimental local trainers to obtain the experimental trainers, sequencing the experimental trainers according to the first sequencing rule of the experimental global trainers, and taking the sequencing result as the serial number of the experimental trainers.
Performing scale distribution of the experimental data sets on each experimental trainer according to a random distribution rule to obtain the distribution scale of the experimental data sets corresponding to each experimental trainer, wherein the random distribution rule is specifically expressed asK' q represents the distribution scale of the corresponding experimental data set of the q-th experimental trainer, q represents the experimental trainer number, q=1, 2.
Setting a training data set and a test data set, setting allocation duty ratio of the training data set and the test data set, marking the allocation duty ratio of the training data set as k Training device , marking the allocation duty ratio of the test data set as k Measuring , and corresponding each experimental trainer of k Training device and k Measuring to the allocation scale of the training data set and the test data set.
The distribution scale of the experimental data set and the distribution scales of the training data set and the test data set are used as the experimental data set distribution strategy.
Preferably, said analyzing said comparative experimental data comprises: and extracting a target tongue image classification rule and an elevated value of each evaluation index corresponding to each reference tongue image classification rule in each experimental data set from the fusion comparison experimental data.
And carrying out corresponding difference on the target tongue image classification rule and each reference tongue image classification rule at the lifting value of each evaluation index corresponding to each experimental data set, and taking the difference value as a lifting value deviation.
If the deviation of the lifting value of the target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and the lifting value of the target tongue image classification rule corresponding to the evaluation index in the experimental data set is greater than 0, taking the reference tongue image classification rule as a first optimized tongue image classification rule, taking the experimental data set as a first optimized experiment data set of the first optimized tongue image classification rule, and taking the evaluation index as an optimized evaluation index of the first optimized experiment data set.
Extracting a lifting value deviation difference U rvl of each optimization evaluation index corresponding to each first optimization experiment data set under each first optimization tongue image classification rule corresponding to the target tongue image classification rule, r represents the number of the first optimization tongue image classification rule,V denotes the first optimization experimental dataset number, v=1, 2,.. l=1, 2,. The..epsilon., and further, the fusion precision optimization trend psi Excellent (excellent) of the target tongue image classification rule is counted.
And screening and extracting a target tongue image classification rule and a descending value of each evaluation index corresponding to each reference tongue image classification rule in each experimental data set from the ablation comparison experimental data, and carrying out homologous statistics according to a statistical mode of phi Excellent (excellent) to obtain an ablation accurate optimization trend phi' Excellent (excellent) of the target tongue image classification rule.
The comprehensive precision optimization trend psi Heald of the target tongue image classification rule is counted,And psi 0、ψ1 is the fusion experiment precise optimized trend degree and the ablation experiment precise optimized trend degree of the set reference respectively, and psi Excellent (excellent) 、ψ′ Excellent (excellent) and psi Heald are taken as comparison experiment analysis results.
Preferably, a specific statistical formula of the fused precise optimization trend of the target tongue image classification rule is as follows: Wherein K Ginseng radix represents the number of tongue image classification rules, U 0 is the set reference evaluation index lifting value,/> Sigma and epsilon are the number of the first optimized tongue image classification rules, the number of the first optimized experimental data sets and the number of the optimized evaluation indexes respectively.
Compared with the prior art, the invention has the following beneficial effects: (1) According to the invention, the static characteristics of the image are extracted by using the residual network, the spatial characteristics between the internal main structures of the image are extracted by using the capsule network, then the dimension reduction of the characteristics of the residual network and the capsule network is realized by using a plurality of layers of convolution, finally the extracted characteristics of the residual network and the capsule network are fused, the residual network and the capsule network are used as basic network architecture of each trainer of the federal learning model, further the federal learning is developed based on the architecture, the problem that the accuracy of classification results is difficult to guarantee when the current data quantity is insufficient is effectively solved, the use scene of tongue image classification is expanded, the scene restriction of the current tongue image classification mode is avoided, and the flexibility, the applicability and the reliability of tongue image classification are further improved.
(2) According to the invention, by adding the predictive label and the real label to the training sample, the possibility and convenience are provided for the development of federal learning under the background of limited case resources, the addition of the pseudo label of the difficult-to-train sample is realized, the subsequent difficult-to-train sample is convenient for repeated training of the federal learning model, and the accuracy and the effectiveness of the subsequent tongue image classification are further improved.
(3) According to the invention, the static features of the image are extracted by using the residual error network, the spatial features between the main structures in the image are extracted by using the capsule network, the defect that the current image features are extracted incompletely is avoided, the spatial feature extraction of different parts of the tongue fur image is realized, and the convenience and feasibility of constructing the federal learning model are improved.
(4) When the invention carries out data transmission between a local trainer and a global trainer, the encryption parameter matrix and the decryption parameter matrix are set based on the federal learning model and the homomorphic encryption model, and meanwhile, the data transmission is carried out in an asynchronous transmission mode, so that the current transmission data volume is effectively reduced, the communication overhead is further reduced, the blocking of a communication link is effectively avoided, the tongue classification efficiency is ensured, and the loss probability and the transmission error probability in the data transmission process are also effectively reduced on the other side.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of the steps of the method of the present invention.
FIG. 2 is a schematic diagram of a federal learning model according to the present invention.
Fig. 3 is a schematic diagram of data exchange transfer according to the present invention.
FIG. 4 is a diagram illustrating the extraction of network structural features of a trainer according to the present invention.
Fig. 5 is a comparison chart of a fusion experiment of tongue image classification rules in a 2CLS dataset according to the present invention.
FIG. 6 is a comparison chart of the fusion experiment of tongue image classification rules of the invention in ZXSFL datasets.
Fig. 7 is a comparison graph of an ablation experiment of the tongue image classification rule of the present invention in a 2CLS dataset.
Fig. 8 is a comparison chart of an ablation experiment with tongue image classification rules of the present invention at ZXSFL dataset.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, the invention provides a multi-model fusion tongue image intelligent classification method based on data enhancement, which comprises the following steps: step 1, taking each tongue fur image as each training sample, forming an original data set, expanding the original data set to obtain a training data set, and adding a prediction label and a real label to each training sample in the training data set.
In a specific embodiment, the expansion mode of expanding the original data set to obtain the training data set is as follows: each training sample in the raw data set is incremented by a number of training samples including, but not limited to, flipping and mirroring, and the incremented number of training samples and the number of training samples are then combined into the training data set.
The embodiment of the invention provides possibility and convenience for the development of federal learning under the background of limited case resources by adding the predictive label and the real label to the training sample, realizes the addition of the pseudo label of the difficult-to-train sample, is convenient for the subsequent difficult-to-train sample to carry out repeated training of the federal learning model, and further improves the accuracy and the effectiveness of the subsequent tongue classification.
And 2, respectively extracting the characteristics of different dimensions of each training sample in the training data set by combining a plurality of network structures, and fusing to obtain fused characteristics.
The specific implementation process of fusion in step 2 includes the following steps: and selecting training samples in the training data set in a random voting mode, and marking each selected training sample as a target sample.
And extracting static characteristics of the target sample through a residual error network.
Spatial features between internal body structures of the target sample are extracted through the capsule network.
And reducing the dimension of the static features extracted from the residual network and the spatial features extracted from the capsule network through a plurality of layers of convolution.
And fusing the static characteristics and the space characteristics after dimension reduction through a fusion algorithm.
In a specific embodiment, a specific fusion process for performing fusion is shown in fig. 4, first, a certain batch of samples are obtained by using a random voting mode, and static features and spatial features of the samples are extracted by using a residual network and a capsule network respectively. Then continuing to carry out 3 times of basic convolution, wherein the convolution kernel sizes are respectively as follows: 3×3,2×2, the step sizes are respectively: and 2 multiplied by 2, reducing the dimension of the characteristics extracted by the residual network and the capsule network through 3-layer simple convolution, and finally fusing the residual network and the capsule network through a fusion algorithm.
According to the embodiment of the invention, the static characteristics of the image are extracted by using the residual network, the spatial characteristics among the main structures in the image are extracted by using the capsule network, the defect that the current image characteristics are extracted incompletely is avoided, the spatial characteristic extraction of different parts of the tongue fur image is practiced, and the convenience and feasibility of constructing the federal learning model are further improved.
Further, the fusion algorithm is specifically expressed as: x is training sample data of a trainer i, i is a trainer number, i=1, 2, &..n, gi, α, β is a network structure representation of the ith trainer, α represents a sharing parameter between trainers, β represents a local private parameter of the trainer, fr, fc respectively represent a static feature and a spatial feature, δ and λ respectively represent classification contribution factors corresponding to the static feature and the spatial feature, and f merge represents a fusion feature.
And 3, selecting the difficult-to-train samples of each trainer, setting a difficult-to-train sample queue according to the difficult-to-train sample, and performing reinforcement training on the difficult-to-train sample queue.
Specifically, the difficult training sample selection is obtained through training sample selection model selection, wherein the training sample selection model specifically comprises the following steps: Wherein j represents the training sample number in the training data set, j=1, 2, &..u, M is a reference minimum cosine value for setting the features and fusion features of the difficult-to-train sample, M j represents the cosine distance between the predicted label and the real label corresponding to the jth training sample in the training data set, H j←x if mj is less than or equal to M, when the cosine distance between the predicted label and the real label corresponding to the jth training sample in the training data set is less than or equal to the reference minimum cosine value for the features and fusion features of the difficult-to-train sample, the training samples are collected into a difficult-to-train sample queue H j, L is the number of labels, and Avg j′ represents the average value of all the features of the real sample corresponding to the real label in the training samples of the trainer i.
It should be noted that, assuming that f ix represents the feature extracted by the trainer i on the sample x, f ix＝Gi,α,β, the calculation formula of Avg' j′ is: Dim represents feature fusion dimensions of each trainer, wherein the dimensions of the feature fusion of each trainer are the same, z represents dimensions of a vector corresponding to a spatial feature, and k represents dimensions of a vector corresponding to a static feature.
And 4, extracting attribute labels of all trainers, respectively marking the trainers with the attribute labels of the overall and the body as the overall trainers and the local trainers, exchanging data between each local trainer and the overall trainers, retraining the exchanged data, and classifying tongue images according to training results.
Specifically, each local trainer exchanges data with a global trainer, including: and B1, taking a residual network and a capsule network as basic network architecture of each trainer under a federal learning model, and encrypting fusion characteristics of each local trainer through a homomorphic encryption model based on the federal learning model to further obtain an encryption parameter matrix P g of each local trainer, wherein g represents the number of the local trainer, and g=1, 2.
In one embodiment, the federal learning model is illustrated with reference to fig. 2, each local trainer has its own private data set, and after training on the private data set, participates in data exchange with the global trainer.
Understandably, the homomorphic encryption model is specifically expressed as: In the/> The method is characterized in that the method comprises the steps of representing an upward rounding symbol, y is a set natural constant, f a,d is a matrix element, a represents the dimension of a matrix to be encrypted, and d represents the length of a vector in the dimension of the matrix to be encrypted.
In a specific embodiment, the encryption precision is completely dependent on the amplification ratio, and the smaller the amplification ratio is, the more precision is lost, and in order to ensure the encryption precision, the specific value of y can be 7.
B2, constructing a restoration parameter matrix W g of each body trainer based on the encryption parameter matrix,
Where the sig function characterizes the sign of an element in the encryption matrix, x 'is the matrix element, x' =f a,d.
B3, setting decryption parameter matrix The parameter Q is the decryption function of homomorphic encryption matrix,/>Representing the inner product of the matrix corresponding elements.
And B4, according to the encryption parameter matrix and the decryption parameter matrix of the fusion characteristics corresponding to each local trainer, each local trainer and each global trainer exchange data in an asynchronous transmission mode, as shown in fig. 3, a dotted line indicates that no communication exists between some local trainers and global trainers in the current communication process.
It should be noted that, the local trainer uploads the encrypted gradient, the global experience pool is used for storing the uploading gradient of the local trainer, after the global experience pool is full, all samples in the global experience pool are emptied, the network parameters of the global trainer are safely aggregated and updated, and the aggregation mode sums the gradients uploaded by the local trainers.
In a specific embodiment, the local trainer may be understood as a client, and the global trainer may be understood as a server, where the lack of communication between the local trainer and the global trainer may be that the computing capability of the local trainer is insufficient, so that the gradient calculation cannot be completed within a specified time, or the global trainer randomly selects a part of the local trainers to broadcast the gradient, so that a number of local trainers cannot update parameters at a time.
When the embodiment of the invention is used for transmitting data of a local trainer and a global trainer, the encryption parameter matrix and the decryption parameter matrix are set based on the federal learning model and the homomorphic encryption model, and meanwhile, the data transmission is carried out in an asynchronous transmission mode, so that the current transmission data amount is effectively reduced, the communication overhead is further reduced, the blocking of a communication link is effectively avoided, the tongue classification efficiency is ensured, and the loss probability and the transmission error probability in the data transmission process are also effectively reduced in the other layer.
And 5, performing a plurality of training data set comparison experiments to obtain comparison experiment data, analyzing the comparison experiment data to obtain comparison experiment results, and further outputting the comparison experiment results.
In the embodiment of the invention, a plurality of training data set comparison experiments are carried out, and the method comprises the following steps: a1, taking the current multi-model fusion tongue image classification as a target tongue image classification rule, and extracting each tongue image classification rule which is currently accumulated and formulated from a tongue image classification database as each reference tongue image classification rule.
In a specific embodiment, the target tongue image classification rule is simplified to AFME algorithm, and CapsNet +LSTM algorithm, resNet + BILSTM algorithm and ResNetblock + CapsNet algorithm are selected as each reference tongue image classification rule.
A3, setting each comparison data set, taking each comparison data set as each experimental data set, and simultaneously formulating each evaluation index.
In a specific embodiment, the medical dataset datasets 2CLS and ZXSFL are selected as each experimental dataset, and the dataset scale is expanded in a manner of simultaneously turning over on the X-axis, the Y-axis, the X-axis and the Y-axis, wherein the basic information of each experimental dataset is shown in table 1.
Table 1 basic information of experimental data set
| Data set name | Data type | Original dataset size | Post expansion dataset size | Category(s) |
| 2CLS | Image data set | 831 | 3324 | 4 |
| ZXSFL | Image data set | 1778 | 7112 | 2 |
In another embodiment, four of Accuracy, precision, recall-Score and F1-Score are selected as evaluation indexes, recall-Score is simplified to Recall, F1-Score is simplified to F1, and the evaluation indexes are calculated by the existing mature technology, which is not described herein, wherein tongue image classification rules and evaluation indexes can be shown in Table 2.
Table 2 comparison algorithm and evaluation index
A2, randomly extracting Y 0 local trainers as local trainers of each experiment, randomly extracting Y 1 global trainers as global trainers of each experiment, and formulating an experiment data set allocation strategy.
Understandably, formulating an experimental dataset distribution strategy includes: a2-1, integrating the number of all experimental global trainers and the number of all experimental local trainers to obtain all experimental trainers, sequencing all experimental trainers according to the prior sequencing rule of all experimental trainers, and taking the sequencing result as the number of all experimental trainers.
A2-2, carrying out scale distribution of the experimental data sets on each experimental trainer according to a random distribution rule to obtain the distribution scale of the experimental data sets corresponding to each experimental trainer, wherein the random distribution rule is specifically expressed asK' q represents the distribution scale of the corresponding experimental data set of the q-th experimental trainer, q represents the experimental trainer number, q=1, 2.
A2-3, setting a training data set and a test data set, setting allocation duty ratio of the training data set and the test data set, marking the allocation duty ratio of the training data set as k Training device , marking the allocation duty ratio of the test data set as k Measuring , and corresponding each experimental trainer of k Training device and k Measuring to the allocation scale of the training data set and the test data set.
A2-4, taking the distribution scale of the experimental data set and the distribution scales of the training data set and the test data set as an experimental data set distribution strategy.
In a specific embodiment, for ease of analysis, the embodiment of the present invention selects one global trainer and three local trainers as experimental trainers, where the experimental trainers' experimental dataset allocation strategy can be referred to in table 3.
Table 3 experimental trainer experimental data set assignments
| Training person | Data set size | Training set | Test set |
| Global experiment trainer | 10％ | 80％ | 20％ |
| Local experiment trainer 1 | 20％ | 80％ | 20％ |
| Local experiment trainer 2 | 30％ | 80％ | 20％ |
| Local experiment trainer 3 | 40％ | 80％ | 20％ |
And A4, carrying out fusion comparison experiments on all experimental data sets based on the target tongue image classification rules and all reference tongue image classification rules, and recording fusion comparison experiment data.
In a specific embodiment, based on the fusion contrast experimental data of the fusion contrast experiment performed on the 2CLS experimental data set by using the target tongue image classification rule and each reference tongue image classification rule shown in fig. 5, fig. 5 (a) shows the F1 value of the fusion contrast experiment performed on the 2CLS experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, fig. 5 (b) shows the Accuracy value of the fusion contrast experiment performed on the 2CLS experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, fig. 5 (c) shows the Precision value of the fusion contrast experiment performed on the 2CLS experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, and fig. 5 (d) shows the Recall value of the fusion contrast experiment performed on the 2CLS experimental data set by using the target tongue image classification rule and each reference tongue image classification rule.
In a specific embodiment, based on the fusion contrast experimental data of the fusion contrast experiment performed on the ZXSFL experimental data set by using the target tongue image classification rule and each reference tongue image classification rule shown in fig. 6, fig. 6 (a) shows the F1 value of the fusion contrast experiment performed on the ZXSFL experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, fig. 6 (b) shows the Accuracy value of the fusion contrast experiment performed on the ZXSFL experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, fig. 6 (c) shows the Precision value of the fusion contrast experiment performed on the ZXSFL experimental data set by using the target tongue image classification rule and each reference tongue image classification rule, and fig. 6 (d) shows the Recall value of the fusion contrast experiment performed on the ZXSFL experimental data set by using the target tongue image classification rule and each reference tongue image classification rule.
In another embodiment, the fusion contrast experimental data are specifically shown with reference to tables 4,5 and 6 for the convenience of subsequent analysis.
Table 4 summary of experiments in 2CLS and ZXSFL experimental data sets
Table 5 summary of the improvement effect of each evaluation index in 2CLS experimental dataset
Table 6 summary of the lifting effects of the evaluation indicators in ZXSFL experimental dataset
It can be understood from tables 5 and 6 that the average accuracy of the algorithm of AFME on the 2CLS experimental dataset is improved by 5.111% at the maximum, 3.795% at the minimum, and compared with the classification rule of each reference tongue image, the accuracy is improved by 8.010% at the maximum, 3.031% at the minimum, 2.953% at the average accuracy of the algorithm of AFME on the ZXSFL dataset is improved by 2.953% at the maximum, 2.265% at the minimum, and 4.972% at the minimum, 8.829% at the maximum, compared with the classification rule of each reference tongue image, so that the algorithm of AFME has certain superiority and robustness.
A5, updating the test data set, taking the updated test data set as an ablation experiment data set, carrying out an ablation contrast experiment on the ablation experiment data set based on the target tongue image classification rule and each reference tongue image classification rule, recording the ablation contrast experiment data,
In a specific embodiment, based on the ablation contrast experiment data of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the 2CLS experiment data set shown in fig. 7, fig. 7 (a) shows the F1 value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the 2CLS experiment data set, fig. 7 (b) shows the Accuracy value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the 2CLS experiment data set, fig. 7 (c) shows the Precision value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the 2CLS experiment data set, and fig. 7 (d) shows the Recall value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the 2CLS experiment data set.
In a specific embodiment, based on the ablation contrast experiment data of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the ZXSFL experiment data set shown in fig. 8, fig. 8 (a) shows the F1 value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the ZXSFL experiment data set, fig. 8 (b) shows the Accuracy value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the ZXSFL experiment data set, fig. 8 (c) shows the Precision value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the ZXSFL experiment data set, and fig. 8 (d) shows the Recall value of the target tongue image classification rule and each reference tongue image classification rule for performing the ablation contrast experiment on the ZXSFL experiment data set.
Ablation contrast experimental data may be as shown in table 7 for ease of subsequent analysis.
Table 7 summary of ablation vs. experimental data
Understandably, from table 7, it can be seen that the AFME algorithm trained without adding the expanded samples in the training set had a decrease in the sample predictive power on the test set, but the rate of decrease was not high. Compared with the experimental result of the algorithm CapsNet +LSTM, the accuracy of CapsNet +LSTM algorithm on the 2CLS experimental data set is reduced by 9.64%, the prediction accuracy on the ZXSFL experimental data set is reduced by 4.75%, namely, the average prediction accuracy on the two experimental data sets is reduced by 7.15%, the accuracy of ResNetblock + CapsNet algorithm on the 2CLS data set is reduced by 8.42%, the prediction accuracy on the ZXSFL experimental data set is reduced by 6.14%, namely, the average prediction accuracy on the two experimental data sets is reduced by 7.25%, the accuracy of ResNet + BILSTM algorithm on the 2CLS data set is reduced by 6.93%, the prediction accuracy on the ZXSFL experimental data set is reduced by 5.25%, the average prediction accuracy on the two experimental data sets is reduced by 6.09%, and the target tongue image classification rule of the invention, namely, the accuracy of the AFME algorithm on the 2CLS data set is reduced by 1.04%, the prediction accuracy on the ZXSFL experimental data set is reduced by 2.42%, and the average prediction accuracy on the two experimental data sets is reduced by 1.73%.
According to the embodiment of the invention, through fusion comparison experiments and ablation comparison experiments, the classification accuracy and robustness of the current tongue image classification mode are intuitively displayed, and through comparison experiments of multiple experimental data sets, multiple evaluation indexes and multiple comparison groups, the scientificity and the referential of the current tongue image classification accuracy evaluation are ensured, data assistance is provided for the construction of a subsequent tongue image classification model, the optimization of the tongue image classification model is facilitated, and the selection of a subsequent user is facilitated.
Further, analyzing the comparative experimental data, comprising: and X1, extracting a target tongue image classification rule from the fusion comparison experimental data, wherein each reference tongue image classification rule corresponds to the lifting value of each evaluation index in each experimental data set.
And X2, carrying out corresponding difference on the target tongue image classification rule and each reference tongue image classification rule at the lifting value of each evaluation index corresponding to each experimental data set, and taking the difference value as a lifting value deviation.
And X3, if the deviation of the lifting value of the target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and the lifting value of the target tongue image classification rule corresponding to the evaluation index in the experimental data set is greater than 0, taking the reference tongue image classification rule as a first optimized tongue image classification rule, taking the experimental data set as a first optimized experiment data set of the first optimized tongue image classification rule, and taking the evaluation index as an optimized evaluation index of the first optimized experiment data set.
X4, extracting a lifting value deviation difference U rvl of each optimization evaluation index corresponding to each first optimization experiment dataset under each first optimization tongue image classification rule corresponding to the target tongue image classification rule, r represents the number of the first optimization tongue image classification rule,V denotes the first optimization experimental data set number, v=1, 2, & gtWherein K Ginseng radix represents the number of tongue image classification rules, U 0 is the set reference evaluation index lifting value,/>Sigma and epsilon are the number of the first optimized tongue image classification rules, the number of the first optimized experimental data sets and the number of the optimized evaluation indexes respectively.
And X5, screening and extracting a target tongue image classification rule and a descending value of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set from the ablation comparison experimental data, and carrying out corresponding difference between the target tongue image classification rule and the descending value of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set to obtain a descending value deviation.
And X6, if the deviation between the descending value of the target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and the descending value of the target tongue image classification rule corresponding to the evaluation index in the experimental data set is smaller than 0, taking the reference tongue image classification rule as a second optimized tongue image classification rule, taking the experimental data set as a second optimized experiment data set of the second optimized tongue image classification rule, and taking the evaluation index as an optimized evaluation index of the second optimized experiment data set.
And X7, carrying out homologous statistics according to a statistical mode of phi Excellent (excellent) to obtain the ablation accurate optimization trend phi' Excellent (excellent) of the target tongue image classification rule.
X8, the comprehensive accurate optimization trend psi Heald of the statistical target tongue image classification rule,And psi 0、ψ1 is the fusion experiment precise optimized trend degree and the ablation experiment precise optimized trend degree of the set reference respectively, and psi Excellent (excellent) 、ψ′ Excellent (excellent) and psi Heald are taken as comparison experiment analysis results.
The foregoing is merely illustrative and explanatory of the principles of this invention, as various modifications and additions may be made to the specific embodiments described, or similar arrangements may be substituted by those skilled in the art, without departing from the principles of this invention or beyond the scope of this invention as defined in the claims.
Claims (8)
1. A multi-model fusion tongue image intelligent classification method based on data enhancement is characterized in that: comprising the following steps:
step 1, taking each tongue fur image as each training sample, forming an original data set, expanding the original data set to obtain a training data set, and adding a prediction label and a real label to each training sample in the training data set;
Step 2, respectively extracting features of different dimensions of each training sample in the training data set by combining a plurality of network structures, and fusing to obtain fused features;
step 3, selecting a difficult-to-train sample for each trainer, setting a difficult-to-train sample queue according to the difficult-to-train sample selection, and performing reinforcement training on the difficult-to-train sample queue;
Step 4, extracting attribute labels of all trainers, respectively marking the trainers with the attribute labels of the overall and the body as the overall trainers and the local trainers, exchanging data between each local trainer and the overall trainer, retraining the exchanged data, and classifying tongue images according to training results;
Step 5, performing a plurality of training data set comparison experiments to obtain comparison experiment data, analyzing the comparison experiment data to obtain comparison experiment results, and outputting the comparison experiment results;
the method for performing the plurality of training data set comparison experiments comprises the following steps:
a1, taking the current multi-model fusion tongue image classification as a target tongue image classification rule, and extracting each tongue image classification rule which is currently accumulated and formulated from a tongue image classification database as each reference tongue image classification rule;
A3, setting each comparison data set, taking each comparison data set as each experimental data set, and simultaneously formulating each evaluation index;
A2, randomly extracting Local trainers are used as local trainers for each experiment, and are randomly extracted/>The global trainers are used as the global trainers of all experiments, and an experiment data set allocation strategy is formulated;
a4, carrying out fusion comparison experiments on all experimental data sets based on the target tongue image classification rules and all reference tongue image classification rules, and recording fusion comparison experiment data;
A5, updating the test data set, taking the updated test data set as an ablation experiment data set, carrying out an ablation comparison experiment on the ablation experiment data set based on the target tongue image classification rule and each reference tongue image classification rule, and recording ablation comparison experiment data;
The making of the experimental data set allocation strategy comprises the following steps:
Integrating the number of the experimental global trainers and the number of the experimental local trainers to obtain the experimental trainers, sequencing the experimental trainers according to the first sequencing rule of the experimental global trainers, and taking the sequencing result as the serial number of the experimental trainers;
Performing scale distribution of the experimental data sets on each experimental trainer according to a random distribution rule to obtain the distribution scale of the experimental data sets corresponding to each experimental trainer, wherein the random distribution rule is specifically expressed as ，/>Represents the/>Distribution scale of corresponding experimental data set of individual experimental trainers,/>The number of the experiment training person is shown,，/>Represents the/>The distribution scale of the corresponding experimental data sets of the individual experimental trainers;
Setting a training data set and a test data set, setting the allocation ratio of the training data set and the test data set, and marking the allocation ratio of the training data set as The allocation duty cycle of the test dataset is noted as/>And will/>And/>Each experimental trainer corresponds to the distribution scale of the training data set and the test data set;
the distribution scale of the experimental data set and the distribution scales of the training data set and the test data set are used as the experimental data set distribution strategy.
2. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: the specific implementation process of fusion in the step 2 comprises the following steps:
training sample selection is carried out in a training data set in a random voting mode, and each selected training sample is marked as a target sample;
extracting static characteristics of a target sample through a residual error network;
extracting spatial features among internal main body structures of a target sample through a capsule network;
realizing dimension reduction of static features extracted from a residual error network and spatial features extracted from a capsule network through a plurality of layers of convolution;
and fusing the static characteristics and the space characteristics after dimension reduction through a fusion algorithm.
3. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 2, which is characterized in that: the fusion algorithm is specifically expressed as:，/> Is the trainer/> Training sample data of,/>Numbering the trainers,/>，/>For/>Network structure representation of individual trainers,/>Representing sharing parameters between trainers,/>Representing the local private parameters of the trainer,/>、/>Respectively represent static characteristics and spatial characteristics,/>And/>Representing the corresponding classification contribution factor of static features and spatial features respectively,/>Representing the fusion characteristics.
4. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: the difficult training sample selection is obtained through training sample selection models, wherein the training sample selection models are specifically expressed as: In the above, the ratio of/> Representing the training sample numbers in the training dataset,，/>To set the reference minimum cosine value of the characteristics and the fusion characteristics of the samples difficult to train,/>Representing the/>, in the training datasetCosine distance between predicted label and real label corresponding to each training sample,/>Representing the/>, in the training datasetWhen the cosine distance between the prediction label corresponding to each training sample and the real label is smaller than or equal to the reference minimum cosine value of the characteristics of the difficult training sample and the fusion characteristics, the training samples are collected to a difficult training sample queue/>In/>The number of tags to be used in the method,Representing the trainer/>The real labels in the training samples correspond to the average value of all real sample characteristics.
5. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 2, which is characterized in that: each local trainer exchanges data with a global trainer, and the method comprises the following steps:
Taking the residual network and the capsule network as basic network architecture of each trainer under the federal learning model, and based on the federal learning model, each local trainer encrypts the fusion characteristics of each local trainer through the homomorphic encryption model to further obtain an encryption parameter matrix of each local trainer ，/>Representing the local trainer number,/>；
Constructing a restoring parameter matrix of each ontology trainer based on encryption parameter matrix，In the above, the ratio of/>The function characterizes the sign of the element in the encryption matrix,/>Representing the dimension of the matrix to be encrypted,/>Representing the length of the vector in the dimension of the matrix to be encrypted,/>To set natural constant,/>Is a matrix element,/>；
Setting decryption parameter matrix，/>Parameter/>Is a decryption function of homomorphic encryption matrix,/>Representing solving inner products of corresponding elements of the matrix;
And according to the encryption parameter matrix and the decryption parameter matrix of the corresponding fusion characteristics of each local trainer, each local trainer and each global trainer exchange data in an asynchronous transmission mode.
6. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 5, which is characterized in that: the homomorphic encryption model is specifically expressed as: In which, in the process, Representing rounding up symbols.
7. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: said analyzing said comparative experimental data comprising:
Extracting a target tongue image classification rule and an elevated value of each evaluation index corresponding to each reference tongue image classification rule in each experimental data set from the fusion comparison experimental data;
performing corresponding difference on the target tongue image classification rule and each reference tongue image classification rule at the lifting value of each evaluation index corresponding to each experimental data set, and taking the difference value as a lifting value deviation;
If the deviation of the lifting value of the target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and the lifting value of the target tongue image classification rule corresponding to the evaluation index in the experimental data set is greater than 0, taking the reference tongue image classification rule as a first optimized tongue image classification rule, taking the experimental data set as a first optimized experiment data set of the first optimized tongue image classification rule, and taking the evaluation index as an optimized evaluation index of the first optimized experiment data set;
Extracting the lifting value deviation difference of each optimization evaluation index corresponding to each first optimization experiment data set under each first optimization tongue image classification rule corresponding to the target tongue image classification rule ，/>Representing the number of the first optimized tongue image classification rule/>,Representing the first optimization experiment dataset numbering,/>，/>To optimize the evaluation index number,/>Further, the fusion precision optimization trend/>, of the target tongue image classification rules is counted；
Screening and extracting target tongue image classification rules and descending values of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set from the ablation comparison experimental data according to the following steps ofThe statistical mode of the target tongue image classification rule is obtained through the same statistics, and the ablation accurate optimization trend/>；
Comprehensive accurate optimization trend degree for statistics of target tongue image classification rules，/>，The trend degree is accurately optimized for the fusion experiment of the set reference and the ablation experiment, and/>、And/>As a result of comparative experimental analysis.
8. The intelligent classification method for the multi-model fusion tongue images based on data enhancement of claim 7 is characterized in that: the specific statistical formula of the fused precise optimization trend of the target tongue image classification rule is as follows: In the above, the ratio of/> Representing tongue image classification rule number,/>To set the reference evaluation index rise value,/>The number of the first optimized tongue image classification rules, the number of the first optimized experiment data sets and the number of the optimized evaluation indexes are respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311513218.2A CN117557844B (en) | 2023-11-14 | 2023-11-14 | Multi-model fusion tongue image intelligent classification method based on data enhancement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311513218.2A CN117557844B (en) | 2023-11-14 | 2023-11-14 | Multi-model fusion tongue image intelligent classification method based on data enhancement |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117557844A CN117557844A (en) | 2024-02-13 |
| CN117557844B true CN117557844B (en) | 2024-04-26 |
Family
ID=89816023
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311513218.2A Active CN117557844B (en) | 2023-11-14 | 2023-11-14 | Multi-model fusion tongue image intelligent classification method based on data enhancement |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117557844B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111191660A (en) * | 2019-12-30 | 2020-05-22 | 浙江工业大学 | Rectal cancer pathology image classification method based on multi-channel collaborative capsule network |
| CN111223553A (en) * | 2020-01-03 | 2020-06-02 | 大连理工大学 | Two-stage deep migration learning traditional Chinese medicine tongue diagnosis model |
| CN111783831A (en) * | 2020-05-29 | 2020-10-16 | 河海大学 | Complex image accurate classification method based on multi-source multi-label shared subspace learning |
| CN114581432A (en) * | 2022-03-18 | 2022-06-03 | 河海大学 | Tongue appearance tongue image segmentation method based on deep learning |
-
2023
- 2023-11-14 CN CN202311513218.2A patent/CN117557844B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111191660A (en) * | 2019-12-30 | 2020-05-22 | 浙江工业大学 | Rectal cancer pathology image classification method based on multi-channel collaborative capsule network |
| CN111223553A (en) * | 2020-01-03 | 2020-06-02 | 大连理工大学 | Two-stage deep migration learning traditional Chinese medicine tongue diagnosis model |
| CN111783831A (en) * | 2020-05-29 | 2020-10-16 | 河海大学 | Complex image accurate classification method based on multi-source multi-label shared subspace learning |
| CN114581432A (en) * | 2022-03-18 | 2022-06-03 | 河海大学 | Tongue appearance tongue image segmentation method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117557844A (en) | 2024-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2023065545A1 (en) | Risk prediction method and apparatus, and device and storage medium | |
| CN113535984B (en) | Knowledge graph relation prediction method and device based on attention mechanism | |
| Li et al. | An efficient manifold regularized sparse non-negative matrix factorization model for large-scale recommender systems on GPUs | |
| CN105825269B (en) | A kind of feature learning method and system based on parallel automatic coding machine | |
| CN107680661A (en) | System and method for estimating medical resource demand | |
| CN113190688A (en) | Complex network link prediction method and system based on logical reasoning and graph convolution | |
| CN115659985B (en) | Electric power knowledge graph entity alignment method and device and computer equipment | |
| CN109815478A (en) | Medicine entity recognition method and system based on convolutional neural networks | |
| CN115588193A (en) | Visual question-answering method and device based on graph attention neural network and visual relation | |
| CN117557844B (en) | Multi-model fusion tongue image intelligent classification method based on data enhancement | |
| CN118036726A (en) | Knowledge graph link prediction method based on heuristic information and graph neural network | |
| CN112925994A (en) | Group recommendation method, system and equipment based on local and global information fusion | |
| CN117273143A (en) | Time sequence knowledge graph reasoning method and system based on Markov logic network | |
| CN111914772A (en) | Method for identifying age, and training method and device of age identification model | |
| CN115905953A (en) | State data processing method and device, computer equipment and storage medium | |
| CN109299291A (en) | A kind of Ask-Answer Community label recommendation method based on convolutional neural networks | |
| CN114861863A (en) | Heterogeneous graph representation learning method based on meta-path multi-level graph attention network | |
| Amene et al. | Variable selection and regression analysis for the prediction of mortality rates associated with foodborne diseases | |
| Nasir et al. | Pandemia Prediction Using Machine Learning | |
| Li et al. | GT-CAM: Game Theory based Class Activation Map for GCN | |
| Gu et al. | Defining and identifying the optimal embedding dimension of networks | |
| Kulakou | Exploration of time-series models on time series data | |
| CN110647856B (en) | Method for recognizing facial expressions based on theory of axiomatic fuzzy set | |
| CN115830400B (en) | Data identification method and system based on federal learning mechanism | |
| Hapsari et al. | Steganography on Color Images Using Least Significant Bit (LSB) Method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |