CN117557844B - Multi-model fusion tongue image intelligent classification method based on data enhancement - Google Patents

Abstract

The present invention belongs to the technical field of tongue image classification, and specifically discloses a multi-model fusion tongue image intelligent classification method based on data enhancement, which includes: constructing an original data set, expanding the original data set to obtain a training data set, adding predicted labels and real labels to each training sample in the training data set; selecting difficult training samples, setting a difficult training sample queue, and performing intensive training; extracting and fusing features of training samples; exchanging data between each local trainer and the global trainer, and training, classifying tongue images accordingly, performing comparative experiments on multiple training data sets, and analyzing comparative experimental data. The present invention effectively solves the problem that it is difficult to ensure the accuracy of classification results when the current data volume is insufficient, expands the use scenarios of tongue image classification, avoids the scenario limitations of the current tongue image classification method, and thus improves the flexibility, applicability and reliability of tongue image classification.

Description

A multi-model fusion tongue image intelligent classification method based on data enhancement

Technical Field

The present invention belongs to the technical field of tongue image classification, and relates to a multi-model fusion tongue image intelligent classification method based on data enhancement.

Background technique

In recent years, with the continuous development of artificial intelligence technology, tongue image intelligent classification has played an increasingly important role in the field of TCM diagnosis. Tongue image is one of the important bases for TCM diagnosis. By observing and analyzing tongue image, the type, degree and change trend of the disease can be judged.

In the task of intelligent tongue image classification, data enhancement methods can be used to expand the training set, which can effectively increase the diversity of training data and improve the robustness of the model. However, the current model has high requirements for sample size. When the sample size is not enough, the parameters are difficult to be effectively trained, which often causes underfitting, making it difficult for the model to achieve good results. Therefore, the current tongue image classification still has the following deficiencies: 1. It is difficult to ensure the accuracy of classification when the amount of data is insufficient: the cases of medical-related organizations are confidential and the structured data that can actually be used for machine learning varies greatly. Large medical organizations may have a lot of case resources, but for small medical organizations, the cases are often limited and there are certain restrictions.

2. It is difficult to carry out federated learning with limited medical record resources, and pseudo labels are not added to difficult-to-train samples for repeated training of the federated model.

3. Incomplete feature extraction: Residual networks, VGG16, LSTM and other algorithms can extract static features of images and texts very well, but it is difficult to effectively model the spatial correlation between different parts of the image.

4. The current data transmission volume is huge, resulting in a large amount of communication overhead, which in turn easily causes communication link congestion and cannot guarantee the efficiency of tongue image classification.

Summary of the invention

In view of this, in order to solve the problems raised in the above background technology, a multi-model fusion tongue image intelligent classification method based on data enhancement is proposed.

The objective of the present invention can be achieved through the following technical scheme: The present invention provides a multi-model fusion tongue image intelligent classification method based on data enhancement, comprising: step 1, taking each tongue coating image as each training sample, and forming an original data set, expanding the original data set to obtain a training data set, and adding predicted labels and true labels to each training sample in the training data set.

Step 2: Combine multiple network structures to extract features of different dimensions of each training sample in the training data set, and fuse them to obtain fused features.

Step 3: Select difficult training samples for each trainer, set up a difficult training sample queue accordingly, and perform intensive training on the difficult training sample queue.

Step 4: extract the attribute labels of each trainer, record the trainers with attribute labels of global and entity as global trainers and local trainers respectively, exchange data between each local trainer and the global trainer, train the exchanged data again, and classify the tongue images according to the training results.

Step 5: Conduct comparative experiments on multiple training data sets to obtain comparative experimental data, analyze the comparative experimental data to obtain comparative experimental results, and then output the comparative experimental results.

Preferably, the specific execution process of the fusion in step 2 includes the following steps: selecting training samples from the training data set by random voting, and recording each selected training sample as a target sample.

The static features of the target sample are extracted through the residual network.

The capsule network is used to extract the spatial features between the main structures inside the target sample.

The dimensionality reduction of the static features extracted by the residual network and the spatial features extracted by the capsule network is achieved through several layers of convolution.

The static features and spatial features after dimensionality reduction are fused through the fusion algorithm.

Preferably, the fusion algorithm is specifically expressed as: x is the training sample data of trainer i, i is the trainer number, i=1,2,...n, Gi,α,β is the network structure representation of the i-th trainer, α represents the shared parameters between trainers, β represents the local private parameters of the trainer, fr and fc represent static features and spatial features respectively, δ and λ represent the classification contribution factors corresponding to static features and spatial features respectively, and f _merge represents fused features.

Preferably, the difficult training samples are selected by a training sample selection model, wherein the training sample selection model is specifically expressed as: In the formula, j represents the training sample number in the training data set, j = 1, 2, ... u, M is the reference minimum cosine value for setting the difficult training sample features and the fusion features, m _j represents the cosine distance between the predicted label and the true label corresponding to the j-th training sample in the training data set, H _j ←x if m _j ≤M means that when the cosine distance between the predicted label and the true label corresponding to the j-th training sample in the training data set is less than or equal to the reference minimum cosine value between the difficult training sample features and the fusion features, the training sample is collected into the difficult training sample queue H _j , L is the number of labels, and Avg _j′ represents the mean of all true sample features corresponding to the true label in the training samples of trainer i.

Preferably, each local trainer exchanges data with the global trainer, including: using the residual network and the capsule network as the basic network architecture of each trainer under the federated learning model, based on the federated learning model, each local trainer encrypts its fusion features through a homomorphic encryption model, and then obtains the encrypted parameter matrix P _g of each local trainer, where g represents the local trainer number, g=1, 2,...h.

Based on the encrypted parameter matrix, the restored parameter matrix W _g of each ontology trainer is constructed.

Wherein, the sig function represents the sign of the element in the encryption matrix, a represents the dimension of the matrix to be encrypted, d represents the length of the vector under the dimension of the matrix to be encrypted, y is a set natural constant, x′ is a matrix element, and x′=fa _,d .

Set decryption parameter matrix Parameter Q is the decryption function of the homomorphic encryption matrix,/> It means to find the inner product of corresponding elements of the matrix.

According to the encryption parameter matrix and decryption parameter matrix of each local trainer corresponding to the fusion feature, each local trainer exchanges data with each global trainer through asynchronous transmission.

Preferably, the homomorphic encryption model is specifically expressed as: In the formula, /> Indicates the round-up symbol.

Preferably, the comparative experiment of multiple training data sets comprises the following steps: A1, taking the current multi-model fusion tongue image classification as the target tongue image classification rule, and extracting the currently accumulated tongue image classification rules from the tongue image classification database as the reference tongue image classification rules.

A3. Set up various comparison data sets and use them as experimental data sets, and formulate various evaluation indicators.

A2. Randomly select Y ₀ local trainers as local trainers for each experiment, and randomly select Y ₁ global trainers as global trainers for each experiment, and formulate an experimental data set allocation strategy.

A4. Based on the target tongue image classification rules and the reference tongue image classification rules, a fusion comparison experiment is carried out on each experimental data set, and the fusion comparison experimental data is recorded.

A5. Update the test data set, use the updated test data set as the ablation experiment data set, conduct an ablation comparison experiment on the ablation experiment data set based on the target tongue image classification rules and each reference tongue image classification rule, and record the ablation comparison experiment data.

Preferably, the formulation of the experimental data set allocation strategy includes: integrating the number of global trainers of each experiment and the number of local trainers of each experiment to obtain trainers of each experiment, sorting each experimental trainer according to the experiment global trainer first sorting rule, and using the sorting result as the number of each experimental trainer.

According to the random allocation rule, the scale of the experimental data set is allocated to each experimental trainer to obtain the allocation scale of the experimental data set corresponding to each experimental trainer. The random allocation rule is specifically expressed as k′ _q represents the allocation scale of the corresponding experimental data set of the qth experimental trainer, q represents the experimental trainer number, q=1, 2,...b, k′ _q-1 represents the allocation scale of the corresponding experimental data set of the q-1th experimental trainer.

Set up a training data set and a test data set, and set the allocation ratio of the training data set and the test data set. The allocation ratio of the training data set is recorded as _ktrain , and the allocation ratio of the test data set is recorded as _ktest . The _ktrain and _ktest experimental trainers correspond to the allocation scales of the training data set and the test data set.

The allocation scale of the experimental dataset and the allocation scale of the training dataset and the test dataset are used as the experimental dataset allocation strategy.

Preferably, the analyzing the comparative experimental data includes: extracting the target tongue image classification rule and the improvement value of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set from the fused comparative experimental data.

The improvement values of the target tongue image classification rule and the reference tongue image classification rules corresponding to each evaluation index in each experimental data set are subtracted, and the difference is used as the improvement value deviation.

If the deviation between the improvement value of a target tongue image classification rule corresponding to a certain evaluation indicator in a certain experimental data set and that of a reference tongue image classification rule corresponding to the evaluation indicator in the experimental data set is greater than 0, the reference tongue image classification rule is used as the first optimized tongue image classification rule, the experimental data set is used as the first optimized experimental data set of the first optimized tongue image classification rule, and the evaluation indicator is used as the optimized evaluation indicator of the first optimized experimental data set.

Extract the improvement value deviation difference U _rvl of each optimization evaluation index corresponding to each first optimization experimental data set under each first optimization tongue image classification rule of the target tongue image classification rule, where r represents the number of the first optimization tongue image classification rule, v represents the first optimization experimental data set number, v=1,2,...σ, l is the optimization evaluation index number, l=1,2,...ε, and then the fusion precise optimization trend degree ψ of the target tongue image classification rule is statistically _analyzed .

The target tongue image classification rule and the reduction values of the reference tongue image classification rules corresponding to each evaluation index in each experimental data set are screened out from the ablation comparison experimental data, and the ablation precision optimization trend degree _ψ′optimal of the target tongue image classification rule is obtained in the same way as the statistical method of _ψoptimal .

The comprehensive and accurate optimization trend degree ψ of the statistical target tongue image classification rule _is ψ ₀ and ψ ₁ are respectively the precise optimization trend of the fusion experiment and the precise optimization trend of the ablation experiment of the set reference, and ψ _optimal , ψ′ _optimal and ψ _{comprehensive} are taken as the comparative experimental analysis results.

Preferably, the specific statistical formula for the fusion precision optimization trend of the target tongue image classification rule is: In the formula, K _parameter represents the number of tongue image classification rules, U ₀ is the reference evaluation index improvement value, /> σ and ε are the number of the first optimized tongue image classification rules, the number of the first optimized experimental data sets, and the number of optimized evaluation indicators, respectively.

Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention uses a residual network to extract static features of an image, uses a capsule network to extract spatial features between the main structures inside the image, and then uses several layers of convolution to achieve dimensionality reduction of the features of the two. Finally, the features extracted by the two are fused, and the residual network and the capsule network are used as the basic network architecture of each trainer of the federated learning model, and then federated learning is carried out based on the architecture, which effectively solves the problem that it is difficult to ensure the accuracy of classification results when the current data volume is insufficient, expands the use scenarios of tongue image classification, avoids the scenario limitations of the current tongue image classification method, and thus improves the flexibility, applicability and reliability of tongue image classification.

(2) The present invention provides the possibility and convenience for the implementation of federated learning in the context of limited case resources by adding predicted labels and true labels to training samples, realizes the addition of pseudo labels for difficult-to-train samples, and facilitates the repeated training of the federated learning model for subsequent difficult-to-train samples, thereby improving the accuracy and effectiveness of subsequent tongue image classification.

(3) The present invention uses a residual network to extract the static features of the image and a capsule network to extract the spatial features between the main structures inside the image, thereby avoiding the shortcomings of the current image feature extraction that is incomplete, and practicing the spatial feature extraction of different parts of the tongue coating image, thereby improving the convenience and feasibility of building a federated learning model.

(4) When the present invention performs data transmission between local trainers and global trainers, the encryption parameter matrix and the decryption parameter matrix are set based on the federated learning model and the homomorphic encryption model, and data is transmitted through asynchronous transmission, which effectively reduces the current transmission data volume, thereby reducing communication overhead, effectively avoiding congestion of the communication link, ensuring the efficiency of tongue image classification, and at another level effectively reducing the loss probability and transmission error probability during data transmission.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

FIG1 is a schematic flow chart of the implementation steps of the method of the present invention.

FIG2 is a schematic diagram of a federated learning model of the present invention.

FIG. 3 is a schematic diagram of data exchange and transmission according to the present invention.

FIG4 is a schematic diagram of the trainer network structure feature extraction of the present invention.

FIG5 is a comparison diagram of the fusion experiment of the tongue image classification rule of the present invention in the 2CLS data set.

FIG. 6 is a comparison diagram of the fusion experiment of the tongue image classification rule of the present invention on the ZXSFL dataset.

FIG. 7 is a comparison diagram of the ablation experiment of the tongue image classification rule of the present invention on the 2CLS dataset.

FIG8 is a comparison diagram of the ablation experiment of the tongue image classification rule of the present invention on the ZXSFL dataset.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Please refer to Figure 1, the present invention provides a multi-model fusion tongue image intelligent classification method based on data enhancement, including: step 1, taking each tongue coating image as a training sample to form an original data set, expanding the original data set to obtain a training data set, and adding predicted labels and true labels to each training sample in the training data set.

In a specific embodiment, the expansion method of expanding the original data set to obtain the training data set is: increasing the number of training samples of each training sample in the original data set by means including but not limited to flipping and mirroring, and then combining the increased number of training samples and the number of training samples into the training data set.

The embodiments of the present invention provide the possibility and convenience for the implementation of federated learning in the context of limited case resources by adding predicted labels and true labels to training samples, realize the addition of pseudo labels for difficult-to-train samples, facilitate the repeated training of the federated learning model for subsequent difficult-to-train samples, and thus improve the accuracy and effectiveness of subsequent tongue image classification.

Step 2: Combine multiple network structures to extract features of different dimensions of each training sample in the training data set, and fuse them to obtain fused features.

Exemplarily, the specific execution process of the fusion in step 2 includes the following steps: selecting training samples from the training data set by random voting, and recording each selected training sample as a target sample.

The static features of the target sample are extracted through the residual network.

The capsule network is used to extract the spatial features between the main structures inside the target sample.

The dimensionality reduction of the static features extracted by the residual network and the spatial features extracted by the capsule network is achieved through several layers of convolution.

The static features and spatial features after dimensionality reduction are fused through the fusion algorithm.

In a specific embodiment, the specific fusion process is shown in FIG4. First, a certain batch of samples is obtained by random voting, and the static features and spatial features of the samples are extracted using the residual network and the capsule network, respectively. Then, three basic convolutions are performed, and the convolution kernel sizes are 3×3, 3×3, and 2×2, and the step sizes are 2×2. The features extracted by the residual network and the capsule network are reduced in dimension through three layers of simple convolution, and finally the two are fused through the fusion algorithm.

The embodiment of the present invention uses a residual network to extract the static features of the image and a capsule network to extract the spatial features between the main structures inside the image, thereby avoiding the shortcomings of the current image feature extraction that are incomplete, and practicing the spatial feature extraction of different parts of the tongue coating image, thereby improving the convenience and feasibility of building a federated learning model.

Furthermore, the fusion algorithm is specifically expressed as: x is the training sample data of trainer i, i is the trainer number, i=1,2,......n, Gi,α,β is the network structure representation of the i-th trainer, α represents the shared parameters between trainers, β represents the local private parameters of the trainer, fr and fc represent static features and spatial features respectively, δ and λ represent the classification contribution factors corresponding to static features and spatial features respectively, and f _merge represents fused features.

Step 3: Select difficult training samples for each trainer, set up a difficult training sample queue accordingly, and perform intensive training on the difficult training sample queue.

Specifically, the difficult training samples are selected by a training sample selection model, wherein the training sample selection model is specifically expressed as: In the formula, j represents the training sample number in the training data set, j = 1, 2, ... u, M is the reference minimum cosine value for setting the difficult training sample features and the fusion features, m _j represents the cosine distance between the predicted label and the true label corresponding to the j-th training sample in the training data set, H _j ←x if m _j ≤M means that when the cosine distance between the predicted label and the true label corresponding to the j-th training sample in the training data set is less than or equal to the reference minimum cosine value between the difficult training sample features and the fusion features, the training sample is collected into the difficult training sample queue H _j , L is the number of labels, and Avg _j′ represents the mean of all true sample features corresponding to the true label in the training samples of trainer i.

It should be noted that, assuming that _fix represents the features extracted by trainer i for sample x, _fix = _Gi,α,β , then the calculation formula of Avg′j _′ is: Dim represents the fusion dimension of the features of each trainer, wherein the dimensions of the features of each trainer after fusion in the present invention are the same, z represents the dimension of the vector corresponding to the spatial feature, and k represents the dimension of the vector corresponding to the static feature.

Step 4: extract the attribute labels of each trainer, record the trainers with attribute labels of global and entity as global trainers and local trainers respectively, exchange data between each local trainer and the global trainer, train the exchanged data again, and classify the tongue images according to the training results.

Specifically, each local trainer exchanges data with the global trainer, including: B1. Using the residual network and capsule network as the basic network architecture of each trainer under the federated learning model, based on the federated learning model, each local trainer encrypts its fusion features through the homomorphic encryption model, and then obtains the encrypted parameter matrix P _g of each local trainer, where g represents the local trainer number, g=1,2,......h.

In a specific embodiment, the federated learning model is shown in FIG. 2 , where each local trainer has its own private data set, and after training on the private data set, participates in data exchange with the global trainer.

Understandably, the homomorphic encryption model is specifically expressed as: In the formula, /> represents the rounding up symbol, y is the set natural constant, f _a,d are matrix elements, a represents the dimension of the matrix to be encrypted, and d represents the length of the vector under the dimension of the matrix to be encrypted.

In a specific embodiment, the encryption accuracy depends entirely on the magnification ratio. The smaller the magnification ratio, the more accuracy is lost. In order to ensure the encryption accuracy, the specific value of y can be 7.

B2. Construct the restored parameter matrix W _g of each ontology trainer based on the encrypted parameter matrix.

Wherein, the sig function represents the sign of the element in the encryption matrix, x′ is the matrix element, and x′=fa _,d .

B3. Set the decryption parameter matrix Parameter Q is the decryption function of the homomorphic encryption matrix,/> It means to find the inner product of corresponding elements of the matrix.

B4. According to the encryption parameter matrix and decryption parameter matrix of the fusion features corresponding to each local trainer, each local trainer exchanges data with each global trainer through asynchronous transmission, as shown in Figure 3. The dotted line indicates that there is no communication between some local trainers and the global trainer during the current communication process.

It should be noted that the local trainer uploads encrypted gradients, and the global experience pool is used to store the uploaded gradients of the local trainer. When the global experience pool is full, all samples in the global experience pool are cleared, and secure aggregation is performed to update the network parameters of the global trainer. The aggregation method sums the gradients uploaded by each local trainer.

In a specific embodiment, the local trainer can be understood as a client, and the global trainer can be understood as a server. The lack of communication between the local trainer and the global trainer may be due to insufficient computing power of the local trainer, resulting in failure to complete gradient calculation within the specified time. It may also be that the global trainer randomly selects some local trainers to broadcast gradients, resulting in several local trainers being unable to update parameters at a single time.

When the embodiment of the present invention performs data transmission between the local trainer and the global trainer, the encryption parameter matrix and the decryption parameter matrix are set based on the federated learning model and the homomorphic encryption model, and the data is transmitted through asynchronous transmission, which effectively reduces the current transmission data volume, thereby reducing the communication overhead, effectively avoiding the congestion of the communication link, ensuring the efficiency of tongue image classification, and at another level, effectively reducing the loss probability and transmission error probability during data transmission.

Step 5: Conduct comparative experiments on multiple training data sets to obtain comparative experimental data, analyze the comparative experimental data to obtain comparative experimental results, and then output the comparative experimental results.

In an embodiment of the present invention, a comparison experiment of multiple training data sets is conducted, including the following steps: A1. The current multi-model fusion tongue image classification is used as the target tongue image classification rule, and the currently accumulated tongue image classification rules are extracted from the tongue image classification database as the reference tongue image classification rules.

In a specific embodiment, the target tongue image classification rule is simplified to the AFME algorithm, and the CapsNet+LSTM algorithm, the ResNet+BILSTM algorithm and the ResNetblock+CapsNet algorithm are selected as the reference tongue image classification rules.

A3. Set up various comparison data sets and use them as experimental data sets, and formulate various evaluation indicators.

In a specific embodiment, medical data sets 2CLS and ZXSFL are selected as experimental data sets, and the data set size is expanded by flipping on the X axis, Y axis, and X axis and Y axis at the same time. The basic information of each experimental data set is shown in Table 1.

Table 1 Basic information of the experimental dataset

Dataset name type of data Original dataset size The size of the expanded dataset category 2CLS Image Datasets 831 3324 4 ZXSFL Image Datasets 1778 7112 2

In another specific embodiment, Accuracy, Precision, Recall-Score, and F1-Score are selected as evaluation indicators, Recall-Score is simplified to Recall, and F1-Score is simplified to F1. The calculation method of the evaluation indicators is an existing relatively mature technology and will not be elaborated here. The tongue image classification rules and evaluation indicators can be referred to as shown in Table 2.

Table 2 Comparison of algorithms and evaluation indicators

A2. Randomly select Y ₀ local trainers as local trainers for each experiment, and randomly select Y ₁ global trainers as global trainers for each experiment, and formulate an experimental data set allocation strategy.

Understandably, the experimental data set allocation strategy is formulated, including: A2-1, integrating the number of global trainers of each experiment and the number of local trainers of each experiment to obtain the trainers of each experiment, sorting the trainers of each experiment according to the first sorting rule of the global trainers of the experiment, and using the sorting result as the number of the trainers of each experiment.

A2-2. Allocate the scale of the experimental data set to each experimental trainer according to the random allocation rule to obtain the allocation scale of the experimental data set corresponding to each experimental trainer. The random allocation rule is specifically expressed as k′ _q represents the allocation scale of the corresponding experimental data set of the qth experimental trainer, q represents the experimental trainer number, q=1, 2,...b, k′ _q-1 represents the allocation scale of the corresponding experimental data set of the q-1th experimental trainer.

A2-3. Set up a training data set and a test data set, and set the allocation ratio of the training data set and the test data set. The allocation ratio of the training data set is recorded as _ktrain , and the allocation ratio of the test data set is recorded as _ktest . The _ktrain and _ktest experimental trainers correspond to the allocation scales of the training data set and the test data set.

A2-4. The allocation scale of the experimental data set and the allocation scale of the training data set and the test data set are used as the experimental data set allocation strategy.

In a specific embodiment, for the convenience of analysis, the embodiment of the present invention selects one global trainer and three local trainers as experimental trainers, wherein the experimental data set allocation strategy of the experimental trainers can be referred to as shown in Table 3.

Table 3 Experimental trainer experimental dataset allocation

trainer Dataset size Training set Test Set Global Experiment Trainer 10% 80% 20% Local Experiment Trainer 1 20% 80% 20% Local Experiment Trainer 2 30% 80% 20% Local Experiment Trainer 3 40% 80% 20%

A4. Based on the target tongue image classification rules and the reference tongue image classification rules, a fusion comparison experiment is carried out on each experimental data set, and the fusion comparison experimental data is recorded.

In a specific embodiment, based on Figure 5, fusion comparison experimental data of the target tongue image classification rule and each reference tongue image classification rule carried out on the 2CLS experimental data set can be obtained, wherein Figure 5(a) represents the F1 value of the target tongue image classification rule and each reference tongue image classification rule carried out on the 2CLS experimental data set, Figure 5(b) represents the Accuracy value of the target tongue image classification rule and each reference tongue image classification rule carried out on the 2CLS experimental data set, Figure 5(c) represents the Precision value of the target tongue image classification rule and each reference tongue image classification rule carried out on the 2CLS experimental data set, and Figure 5(d) represents the Recall value of the target tongue image classification rule and each reference tongue image classification rule carried out on the 2CLS experimental data set.

In a specific embodiment, based on FIG6 , fusion comparison experimental data of the target tongue image classification rule and each reference tongue image classification rule carried out on the ZXSFL experimental data set can be obtained, wherein FIG6(a) represents the F1 value of the target tongue image classification rule and each reference tongue image classification rule carried out on the ZXSFL experimental data set for fusion comparison experiment, FIG6(b) represents the Accuracy value of the target tongue image classification rule and each reference tongue image classification rule carried out on the ZXSFL experimental data set for fusion comparison experiment, FIG6(c) represents the Precision value of the target tongue image classification rule and each reference tongue image classification rule carried out on the ZXSFL experimental data set for fusion comparison experiment, and FIG6(d) represents the Recall value of the target tongue image classification rule and each reference tongue image classification rule carried out on the ZXSFL experimental data set for fusion comparison experiment.

In another specific embodiment, in order to facilitate subsequent analysis, the fusion comparison experimental data can be specifically referred to as shown in Table 4, Table 5 and Table 6.

Table 4 Experimental summary on 2CLS and ZXSFL experimental datasets

Table 5 Summary of the improvement effect of each evaluation index in the 2CLS experimental dataset

Table 6 Summary of the improvement effect of each evaluation index in the ZXSFL experimental dataset

Understandably, it can be seen from Tables 5 and 6 that the target tongue image rule of the present invention, i.e., the average accuracy of the AFME algorithm on the 2CLS experimental data set is improved by a maximum of 5.111% and a minimum of 3.795%, and relative to each reference tongue image classification rule, the accuracy is improved by a maximum of 8.010% and a minimum of 3.031%. The average accuracy of the AFME algorithm on the ZXSFL data set is improved by a maximum of 2.953% and a minimum of 2.265%, and relative to each reference tongue image classification rule, the accuracy is improved by a minimum of 4.972% and a maximum of 8.829%. It can be seen that the AFME algorithm has certain superiority and robustness.

A5. Update the test data set, use the updated test data set as the ablation experiment data set, conduct ablation comparison experiments on the ablation experiment data set based on the target tongue image classification rules and the reference tongue image classification rules, and record the ablation comparison experiment data.

In a specific embodiment, based on FIG7 , ablation comparison experiment data of the target tongue image classification rule and each reference tongue image classification rule in the 2CLS experimental data set can be obtained, wherein FIG7(a) represents the F1 value of the ablation comparison experiment of the target tongue image classification rule and each reference tongue image classification rule in the 2CLS experimental data set, FIG7(b) represents the Accuracy value of the ablation comparison experiment of the target tongue image classification rule and each reference tongue image classification rule in the 2CLS experimental data set, FIG7(c) represents the Precision value of the ablation comparison experiment of the target tongue image classification rule and each reference tongue image classification rule in the 2CLS experimental data set, and FIG7(d) represents the Recall value of the ablation comparison experiment of the target tongue image classification rule and each reference tongue image classification rule in the 2CLS experimental data set.

In a specific embodiment, based on FIG8 , ablation comparison experiment data of the target tongue image classification rule and each reference tongue image classification rule in the ZXSFL experimental dataset can be obtained, wherein FIG8(a) represents the F1 value of the target tongue image classification rule and each reference tongue image classification rule in the ZXSFL experimental dataset, FIG8(b) represents the Accuracy value of the target tongue image classification rule and each reference tongue image classification rule in the ZXSFL experimental dataset, FIG8(c) represents the Precision value of the target tongue image classification rule and each reference tongue image classification rule in the ZXSFL experimental dataset, and FIG8(d) represents the Recall value of the target tongue image classification rule and each reference tongue image classification rule in the ZXSFL experimental dataset.

To facilitate subsequent analysis, the ablation comparison experimental data can be found in Table 7.

Table 7. Summary of ablation comparison experimental data

Understandably, it can be seen from Table 7 that when the AFME algorithm trained without adding the expanded samples to the training set, the prediction ability of the samples in the test set is reduced, but the proportion of the reduction is not high. Compared with the experimental results of the CapsNet+LSTM algorithm, the accuracy of the CapsNet+LSTM algorithm on the 2CLS experimental dataset decreased by 9.64%, and the prediction accuracy on the ZXSFL experimental dataset decreased by 4.75%, that is, the prediction accuracy on the two experimental datasets decreased by 7.15% on average; the accuracy of the ResNetblock+CapsNet algorithm on the 2CLS dataset decreased by 8.42%, and the prediction accuracy on the ZXSFL experimental dataset decreased by 6.14%, that is, the prediction accuracy on the two experimental datasets decreased by 7.25% on average; the accuracy of the ResNet+BILSTM algorithm on the 2CLS dataset decreased by 6.93%, and the prediction accuracy on the ZXSFL experimental dataset decreased by 5.25%, and the prediction accuracy on the two experimental datasets decreased by 6.09% on average; the target tongue image classification rule of the present invention, that is, the AFME algorithm, decreased by 1.04% on the 2CLS dataset, decreased by 2.42% on the ZXSFL experimental dataset, and decreased by 1.73% on average on the two experimental datasets.

The embodiments of the present invention intuitively demonstrate the classification accuracy and robustness of the current tongue image classification method by conducting fusion comparison experiments and ablation comparison experiments. At the same time, through comparison experiments of multiple experimental data sets, multiple evaluation indexes and multiple control groups, the scientificity and reference nature of the current tongue image classification accuracy evaluation are ensured, which provides data assistance for the construction of subsequent tongue image classification models, facilitates the optimization of tongue image classification models, and is also convenient for subsequent users to select.

Further, analyzing the comparative experimental data includes: X1, extracting the target tongue image classification rule and the improvement value of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set from the fused comparative experimental data.

X2. Subtract the improvement values of the target tongue image classification rule and the reference tongue image classification rules in each experimental data set corresponding to each evaluation indicator, and use the difference as the improvement value deviation.

X3. If the deviation between the improvement value of a target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and that of a reference tongue image classification rule corresponding to the evaluation index in the experimental data set is greater than 0, the reference tongue image classification rule is used as the first optimized tongue image classification rule, the experimental data set is used as the first optimized experimental data set of the first optimized tongue image classification rule, and the evaluation index is used as the optimized evaluation index of the first optimized experimental data set.

X4, extracting the improvement value deviation difference U _rvl of each optimization evaluation index corresponding to each first optimization experimental data set under each first optimization tongue image classification rule of the target tongue image classification rule, where r represents the number of the first optimization tongue image classification rule, v represents the number of the first optimization experiment data set, v = 1, 2, ... σ, l is the number of the optimization evaluation index, l = 1, 2, ... ε, and then the fusion precision optimization trend _degree ψ of the target tongue image classification rule is statistically analyzed, /> In the formula, K _parameter represents the number of tongue image classification rules, U ₀ is the reference evaluation index improvement value, /> σ and ε are the number of the first optimized tongue image classification rules, the number of the first optimized experimental data sets, and the number of optimized evaluation indicators, respectively.

X5. Filter out the target tongue image classification rules and the reduction values of each reference tongue image classification rules corresponding to each evaluation index in each experimental data set from the ablation comparison experimental data, and make a corresponding difference between the reduction values of each evaluation index corresponding to the target tongue image classification rules and each reference tongue image classification rules in each experimental data set to obtain the reduction value deviation.

X6. If the decrease value deviation between a target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and a reference tongue image classification rule corresponding to the evaluation index in the experimental data set is less than 0, the reference tongue image classification rule is used as the second optimized tongue image classification rule, the experimental data set is used as the second optimized experimental data set of the second optimized tongue image classification rule, and the evaluation index is used as the optimized evaluation index of the second optimized experimental data set.

X7. According to the statistical method of _ψoptimal , the ablation precision optimization trend degree _ψ′optimal of the target tongue image classification rule is obtained.

X8. Comprehensive and accurate optimization trend degree of statistical target tongue image classification rules ψ _{comprehensive} , ψ ₀ and ψ ₁ are respectively the precise optimization trend of the fusion experiment and the precise optimization trend of the ablation experiment of the set reference, and ψ _optimal , ψ′ _optimal and ψ _{comprehensive} are taken as the comparative experimental analysis results.

The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they should all fall within the protection scope of the present invention.

Claims (8)

1. A multi-model fusion tongue image intelligent classification method based on data enhancement is characterized in that: comprising the following steps:

step 1, taking each tongue fur image as each training sample, forming an original data set, expanding the original data set to obtain a training data set, and adding a prediction label and a real label to each training sample in the training data set;

Step 2, respectively extracting features of different dimensions of each training sample in the training data set by combining a plurality of network structures, and fusing to obtain fused features;

step 3, selecting a difficult-to-train sample for each trainer, setting a difficult-to-train sample queue according to the difficult-to-train sample selection, and performing reinforcement training on the difficult-to-train sample queue;

Step 4, extracting attribute labels of all trainers, respectively marking the trainers with the attribute labels of the overall and the body as the overall trainers and the local trainers, exchanging data between each local trainer and the overall trainer, retraining the exchanged data, and classifying tongue images according to training results;

Step 5, performing a plurality of training data set comparison experiments to obtain comparison experiment data, analyzing the comparison experiment data to obtain comparison experiment results, and outputting the comparison experiment results;

the method for performing the plurality of training data set comparison experiments comprises the following steps:

a1, taking the current multi-model fusion tongue image classification as a target tongue image classification rule, and extracting each tongue image classification rule which is currently accumulated and formulated from a tongue image classification database as each reference tongue image classification rule;

A3, setting each comparison data set, taking each comparison data set as each experimental data set, and simultaneously formulating each evaluation index;

A2, randomly extracting Local trainers are used as local trainers for each experiment, and are randomly extracted/>The global trainers are used as the global trainers of all experiments, and an experiment data set allocation strategy is formulated;

a4, carrying out fusion comparison experiments on all experimental data sets based on the target tongue image classification rules and all reference tongue image classification rules, and recording fusion comparison experiment data;

A5, updating the test data set, taking the updated test data set as an ablation experiment data set, carrying out an ablation comparison experiment on the ablation experiment data set based on the target tongue image classification rule and each reference tongue image classification rule, and recording ablation comparison experiment data;

The making of the experimental data set allocation strategy comprises the following steps:

Integrating the number of the experimental global trainers and the number of the experimental local trainers to obtain the experimental trainers, sequencing the experimental trainers according to the first sequencing rule of the experimental global trainers, and taking the sequencing result as the serial number of the experimental trainers;

Performing scale distribution of the experimental data sets on each experimental trainer according to a random distribution rule to obtain the distribution scale of the experimental data sets corresponding to each experimental trainer, wherein the random distribution rule is specifically expressed as ，/>Represents the/>Distribution scale of corresponding experimental data set of individual experimental trainers,/>The number of the experiment training person is shown,，/>Represents the/>The distribution scale of the corresponding experimental data sets of the individual experimental trainers;

Setting a training data set and a test data set, setting the allocation ratio of the training data set and the test data set, and marking the allocation ratio of the training data set as The allocation duty cycle of the test dataset is noted as/>And will/>And/>Each experimental trainer corresponds to the distribution scale of the training data set and the test data set;

the distribution scale of the experimental data set and the distribution scales of the training data set and the test data set are used as the experimental data set distribution strategy.

2. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: the specific implementation process of fusion in the step 2 comprises the following steps:

training sample selection is carried out in a training data set in a random voting mode, and each selected training sample is marked as a target sample;

extracting static characteristics of a target sample through a residual error network;

extracting spatial features among internal main body structures of a target sample through a capsule network;

realizing dimension reduction of static features extracted from a residual error network and spatial features extracted from a capsule network through a plurality of layers of convolution;

and fusing the static characteristics and the space characteristics after dimension reduction through a fusion algorithm.

3. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 2, which is characterized in that: the fusion algorithm is specifically expressed as:，/> Is the trainer/> Training sample data of,/>Numbering the trainers,/>，/>For/>Network structure representation of individual trainers,/>Representing sharing parameters between trainers,/>Representing the local private parameters of the trainer,/>、/>Respectively represent static characteristics and spatial characteristics,/>And/>Representing the corresponding classification contribution factor of static features and spatial features respectively,/>Representing the fusion characteristics.

4. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: the difficult training sample selection is obtained through training sample selection models, wherein the training sample selection models are specifically expressed as: In the above, the ratio of/> Representing the training sample numbers in the training dataset,，/>To set the reference minimum cosine value of the characteristics and the fusion characteristics of the samples difficult to train,/>Representing the/>, in the training datasetCosine distance between predicted label and real label corresponding to each training sample,/>Representing the/>, in the training datasetWhen the cosine distance between the prediction label corresponding to each training sample and the real label is smaller than or equal to the reference minimum cosine value of the characteristics of the difficult training sample and the fusion characteristics, the training samples are collected to a difficult training sample queue/>In/>The number of tags to be used in the method,Representing the trainer/>The real labels in the training samples correspond to the average value of all real sample characteristics.

5. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 2, which is characterized in that: each local trainer exchanges data with a global trainer, and the method comprises the following steps:

Taking the residual network and the capsule network as basic network architecture of each trainer under the federal learning model, and based on the federal learning model, each local trainer encrypts the fusion characteristics of each local trainer through the homomorphic encryption model to further obtain an encryption parameter matrix of each local trainer ，/>Representing the local trainer number,/>；

Constructing a restoring parameter matrix of each ontology trainer based on encryption parameter matrix，In the above, the ratio of/>The function characterizes the sign of the element in the encryption matrix,/>Representing the dimension of the matrix to be encrypted,/>Representing the length of the vector in the dimension of the matrix to be encrypted,/>To set natural constant,/>Is a matrix element,/>；

Setting decryption parameter matrix，/>Parameter/>Is a decryption function of homomorphic encryption matrix,/>Representing solving inner products of corresponding elements of the matrix;

And according to the encryption parameter matrix and the decryption parameter matrix of the corresponding fusion characteristics of each local trainer, each local trainer and each global trainer exchange data in an asynchronous transmission mode.

6. The intelligent classification method of the multi-model fusion tongue images based on data enhancement according to claim 5, which is characterized in that: the homomorphic encryption model is specifically expressed as: In which, in the process, Representing rounding up symbols.

7. The intelligent classification method of the multi-model fusion tongue images based on data enhancement of claim 1 is characterized in that: said analyzing said comparative experimental data comprising:

Extracting a target tongue image classification rule and an elevated value of each evaluation index corresponding to each reference tongue image classification rule in each experimental data set from the fusion comparison experimental data;

performing corresponding difference on the target tongue image classification rule and each reference tongue image classification rule at the lifting value of each evaluation index corresponding to each experimental data set, and taking the difference value as a lifting value deviation;

If the deviation of the lifting value of the target tongue image classification rule corresponding to a certain evaluation index in a certain experimental data set and the lifting value of the target tongue image classification rule corresponding to the evaluation index in the experimental data set is greater than 0, taking the reference tongue image classification rule as a first optimized tongue image classification rule, taking the experimental data set as a first optimized experiment data set of the first optimized tongue image classification rule, and taking the evaluation index as an optimized evaluation index of the first optimized experiment data set;

Extracting the lifting value deviation difference of each optimization evaluation index corresponding to each first optimization experiment data set under each first optimization tongue image classification rule corresponding to the target tongue image classification rule ，/>Representing the number of the first optimized tongue image classification rule/>,Representing the first optimization experiment dataset numbering,/>，/>To optimize the evaluation index number,/>Further, the fusion precision optimization trend/>, of the target tongue image classification rules is counted；

Screening and extracting target tongue image classification rules and descending values of each reference tongue image classification rule corresponding to each evaluation index in each experimental data set from the ablation comparison experimental data according to the following steps ofThe statistical mode of the target tongue image classification rule is obtained through the same statistics, and the ablation accurate optimization trend/>；

Comprehensive accurate optimization trend degree for statistics of target tongue image classification rules，/>，The trend degree is accurately optimized for the fusion experiment of the set reference and the ablation experiment, and/>、And/>As a result of comparative experimental analysis.

8. The intelligent classification method for the multi-model fusion tongue images based on data enhancement of claim 7 is characterized in that: the specific statistical formula of the fused precise optimization trend of the target tongue image classification rule is as follows: In the above, the ratio of/> Representing tongue image classification rule number,/>To set the reference evaluation index rise value,/>The number of the first optimized tongue image classification rules, the number of the first optimized experiment data sets and the number of the optimized evaluation indexes are respectively.