CN116844687A - Prescription recommendation method and system based on tongue images and knowledge patterns - Google Patents
Prescription recommendation method and system based on tongue images and knowledge patterns Download PDFInfo
- Publication number
- CN116844687A CN116844687A CN202310589649.0A CN202310589649A CN116844687A CN 116844687 A CN116844687 A CN 116844687A CN 202310589649 A CN202310589649 A CN 202310589649A CN 116844687 A CN116844687 A CN 116844687A
- Authority
- CN
- China
- Prior art keywords
- prescription
- tongue
- tongue image
- chinese medicine
- traditional chinese
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 239000003814 drug Substances 0.000 claims abstract description 88
- 238000003745 diagnosis Methods 0.000 claims abstract description 76
- 238000012549 training Methods 0.000 claims abstract description 61
- 238000003062 neural network model Methods 0.000 claims abstract description 52
- 238000012360 testing method Methods 0.000 claims abstract description 44
- 238000007781 pre-processing Methods 0.000 claims abstract description 17
- 230000006870 function Effects 0.000 claims abstract description 12
- 230000004927 fusion Effects 0.000 claims description 36
- 238000010276 construction Methods 0.000 claims description 17
- 239000000463 material Substances 0.000 claims description 17
- 239000013598 vector Substances 0.000 claims description 15
- 238000004891 communication Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 8
- 238000013461 design Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 5
- 230000003993 interaction Effects 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 201000010099 disease Diseases 0.000 description 6
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 6
- 238000007689 inspection Methods 0.000 description 6
- 208000011580 syndromic disease Diseases 0.000 description 6
- 239000011248 coating agent Substances 0.000 description 5
- 238000000576 coating method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 244000246386 Mentha pulegium Species 0.000 description 2
- 235000016257 Mentha pulegium Nutrition 0.000 description 2
- 235000004357 Mentha x piperita Nutrition 0.000 description 2
- 208000031975 Yang Deficiency Diseases 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 229940079593 drug Drugs 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 235000001050 hortel pimenta Nutrition 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000132012 Atractylodes Species 0.000 description 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 208000031971 Yin Deficiency Diseases 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 230000000857 drug effect Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 241000411851 herbal medicine Species 0.000 description 1
- 210000003734 kidney Anatomy 0.000 description 1
- 210000004185 liver Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001717 pathogenic effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 229940126532 prescription medicine Drugs 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000452 restraining effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 210000000952 spleen Anatomy 0.000 description 1
- 210000002784 stomach Anatomy 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 210000003462 vein Anatomy 0.000 description 1
- 210000001835 viscera Anatomy 0.000 description 1
- 238000009736 wetting Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/10—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to drugs or medications, e.g. for ensuring correct administration to patients
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/90—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to alternative medicines, e.g. homeopathy or oriental medicines
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
Abstract
The invention discloses a prescription recommendation method and a system based on tongue images and knowledge maps, wherein the method comprises the following steps: constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set, and preprocessing; constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning; constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for carrying out iterative training on the neural network model by utilizing a tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph, and testing by utilizing a tongue image prescription testing set to obtain an excellent neural network model; and identifying the tongue image to be processed by adopting the optimal neural network model to obtain a corresponding recommended prescription. The method can realize the auxiliary diagnosis and treatment function of recommending the traditional Chinese medicine prescription according to the tongue image of the patient. The method is developed into a software program without depending on specific medical equipment, and has certain popularization value.
Description
Technical Field
The invention relates to the technical field of computer image recognition and computer vision, in particular to a prescription recommendation method and system based on tongue images and knowledge maps.
Background
As the first of four diagnostic methods of TCM, inspection, smelling, asking and cutting, inspection is an important component of diagnosis and treatment in TCM. Inspection is performed by observing typical mapping areas such as face, ear, tongue and the like, judging the health condition of corresponding viscera according to the body surface characteristics, and diagnosing diseases. Inspection mainly focuses on facial inspection and tongue inspection. The tongue diagnosis is a method for judging the health condition of a human body by observing tongue features such as tongue body, tongue quality, tongue coating, sublingual vein and the like, has the characteristic that the tongue observation can check yin and yang deficiency and excess of the body, and the tongue scale can know the depth of pathogenic cold and heat, is an essential reference basis for the diagnosis and treatment of traditional Chinese medicine, and has important guiding function on clinical medication.
Traditional Chinese medicine tongue diagnosis relies on the knowledge and diagnosis skills of expert doctors, is easily affected by subjective factors, and faces the scarce pressure of the expert doctors. With the rapid development of computer science and artificial intelligence, many researchers combine artificial intelligence with traditional Chinese medicine diagnosis and treatment, for example, in tongue diagnosis, modern computer vision technology is introduced to extract signals in tongue coating images and capture characteristic information of the signals, so that processing and analysis of tongue coating images are realized, and medical tasks such as diagnosis and the like are completed, and even certain specific tasks have a level exceeding that of human top doctors. The artificial intelligence can be said to be effectively applied to various fields of traditional Chinese medicine, so that the traditional Chinese medicine diagnosis and treatment efficiency can be greatly improved, the scarce pressure of medical resources can be relieved, and the modern traditional Chinese medicine diagnosis and treatment can be further promoted.
The recommendation system is an important research direction in the field of computers, and is mainly used for solving the problems of information overload, data noise flooding and the like caused by massive internet data. The recommendation system realizes personalized recommendation of the user, and achieves the effect of information screening by analyzing the behavioral interests of the user and customizing a recommendation item list of the user according to an algorithm.
In recent years, research for alleviating problems of sparse data, missing key information and the like faced by a recommendation system by combining information fusion, knowledge graph and other technologies is also receiving more and more attention. The personalized customization features of the recommender system make it a solution to the task of prescribing.
Therefore, how to implement prescription recommendation according to tongue images based on computer vision technology is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the invention provides a prescription recommendation method and a system based on tongue images and knowledge maps, which can recommend corresponding traditional Chinese medicine prescriptions according to tongue image conditions and provide intelligent assistance for tongue diagnosis of traditional Chinese medicine.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
in a first aspect, an embodiment of the present invention provides a prescription recommendation method based on tongue images and knowledge maps, including the following steps:
constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set, and preprocessing;
constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning;
constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
performing iterative training on the neural network model by using the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph, and performing testing by using a tongue image prescription testing set to obtain an optimal neural network model;
and identifying the tongue image to be processed by adopting the optimal neural network model to obtain a corresponding recommended prescription.
Further, the tongue image prescription data set comprises a plurality of tongue images and corresponding prescription information; the prescription information includes: medicinal materials and dosage information.
Further, dividing and preprocessing the tongue image prescription training set and the test set, including:
dividing a tongue image prescription training set and a tongue image prescription test set according to a preset proportion;
preprocessing the tongue image prescription training set comprises random cutting, size scaling, random horizontal and vertical overturning and normalization operation;
preprocessing the tongue prescription test set is size scaling, centering clipping and normalizing operation.
Further, constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning; comprising the following steps:
the medical resource database, the traditional Chinese medicine literature and the medical hundred degree encyclopedia are used as data sources of the tongue diagnosis knowledge graph of the traditional Chinese medicine;
adopting a top-down construction mode, extracting entity, relationship and attribute information from the data sources according to a top-level data mode of a design knowledge graph, and obtaining triple knowledge;
and storing through a graph database Neo4j to complete the construction of the tongue diagnosis knowledge graph of the traditional Chinese medicine and perform knowledge representation learning.
Further, the prescription characteristic module inputs prescription information, converts the prescription information into a single thermal vector, and reduces dimensions through a full connection layer to obtain prescription characteristic representation in a text mode.
Further, the processing procedure of the prescription recommendation module comprises:
the tongue image characteristics and the tongue image node characteristics in the traditional Chinese medicine tongue diagnosis knowledge graph are fused by adopting an addition operation, so that tongue image fusion characteristics are formed;
the prescription characteristic and the traditional Chinese medicine tongue diagnosis knowledge graph traditional Chinese medicine node characteristic are fused by adopting an adding operation to form a prescription fusion characteristic;
the tongue image fusion characteristics and the prescription fusion characteristics are subjected to standardized processing and spliced, the internal interaction between the tongue image fusion characteristics and the prescription fusion characteristics is input into a multi-layer perceptron to learn, and the recommendation score predicted by the model is output; and arranging different prescriptions in a descending order according to the recommended scores, and finally obtaining the recommended prescriptions and the corresponding recommended scores.
Further, the neural network model is subjected to iterative training by utilizing the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph; comprising the following steps:
training a neural network model by taking tongue images-prescription pair samples in the tongue image prescription training set and tongue image node characteristics and medicinal material node characteristics corresponding to the tongue diagnosis knowledge graph of the traditional Chinese medicine as training sample data;
and adopting a cross entropy loss function, adopting a random gradient descent method by an optimization algorithm, and carrying out multiple iterations to optimally train the neural network model part.
In a second aspect, an embodiment of the present invention further provides a prescription recommendation system based on tongue images and knowledge maps, including:
the first construction module is used for constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set and preprocessing the tongue image prescription training set and the testing set;
the second construction module is used for constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine and carrying out knowledge representation learning;
the third construction module is used for constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
the training test module is used for carrying out iterative training and testing on the neural network model by utilizing the tongue image prescription training set, the testing set and the traditional Chinese medicine tongue diagnosis knowledge graph to obtain an optimal neural network model;
and the recommending module is used for identifying the tongue image to be processed by adopting an optimal neural network model to obtain a corresponding recommending prescription.
In a third aspect, an embodiment of the present invention further provides a computer device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
the processor is configured to implement the method for recommending prescriptions based on tongue images and knowledge patterns according to any one of the first aspect when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present invention further provides a storage medium, where at least one instruction is stored, where the instruction is loaded and executed by a processor to implement the method for recommending prescriptions based on tongue images and knowledge patterns according to any one of the first aspect.
The description of the second to fourth aspects of the present invention may refer to the detailed description of the first aspect; also, the advantageous effects described in the second aspect to the third aspect may refer to the advantageous effect analysis of the first aspect, and are not described herein.
Compared with the prior art, the prescription recommendation method based on tongue images and knowledge patterns is provided:
1. the method adopts a deep learning recommendation method and is applied to prescription making scenes of tongue diagnosis of traditional Chinese medicine. The characteristics of the tongue image of the patient and the prescription information can be extracted and mined through the network model, the internal interaction relation between the characteristics and the prescription information can be learned, namely whether the current prescription is matched with the tongue image or not, and the final recommendation result is predicted according to the matching degree.
2. According to the method, related theoretical knowledge of traditional Chinese medicine tongue diagnosis is introduced in the form of the knowledge graph, the knowledge graph of the traditional Chinese medicine tongue diagnosis is constructed, knowledge representation learning is carried out on the knowledge graph, and feature representation fused with knowledge information is obtained, so that auxiliary information of different modes can be provided.
3. The method is based on the tongue image of the patient and the traditional Chinese medicine prescription data set, and applies the deep learning recommendation technology and the constructed traditional Chinese medicine tongue diagnosis knowledge graph to the traditional Chinese medicine diagnosis and treatment field, so that the auxiliary diagnosis and treatment function of recommending the traditional Chinese medicine prescription according to the tongue image of the patient is realized. Compared with traditional Chinese medicine tongue diagnosis which relies on expert doctors, the method reduces the cost of repeated work and manpower time, relieves the scarce pressure of medical resources, does not depend on specific equipment, and has certain popularization value.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a prescription recommendation method based on tongue images and knowledge patterns provided by the invention;
FIG. 2 is a flow chart of constructing a tongue prescription dataset provided by the present invention;
FIG. 3 is a schematic diagram of a neural network model according to the present invention;
FIG. 4 is a flow chart of training and testing of a neural network model provided by the present invention;
FIG. 5 is a block diagram of a prescription recommendation system based on tongue images and knowledge patterns provided by the invention;
fig. 6 is a block diagram of a computer device provided by the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1:
the embodiment of the invention discloses a prescription recommendation method based on tongue images and knowledge maps, which comprises the following steps:
s10, constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set, and preprocessing;
s20, constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning;
s30, constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
s40, performing iterative training on the neural network model by using the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph, and performing testing by using a tongue image prescription testing set to obtain an optimal neural network model;
and S50, identifying the tongue image to be processed by adopting the optimal neural network model, and obtaining a corresponding recommended prescription.
In the embodiment, the tongue image prescription data set, the traditional Chinese medicine tongue diagnosis knowledge graph and the neural network model are constructed, and the neural network is optimized and trained based on the tongue image prescription data set and the traditional Chinese medicine tongue diagnosis knowledge graph; based on the trained neural network model, the auxiliary diagnosis and treatment function of recommending the traditional Chinese medicine prescription according to the tongue image of the patient can be realized. The method is developed into a software program without depending on specific medical equipment, and has certain popularization value.
The above steps are described in detail below.
In step S10, a tongue image prescription data set is constructed, for example, the data set used in this embodiment includes 26422 tongue images of patients and 26422 corresponding Chinese medicine prescriptions, and the types of prescription medicines collected are 577 types and corresponding dosage information. The tongue image sources of the data set are all a plurality of cooperative medical institutions in Guangzhou city in Guangdong, and the data acquisition is completed by trained medical assistants and android mobile terminals with tongue image acquisition systems deployed. The equipment held by the medical assistant is a mobile terminal camera during collection. During the acquisition process, the medical assistant will do the following: an image of the tongue coating of the patient is acquired and prescription information prescribed by a specialist physician for the patient is recorded. The present example uses the Python language to implement the code and operation of the embodiment on the Ubuntu system based on the PyTorch deep learning framework.
The acquired patient tongue image data was divided into training and test sets at a ratio of 8:2 as shown in fig. 2. The data preprocessing mode of the training set is as follows: cutting an original image according to a proportion, scaling the cut image to 224×224, then carrying out random horizontal overturning operation, respectively carrying out normalization and standardization treatment on RGB channels of the image, namely traversing each pixel of an image matrix in a data set to obtain a mean mu and a variance sigma of each channel, and obtaining a standardized numerical value through a z-score formula
x`＝(x-μ)/σ
The data preprocessing mode of the test set is centered clipping, scaling to 224×224 size, and carrying out normalization and standardization processing on the RGB channels of the image respectively. Prescription information in the data set is recorded in the form of a medicinal material list.
The data in the data set of this embodiment is in the form of tongue image-traditional Chinese medicine prescription sample pair, and the label is the cosine similarity of the sample prescription and the real prescription (i.e. input tongue image-prescription sample pair, output prediction score; label refers to real score), i.e. for one tongue image t i And a prescription p j The real prescription recorded by the tongue image during data acquisition is p i The label of the sample pair is marked p j And p is as follows i The higher the cosine similarity of (2), the p j And p is as follows i The more similar the types of the medicinal materials contained in the Chinese herbal medicines, p j The more suitable it is for recommending tongue picture t i Is a patient of (a).
In step S20, a tongue diagnosis knowledge graph of the traditional Chinese medicine is constructed, and knowledge representation learning is performed;
the sources of the knowledge graph data of the tongue diagnosis of the traditional Chinese medicine constructed in the embodiment are a medical resource database, a traditional Chinese medicine literature and a medical hundred degree encyclopedia, a top-down construction mode is adopted, for example, a top-level data mode of the knowledge graph is designed by an expert doctor, the top-level data mode comprises 7 entity types and 6 relation types, and detailed designs are respectively shown in table 1 and table 2:
table 1A physical form of knowledge graph of tongue diagnosis in TCM
| Entity type | Entity examples | Quantity of |
| Medicinal material | Herba Ephedrae; white atractylodes rhizome; ramulus Cinnamomi; peppermint (peppermint) | 577 |
| Efficacy of | Tonifying qi; dispelling wind; defervescing; relieving exterior syndrome | 260 |
| Disease type | Yang deficiency; stagnation of qi; wetting; heat of the body | 33 |
| Syndrome of pattern of wind and wind | Heart qi deficiency syndrome; disturbance of heat fireHeart syndrome | 331 |
| Site of action | Liver; spleen; a kidney; stomach | 12 |
| Drug properties | Cold; cooling; leveling; micro temperature | 9 |
| Tongue picture | Pale tongue; a cracked tongue; bao Tai; moistening the moss; | 27 |
TABLE 2 knowledge graph relationship type table for tongue diagnosis in traditional Chinese medicine
| Relationship type | Description of the relationship | Quantity of |
| The drug effect is that | (medicinal materials, efficacy is that of the efficacy) | 3404 |
| Relationships of treatment | (efficacy, treatment relationship, syndrome) | 2706 |
| Relation of disease nature | (syndrome, disease-nature relationship, disease-nature) | 644 |
| Is expressed as | (the nature of the disease, manifests as a tongue pattern) | 54 |
| The medicine property is | (medicinal materials, property of medicine is that of medicine) | 579 |
| Acting on | (medicinal material, action site) | 1372 |
After the design of the top-level data mode of the atlas is completed, entity and relation information is extracted from specific data to form triplet knowledge, the triplet knowledge is stored through a graph database Neo4j, a TransR model is adopted, so that the entity and the relation in the knowledge atlas are conveniently embedded into a low-dimensional vector space, and the method specifically operates as follows:
an example of a triplet in the knowledge graph is (head, relation, tail), where tail entity is a translation of head entity head through relation, expressed as: h+r.apprxeq.t. By continuously adjusting the vectors of h, r and t to be satisfied so as to satisfy the constraint function
The TransR model passes through a projection matrix M before restraining the triplets r Projecting the entity vector into a relationship space:
h r ＝hM r
t r ＝tM r
will throw intoThe hatched entity vector h r 、t r Substituting into constraint function to obtain
Obtaining a new triplet vector through loss training to represent a knowledge graph:
s is a ternary combination set in the constructed tongue diagnosis knowledge graph of the traditional Chinese medicine, S ' is a negative sample ternary combination set which is formed by randomly extracting entities h ' and t ' and a relation r and does not belong to S, and gamma is a threshold value.
In step S30, a neural network model is constructed, and the architecture is shown in fig. 3; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-mode feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph.
Wherein:
1) The tongue image feature module is used for extracting tongue image features based on a convolutional neural network;
specifically, in this embodiment, a convolutional neural network based on tongue images may be built, where the network structure is a residual network res net18, and the input tongue coating images are first 7×7 rolled and 3×3 pooled, and then each block is composed of two residual blocks through four residual modules, and the output sizes are 56×56×64, 28×28×128, and 14×14× 256,7 ×7×512 respectively. And connecting an average pooling layer and a full connection layer after passing through the residual error module to obtain the characteristic representation of the input tongue image.
2) The prescription feature module is used for representing the prescription by using the multi-label vector and extracting the features of the prescription;
specifically, in this embodiment, a prescription input in the form of a list of medicinal materials is converted into One-Hot vector by single-Hot encoding, each dimension 0/1 in the vector indicates whether the prescription contains the medicinal material, and a layer of full-connection layer is input for vector dimension compression, so as to obtain 128-dimensional prescription characteristic representation.
3) The prescription recommendation module fuses tongue images, prescriptions and multi-modal feature vectors of the atlas and outputs recommendation scores;
specifically, the embodiment constructs a traditional Chinese medicine prescription recommendation module and uses tongue image characteristics X t Prescription characteristic X p Respectively fusing the information under three different modes with the map features G, and predicting a recommended result according to the fused new features, wherein the specific steps are as follows:
s301, representing tongue image characteristics X t Node characteristic representation of tongue picture corresponding to knowledge graph The vector of (2) is subjected to point-to-point addition operation to obtain the characteristic fusion output based on the two modes as
Wherein g t,i For the characteristic representation of the ith tongue picture in the knowledge graph, n t The total number of tongue images.
S302, expressing X for prescription characteristics p Node characteristic representation of corresponding medicinal materials in knowledge graph The vector of (2) is subjected to point-to-point addition operation to obtain the characteristic fusion output based on the two modes as
Wherein g h,j For the characteristic representation of the jth medicinal material in the knowledge graph, n p Is the total number of medicinal materials in the prescription.
S303, finally fusing tongue picture with feature e t Prescription fusion feature e p The features after the two are recombined are subjected to standardization and connection operation (connection), and then are input into a three-layer perceptron, wherein the output dimension of each layer is 128/64/1, and the output of the network is [0,1 ] by using a Softmax activation function]Is a recommendation score of (1). And (5) arranging different prescriptions in a descending order according to the recommendation score, and finally obtaining the prescriptions recommended by the model for the patient.
In step S40, as shown in fig. 4, the neural network model is iteratively trained by using the tongue image prescription training set and the knowledge graph of tongue diagnosis of the traditional Chinese medicine, and tested by using the tongue image prescription test set, so as to obtain an optimal neural network model.
Firstly, training a model by using a tongue image-prescription pair sample in a tongue image prescription training set;
specifically, the model training process of the present embodiment is: the tongue image-prescription sample pair of the training set and the traditional Chinese medicine tongue diagnosis knowledge graph are input, the model outputs the recommended score of the prescription, the loss of the prescription and a real label is calculated, a counter-propagation algorithm is adopted, the weight of the parameters is adjusted, and the model is optimized in an iterative mode. In this embodiment, a random gradient descent optimization algorithm is adopted, the loss function is a cross entropy loss function, the initial learning rate is 0.1, the training batch size (batch size) is 64, the iteration round number is 200, the number of iterations is 50, 100 and 160 respectively, the number of iterations is reduced to 0.1, and the weight attenuation is 0.0001.
Secondly, inputting test samples in the tongue image prescription test set into a trained network model, and recommending corresponding prescriptions by the model.
Specifically, matching the tongue images of the patients in the preprocessed test set with prescriptions, and inputting a trained network model to obtain the predicted recommended score of the corresponding prescriptions. All prescriptions are orderly arranged from high to low according to the prediction scores, and the prescriptions corresponding to the first K items are recommended prescriptions of the patient; comparing the real prescriptions corresponding to the tongue images in the test set with the recommended prescriptions, and stopping iterative training to serve as an optimal model if the expectations are met.
In step S50, finally, an optimal neural network model generated by training is adopted to identify the tongue image to be processed, and a corresponding recommended prescription is obtained; the prescription with the highest recommended score is obtained.
Example 2:
based on the same inventive concept, referring to fig. 5, an embodiment of the present invention further provides a prescription recommendation system based on tongue images and knowledge maps, including:
a first construction module 51, configured to construct a tongue image prescription data set, divide a tongue image prescription training set and a test set, and perform preprocessing;
the second construction module 52 is configured to construct a knowledge graph of tongue diagnosis of traditional Chinese medicine and perform knowledge representation learning;
a third construction module 53, configured to construct a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
the training test module 54 performs iterative training and testing on the neural network model by using the tongue image prescription training set, the testing set and the traditional Chinese medicine tongue diagnosis knowledge graph to obtain an optimal neural network model;
the recommendation module 55 adopts an optimal neural network model to identify the tongue image to be processed, and obtains a corresponding recommendation prescription.
Further, the tongue image prescription data set in the first construction module 51 includes a plurality of tongue images and corresponding prescription information; the prescription information includes: medicinal materials and dosage information.
Dividing and preprocessing a tongue image prescription training set and a test set, wherein the method comprises the following steps:
dividing a tongue image prescription training set and a tongue image prescription test set according to a preset proportion;
preprocessing the tongue image prescription training set comprises random cutting, size scaling, random horizontal and vertical overturning and normalization operation;
preprocessing the tongue prescription test set is size scaling, centering clipping and normalizing operation.
Further, the second building module 52 specifically includes:
the medical resource database, the traditional Chinese medicine literature and the medical hundred degree encyclopedia are used as data sources of the tongue diagnosis knowledge graph of the traditional Chinese medicine;
adopting a top-down construction mode, extracting entity, relationship and attribute information from the data sources according to a top-level data mode of a design knowledge graph, and obtaining triple knowledge;
and storing through a graph database Neo4j to complete the construction of the tongue diagnosis knowledge graph of the traditional Chinese medicine and perform knowledge representation learning.
Further, the prescription feature module in the third building module 53 is configured to input prescription information, convert the prescription information into a unique heat vector, and reduce the dimension through the full connection layer to obtain a prescription feature representation in the text mode.
The processing procedure of the prescription recommendation module comprises the following steps:
the tongue image characteristics and the tongue image node characteristics in the traditional Chinese medicine tongue diagnosis knowledge graph are fused by adopting an addition operation, so that tongue image fusion characteristics are formed;
the prescription characteristic and the traditional Chinese medicine tongue diagnosis knowledge graph traditional Chinese medicine node characteristic are fused by adopting an adding operation to form a prescription fusion characteristic;
the tongue image fusion characteristics and the prescription fusion characteristics are subjected to standardized processing and spliced, the internal interaction between the tongue image fusion characteristics and the prescription fusion characteristics is input into a multi-layer perceptron to learn, and the recommendation score predicted by the model is output; and arranging different prescriptions in a descending order according to the recommended scores, and finally obtaining the recommended prescriptions and the corresponding recommended scores.
Further, the training test module 54, the training part includes:
training a neural network model by taking tongue images-prescription pair samples in the tongue image prescription training set and tongue image node characteristics and medicinal material node characteristics corresponding to the tongue diagnosis knowledge graph of the traditional Chinese medicine as training sample data; and adopting a cross entropy loss function, adopting a random gradient descent method by an optimization algorithm, and carrying out multiple iterations to optimally train the neural network model part.
Example 3:
based on the same inventive concept, the embodiment of the present invention further provides a computer device, as shown in fig. 6, including a processor 61, a communication interface 62, a memory 63, and a communication bus 64, where the processor 61, the communication interface 62, and the memory 63 complete communication with each other through the communication bus 64;
a memory 63 for storing a computer program;
the processor 61 is configured to execute the program stored in the memory 63, thereby realizing the recipe recommendation method based on the tongue image and the knowledge map as in embodiment 1.
Example 4:
the embodiment of the present invention further provides a storage medium, where at least one instruction is stored, where the instruction is loaded and executed by a processor to implement the prescription recommendation method based on tongue images and knowledge patterns as in embodiment 1.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A prescription recommendation method based on tongue images and knowledge maps is characterized by comprising the following steps:
constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set, and preprocessing;
constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning;
constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
performing iterative training on the neural network model by using the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph, and performing testing by using a tongue image prescription testing set to obtain an optimal neural network model;
and identifying the tongue image to be processed by adopting the optimal neural network model to obtain a corresponding recommended prescription.
2. The method for recommending prescriptions based on tongue images and knowledge patterns according to claim 1, wherein the tongue image prescription dataset comprises a plurality of tongue images and corresponding prescription information; the prescription information includes: medicinal materials and dosage information.
3. The tongue image and knowledge graph based prescription recommendation method as claimed in claim 1, wherein the tongue image prescription training set and the test set are divided and preprocessed, and the method comprises the following steps:
dividing a tongue image prescription training set and a tongue image prescription test set according to a preset proportion;
preprocessing the tongue image prescription training set comprises random cutting, size scaling, random horizontal and vertical overturning and normalization operation;
preprocessing the tongue prescription test set is size scaling, centering clipping and normalizing operation.
4. The prescription recommendation method based on tongue images and knowledge patterns according to claim 1, wherein the tongue diagnosis knowledge patterns of the traditional Chinese medicine are constructed and knowledge representation learning is performed; comprising the following steps:
the medical resource database, the traditional Chinese medicine literature and the medical hundred degree encyclopedia are used as data sources of the tongue diagnosis knowledge graph of the traditional Chinese medicine;
adopting a top-down construction mode, extracting entity, relationship and attribute information from the data sources according to a top-level data mode of a design knowledge graph, and obtaining triple knowledge;
and storing through a graph database Neo4j to complete the construction of the tongue diagnosis knowledge graph of the traditional Chinese medicine and perform knowledge representation learning.
5. The prescription recommendation method based on tongue images and knowledge patterns according to claim 1, wherein the prescription characteristic module is used for inputting prescription information, converting the prescription information into independent heat vectors, and obtaining prescription characteristic representation in a text mode through dimension reduction of a full-connection layer.
6. The method for recommending prescriptions based on tongue images and knowledge patterns according to claim 1, wherein the processing procedure of the prescription recommending module comprises:
the tongue image characteristics and the tongue image node characteristics in the traditional Chinese medicine tongue diagnosis knowledge graph are fused by adopting an addition operation, so that tongue image fusion characteristics are formed;
the prescription characteristic and the traditional Chinese medicine tongue diagnosis knowledge graph traditional Chinese medicine node characteristic are fused by adopting an adding operation to form a prescription fusion characteristic;
the tongue image fusion characteristics and the prescription fusion characteristics are subjected to standardized processing and spliced, the internal interaction between the tongue image fusion characteristics and the prescription fusion characteristics is input into a multi-layer perceptron to learn, and the recommendation score predicted by the model is output; and arranging different prescriptions in a descending order according to the recommended scores, and finally obtaining the recommended prescriptions and the corresponding recommended scores.
7. The tongue image and knowledge graph-based prescription recommendation method according to claim 1, wherein the neural network model is iteratively trained by using the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph; comprising the following steps:
training a neural network model by taking tongue images-prescription pair samples in the tongue image prescription training set and tongue image node characteristics and medicinal material node characteristics corresponding to the tongue diagnosis knowledge graph of the traditional Chinese medicine as training sample data;
and adopting a cross entropy loss function, adopting a random gradient descent method by an optimization algorithm, and carrying out multiple iterations to optimally train the neural network model part.
8. A prescription recommendation system based on tongue images and knowledge maps, comprising:
the first construction module is used for constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set and preprocessing the tongue image prescription training set and the testing set;
the second construction module is used for constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine and carrying out knowledge representation learning;
the third construction module is used for constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for extracting tongue image features and prescription features; and fusing the first multi-modal feature and the second multi-modal feature to obtain prescription recommendation and score; the first multi-modal feature is a fusion result of the tongue image feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph; the second multi-modal feature is a fusion result of the prescription feature and a corresponding node in the traditional Chinese medicine tongue diagnosis knowledge graph;
the training test module is used for carrying out iterative training and testing on the neural network model by utilizing the tongue image prescription training set, the testing set and the traditional Chinese medicine tongue diagnosis knowledge graph to obtain an optimal neural network model;
and the recommending module is used for identifying the tongue image to be processed by adopting an optimal neural network model to obtain a corresponding recommending prescription.
9. A computer device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
a memory for storing a computer program;
a processor, configured to execute a program stored in a memory, and implement the tongue image and knowledge graph-based prescription recommendation method according to any one of claims 1 to 7.
10. A storage medium having stored therein at least one instruction loaded and executed by a processor to implement the tongue and knowledge-graph based prescription recommendation method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310589649.0A CN116844687A (en) | 2023-05-24 | 2023-05-24 | Prescription recommendation method and system based on tongue images and knowledge patterns |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310589649.0A CN116844687A (en) | 2023-05-24 | 2023-05-24 | Prescription recommendation method and system based on tongue images and knowledge patterns |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116844687A true CN116844687A (en) | 2023-10-03 |
Family
ID=88167879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310589649.0A Pending CN116844687A (en) | 2023-05-24 | 2023-05-24 | Prescription recommendation method and system based on tongue images and knowledge patterns |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116844687A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117611581A (en) * | 2024-01-18 | 2024-02-27 | 之江实验室 | Tongue picture identification method and device based on multi-mode information and electronic equipment |
-
2023
- 2023-05-24 CN CN202310589649.0A patent/CN116844687A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117611581A (en) * | 2024-01-18 | 2024-02-27 | 之江实验室 | Tongue picture identification method and device based on multi-mode information and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108446730B (en) | CT pulmonary nodule detection device based on deep learning | |
| Tang et al. | CapSurv: Capsule network for survival analysis with whole slide pathological images | |
| CN110298383B (en) | Multi-mode deep learning-based pathology classification method and system | |
| Vu et al. | DFDL: Discriminative feature-oriented dictionary learning for histopathological image classification | |
| Guo et al. | Classification of thyroid ultrasound standard plane images using ResNet-18 networks | |
| CN113850753B (en) | Medical image information computing method, device, edge computing equipment and storage medium | |
| CN115410046A (en) | Skin disease tongue picture classification model based on deep learning, establishing method and application | |
| Muhaba et al. | Automatic skin disease diagnosis using deep learning from clinical image and patient information | |
| CN108877923A (en) | A method of the tongue fur based on deep learning generates prescriptions of traditional Chinese medicine | |
| CN116844687A (en) | Prescription recommendation method and system based on tongue images and knowledge patterns | |
| Liu et al. | Classification of breast cancer histology images using MSMV-PFENet | |
| Xiang et al. | A novel weight pruning strategy for light weight neural networks with application to the diagnosis of skin disease | |
| Wang et al. | A cell phone app for facial acne severity assessment | |
| Bedeir et al. | Automated multi-class skin cancer classification through concatenated deep learning models | |
| Junayed et al. | A transformer-based versatile network for acne vulgaris segmentation | |
| Dhanagopal et al. | An efficient retinal segmentation-based deep learning framework for disease prediction | |
| CN116797817A (en) | Autism disease prediction technology based on self-supervision graph convolution model | |
| Zhang et al. | Hepatocellular carcinoma histopathological images grading with a novel attention-sharing hybrid network based on multi-feature fusion | |
| CN110992309B (en) | Fundus image segmentation method based on deep information transfer network | |
| Wang et al. | Diagnosis of cognitive and motor disorders levels in stroke patients through explainable machine learning based on MRI | |
| CN112990339A (en) | Method and device for classifying stomach pathological section images and storage medium | |
| CN113077894A (en) | System, method, apparatus and medium for skin diagnosis based on graph convolution neural network | |
| Wu et al. | PCAF‐Net: A liver segmentation network based on deep learning | |
| Su et al. | Whole slide cervical image classification based on convolutional neural network and random forest | |
| Rani et al. | Construction of Deep Learning Model using RESNET 50 for Schizophrenia Prediction from rsFMRI Images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |