CN116386862A - Multi-modal cognitive impairment evaluation method, device, equipment and storage medium - Google Patents
Multi-modal cognitive impairment evaluation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116386862A CN116386862A CN202310143687.3A CN202310143687A CN116386862A CN 116386862 A CN116386862 A CN 116386862A CN 202310143687 A CN202310143687 A CN 202310143687A CN 116386862 A CN116386862 A CN 116386862A
- Authority
- CN
- China
- Prior art keywords
- features
- cognitive
- prediction
- fusion
- test data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 208000010877 cognitive disease Diseases 0.000 title claims abstract description 158
- 208000028698 Cognitive impairment Diseases 0.000 title claims abstract description 125
- 238000011156 evaluation Methods 0.000 title claims abstract description 47
- 230000004927 fusion Effects 0.000 claims abstract description 127
- 238000012360 testing method Methods 0.000 claims abstract description 123
- 230000001149 cognitive effect Effects 0.000 claims abstract description 109
- 238000000034 method Methods 0.000 claims abstract description 67
- 238000000605 extraction Methods 0.000 claims abstract description 35
- 238000013210 evaluation model Methods 0.000 claims abstract description 32
- 230000003920 cognitive function Effects 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000012911 target assessment Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 12
- 238000013473 artificial intelligence Methods 0.000 abstract description 8
- 238000003745 diagnosis Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 17
- 230000019771 cognition Effects 0.000 description 9
- 230000001771 impaired effect Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 201000007201 aphasia Diseases 0.000 description 2
- 230000003542 behavioural effect Effects 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 230000008451 emotion Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 206010012289 Dementia Diseases 0.000 description 1
- 241001112258 Moca Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003930 cognitive ability Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007636 ensemble learning method Methods 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 208000011977 language disease Diseases 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 230000031864 metaphase Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012030 stroop test Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/40—Detecting, measuring or recording for evaluating the nervous system
- A61B5/4076—Diagnosing or monitoring particular conditions of the nervous system
- A61B5/4088—Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Neurology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Hospice & Palliative Care (AREA)
- General Physics & Mathematics (AREA)
- Developmental Disabilities (AREA)
- Psychiatry (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Human Resources & Organizations (AREA)
- Heart & Thoracic Surgery (AREA)
- Veterinary Medicine (AREA)
- Child & Adolescent Psychology (AREA)
- Economics (AREA)
- Data Mining & Analysis (AREA)
- Strategic Management (AREA)
- Psychology (AREA)
- Educational Technology (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Primary Health Care (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Epidemiology (AREA)
- Databases & Information Systems (AREA)
- Development Economics (AREA)
Abstract
The application relates to an artificial intelligence technology and provides a multi-modal cognitive impairment assessment method, device, equipment and storage medium, wherein the method comprises the following steps: acquiring multi-mode test data of a target object in a cognitive function test; and carrying out feature extraction on each test data by utilizing the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, carrying out first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, carrying out cognitive impairment prediction on the target object according to different prediction features to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target evaluation result. According to the method, the object to be evaluated is comprehensively analyzed and comprehensively evaluated from the multidimensional characteristics, the accuracy and the reliability of the prediction and the diagnosis of the cognitive disorder are improved, and the method is widely applied to digital medical treatment.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a method, an apparatus, a device, and a storage medium for evaluating a multi-modal cognitive impairment.
Background
In the medical field, conventional cognitive impairment assessment is generally to perform one-to-one paper quality table or digital online assessment between an evaluator and a subject according to different cognitive fields, such as logic memory capacity, language function, computing capacity, vision space function, executive function, and the like. However, the current cognitive evaluation process requires professional testing and evaluation personnel, is time-consuming and labor-consuming, is not suitable for large-scale popularization and application, and has subjectivity and inconsistency based on manual evaluation and scoring results, so that the cognitive function evaluation efficiency and accuracy are not high.
In the prior art, cognitive impairment is predicted by an artificial intelligence technology, but in the prior art, the evaluation results are inaccurate due to single evaluation of the expression, the movement data and the like of the person.
Disclosure of Invention
The method aims at solving the technical problems of low cognitive impairment evaluation efficiency and low accuracy in the prior art. The application provides a multi-modal cognitive disorder assessment method, device, equipment and storage medium, which mainly aim to realize comprehensive analysis and comprehensive judgment of behavior characteristics of an object to be assessed and improve accuracy and reliability of cognitive disorder prediction and diagnosis.
To achieve the above object, the present application provides a method for evaluating a multimodal cognitive disorder, the method comprising:
acquiring multi-mode test data of a target object in a cognitive function test, wherein the multi-mode test data comprises a first number of test data in different modes, and the first number is not less than 2;
inputting multi-mode test data into a trained cognitive impairment evaluation model, carrying out feature extraction on each test data by utilizing the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, carrying out first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, carrying out cognitive impairment prediction on a target object according to different prediction features to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target evaluation result, wherein the prediction features comprise a fifth number of first features and all fusion features, each fusion feature is obtained by fusing at least two different first features, the second number is not less than the first number, and the fifth number is not more than the second number.
In addition, in order to achieve the above object, the present application further provides an evaluation device for multi-modal cognitive impairment, the device comprising:
the first acquisition module is used for acquiring multi-mode test data of the target object in the cognitive function test, wherein the multi-mode test data comprise test data of a first number of different modes, and the first number is not less than 2;
the prediction module is used for inputting the multi-mode test data into the trained cognitive impairment evaluation model, extracting the characteristics of each test data by utilizing the trained cognitive impairment evaluation model to obtain a second number of first characteristics with different dimensions, carrying out first fusion on the second number of first characteristics with different dimensions according to a preset fusion rule to obtain a third number of fusion characteristics, carrying out cognitive impairment prediction on the target object according to different prediction characteristics to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target evaluation result, wherein the prediction characteristics comprise a fifth number of first characteristics and all fusion characteristics, each fusion characteristic is obtained by fusing at least two different first characteristics, the second number is not less than the first number, and the fifth number is not more than the second number.
To achieve the above object, the present application further provides a computer device comprising a memory, a processor and computer readable instructions stored on the memory and executable on the processor, the processor executing the steps of the method for evaluating multimodal cognitive impairment as in any one of the preceding claims.
To achieve the above object, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which when executed by a processor, cause the processor to perform the steps of the method for evaluating multimodal cognitive impairment as in any one of the preceding claims.
According to the multi-mode cognitive disorder assessment method, device, equipment and storage medium, feature extraction is carried out on test data of different modes, feature fusion is carried out on the extracted first features through different fusion modes, cognitive disorder prediction is carried out on the extracted first features according to the obtained fusion features and part of the first features or all the first features respectively to obtain a corresponding number of cognitive prediction results, finally all the cognitive prediction results are fused to obtain a final target assessment result, and the cognitive disorder assessment is carried out on the cognitive disorder assessment through multi-dimensional features and multi-fusion features, so that comprehensive analysis and comprehensive assessment on the behavior features of an object to be assessed are achieved, the defect that the prior art can only carry out disease prediction from single features is overcome, accuracy and reliability of cognitive disorder prediction and diagnosis are improved, intelligence of cognitive disorder assessment is achieved, and assessment efficiency is improved.
Drawings
FIG. 1 is a flow chart of a method for evaluating multimodal cognitive impairment according to an embodiment of the present application;
FIG. 2 is a block diagram of a multi-modal cognitive impairment assessment device according to one embodiment of the present application;
fig. 3 is a block diagram showing an internal structure of a computer device according to an embodiment of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present application based on the embodiments herein. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
Fig. 1 is a flow chart of a method for evaluating a multi-modal cognitive impairment according to an embodiment of the present application, and referring to fig. 1, the method for evaluating a multi-modal cognitive impairment includes the following steps S100-S200.
S100: the method comprises the steps of obtaining multi-mode test data of a target object in cognitive function test, wherein the multi-mode test data comprise test data of a first number of different modes, and the first number is not less than 2.
Specifically, cognition impaired patients reflect a variety of behavioral aspects such as slow-acting, slow-reacting, slow-acting, loss of recognition, naming difficulties, impaired spoken and written expression, fluent but empty speech and semantic aphasia, etc. Thus, the present application makes assessment or prediction of cognitive impairment through test data of various modalities.
The test data of different modalities specifically refer to different types of data, for example, video data, voice data, text data, and the like, without being limited thereto. The test data of different modes in this embodiment includes test data of at least two different modes.
S200: inputting multi-mode test data into a trained cognitive impairment evaluation model, carrying out feature extraction on each test data by utilizing the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, carrying out first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, carrying out cognitive impairment prediction on a target object according to different prediction features to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target evaluation result, wherein the prediction features comprise a fifth number of first features and all fusion features, each fusion feature is obtained by fusing at least two different first features, the second number is not less than the first number, and the fifth number is not more than the second number.
In particular, the trained cognitive impairment assessment model may perform a plurality of feature extraction on the input multimodal test data, wherein each test data may extract a first feature of at least one dimension, such that the second number is not less than the first number.
The first fusion is specifically fusion among the features, and a preset fusion rule is used for specifying a fusion mode, namely, which first features are subjected to the first fusion. The fusion rule may specify that the first features are fused two by two or that some three first features are fused or that some four first features are fused, or that some first features are fused two by two, some first features are fused once every three, some first features are fused once every four, etc. The fusion rules may also specify that some first features may participate in multiple different fusions and some first features may not participate in any one fusion. The present application is not limited to this, particularly according to the practical application configuration. The third number of fused features may be equal to the second number of first features, or the third number may be greater than the second number, or the third number may be less than the second number.
The trained cognitive impairment evaluation model can conduct cognitive impairment prediction on the target object according to the fusion characteristics, and can conduct cognitive impairment prediction on the target object according to the first characteristics. The features on which each cognitive disorder prediction is based are different, and thus the cognitive prediction results obtained are not necessarily the same. The features used for cognitive disorder prediction are prediction features, wherein the prediction features comprise fusion features and part or all of first features, and one prediction feature corresponds to one cognitive prediction result. Wherein the fourth number is equal to the sum of the third number and the fifth number. The first features selected as the prediction features are important features which can reflect the cognitive level in all the first features, and particularly the selection of the prediction features is configured according to the actual application scene.
The second fusion is specifically the fusion of cognitive prediction results, and the trained cognitive impairment assessment model is also used for fusing all cognitive prediction results into target assessment results.
According to the method, feature extraction is carried out on test data of different modes, feature fusion is carried out on the extracted first features through different fusion modes, cognitive disorder prediction is carried out on the extracted first features according to the obtained fusion features and part of or all of the first features, a corresponding number of cognitive disorder prediction results are obtained, finally all the cognitive prediction results are fused to obtain a final target evaluation result, and the cognitive disorder is evaluated through multidimensional features and multi-fusion features, so that comprehensive analysis and comprehensive evaluation on the behavior features of an object to be evaluated are realized, the defect that the prior art can only carry out disease prediction from a single feature is overcome, the accuracy and reliability of cognitive disorder prediction and diagnosis are improved, the intelligence of cognitive disorder evaluation is realized, and the evaluation efficiency is improved.
In one embodiment, the trained cognitive impairment assessment model comprises a decision module, a second number of feature extraction networks, a third number of feature fusion networks, and a fourth number of prediction networks;
The step S200 specifically includes:
respectively inputting a first number of test data into corresponding feature extraction networks, and extracting features of one dimension in the input test data by utilizing each feature extraction network to obtain first features of the corresponding dimension, wherein each feature extraction network inputs one test data;
inputting the obtained second number of first features into corresponding feature fusion networks respectively, and fusing at least two different first features which are input by utilizing each feature fusion network once to obtain corresponding fusion features;
respectively inputting the obtained third number of fusion features and the fifth number of first features serving as prediction features into corresponding prediction networks, and performing cognitive impairment prediction on a target object by utilizing each prediction network according to the input prediction features to obtain corresponding cognitive prediction results, wherein each prediction network inputs one prediction feature;
and taking all the cognitive prediction results as the input of the decision module, and performing second fusion on all the cognitive prediction results by using the decision module to obtain a target evaluation result.
Specifically, the trained cognitive impairment assessment model is provided with a second number of feature extraction networks, each of which has an input of one type of test data for extracting features of one dimension of one type of test data. The same test data may be input to a plurality of feature extraction networks for extracting first features of different dimensions.
The trained cognitive impairment evaluation model is provided with a third number of feature fusion networks, each feature fusion network being connected to a corresponding feature extraction network, some feature extraction networks being possibly connected to a plurality of different feature fusion networks, and some feature fusion networks being possibly connected to different feature extraction networks. The number and types of the first features input by each feature fusion network are not identical, some feature fusion networks may input two first features, some feature fusion networks may input three first features, some feature fusion networks may input four first features, etc. are not limited thereto.
The trained cognitive impairment evaluation model is further provided with a fourth number of prediction networks, wherein one prediction network is connected with one feature extraction network or one feature fusion network and is used for performing cognitive impairment prediction according to input prediction features. Each prediction network inputs a prediction feature, and performs a cognitive impairment prediction to obtain a cognitive prediction result.
The trained cognitive impairment evaluation model is provided with a decision module, and cognitive prediction results of all prediction networks are input into the decision module. The decision module can be a decision network constructed based on a neural network or a non-neural network.
If the decision module is a decision network constructed based on a neural network, the decision module learns the prediction capability of each prediction network, and specifically learns how to fuse the cognitive prediction results of all prediction networks to obtain accurate results close to reality. More specifically, the decision network learns the weights of each prediction network on the prediction results. The more accurate the cognitive prediction result, the higher the weight is given to the prediction network.
According to the method, the device and the system, the characteristic extraction network is utilized to extract the characteristics, the characteristic fusion network is utilized to carry out characteristic fusion, the prediction network is utilized to carry out cognition prediction, and the decision module is utilized to fuse the cognition prediction result to obtain a final prediction result.
In one embodiment, each cognitive prediction result includes a probability of each class of cognitive level classification;
and performing second fusion on all the cognitive prediction results by using the decision module to obtain target evaluation results, wherein the second fusion comprises the following steps:
Calculating the average value of the probabilities of the same cognitive level classification in all the cognitive prediction results by utilizing a decision module, and taking the cognitive level classification with the highest average value as a target evaluation result;
or alternatively, the first and second heat exchangers may be,
calculating probability sums of probabilities of the same cognitive level classification in all cognitive prediction results by utilizing a decision module, and taking the probability sum of the probabilities and the highest cognitive level classification as a target evaluation result;
or alternatively, the first and second heat exchangers may be,
and determining the cognitive level classification with the highest probability in each cognitive prediction result as the obtained vote classification by using a decision module by adopting a minority-subject majority voting mechanism, counting the same obtained vote classification, and taking the cognitive level classification corresponding to the obtained vote classification with the highest counting times as a target evaluation result.
Specifically, the cognitive level classification specifically includes normal populations and cognition impaired patients. Alternatively, the cognitive level classification specifically includes patients with advanced cognitive impairment, patients with intermediate cognitive impairment, patients with early cognitive impairment, normal populations. Different cognitive level classifications may be refined according to different application scenarios, to which the present application is not limited.
The cognitive prediction result obtained by each prediction module comprises the probability of each class of cognitive level classification, for example, if the cognitive level classification is two classes, the cognitive level classification comprises the first probability of a normal population and the second probability of a cognitive disorder patient, and the sum of the first probability and the second probability is 1. If the classification is multi-classification, the first probability of the patients with advanced cognitive impairment, the second probability of the patients with medium cognitive impairment, the third probability of the patients with early cognitive impairment and the fourth probability of the normal population are included, and the sum of the first probability, the second probability, the third probability and the fourth probability is 1.
The manner of averaging is specifically, for example: the first prediction network predicts that the probability of being a normal crowd is 0.3 and the probability of being a cognitive disorder patient is 0.7; the probability of the second prediction network predicting the normal population is 0.8, and the probability of the second prediction network predicting the cognitive disorder patient is 0.2; the third prediction network predicts that the probability of being a normal population is 0.5 and the probability of being a cognitive disorder patient is 0.5; the fourth predictive network predicts a probability of 0.6 for a normal population and 0.4 for a cognition impaired patient.
The average of all probabilities for normal population is calculated: (0.3+0.8+0.5+0.6)/4=0.55, all mean values of probabilities for cognition impaired patients: (0.7+0.2+0.5+0.4)/4=0.45. 0.55 is greater than 0.45, and therefore, the normal population is judged as the target evaluation result.
Taking the above example as an example, the way of calculating the probability sum is specifically, for example: the probability sum of all the probabilities of normal population is calculated: 0.3+0.8+0.5+0.6=2.2, the probability sum of the cognitive impairment patient probabilities is calculated: 0.7+0.2+0.5+0.4=1.8. 2.2 is greater than 1.8, and therefore, normal population is judged as the target evaluation result.
For another example, there are 4 prediction networks and the first prediction network predicts a probability of 0.1 for patients with advanced cognitive impairment, 0.2 for patients with intermediate cognitive impairment, 0.4 for patients with early cognitive impairment, and 0.3 for normal populations, and is multi-classified.
The second prediction network predicts a probability of 0.2 for patients with advanced cognitive impairment, a probability of 0.2 for patients with intermediate cognitive impairment, a probability of 0.3 for patients with early cognitive impairment, and a probability of 0.3 for normal population.
The third prediction network predicts a probability of 0.3 for patients with advanced cognitive impairment, a probability of 0.2 for patients with intermediate cognitive impairment, a probability of 0.1 for patients with early cognitive impairment, and a probability of 0.4 for normal population.
The fourth prediction network predicts a probability of 0.4 for patients with advanced cognitive impairment, a probability of 0.2 for patients with intermediate cognitive impairment, a probability of 0.1 for patients with early cognitive impairment, and a probability of 0.3 for normal population.
The mean value is calculated:
the average of all probabilities for patients with advanced cognitive impairment is calculated: (0.1+0.2+0.3+0.4)/4=0.25 the average of the probabilities for all patients with metaphase cognitive impairment is calculated: (0.2+0.2+0.2+0.2)/4=0.2
The average of all probabilities for early cognitive impairment patients was calculated: (0.4+0.3+0.1+0.1)/4=0.225
The average of all probabilities for normal population is calculated: (0.3+0.3+0.4+0.3)/4=0.325.
0.325 is the maximum average value, and thus the normal population is determined as the target evaluation result.
Voting mechanism:
taking the above two classification examples as an example, 0.7 is greater than 0.3 according to a minority compliance majority rule, so the first predictive network votes for cognition impaired patients; 0.8 is greater than 0.2, so the second predictive network votes for the normal population; 0.6 is greater than 0.4, so the fourth predictive network votes for the normal population. 0.5 is equal to 0.5, so the third prediction network does not vote. In conclusion, the normal crowd is 2 tickets, and the cognitive disorder patient is 1 ticket, so that the normal crowd is judged to be the target evaluation result.
The multi-class example is similar to the two-class example and will not be described again here.
The average value calculation or voting mechanism is to fuse the prediction results of different networks or models by using an integrated learning mode in machine learning to obtain a final prediction result.
The embodiment realizes the fusion of the prediction results of a plurality of models or networks by using an ensemble learning method.
In one embodiment, each cognitive prediction result includes a probability of each class of cognitive level classification;
and performing second fusion on all the cognitive prediction results by using the decision module to obtain target evaluation results, wherein the second fusion comprises the following steps:
and respectively carrying out weight calculation on the probabilities of the same class of cognitive level classification in all the cognitive prediction results by utilizing a decision module to obtain the fusion probability of each class of cognitive level classification, and taking the cognitive level classification with the highest fusion probability as a target evaluation result.
Specifically, the decision module in this embodiment is a decision network constructed based on a neural network.
For example: the cognitive impairment prediction system comprises 4 prediction networks, wherein the prediction networks are classified into two types, the probability of the first prediction network predicting that the cognitive impairment prediction system is a normal crowd is 0.3, the probability of the first prediction network predicting that the cognitive impairment prediction system is a cognitive impairment patient is 0.7, and the decision module distributes a weight of 0.1 to the first prediction network through learning; the probability of the second prediction network predicting the cognitive impairment patient as a normal crowd is 0.8, the probability of the second prediction network predicting the cognitive impairment patient as a cognitive impairment patient is 0.2, and the decision module assigns the weight of the second prediction network as 0.2 through learning; the third prediction network predicts that the probability of the normal crowd is 0.5 and the probability of the cognitive impairment patient is 0.5, and the decision module learns that the weight assigned to the third prediction network is 0.3; the fourth prediction network predicts that the probability of being a normal crowd is 0.6 and the probability of being a cognitive impairment patient is 0.4, and the decision module assigns a weight of 0.4 to the fourth prediction network through learning.
The probability of normal crowd is weighted, and the obtained fusion probability is as follows:
0.3*0.1+0.8*0.2+0.5*0.3+0.6*0.4＝0.58
the probability of the cognitive disorder patient is subjected to weight calculation, and the obtained fusion probability is as follows:
0.7*0.1+0.2*0.2+0.5*0.3+0.4*0.4＝0.42
and 0.58 is greater than 0.42, and thus normal population is judged as the target evaluation result.
The multi-class example is similar to the two-class example and will not be described again here.
In the embodiment, the prediction results of all the prediction networks are fused by learning the prediction capability of each prediction network through the decision module based on the neural network model. The accuracy and reliability of cognitive disorder disease prediction are improved.
In one embodiment, the multimodal test data includes test data of at least two modalities of voice data, drawing data, video data, and text writing data;
if the multimodal test data comprises voice data, the first features extracted by the trained cognitive impairment evaluation model comprise at least one of voice features and text features, wherein the text features are extracted from voice recognition texts corresponding to the voice data;
if the multimodal test data includes drawing data, the first features extracted using the trained cognitive impairment assessment model include drawing features;
if the multimodal test data includes video data, the first features extracted using the trained cognitive impairment assessment model include video features;
if the multimodal test data includes written text data, the first feature extracted using the trained cognitive impairment assessment model includes a written feature.
Specifically, at least two of videos, voices, drawings, written characters and the like of a target object under various test items in the cognitive function evaluation test are collected to form multi-mode test data. In the cognitive function test, the evaluation is usually carried out according to different cognitive fields. The cognitive function assessment test may specifically be performed according to test items in the medical community general brief mental state scale (MMSE scale) and/or the montreal cognitive assessment scale (MoCA scale). The brief mental condition scale and the Montreal cognitive assessment scale are the most common cognitive function screening scales applied clinically at present, and are mainly used for primary screening of various types of cognitive disorders and dementia. The test items in the cognitive function assessment test of the present embodiment may include one or more of a logical memory capability test, a language function test, a computing capability test, a visual space capability test, and an executive capability test and an attention capability test.
The memory capacity adopts an auditory word learning test or a logic memory test, the language capacity adopts a Boston naming test or a speech fluency test, the attention capacity adopts a digital breadth test or a digital symbol conversion test, the viewing space capacity adopts a line direction judging test or a complex picture imitating test, and the execution capacity adopts a line connecting test or a Stroop word test and the like. In the test process, a microphone and a camera are adopted to collect voice data and video data of the target object, and pen drawing patterns, characters and the like of the target object can be uploaded.
Preferably, the voice data and the video data may be preprocessed. The preprocessing includes, but is not limited to, at least one of speech noise reduction processing, signal enhancement processing, volume normalization processing, and speech endpoint detection processing.
In one embodiment, the speech features include at least one of mel-frequency cepstral coefficients, fundamental frequencies, formants, fundamental frequency perturbations, amplitude perturbations;
the text features comprise at least one of TF-IDF, word frequency, sentence logic features and sentence pauses;
the drawing features comprise at least one of pattern features and drawing scores;
the video features comprise at least one of micro-expression features, motion delay features, drawing speed and writing speed;
The writing characteristics include at least one of handwriting characteristics, writing accuracy.
Specifically, the normal population and the cognition impaired patient differ in various behavioral activities. TF-IDF is known as term frequency-inverse document frequency, chinese is word frequency-reverse document frequency, and is used to evaluate the importance of a word to one of a document set or a corpus. TF is the Term Frequency (Term Frequency) and IDF is the inverse text Frequency index (Inverse Document Frequency). The speaking content or meaning of the normal crowd is logically clear and orderly, emotion is stable, intention is obvious, speaking sentences are logically compliant, and in the cognitive function test, the spoken words are fluent, the pauses are reasonable, the emotion is carried, the existing barriers are fewer, and repeated words are fewer. While cognitive impairment patients can suffer from various language disorders such as slow response, unreasonable long-term pauses, aphasia, repeated utterances, etc. caused by impaired understanding ability to different degrees.
The normal crowd is normal in drawing and writing ability, and the drawing pattern and written characters are also higher in accuracy, and the drawing pattern and written characters of cognitive impairment patient can appear incomplete, serious defect, serious error scheduling problem.
The microexpressions of normal people are sensitive and react normally, while the microexpressions of cognition disorder patients are slow, slow and strange, so that the movement delay is larger. Drawing speed and writing speed can also be affected, but slow and sluggish, and unfavorable and smooth.
In one embodiment, prior to step S200, the method further comprises:
acquiring multi-mode sample test data of different subjects to form a sample set, wherein the multi-mode sample test data is marked with a real cognitive level label of the corresponding subject;
using a strategy of back propagation minimization loss function, performing iterative update training on a decision module, a feature extraction network, a feature fusion network and a prediction network in a cognitive impairment evaluation model to be trained by using a sample set to obtain a trained cognitive impairment evaluation model,
or alternatively, the first and second heat exchangers may be,
and performing iterative update training on a feature extraction network, a feature fusion network and a prediction network in the cognitive impairment evaluation model to be trained by using a strategy of back propagation minimization loss function by using a sample set to obtain a trained cognitive impairment evaluation model.
Specifically, a sample set is obtained, wherein the sample set comprises multi-modal sample test data marked with a plurality of different cognitive level labels, and each group of multi-modal sample test data is marked with a cognitive level label of a corresponding subject.
If the decision module is not a decision network constructed based on a neural network, the model parameter iteration update of the decision module is not needed in the model training process. If the decision module is a decision network constructed based on a neural network, the decision module is updated in a parameter iteration mode in the model training process.
If the decision module is a decision network constructed based on a neural network and the decision network is trained in advance, the weight set by the decision network for the prediction capacity of each prediction network is a preset value, and model parameter iterative update of the decision module is not needed in the model training process. The preset values corresponding to all the prediction networks can be equal to each other and are 1/fourth number. The weight is an assessment of the predictive ability of the predictive network by the decision module.
The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.
Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.
According to the method, through collecting the feedback results of the voice, the picture, the video and the test items of the subject, the input information of multiple modes is used to obtain the multidimensional characteristics, the cognitive ability of the cognitive patient is evaluated based on the multi-mode data, and the integrated learning is adopted to fuse the prediction results of the multiple models, so that the final prediction accuracy is improved. The multi-modal cognitive disorder assessment scheme can be applied to the field of digital medical treatment, and can be used for accurately diagnosing and predicting cognitive disorder symptoms.
Fig. 2 is a block diagram of a multi-modal cognitive impairment evaluation apparatus according to an embodiment of the present application. Referring to fig. 3, the multi-modal cognitive impairment assessment apparatus includes:
the first obtaining module 100 is configured to obtain multi-mode test data of the target object in a cognitive function test, where the multi-mode test data includes a first number of test data of different modes, and the first number is not less than 2;
The prediction module 200 is configured to input multi-modal test data into a trained cognitive impairment evaluation model, perform feature extraction on each test data by using the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, perform first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, perform cognitive impairment prediction on a target object according to different prediction features to obtain a fourth number of cognitive prediction results, and perform second fusion on all obtained cognitive prediction results to obtain a target evaluation result, where the prediction features include a fifth number of first features and all fusion features, each fusion feature is obtained by fusing at least two different first features, the second number is not less than the first number, and the fifth number is not greater than the second number.
In one embodiment, the trained cognitive impairment assessment model comprises a decision module, a second number of feature extraction networks, a third number of feature fusion networks, and a fourth number of prediction networks;
the prediction module 200 specifically includes:
the feature extraction module is used for respectively inputting the first quantity of test data into the corresponding feature extraction networks, extracting the feature of one dimension in the input test data by utilizing each feature extraction network to obtain the first feature of the corresponding dimension, wherein each feature extraction network inputs one test data;
The first fusion module is used for inputting the obtained second number of first features into corresponding feature fusion networks respectively, and carrying out primary fusion on at least two different first features input by utilizing each feature fusion network to obtain corresponding fusion features;
the sub-prediction module is used for respectively inputting the obtained third number of fusion features and the fifth number of first features serving as prediction features into corresponding prediction networks, and performing cognitive impairment prediction on the target object by utilizing each prediction network according to the input prediction features to obtain a corresponding cognitive prediction result, wherein each prediction network inputs one prediction feature;
and the second fusion module is used for taking all the cognitive prediction results as the input of the decision module, and performing second fusion on all the cognitive prediction results by using the decision module to obtain a target evaluation result.
In one embodiment, each cognitive prediction result includes a probability of each class of cognitive level classification;
the second fusion module specifically comprises:
the first fusion unit is used for calculating the average value of the probabilities of the same cognitive level classification in all the cognitive prediction results by utilizing the decision module, and taking the cognitive level classification with the highest average value as a target evaluation result;
Or alternatively, the first and second heat exchangers may be,
the second fusion unit is used for calculating the probability sum of the probabilities of the same cognitive level classification in all the cognitive prediction results by utilizing the decision module, and taking the probability sum of the probabilities of the same cognitive level classification as a target evaluation result;
or alternatively, the first and second heat exchangers may be,
and the third fusion unit is used for determining the cognitive level classification with the highest probability in each cognitive prediction result as the obtained vote classification by using a decision module by adopting a minority voting mechanism obeying majority, counting the same obtained vote classification, and taking the cognitive level classification corresponding to the obtained vote classification with the highest counting times as the target evaluation result.
In one embodiment, each cognitive prediction result includes a probability of each class of cognitive level classification;
the second fusion module specifically comprises:
and the fourth fusion unit is used for respectively carrying out weight calculation on the probabilities of the same class of cognitive level classification in all the cognitive prediction results by utilizing the decision module to obtain fusion probability of each class of cognitive level classification, and taking the cognitive level classification with the highest fusion probability as a target evaluation result.
In one embodiment, the multi-modal test data includes test data of at least two modalities of speech data, drawing data, video data, text writing data, and test item feedback results;
If the multimodal test data comprises voice data, the first features extracted by the trained cognitive impairment evaluation model comprise at least one of voice features and text features, wherein the text features are extracted from voice recognition texts corresponding to the voice data;
if the multimodal test data includes drawing data, the first features extracted using the trained cognitive impairment assessment model include drawing features;
if the multimodal test data includes video data, the first features extracted using the trained cognitive impairment assessment model include video features;
if the multimodal test data includes written text data, the first features extracted using the trained cognitive impairment assessment model include written features;
if the multimodal test data includes test item feedback results, the first features extracted using the trained cognitive impairment assessment model include feedback results for each test item.
In one embodiment, the speech features include at least one of mel-frequency cepstral coefficients, fundamental frequencies, formants, fundamental frequency perturbations, amplitude perturbations;
the text features comprise at least one of TF-IDF, word frequency, sentence logic features and sentence pauses;
The drawing features comprise at least one of pattern features and drawing scores;
the video features comprise at least one of micro-expression features, motion delay features, drawing speed and writing speed;
the writing characteristics include at least one of handwriting characteristics, writing accuracy.
In one embodiment, the apparatus further comprises:
the sample acquisition module is used for acquiring multi-mode sample test data of different subjects to form a sample set, wherein the multi-mode sample test data is marked with a real cognition level label corresponding to the subjects;
a first model training module for performing iterative update training on a decision module, a feature extraction network, a feature fusion network and a prediction network in a cognitive impairment evaluation model to be trained by using a strategy of back propagation minimization loss function by using a sample set to obtain a trained cognitive impairment evaluation model,
or alternatively, the first and second heat exchangers may be,
and the second model training module is used for carrying out iterative update training on the feature extraction network, the feature fusion network and the prediction network in the cognitive impairment evaluation model to be trained by using a strategy of back propagation minimization loss function and utilizing a sample set to obtain a trained cognitive impairment evaluation model.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way.
The meaning of "first" and "second" in the above modules/units is merely to distinguish different modules/units, and is not used to limit which module/unit has higher priority or other limiting meaning. Furthermore, the terms "comprises," "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules that are expressly listed or inherent to such process, method, article, or apparatus, but may include other steps or modules that may not be expressly listed or inherent to such process, method, article, or apparatus, and the partitioning of such modules by means of such elements is only a logical partitioning and may be implemented in a practical application.
For specific limitations on the device for assessing multimodal cognitive impairment, reference may be made to the above limitations on the method for assessing multimodal cognitive impairment, which are not described in detail herein. The above-mentioned various modules in the multi-modal cognitive impairment assessment device may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Fig. 3 is a block diagram showing an internal structure of a computer device according to an embodiment of the present application. As shown in fig. 3, the computer device includes a processor, a memory, a network interface, an input device, and a display screen connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory includes a storage medium and an internal memory. The storage medium may be a nonvolatile storage medium or a volatile storage medium. The storage medium stores an operating system and may further store computer readable instructions that, when executed by the processor, cause the processor to implement a method of evaluating multimodal cognitive impairment. The internal memory provides an environment for the execution of an operating system and computer-readable instructions in the storage medium. The internal memory may also have stored therein computer readable instructions that, when executed by the processor, cause the processor to perform a method for evaluating a multimodal cognitive impairment. The network interface of the computer device is for communicating with an external server via a network connection. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
In one embodiment, a computer device is provided that includes a memory, a processor, and computer readable instructions (e.g., a computer program) stored on the memory and executable on the processor, which when executed by the processor implement the steps of the method for assessing multimodal cognitive impairment of the above embodiments, such as steps S100 through S200 shown in FIG. 1 and other extensions of the method and related steps. Alternatively, the processor executes computer readable instructions to implement the functions of the modules/units of the multi-modal cognitive impairment assessment apparatus of the above embodiments, such as the functions of modules 100-200 shown in fig. 2. In order to avoid repetition, a description thereof is omitted.
The processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being a control center of the computer device, and the various interfaces and lines connecting the various parts of the overall computer device.
The memory may be used to store computer-readable instructions and/or modules that, by being executed or executed by the processor, implement various functions of the computer device by invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, video data, etc.) created according to the use of the cellular phone, etc.
The memory may be integrated with the processor or may be separate from the processor.
It will be appreciated by those skilled in the art that the structure shown in fig. 3 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer readable storage medium is provided, on which computer readable instructions are stored, which when executed by a processor, implement the steps of the method for assessing multimodal cognitive impairment of the above embodiments, such as steps S100 to S200 shown in FIG. 1 and other extensions of the method and related steps. Alternatively, the computer readable instructions, when executed by the processor, implement the functions of the modules/units of the multi-modal cognitive impairment assessment apparatus of the above embodiments, such as the functions of modules 100-200 shown in fig. 2. In order to avoid repetition, a description thereof is omitted.
Those of ordinary skill in the art will appreciate that implementing all or part of the processes of the above described embodiments may be accomplished by instructing the associated hardware by way of computer readable instructions stored in a computer readable storage medium, which when executed, may comprise processes of embodiments of the above described methods. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, apparatus, article or method that comprises the element.
The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments. From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) as described above, including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method described in the embodiments of the present application.
The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the claims, and all equivalent structures or equivalent processes using the descriptions and drawings of the present application, or direct or indirect application in other related technical fields are included in the scope of the claims of the present application.
Claims (10)
1. A method of assessing a multimodal cognitive disorder, the method comprising:
acquiring multi-mode test data of a target object in a cognitive function test, wherein the multi-mode test data comprises a first number of test data in different modes, and the first number is not less than 2;
inputting the multi-modal test data into a trained cognitive impairment evaluation model, carrying out feature extraction on each test data by utilizing the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, carrying out first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, respectively carrying out cognitive impairment prediction on the target object according to different prediction features to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target evaluation result, wherein the prediction features comprise a fifth number of first features and all fusion features, each fusion feature is obtained by fusion according to at least two different first features, the second number is not less than the first number, and the fifth number is not more than the second number.
2. The method of claim 1, wherein the trained cognitive impairment assessment model comprises a decision module, a second number of feature extraction networks, a third number of feature fusion networks, and a fourth number of prediction networks;
performing feature extraction on each test data by using the trained cognitive impairment evaluation model to obtain a second number of first features with different dimensions, performing first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, performing cognitive impairment prediction on the target object according to different prediction features to obtain a fourth number of cognitive prediction results, performing second fusion on all obtained cognitive prediction results to obtain a target evaluation result, and performing second fusion on the target object according to the different prediction features to obtain a third number of cognitive impairment prediction results, wherein the method comprises the following steps:
respectively inputting the first number of test data to corresponding feature extraction networks, and extracting features of one dimension in the input test data by utilizing each feature extraction network to obtain first features of the corresponding dimension, wherein each feature extraction network inputs one test data;
inputting the obtained second number of first features into corresponding feature fusion networks respectively, and fusing at least two different first features which are input by utilizing each feature fusion network once to obtain corresponding fusion features;
Respectively inputting the obtained third number of fusion features and the fifth number of first features serving as prediction features into corresponding prediction networks, and performing cognitive impairment prediction on the target object by utilizing each prediction network according to the input prediction features to obtain corresponding cognitive prediction results, wherein each prediction network inputs one prediction feature;
and taking all the cognitive prediction results as the input of the decision module, and performing second fusion on all the cognitive prediction results by using the decision module to obtain a target evaluation result.
3. The method of claim 2, wherein each cognitive prediction result comprises a probability of each class of cognitive level classification;
and performing second fusion on all the cognitive prediction results by using the decision module to obtain a target evaluation result, wherein the second fusion comprises the following steps:
calculating the average value of the probabilities of the same cognitive level classification in all the cognitive prediction results by using the decision module, and taking the cognitive level classification with the highest average value as a target evaluation result;
or alternatively, the first and second heat exchangers may be,
calculating probability sums of probabilities of the same cognitive level classification in all cognitive prediction results by using the decision module, and taking the probability sum of the probabilities and the highest cognitive level classification as a target evaluation result;
Or alternatively, the first and second heat exchangers may be,
and determining the cognitive level classification with the highest probability in each cognitive prediction result as the obtained vote classification by using the decision module by adopting a minority-subject majority voting mechanism, counting the same obtained vote classification, and taking the cognitive level classification corresponding to the obtained vote classification with the highest counting times as a target evaluation result.
4. The method of claim 2, wherein each cognitive prediction result comprises a probability of each class of cognitive level classification;
and performing second fusion on all the cognitive prediction results by using the decision module to obtain a target evaluation result, wherein the second fusion comprises the following steps:
and respectively carrying out weight calculation on the probabilities of the same class of cognitive level classification in all the cognitive prediction results by using the decision module to obtain the fusion probability of each class of cognitive level classification, and taking the cognitive level classification with the highest fusion probability as a target evaluation result.
5. The method of claim 1, wherein the multi-modal test data comprises at least two modalities of test data, including voice data, drawing data, video data, text writing data, and test item feedback results;
if the multimodal test data comprises voice data, extracting first features by using the trained cognitive impairment evaluation model, wherein the first features comprise at least one of voice features and text features, and the text features are extracted from voice recognition texts corresponding to the voice data;
If the multimodal test data includes drawing data, the first features extracted using the trained cognitive impairment assessment model include drawing features;
if the multimodal test data includes video data, the first features extracted using the trained cognitive impairment assessment model include video features;
if the multimodal test data includes written text data, the first features extracted using the trained cognitive impairment assessment model include written features;
and if the multi-mode test data comprises test item feedback results, the first characteristics extracted by using the trained cognitive impairment evaluation model comprise feedback results of each test item.
6. The method of claim 5, wherein the speech features include at least one of mel-frequency cepstral coefficients, fundamental frequencies, formants, fundamental frequency perturbations, amplitude perturbations;
the text features comprise at least one of TF-IDF, word frequency, sentence logic features and sentence pauses;
the drawing features comprise at least one of pattern features and drawing scores;
the video features comprise at least one of micro-expression features, motion delay features, drawing speed and writing speed;
The writing characteristics include at least one of handwriting characteristics, writing accuracy.
7. The method of claim 2, wherein prior to the inputting the multimodal test data into the trained cognitive impairment assessment model, the method further comprises:
acquiring multi-mode sample test data of different subjects to form a sample set, wherein the multi-mode sample test data is marked with a real cognitive level label of the corresponding subjects;
using a strategy of back propagation minimization loss function, performing iterative update training on a decision module, a feature extraction network, a feature fusion network and a prediction network in the cognitive impairment evaluation model to be trained by using the sample set to obtain a trained cognitive impairment evaluation model,
or alternatively, the first and second heat exchangers may be,
and performing iterative update training on the feature extraction network, the feature fusion network and the prediction network in the cognitive impairment evaluation model to be trained by using a strategy of back propagation minimization loss function by using the sample set to obtain the trained cognitive impairment evaluation model.
8. An apparatus for assessing a multimodal cognitive disorder, the apparatus comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring multi-mode test data of a target object in a cognitive function test, the multi-mode test data comprise test data of a first number of different modes, and the first number is not less than 2;
The prediction module is used for inputting the multi-mode test data into a trained cognitive disorder assessment model, extracting features of each test data by utilizing the trained cognitive disorder assessment model to obtain a second number of first features with different dimensions, carrying out first fusion on the second number of first features with different dimensions according to a preset fusion rule to obtain a third number of fusion features, carrying out cognitive disorder prediction on the target object according to different prediction features to obtain a fourth number of cognitive prediction results, and carrying out second fusion on all obtained cognitive prediction results to obtain a target assessment result, wherein the prediction features comprise a fifth number of first features and all fusion features, each fusion feature is obtained by fusion according to at least two different first features, the second number is not smaller than the first number, and the fifth number is not larger than the second number.
9. A computer device comprising a memory, a processor and computer readable instructions stored on the memory and executable on the processor, wherein the processor, when executing the computer readable instructions, performs the steps of the method of assessing multimodal cognitive impairment as claimed in any one of claims 1 to 7.
10. A computer readable storage medium having computer readable instructions stored thereon, which when executed by a processor, cause the processor to perform the steps of the method of assessing multimodal cognitive impairment according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310143687.3A CN116386862A (en) | 2023-02-10 | 2023-02-10 | Multi-modal cognitive impairment evaluation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310143687.3A CN116386862A (en) | 2023-02-10 | 2023-02-10 | Multi-modal cognitive impairment evaluation method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116386862A true CN116386862A (en) | 2023-07-04 |
Family
ID=86973965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310143687.3A Pending CN116386862A (en) | 2023-02-10 | 2023-02-10 | Multi-modal cognitive impairment evaluation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116386862A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116881853A (en) * | 2023-09-08 | 2023-10-13 | 小舟科技有限公司 | Attention assessment method, system, equipment and medium based on multi-mode fusion |
| CN117198537A (en) * | 2023-11-07 | 2023-12-08 | 北京无疆脑智科技有限公司 | Task completion data analysis method and device, electronic equipment and storage medium |
| CN117476208A (en) * | 2023-11-06 | 2024-01-30 | 无锡市惠山区人民医院 | Intelligent auxiliary recognition system for cognitive dysfunction based on medical images of time sequence |
-
2023
- 2023-02-10 CN CN202310143687.3A patent/CN116386862A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116881853A (en) * | 2023-09-08 | 2023-10-13 | 小舟科技有限公司 | Attention assessment method, system, equipment and medium based on multi-mode fusion |
| CN116881853B (en) * | 2023-09-08 | 2024-01-05 | 小舟科技有限公司 | Attention assessment method, system, equipment and medium based on multi-mode fusion |
| CN117476208A (en) * | 2023-11-06 | 2024-01-30 | 无锡市惠山区人民医院 | Intelligent auxiliary recognition system for cognitive dysfunction based on medical images of time sequence |
| CN117198537A (en) * | 2023-11-07 | 2023-12-08 | 北京无疆脑智科技有限公司 | Task completion data analysis method and device, electronic equipment and storage medium |
| CN117198537B (en) * | 2023-11-07 | 2024-03-26 | 北京无疆脑智科技有限公司 | Task completion data analysis method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Kumar et al. | Get it scored using autosas—an automated system for scoring short answers | |
| KR102040400B1 (en) | System and method for providing user-customized questions using machine learning | |
| CN116386862A (en) | Multi-modal cognitive impairment evaluation method, device, equipment and storage medium | |
| Kukkar et al. | Prediction of student academic performance based on their emotional wellbeing and interaction on various e-learning platforms | |
| AI-Atroshi et al. | Automated speech based evaluation of mild cognitive impairment and Alzheimer’s disease detection using with deep belief network model | |
| Bikku et al. | Deep learning approaches for classifying data: a review | |
| Hung et al. | Improving predictive power through deep learning analysis of K-12 online student behaviors and discussion board content | |
| CN116130092A (en) | Method and device for training multi-language prediction model and predicting Alzheimer's disease | |
| Yadav et al. | A novel automated depression detection technique using text transcript | |
| Wang et al. | Utilizing artificial intelligence to support analyzing self-regulated learning: A preliminary mixed-methods evaluation from a human-centered perspective | |
| Hendricks et al. | Generating visual explanations with natural language | |
| Yadav et al. | Review of automated depression detection: Social posts, audio and video, open challenges and future direction | |
| CN112669936A (en) | Social network depression detection method based on texts and images | |
| CN116269223A (en) | Alzheimer's disease prediction method, device, equipment and storage medium | |
| Hacine-Gharbi et al. | Prosody based Automatic Classification of the Uses of French ‘Oui’as Convinced or Unconvinced Uses | |
| TJ et al. | D-ResNet-PVKELM: deep neural network and paragraph vector based kernel extreme machine learning model for multimodal depression analysis | |
| JP7303243B2 (en) | Exam question prediction system and exam question prediction method | |
| Tang et al. | Analysis on Gated Recurrent Unit Based Question Detection Approach. | |
| Veinović | Apparent Personality Analysis Based on Aggregation Model | |
| Alrajhi et al. | Plug & Play with Deep Neural Networks: Classifying Posts that Need Urgent Intervention in MOOCs | |
| Senevirathne et al. | Smart Personal Intelligent Assistant for Candidates of IELTS Exams | |
| Jingning | Speech recognition based on mobile sensor networks application in English education intelligent assisted learning system | |
| Biswas et al. | Automatic judgement of neural network-generated image captions | |
| Goto | A hierarchical neural network model for Japanese toward detecting mild cognitive impairment | |
| Siew et al. | Phonological similarity judgments of word pairs reflect sensitivity to large-scale structure of the phonological lexicon. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |