WO2024024062A1 - 症状検出プログラム、症状検出方法および症状検出装置 - Google Patents
症状検出プログラム、症状検出方法および症状検出装置 Download PDFInfo
- Publication number
- WO2024024062A1 WO2024024062A1 PCT/JP2022/029199 JP2022029199W WO2024024062A1 WO 2024024062 A1 WO2024024062 A1 WO 2024024062A1 JP 2022029199 W JP2022029199 W JP 2022029199W WO 2024024062 A1 WO2024024062 A1 WO 2024024062A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- patient
- occurrence
- symptom detection
- machine learning
- learning model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/168—Evaluating attention deficit, hyperactivity
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
- G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/70—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mental therapies, e.g. psychological therapy or autogenous training
Definitions
- the present invention relates to a symptom detection program, a symptom detection method, and a symptom detection device.
- CT Computer Tomography
- blood tests to diagnose dementia patients who are unable to perform basic movements such as eating and bathing, and those who are able to perform basic movements but can perform complex movements such as shopping and housework. It is known that the diagnosis of mild cognitive impairment cannot be made.
- One aspect of the present invention is to provide a symptom detection program, a symptom detection method, and a symptom detection device that can detect symptoms related to dementia at an early stage.
- the symptom detection program causes a computer to acquire video data including the patient's face while performing a specific task, and analyzes the acquired video data to detect the patient's face. Detecting the occurrence intensity of each included action unit, and detecting symptoms related to dementia of the patient based on temporal changes in the occurrence intensity of each of the plurality of detected action units. It is characterized by
- symptoms related to dementia can be detected early.
- FIG. 1 is a diagram illustrating a symptom detection device according to a first embodiment.
- FIG. 2 is a diagram illustrating the reference technology.
- FIG. 3 is a functional block diagram showing the functional configuration of the symptom detection device according to the first embodiment.
- FIG. 4 is a diagram illustrating an example of generation of the first machine learning model.
- FIG. 5 is a diagram showing an example of arrangement of cameras.
- FIG. 6 is a diagram illustrating the movement of the marker.
- FIG. 7 is a diagram illustrating training of the second machine learning model.
- FIG. 8 is a diagram illustrating an example of a specific task.
- FIG. 9 is a diagram illustrating generation of training data for the second machine learning model.
- FIG. 10 is a diagram illustrating detection of longitudinal cognitive dysfunction.
- FIG. 10 is a diagram illustrating detection of longitudinal cognitive dysfunction.
- FIG. 11 is a diagram illustrating details of detection of mild cognitive dysfunction.
- FIG. 12 is a flowchart showing the flow of pre-processing.
- FIG. 13 is a flowchart showing the flow of the detection process.
- FIG. 14 is a diagram illustrating another example of training data for the second machine learning model.
- FIG. 15 is a diagram illustrating an example of how the symptom detection application is used.
- FIG. 16 is a diagram illustrating an example of a hardware configuration.
- Embodiments of the symptom detection program, symptom detection method, and symptom detection device according to the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this example. Moreover, each embodiment can be combined as appropriate within a consistent range.
- FIG. 1 is a diagram illustrating a symptom detection device 10 according to a first embodiment.
- the symptom detection device 10 shown in FIG. 1 is an example of a computer that uses facial expression recognition technology to detect dementia or mild cognitive dysfunction at an early stage.
- FIG. 2 is a diagram illustrating the reference technology.
- an on-duty doctor at hospital 1 performs a diagnosis using CT and blood tests.
- early stage dementia and mild cognitive impairment are difficult to diagnose by anyone other than a specialist, as symptoms do not easily appear through tests. For this reason, if the doctor on duty is not a specialist, there is a high possibility that misdiagnosis will occur and subsequent treatment will be adversely affected.
- the symptom detection device 10 uses video data when a patient executes a specific task (application) that tests the cognitive function by placing a load on the cognitive function to detect dementia or mild cognitive function. Achieve early detection of failures.
- a specific task application
- the symptom detection device 10 executes both a specific task and symptom detection, but they may be executed by separate devices.
- the symptom detection device 10 in the learning phase, the symptom detection device 10 generates each machine learning model used to detect symptoms of dementia and mild cognitive impairment. For example, as shown in FIG. 1, in the learning phase, the symptom detection device 10 uses a first machine learning model that outputs the strength of each action unit (AU) from image data, and a first machine learning model that outputs the intensity of each action unit (AU) from image data, and A second machine learning model is generated that outputs a detection result of the presence or absence of mild cognitive impairment based on the task score.
- AU action unit
- a second machine learning model is generated that outputs a detection result of the presence or absence of mild cognitive impairment based on the task score.
- the symptom detection device 10 inputs training data in which image data showing the patient's face is used as an explanatory variable and the occurrence intensity (value) of each AU is used as an objective variable to the first machine learning model,
- the first machine learning model is generated by training the parameters of the first machine learning model so that error information between the output result of the first machine learning model and the objective variable is minimized.
- the symptom detection device 10 also includes explanatory variables including temporal changes in the intensity of occurrence of each AU when the patient is performing a specific task and a score that is the execution result of the specific task, and occurrence of mild cognitive impairment.
- the training data with presence or absence as the objective variable is input to the second machine learning model, and the parameters of the second machine learning model are set so that the error information between the output result of the second machine learning model and the objective variable is minimized.
- a second machine learning model is generated by training.
- the symptom detection device 10 uses video data when the patient performs a specific task and each trained machine learning model to detect the occurrence of symptoms related to dementia.
- the symptom detection device 10 acquires video data of a patient performing a specific task, and applies each frame (image data) in the video data to a first machine learning model as a feature quantity. input, and obtain the occurrence intensity of each AU for each frame. In this way, the symptom detection device 10 obtains changes (change patterns) in the intensity of occurrence of each AU of a patient performing a specific task. The symptom detection device 10 also acquires the score of a specific task after completing the specific task. Thereafter, the symptom detection device 10 inputs the temporal change in the intensity of occurrence of each AU of the patient and the score as feature quantities to the second machine learning model, and obtains a detection result as to whether mild cognitive impairment has occurred.
- the symptom detection device 10 is able to detect subtle changes in facial expressions with few individual differences, and can detect mild cognitive dysfunction at an early stage.
- mild cognitive dysfunction is illustrated here as an example of symptoms related to dementia in patients, this is not limited to this, and it can be similarly applied to other symptoms such as dementia by setting the objective variable. can do.
- FIG. 3 is a functional block diagram showing the functional configuration of the symptom detection device 10 according to the first embodiment.
- the symptom detection device 10 includes a communication section 11, a display section 12, an imaging section 13, a storage section 20, and a control section 30.
- the communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface.
- the communication unit 11 receives video data and the score of a specific task, which will be described later, and transmits the processing results to a destination specified in advance by the control unit 30, which will be described later.
- the display unit 12 is a processing unit that displays and outputs various information, and is realized by, for example, a display or a touch panel. For example, the display unit 12 outputs a specific task and receives answers to the specific task.
- the imaging unit 13 is a processing unit that captures a video to obtain video data, and is realized by, for example, a camera.
- the imaging unit 13 captures an image including the patient's face while the patient is performing a specific task, and stores the captured image in the storage unit 20 as image data.
- the storage unit 20 is a processing unit that stores various data and programs executed by the control unit 30, and is realized by, for example, a memory or a hard disk.
- This storage unit 20 stores a training data DB 21, a video data DB 22, a first machine learning model 23, and a second machine learning model 24.
- the training data DB 21 is a database that stores various training data used to generate the first machine learning model 23 and the second machine learning model 24.
- the training data stored here can include supervised training data to which correct answer information is added and unsupervised training data to which correct answer information is not added.
- the video data DB 22 is a database that stores video data captured by the imaging unit 13.
- the video data DB 22 stores, for each patient, video data including the patient's face while performing a specific task.
- the video data includes a plurality of time-series frames. A frame number is assigned to each frame in ascending chronological order.
- One frame is image data of a still image captured by the imaging unit 13 at a certain timing.
- the first machine learning model 23 is a machine learning model that outputs the occurrence strength of each AU according to the input of each frame (image data) included in the video data. Specifically, the first machine learning model 23 estimates AU, which is a method of decomposing and quantifying facial expressions based on facial parts and facial muscles. This first machine learning model 23 is based on the input of image data, and is based on "AU1:2, Outputs facial expression recognition results such as "AU2:5, AU3:1, . . .”. For example, the first machine learning model 23 can employ various algorithms such as neural networks and random forests.
- the second machine learning model 24 is a machine learning model that outputs the presence or absence of mild cognitive dysfunction according to the input of the feature amount. For example, the second machine learning model 24 determines whether or not mild cognitive impairment has occurred, depending on the input of features including temporal changes (change patterns) in the intensity of occurrence of each AU and the score of a specific task. Output detection results including For example, the second machine learning model 24 can employ various algorithms such as neural networks and random forests.
- the control unit 30 is a processing unit that controls the entire symptom detection device 10, and is realized by, for example, a processor.
- This control section 30 has a preprocessing section 40 and an operation processing section 50.
- the pre-processing unit 40 and the operation processing unit 50 are realized by an electronic circuit included in a processor, a process executed by the processor, or the like.
- the pre-processing unit 40 is a processing unit that generates each model using training data stored in the storage unit 20 prior to operation of detecting symptoms related to dementia.
- the pre-processing section 40 includes a first training section 41 and a second training section 42.
- the first training unit 41 is a processing unit that generates the first machine learning model 23 through training using training data. Specifically, the first training unit 41 generates the first machine learning model 23 by supervised learning using training data with correct answer information (labels).
- FIG. 4 is a diagram illustrating an example of generation of the first machine learning model 23.
- the first training unit 41 generates training data and Run machine learning.
- the RGB camera 25a and the IR camera 25b are directed at the face of the person to whom the marker is attached.
- the RGB camera 25a is a general digital camera that receives visible light and generates an image.
- the IR camera 25b senses infrared rays.
- the marker is, for example, an IR reflection (retroreflection) marker.
- the IR camera 25b can perform motion capture using IR reflection from the marker.
- the person to be imaged will be referred to as a subject.
- the first training unit 41 acquires image data captured by the RGB camera 25a and the results of motion capture by the IR camera 25b. Then, the first training unit 41 generates image data 122 by removing markers from the AU occurrence intensity 121 and the captured image data through image processing.
- the occurrence intensity 121 is data in which the occurrence intensity of each AU is expressed in a five-level evaluation from A to E, and annotated as "AU1:2, AU2:5, AU3:1, --. It's okay.
- the first training unit 41 performs machine learning using the image data 122 and the AU occurrence intensity 121 output from the training data generation process, and performs machine learning to estimate the AU occurrence intensity from the image data.
- a first machine learning model 23 is generated.
- the first training unit 41 can use the intensity of AU occurrence as a label.
- FIG. 5 is a diagram showing an example of arrangement of cameras.
- a plurality of IR cameras 25b may constitute a marker tracking system.
- the marker tracking system can detect the position of the IR reflective marker by stereo imaging. Further, it is assumed that the relative positional relationship between each of the plurality of IR cameras 25b has been corrected in advance by camera calibration.
- a plurality of markers are attached to the subject's face to be imaged so as to cover AU1 to AU28.
- the position of the marker changes according to changes in the subject's facial expression.
- the marker 401 is placed near the base of the eyebrow.
- the marker 402 and the marker 403 are placed near the Torei Line.
- Markers may be placed on the skin corresponding to movements of one or more AUs and facial muscles.
- the marker may be placed avoiding areas on the skin where the texture changes greatly due to wrinkles or the like.
- the subject wears an instrument 25c with a reference point marker attached outside the contour of the face. It is assumed that even if the facial expression of the subject changes, the position of the reference point marker attached to the instrument 25c does not change. Therefore, the first training unit 41 can detect a change in the position of the marker attached to the face based on a change in the position relative to the reference point marker. Furthermore, by setting the number of reference markers to three or more, the first training unit 41 can specify the positions of the markers in the three-dimensional space.
- the device 25c is, for example, a headband. Further, the equipment 25c may be a VR headset, a mask made of hard material, or the like. In that case, the first training section 41 can utilize the rigid surface of the instrument 25c as a reference point marker.
- the subject's facial expression changes. With this, it is possible to obtain an image of how facial expressions change over time. Further, the RGB camera 25a may capture a moving image. A video can be considered as multiple still images arranged in chronological order. Furthermore, the subject may change their facial expressions freely or may change their facial expressions according to a predetermined scenario.
- the intensity of AU occurrence can be determined by the amount of movement of the marker.
- the first training unit 41 can determine the intensity of occurrence based on the amount of movement of the marker calculated based on the distance between the position of the marker and a position set in advance as a determination criterion.
- FIG. 6 is a diagram illustrating the movement of the marker.
- (a), (b), and (c) of FIG. 6 are images captured by the RGB camera 25a. Further, it is assumed that the images are taken in the order of (a), (b), and (c).
- (a) is an image when the subject is expressionless.
- the first training unit 41 can regard the position of the marker in the image (a) as a reference position with a movement amount of 0.
- the subject's expression was as if he was frowning.
- the position of the marker 401 is moving downward as the facial expression changes.
- the distance between the position of the marker 401 and the reference marker attached to the instrument 25c is increased.
- the first training unit 41 identifies the image data showing a certain expression of the subject and the intensity of each marker at that expression, and sets the explanatory variable "image data" and the objective variable "intensity of each marker”. ” to generate training data.
- the first training unit 41 then generates the first machine learning model 23 through supervised learning using the generated training data.
- the first machine learning model 23 is a neural network.
- the first training unit 41 changes the parameters of the neural network by executing machine learning using the first machine learning model 23.
- the first training unit 41 inputs explanatory variables into the neural network.
- the first training unit 41 generates a machine learning model in which the parameters of the neural network are changed so that the error between the output result output from the neural network and the correct data that is the target variable is reduced.
- the generation of the first machine learning model 23 is just an example, and other methods can be used. Further, as the first machine learning model 23, a model disclosed in Japanese Patent Application Publication No. 2021-111114 can also be used. Furthermore, the direction of the face can also be learned using a similar method.
- the second training unit 42 is a processing unit that generates the second machine learning model 24 through training using training data. Specifically, the second training unit 42 generates the second machine learning model 24 through supervised learning using training data with correct answer information (labels).
- FIG. 7 is a diagram illustrating training of the second machine learning model 24.
- the second training unit 42 uses previously prepared training data or video data of a patient performing a specific task and the trained first machine learning model 23. It is also possible to train the second machine learning model 24 using the training data generated.
- the second training unit 42 acquires "presence or absence of longitudinal cognitive dysfunction" as the doctor's diagnosis result for the patient.
- the second training unit 42 also transmits video data including a score as a result of the patient performing a specific task and the patient's face imaged while the patient is performing the specific task to the first machine.
- the occurrence intensity and face direction of each AU obtained by inputting it into the learning model 23 are acquired.
- the second training unit 42 includes "presence or absence of longitudinal cognitive dysfunction” as “correct answer information” and “temporal changes in the occurrence intensity of each AU, temporal changes in face orientation, and specific tasks” as “features”. Generate training data containing 'scores of '. Then, the second training unit 42 inputs the feature amount of the training data to the second machine learning model 24, and performs the second training so that the error between the output result of the second machine learning model 24 and the correct answer information is small. The parameters of the machine learning model 24 are updated.
- FIG. 8 is a diagram illustrating an example of a specific task.
- the specific task shown in FIG. 8 is an example of an application or an interactive application that tests cognitive function by placing a load on cognitive function.
- the specific task shown in FIG. 8(a) is a task that allows the patient to select today's date. Selection is in the form of radio buttons, and the year, month, day, and day of the week are selected in order starting from the year. The session ends when you finish answering or the time limit is exceeded. The answer completion time and answer are registered as a score. In addition, if the time is up, the mid-way answers and the time limit will be used as the score.
- the specific task shown in FIG. 8(b) is a task in which numbers are randomly arranged and displayed, and the numbers are selected in order starting from "1". If you click 1, click 2 becomes valid, and if you click 2, click 3 becomes valid. The selected number is displayed in a different color, and the current search collection number and remaining time are displayed outside the task frame. If the number of displays is XX, XX items will be completed, but if the time limit of YY seconds is exceeded, the process will end. The completion time and number of achievements (number of correct answers) are registered as a score.
- the specific task shown in FIG. 8(c) is a task that displays 100 and subtracts 7 in turn.
- the item currently being input is displayed in a different color, and the process ends after the maximum number of calculations (XX).
- the process ends when XX calculations are completed or the time limit of YY seconds is exceeded.
- the finishing time and answer will be registered as a score.
- the mid-way answers and the time limit will be used as the score.
- FIG. 9 is a diagram illustrating generation of training data for the second machine learning model 24.
- the second training unit 42 acquires video data captured from the start to the end of a specific task from a camera or the like, and from each frame of the video data, "occurrence intensity of each AU" is used. ” and “face direction”.
- the second training unit 42 inputs the image data of frame 1 to the trained first machine learning model 23, and inputs "AU1:2, AU2:5" and "face orientation: A”. get.
- the second training unit 42 inputs the image data of frame 2 to the trained first machine learning model 23, and inputs "AU1:2, AU2:6" and "face orientation: A”. get.
- the second training unit 42 identifies temporal changes in each AU of the patient and temporal changes in the orientation of the patient's face from the video data.
- the second training unit 42 obtains the score "XX” that is output after the completion of a specific task. Further, the second training unit 42 obtains the doctor's diagnosis result of “longitudinal cognitive dysfunction: present” for the patient who has performed the specific task from the doctor, electronic medical record, or the like.
- the second training unit 42 uses "occurrence intensity of each AU", “face direction”, and “score (XX)" acquired using each frame as explanatory variables, and "longitudinal cognitive dysfunction: present". Training data with the target variable is generated, and the second machine learning model 24 is generated. In other words, the second machine learning model 24 calculates the relationship between "the change pattern of the temporal change in the intensity of each AU, the change pattern of the temporal change of the face orientation, and the score" and the "presence or absence of the occurrence of longitudinal cognitive dysfunction". Learn.
- the operation processing unit 50 includes a task execution unit 51, a video acquisition unit 52, an AU detection unit 53, and a symptom detection unit 54, and uses each model prepared in advance by the preprocessing unit 40 to This is a processing unit that detects whether or not a longitudinal cognitive dysfunction has occurred in a person (patient) depicted in video data.
- FIG. 10 is a diagram illustrating detection of longitudinal cognitive dysfunction.
- the operation processing unit 50 inputs video data including the face of a patient performing a specific task to the trained first machine learning model 23, and calculates the time of each AU of the patient. Identify changes and temporal changes in the patient's facial orientation.
- the operation processing unit 50 also obtains the score of a specific task. Then, the operation processing unit 50 inputs the temporal change in AU, the temporal change in face orientation, and the score to the second machine learning model 24 to detect whether or not longitudinal cognitive dysfunction has occurred.
- the task execution unit 51 is a processing unit that executes a specific task on a patient and obtains a score. For example, the task execution unit 51 executes a specific task by displaying one of the tasks shown in FIG. 8 on the display unit 12 and accepting a response (input) from the patient. Thereafter, when the specific task is completed, the task execution unit 51 acquires a score and outputs it to the symptom detection unit 54 or the like.
- the image acquisition unit 52 is a processing unit that acquires image data including the face of a patient performing a specific task. For example, when a specific task is started, the video acquisition unit 52 starts imaging by the imaging unit 13, and when the specific task is finished, the video acquisition unit 52 ends the imaging by the imaging unit 13, and the video acquisition unit 52 stops the imaging by the imaging unit 13 when the specific task is completed. The image data in between is acquired from the imaging unit 13. Then, the video acquisition unit 52 stores the acquired video data in the video data DB 22 and outputs it to the AU detection unit 53.
- the AU detection unit 53 is a processing unit that detects the intensity of occurrence of each AU included in the patient's face by inputting the video data acquired by the video acquisition unit 52 to the first machine learning model 23. For example, the AU detection unit 53 extracts each frame from the video data, inputs each frame to the first machine learning model 23, and detects the intensity of AU occurrence and the orientation of the patient's face for each frame. The AU detection unit 53 then outputs the AU occurrence intensity and the patient's face direction to the symptom detection unit 54 for each detected frame. Note that the direction of the face can also be specified from the intensity of AU occurrence.
- the symptom detection unit 54 detects the occurrence of dementia-related symptoms in the patient using the temporal changes in the intensity of occurrence of each AU, the temporal changes in the patient's face orientation, and the score of a specific task as feature quantities.
- This is a processing section that performs
- the symptom detection unit 54 may generate a score that is acquired by the task execution unit 51 and an occurrence intensity of each AU detected by the AU detection unit for each frame in order of time. "Temporal change in face orientation” and “Temporal change in face orientation” obtained by connecting similarly detected “face orientations” in time order are input to the second machine learning model 24 as feature quantities.
- the symptom detection unit 54 obtains the output result of the second machine learning model 24, and selects between the probability value (reliability) of the occurrence of the symptom and the probability value of the probability value of the absence of the symptom included in the output result. The one with the higher probability value is obtained as the detection result. Thereafter, the symptom detection section 54 displays the detection results on the display section 12 and stores them in the storage section 20.
- FIG. 11 is a diagram illustrating details of detection of mild cognitive dysfunction.
- the operation processing unit 50 acquires video data captured from the start to the end of a specific task, and from each frame of the video data, the "occurrence intensity of each AU" and "face 'Orientation'.
- the operation processing unit 50 inputs the image data of frame 1 to the trained first machine learning model 23 and obtains "AU1:2, AU2:5" and "face orientation: A”. do. Similarly, the operation processing unit 50 inputs the image data of frame 2 to the trained first machine learning model 23, and inputs "AU1:2, AU2:5" and "face orientation: A”. get. In this way, the operation processing unit 50 identifies temporal changes in each AU of the patient and temporal changes in the orientation of the patient's face from the video data.
- the operation processing unit 50 acquires the score "YY" of the specific task and "time change of each AU of the patient (AU1:2, AU2:5..., AU1:2, AU2:5... ), temporal change in patient's face orientation (face orientation: A, face orientation A, ...), score (YY)" are input into the second machine learning model 24 as feature quantities, and Detects the occurrence of cognitive dysfunction.
- FIG. 12 is a flowchart showing the flow of pre-processing. As shown in FIG. 12, when the pre-processing unit 40 is instructed to start processing (S101: Yes), it generates the first machine learning model 23 using training data (S102).
- the pre-processing unit 40 acquires video data (S104). Then, the pre-processing unit 40 inputs each frame of the video data to the first machine learning model 23, and obtains the occurrence intensity and face orientation of each AU for each frame (S105).
- the pre-processing unit 40 acquires the score (S107). Further, the preprocessing unit 40 obtains the patient's diagnosis result by the doctor (S108).
- the pre-processing unit 40 generates training data including temporal changes in the occurrence intensity of each AU, temporal changes in face orientation, and scores (S109), and uses the training data to generate the second machine learning model 24. Generate (S110).
- FIG. 13 is a flowchart showing the flow of the detection process. As shown in FIG. 13, when the operation processing unit 50 is instructed to start processing (S201: Yes), it executes a specific task for the patient (S202) and starts acquiring video data (S203). .
- the operation processing unit 50 acquires the score and ends the acquisition of the video data (S205).
- the operation processing unit 50 inputs each frame of video data to the first machine learning model 23, and obtains the occurrence intensity and face orientation of each AU for each frame (S206).
- the operation processing unit 50 specifies the temporal change of each AU and the temporal change of the face orientation based on the occurrence strength of each AU and the face orientation for each frame, and determines the temporal change of each AU and the temporal change of the face orientation.
- "Time change in orientation, score” is generated as a feature quantity (S207).
- the operation processing unit 50 inputs the feature amount to the second machine learning model 24, obtains the detection result by the second machine learning model 24 (S208), and outputs the detection result to the display unit 12 or the like (S208). S209).
- the symptom detection device 10 of Example 1 can detect the presence or absence of symptoms related to dementia, mild cognitive impairment, etc., even without the specialized knowledge of a doctor.
- the symptom detection device 10 can detect small changes in facial expressions with little individual variation, and can detect symptoms related to dementia, mild cognitive impairment, etc. at an early stage. can.
- training data In the first embodiment described above, an example was explained in which the time change of each AU, the time change of face orientation, and the score are used as feature quantities (explanatory variables) as training data for the second machine learning model 24. It is not limited.
- FIG. 14 is a diagram illustrating another example of training data for the second machine learning model 24.
- the symptom detection device 10 may use only the temporal change of each AU as an explanatory variable, or may use the temporal change of each AU and the temporal change of the face orientation as explanatory variables, for example.
- the time change and score of each AU may be used as explanatory variables.
- a binary value of presence and absence of symptoms of mild cognitive dysfunction is used as a target variable, but the present invention is not limited to this.
- a binary value of presence or absence of symptoms of dementia can be used, or a four-value value of presence or absence of symptoms of dementia and presence or absence of symptoms of mild cognitive impairment can also be used. .
- the symptom detection device 10 can determine the feature amounts to be used for training and detection according to accuracy and cost, so it can provide a simple symptom detection service and further improve doctor's diagnosis. We can also provide detailed support services.
- the second machine learning model 24 is used to detect the presence or absence of symptoms of mild cognitive dysfunction, but the present invention is not limited to this.
- the presence or absence of symptoms of mild cognitive dysfunction can be detected using a detection rule that associates the temporal change pattern of each AU with the presence or absence of symptoms of mild cognitive dysfunction.
- each AU is not limited to processing using the first machine learning model 23, but can also be detected by analyzing video data. For example, it is also possible to set each AU for the face area of each frame in the video data, and detect changes in each AU in the entire video data.
- FIG. 15 is a diagram illustrating an example of how the symptom detection application is used.
- the application server 70 includes a first machine learning model 23 and a second machine learning model 24 trained by the preprocessing unit 40, and performs the same processing as the operation processing unit 50.
- a detection application (hereinafter referred to as an application) 71 is held.
- the user purchases the app 71 at any location such as home, downloads the app 71 from the application server 70, and installs it on his/her smartphone 60 or the like.
- the user then uses his or her smartphone 60 to execute the same process as the operation processing unit 50 described in the first embodiment, and obtains the symptom detection results.
- the hospital can conduct the examination with the simple detection results obtained, allowing for early confirmation of the disease name and symptoms, and treatment. This can be useful for early start.
- each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. That is, the specific form of distributing and integrating each device is not limited to what is shown in the drawings. In other words, all or part of them can be functionally or physically distributed and integrated into arbitrary units depending on various loads and usage conditions.
- the pre-processing section 40 and the operation processing section 50 can be realized by separate devices.
- each processing function performed by each device can be realized by a CPU and a program that is analyzed and executed by the CPU, or can be realized as hardware using wired logic.
- FIG. 16 is a diagram illustrating an example of a hardware configuration.
- the symptom detection device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Furthermore, the parts shown in FIG. 16 are interconnected by a bus or the like. Note that in addition to these, a display, a touch panel, etc. can be included.
- the communication device 10a is a network interface card or the like, and communicates with other devices.
- the HDD 10b stores programs and DB that operate the functions shown in FIG.
- the processor 10d reads a program that executes the same processing as each processing unit shown in FIG. 3 from the HDD 10b, etc., and deploys it in the memory 10c, thereby operating a process that executes each function described in FIG. 3, etc. For example, this process executes the same functions as each processing unit included in the symptom detection device 10. Specifically, the processor 10d reads a program having the same functions as the preprocessing section 40, the operation processing section 50, etc. from the HDD 10b. The processor 10d then executes a process that executes the same processing as the preprocessing unit 40, the operation processing unit 50, and the like.
- the symptom detection device 10 operates as an information processing device that executes a symptom detection method by reading and executing a program. Moreover, the symptom detection device 10 can also realize the same functions as the above-described embodiments by reading the program from the recording medium using the medium reading device and executing the read program. Note that the programs in other embodiments are not limited to being executed by the symptom detection device 10. For example, the above embodiments may be similarly applied to cases where another computer or server executes a program, or where these computers or servers cooperate to execute a program.
- This program may be distributed via a network such as the Internet.
- this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and is read from the recording medium by the computer. It may also be executed by being read.
- Symptom detection device 11 Communication unit 12 Display unit 13 Imaging unit 20 Storage unit 21 Training data DB 22 Video data DB 23 First machine learning model 24 Second machine learning model 30 Control unit 40 Pre-processing unit 41 First training unit 42 Second training unit 50 Operation processing unit 51 Task execution unit 52 Video acquisition unit 53 AU detection unit 54 Symptom detection Department
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Pathology (AREA)
- Physics & Mathematics (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Veterinary Medicine (AREA)
- Heart & Thoracic Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Psychiatry (AREA)
- Artificial Intelligence (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Child & Adolescent Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Social Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Psychology (AREA)
- Radiology & Medical Imaging (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Physiology (AREA)
- Signal Processing (AREA)
- Fuzzy Systems (AREA)
- Quality & Reliability (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP22953151.2A EP4563092A4 (en) | 2022-07-28 | 2022-07-28 | SYMPTOM DETECTION PROGRAM, SYMPTOM DETECTION METHOD AND SYMPTOM DETECTION DEVICE |
| JP2024536711A JP7764966B2 (ja) | 2022-07-28 | 2022-07-28 | 症状検出プログラム、症状検出方法および症状検出装置 |
| PCT/JP2022/029199 WO2024024062A1 (ja) | 2022-07-28 | 2022-07-28 | 症状検出プログラム、症状検出方法および症状検出装置 |
| US19/035,509 US20250166845A1 (en) | 2022-07-28 | 2025-01-23 | Computer-readable recording medium storing symptom detection program, symptom detection method, and symptom detection device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2022/029199 WO2024024062A1 (ja) | 2022-07-28 | 2022-07-28 | 症状検出プログラム、症状検出方法および症状検出装置 |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US19/035,509 Continuation US20250166845A1 (en) | 2022-07-28 | 2025-01-23 | Computer-readable recording medium storing symptom detection program, symptom detection method, and symptom detection device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2024024062A1 true WO2024024062A1 (ja) | 2024-02-01 |
Family
ID=89705874
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2022/029199 Ceased WO2024024062A1 (ja) | 2022-07-28 | 2022-07-28 | 症状検出プログラム、症状検出方法および症状検出装置 |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US20250166845A1 (cg-RX-API-DMAC7.html) |
| EP (1) | EP4563092A4 (cg-RX-API-DMAC7.html) |
| JP (1) | JP7764966B2 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2024024062A1 (cg-RX-API-DMAC7.html) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2021058231A (ja) * | 2019-10-02 | 2021-04-15 | 株式会社エクサウィザーズ | 認知機能推定方法、コンピュータプログラム及び認知機能推定装置 |
| US20210188291A1 (en) * | 2010-06-07 | 2021-06-24 | Affectiva, Inc. | In-vehicle drowsiness analysis using blink rate |
| JP2021111114A (ja) | 2020-01-09 | 2021-08-02 | 富士通株式会社 | 学習データ生成プログラム、学習データ生成方法及び推定装置 |
| WO2022024272A1 (ja) * | 2020-07-29 | 2022-02-03 | 日本電気株式会社 | 情報処理システム、データ蓄積装置、データ生成装置、情報処理方法、データ蓄積方法、データ生成方法及び記録媒体、並びに、データベース |
| JP2022061587A (ja) | 2020-10-07 | 2022-04-19 | キヤノンメディカルシステムズ株式会社 | 核医学診断装置 |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3921850A1 (en) * | 2019-02-06 | 2021-12-15 | AIC Innovations Group, Inc. | Biomarker identification |
-
2022
- 2022-07-28 JP JP2024536711A patent/JP7764966B2/ja active Active
- 2022-07-28 EP EP22953151.2A patent/EP4563092A4/en not_active Withdrawn
- 2022-07-28 WO PCT/JP2022/029199 patent/WO2024024062A1/ja not_active Ceased
-
2025
- 2025-01-23 US US19/035,509 patent/US20250166845A1/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210188291A1 (en) * | 2010-06-07 | 2021-06-24 | Affectiva, Inc. | In-vehicle drowsiness analysis using blink rate |
| JP2021058231A (ja) * | 2019-10-02 | 2021-04-15 | 株式会社エクサウィザーズ | 認知機能推定方法、コンピュータプログラム及び認知機能推定装置 |
| JP2021111114A (ja) | 2020-01-09 | 2021-08-02 | 富士通株式会社 | 学習データ生成プログラム、学習データ生成方法及び推定装置 |
| WO2022024272A1 (ja) * | 2020-07-29 | 2022-02-03 | 日本電気株式会社 | 情報処理システム、データ蓄積装置、データ生成装置、情報処理方法、データ蓄積方法、データ生成方法及び記録媒体、並びに、データベース |
| JP2022061587A (ja) | 2020-10-07 | 2022-04-19 | キヤノンメディカルシステムズ株式会社 | 核医学診断装置 |
Non-Patent Citations (1)
| Title |
|---|
| See also references of EP4563092A4 |
Also Published As
| Publication number | Publication date |
|---|---|
| JPWO2024024062A1 (cg-RX-API-DMAC7.html) | 2024-02-01 |
| EP4563092A1 (en) | 2025-06-04 |
| US20250166845A1 (en) | 2025-05-22 |
| EP4563092A4 (en) | 2025-09-10 |
| JP7764966B2 (ja) | 2025-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110738192B (zh) | 人体运动功能辅助评估方法、装置、设备、系统及介质 | |
| CN112890815B (zh) | 一种基于深度学习的孤独症辅助评估系统和方法 | |
| US12579795B2 (en) | Diagnosis support system, diagnosis support method, and storage medium | |
| JP7057589B2 (ja) | 医療情報処理システム、歩行状態定量化方法およびプログラム | |
| JP2020533701A (ja) | 対象者の識別のためのカメラ及び画像校正 | |
| JP6199791B2 (ja) | ペット健康診断装置、ペット健康診断方法及びプログラム | |
| JP6433632B1 (ja) | コンピュータシステム、被験者診断支援方法及びプログラム | |
| Den Boer et al. | Volumetric image interpretation in radiology: scroll behavior and cognitive processes | |
| CN111448589B (zh) | 用于检测患者的身体移动的设备、系统和方法 | |
| Chen et al. | Surgical wounds assessment system for self-care | |
| IL288866B1 (en) | Diagnosing skin diseases using machine learning models | |
| JP2023122528A (ja) | 認知症診断に必要な情報の提供方法及び装置 | |
| Hartmann et al. | Gait parameter estimation from a single depth sensor | |
| Liu et al. | Deep neural network-based video processing to obtain dual-task upper-extremity motor performance toward assessment of cognitive and motor function | |
| US20240257945A1 (en) | Autonomous image acquisition start-stop managing system | |
| JP2023122529A (ja) | 認知症診断支援情報の提供装置及びその方法 | |
| JP2023109385A (ja) | 保険提案用システム、保険提案用システムの動作方法 | |
| JP7764966B2 (ja) | 症状検出プログラム、症状検出方法および症状検出装置 | |
| JP2022184272A (ja) | 情報処理装置、情報処理方法及びプログラム | |
| JP7754325B2 (ja) | 推定プログラム、推定方法および推定装置 | |
| EP4342168B1 (en) | Image capturing method | |
| WO2018062366A1 (ja) | 認知症介護負担度判定装置、認知症介護負担度判定方法、認知症介護負担度判定プログラム、認知症治療効果判定装置、認知症治療効果判定方法、及び認知症治療効果判定プログラム | |
| US20230394666A1 (en) | Information processing apparatus, information processing method and information processing program | |
| WO2014104357A1 (ja) | 動作情報処理システム、動作情報処理装置及び医用画像診断装置 | |
| Bai et al. | Convolutional network for plantar pressure parsing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22953151 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2024536711 Country of ref document: JP |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2022953151 Country of ref document: EP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| ENP | Entry into the national phase |
Ref document number: 2022953151 Country of ref document: EP Effective date: 20250228 |
|
| WWP | Wipo information: published in national office |
Ref document number: 2022953151 Country of ref document: EP |
|
| WWW | Wipo information: withdrawn in national office |
Ref document number: 2022953151 Country of ref document: EP |