CN110675922A - Multi-mode-based intelligent follow-up method and system - Google Patents
Multi-mode-based intelligent follow-up method and system Download PDFInfo
- Publication number
- CN110675922A CN110675922A CN201910897892.2A CN201910897892A CN110675922A CN 110675922 A CN110675922 A CN 110675922A CN 201910897892 A CN201910897892 A CN 201910897892A CN 110675922 A CN110675922 A CN 110675922A
- Authority
- CN
- China
- Prior art keywords
- follow
- video
- ecrf
- audio
- patient
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
- G16H10/20—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for electronic clinical trials or questionnaires
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H80/00—ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Pathology (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The invention relates to the technical field of medical follow-up visits, in particular to an intelligent follow-up visit method and system based on multiple modes; the method comprises the following steps: starting follow-up; positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching; the follow-up reading questions in sequence and answering by the patient; the follow-up client collects the audio and video in real time and transmits the audio and video to a follow-up platform for recording; the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video; extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching; and the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database. The multimode-based intelligent follow-up method and the multimode-based intelligent follow-up system realize the fusion work of audio, video and text information, can flexibly select and combine according to the situation, and improve the follow-up success rate and the satisfaction degree.
Description
Technical Field
The invention relates to the technical field of medical follow-up visits, in particular to an intelligent follow-up visit method and system based on multiple modes.
Background
The follow-up refers to an observation method for the hospital to regularly know the disease condition change of the patient and guide the patient to recover by communication or other means with the patient who has been in a visit in the hospital. The medical service level can be improved through follow-up visits, doctors can conveniently track and observe patients, first-hand data is mastered to carry out statistical analysis and experience accumulation, development of medical scientific research work and improvement of the service level of medical workers are facilitated, and therefore the medical service system can better serve the patients.
The follow-up is the work that a hospital contacts a patient after diagnosis and treatment or requires the patient to go to the hospital for reexamination regularly according to the needs of medical treatment, scientific research and teaching, and continuously tracks and observes the curative effect and the development condition of the disease of the patient, and is also called follow-up (follow-up). In short, the patient is continuously tracked and visited after treatment.
The current follow-up system basically adopts 2 methods, namely 1) filling of a paper scale and 2) a telephone mode. The method 1 needs manual filling by a user on line, the working efficiency is low, meanwhile, the statistical analysis of the paper files needs a large amount of workload and a large amount of storage space, and the subsequent traceability work needs manual file retrieval, so that the user requirements under the development condition of the mobile internet cannot be met. The telephone dialing mode has the problems that a follow-up staff is required to manually fill in an electronic scale eCRT, the electronic scale eCRT and a recording file are required to be manually associated, and the like, and has the defect of incomplete data in clinical research.
Chinese patent CN201710121626.1 discloses a patient follow-up system based on wechat public numbers, belonging to the technical field of electronic information management, comprising: the patient terminals are used for providing access to the WeChat public number for the patient, browsing the follow-up information and asking questions in a message board under the page where the follow-up information is located; the plurality of doctor terminals are used for providing doctors with access to the WeChat public number and performing instant reply on questions; the hospital management platform, with little letter public account information interaction, include: the information pushing module is used for pushing follow-up information to the WeChat public number; and the information interaction module is used for receiving the questions and sending the questions and the patient wechat account numbers related to the questions to the corresponding doctor terminals through the wechat public numbers. The beneficial effects of the above technical scheme are: the health propaganda and education knowledge is pushed to the WeChat public number, the patient can browse by himself and can communicate with the doctor on line in real time through the WeChat platform, and real-time, efficient and convenient follow-up service is provided for the patient. However, the data source of the scheme is single, and the reliability of the evidence chain data is not high.
Therefore, in order to solve the above problems, it is urgently needed to invent a new multi-modal based intelligent follow-up method and system.
Disclosure of Invention
The invention aims to: the method and the system adopt artificial intelligence multi-mode technology (a plurality of modes such as audio, video, text and the like) and utilize mobile internet technology to realize audio and video communication and automatic filling of an electronic scale eCRF on an APP so as to realize intelligent follow-up.
The invention provides the following scheme:
an intelligent follow-up method based on multiple modes comprises the following steps:
opening a follow-up client, establishing a video call connection, and starting follow-up;
after the video call is connected, positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching;
after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient;
during the follow-up visit, the follow-up visit client acquires audio and video in real time and transmits the audio and video to the follow-up visit platform for recording;
the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video;
extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching;
and after all follow-up questions are answered, the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database.
Preferably, the multimodal based intelligent follow-up method further comprises:
after the eCRF data set is generated and saved, the evidence chain information of the eCRF data set is continuously updated.
Preferably, the step of converting and analyzing the text content in real time while recording the audio and video includes: automatically filling the answer when the analyzed result can be matched with the answer of the eCRF question, otherwise, judging whether the patient needs to repeat once by the follow-up staff, and if the patient does not need to repeat, automatically filling the answer by the follow-up staff; the automatically filled answers require the accuracy to be checked by the visitors, and if incorrect, manually corrected, the final audio content and eCRF scale results are also handed to ASR and NLP for learning and training.
Preferably, the chain of evidence information is divided into a plurality of types: the method comprises the following steps:
after the follow-up answer is finished, the follow-up platform actively pushes a satisfaction survey questionnaire to the APP of the patient, and the patient autonomously fills, submits and updates the satisfaction survey questionnaire to an evidence chain of an eCRF data set;
after the follow-up answer is finished, the audio and video original file recorded by the streaming media is stored in a specified storage server, and the storage path is updated to an evidence chain of an eCRF data set;
the emotional state number of the patient during the follow-up period is updated to an evidence chain of an eCRF data set after the follow-up answer is finished;
and the follow-up positioning coordinate and specific address data of the patient are updated into an evidence chain of the eCRF data set after the follow-up answer is finished.
Further, the invention also provides an intelligent follow-up system based on multiple modes, which comprises:
the follow-up client is used for acquiring follow-up data of the patient;
the follow-up platform is used for follow-up data management, intelligent data analysis and data structured standardization.
Preferably, the follow-up platform comprises:
the audio and video intelligent analysis subsystem is used for extracting the facial features in each frame and identifying the expressions in the recording process of the video, calculating the emotional state in a window period and caching the emotional state;
and the follow-up table eCRF recording subsystem is used for converting and analyzing the text content in real time during the recording process of the audio and video.
Preferably, the follow-up platform further comprises:
and the structuring and standardizing subsystem is used for generating an eCRF data set after all follow-up questions are answered, automatically performing structuring and standardizing treatment and storing the eCRF data set in a database.
Preferably, the follow-up platform further comprises:
a data storage subsystem for storing the generated eCRF data set;
and the data integration subsystem is used for continuously updating the evidence chain information of the eCRF data set after the eCRF data set is generated and stored.
Preferably, the follow-up platform further comprises:
the service routing gateway is used for realizing wireless communication between the follow-up client and the follow-up platform;
and the audio and video stream transmission subsystem is used for transmitting the audio and video acquired by the follow-up client in real time to the follow-up platform for recording during the follow-up period.
Preferably, the follow-up client, the service routing gateway, the audio and video stream transmission subsystem, the audio and video intelligent analysis subsystem, the follow-up table eCRF logging subsystem, the structuring and standardization subsystem, the data storage subsystem and the data integration subsystem are electrically connected in sequence.
The invention has the following beneficial effects:
the invention discloses a multi-mode-based intelligent follow-up method and a system, wherein the method comprises the following steps: opening a follow-up client, establishing a video call connection, and starting follow-up; after the video call is connected, positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching; after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient; during the follow-up visit, the follow-up visit client acquires audio and video in real time and transmits the audio and video to the follow-up visit platform for recording; the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video; extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching; after all follow-up questions are answered, the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database; the on-line intelligent follow-up visit by the internet technology is relied on, so that the follow-up visit efficiency is greatly improved; the fusion work of audio, video and text information is realized, flexible selection and combination can be realized according to the situation, and the follow-up success rate and the satisfaction degree are improved; enabling the AI, and improving the working efficiency of the follow-up staff; the eCRF follow-up table is structured and normalized, so that the analysis of clinical data and subsequent scientific research are facilitated; enabling AI, mastering the experience and emotion of a patient in operation and use, and pre-judging the falling probability of the patient; by combining the video and the positioning function of the mobile phone, the real follow-up environment is recorded while the prognosis condition of the patient is recorded, so that the scientific research requirement of the real world is met; the trace is left in the whole process, the source can be traced, the follow-up records form a completed data evidence chain, and the medical supervision requirements of the state are met.
Drawings
FIG. 1 is a flow chart of the multi-modal based intelligent follow-up method of the present invention.
Fig. 2 is a block diagram of the multi-modal based intelligent follow-up system of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Referring to fig. 1, an intelligent multi-modality based follow-up method includes the following steps:
opening a follow-up client, establishing a video call connection, and starting follow-up;
after the video call is connected, positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching;
after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient;
during the follow-up visit, the follow-up visit client acquires audio and video in real time and transmits the audio and video to the follow-up visit platform for recording;
the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video;
extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching;
and after all follow-up questions are answered, the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database.
The intelligent multi-modal-based follow-up method further comprises the following steps:
after the eCRF data set is generated and saved, the evidence chain information of the eCRF data set is continuously updated.
Preferably, the step of converting and analyzing the text content in real time while recording the audio and video includes: automatically filling the answer when the analyzed result can be matched with the answer of the eCRF question, otherwise, judging whether the patient needs to repeat once by the follow-up staff, and if the patient does not need to repeat, automatically filling the answer by the follow-up staff; the automatically filled answers require the accuracy to be checked by the visitors, and if incorrect, manually corrected, the final audio content and eCRF scale results are also handed to ASR and NLP for learning and training.
Evidence chain information is divided into a variety of categories: the method comprises the following steps:
after the follow-up answer is finished, the follow-up platform actively pushes a satisfaction survey questionnaire to the APP of the patient, and the patient autonomously fills, submits and updates the satisfaction survey questionnaire to an evidence chain of an eCRF data set;
after the follow-up answer is finished, the audio and video original file recorded by the streaming media is stored in a specified storage server, and the storage path is updated to an evidence chain of an eCRF data set;
the emotional state number of the patient during the follow-up period is updated to an evidence chain of an eCRF data set after the follow-up answer is finished;
and the follow-up positioning coordinate and specific address data of the patient are updated into an evidence chain of the eCRF data set after the follow-up answer is finished.
Referring to fig. 2, a multimodal based intelligent follow-up system includes:
the follow-up client is used for acquiring follow-up data of the patient;
the follow-up platform is used for follow-up data management, intelligent data analysis and data structured standardization.
Preferably, the follow-up platform comprises:
the audio and video intelligent analysis subsystem is used for extracting the facial features in each frame and identifying the expressions in the recording process of the video, calculating the emotional state in a window period and caching the emotional state;
and the follow-up table eCRF recording subsystem is used for converting and analyzing the text content in real time during the recording process of the audio and video.
The follow-up platform further comprises:
and the structuring and standardizing subsystem is used for generating an eCRF data set after all follow-up questions are answered, automatically performing structuring and standardizing treatment and storing the eCRF data set in a database.
The follow-up platform further comprises:
a data storage subsystem for storing the generated eCRF data set;
and the data integration subsystem is used for continuously updating the evidence chain information of the eCRF data set after the eCRF data set is generated and stored.
The follow-up platform further comprises:
the service routing gateway is used for realizing wireless communication between the follow-up client and the follow-up platform;
and the audio and video stream transmission subsystem is used for transmitting the audio and video acquired by the follow-up client in real time to the follow-up platform for recording during the follow-up period.
The follow-up client, the service routing gateway, the audio and video stream transmission subsystem, the audio and video intelligent analysis subsystem, the follow-up table eCRF recording subsystem, the structuring and standardization subsystem, the data storage subsystem and the data integration subsystem are electrically connected in sequence.
In the embodiment, the intelligent follow-up method and system based on multiple modes comprises the following steps: opening a follow-up client, establishing a video call connection, and starting follow-up; after the video call is connected, positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching; after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient; during the follow-up visit, the follow-up visit client acquires audio and video in real time and transmits the audio and video to the follow-up visit platform for recording; the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video; extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching; after all follow-up questions are answered, the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database; the on-line intelligent follow-up visit by the internet technology is relied on, so that the follow-up visit efficiency is greatly improved; the fusion work of audio, video and text information is realized, flexible selection and combination can be realized according to the situation, and the follow-up success rate and the satisfaction degree are improved; enabling the AI, and improving the working efficiency of the follow-up staff; the eCRF follow-up table is structured and normalized, so that the analysis of clinical data and subsequent scientific research are facilitated; enabling AI, mastering the experience and emotion of a patient in operation and use, and pre-judging the falling probability of the patient; by combining the video and the positioning function of the mobile phone, the real follow-up environment is recorded while the prognosis condition of the patient is recorded, so that the scientific research requirement of the real world is met; the trace is left in the whole process, the source can be traced, the follow-up records form a completed data evidence chain, and the medical supervision requirements of the state are met.
In the multimode-based intelligent follow-up method and system, through the adoption of an artificial intelligence multimode technology (multiple modalities such as audio, video and text), the mobile internet technology is utilized, audio and video communication and automatic filling of an electronic scale eCRF are realized on an APP, and intelligent follow-up is realized. The follow-up object communicates with follow-up staff through audio and video, the audio and video assists the seat staff to automatically fill and perfect electronic eCRF through a speech semantic recognition understanding technology at the cloud, facial expressions of patients in videos are recognized through a convolutional neural network to achieve emotion change tracking of the patients, and expression recognition refers to selecting expression states (six basic expressions of human beings, namely happiness, anger, surprise, fear, disgust and sadness) from static photos or video sequences, so that emotion and psychological changes of people are determined. The whole process of the follow-up method is adopted to record the trace, so that the integrity and traceability of the final eCRF data set are ensured.
Compared with the traditional follow-up mode, the multi-mode-based intelligent follow-up method and system in the embodiment have the following advantages: 1) on-line intelligent follow-up by relying on the internet technology, the follow-up efficiency is greatly improved, 2) the fusion work of audio, video and text information is realized, the follow-up efficiency and the follow-up satisfaction are improved, 3) the enabling of AI is improved, 4) the working efficiency of the follow-up staff is improved, the structuring and the standardization of an eCRT follow-up table are facilitated, the analysis of clinical data and the enabling of follow-up scientific research 5) AI are facilitated, the experience and the mood of a patient in operation and use are mastered, the falling probability of the patient is pre-judged, 6) the real follow-up environment is recorded while the prognosis condition of the patient is recorded, the whole-course trace-leaving and traceability are met, the follow-up record forms a completed data evidence chain, and the medical supervision requirement of the state is met.
In this embodiment, the multimodal-based intelligent follow-up method and system includes a follow-up platform and an APP (follow-up client), the APP realizes data (text, audio, video, position, etc.) collection, and the follow-up platform realizes follow-up management, intelligent data analysis and data structured standardization.
In the multi-mode-based intelligent follow-up method and system, the APP is a follow-up terminal, as shown in fig. 2, the APP is downloaded and installed in a mobile phone before the patient is discharged from the hospital, and the APP can be registered by the patient through the mobile phone number registered by the patient in the hospital due to interconnection and intercommunication between the follow-up platform and an internal system of the hospital.
In the multi-modal-based intelligent follow-up method and system of the present embodiment, the follow-up platform is a background integrated system integrating management and analysis, and includes a series of subsystems, as shown in fig. 2. The follow-up platform is integrated with the hospital system through the data integration subsystem, and the patient information in the hospital system is synchronized; there are two ways to communicate with APP: 1) the method adopts a streaming media technology and APP to realize real-time video call 2) externally provides a standard restful API interface depending on the Internet and the mobile Internet technology and communicates with the APP by an HTTPS protocol (HTTP communication protocol based on SSL encryption). The follow-up platform adopts a distributed architecture design of SpringCloud micro-services, and has low coupling degree between services, high availability and strong expansibility.
In the multi-modal-based intelligent follow-up method and system of the present embodiment, the multi-modal-based intelligent follow-up implementation method is shown in the flowchart of fig. 1:
the follow-up process is started when the follow-up staff and the patient communicate in advance for a determined time, the follow-up staff firstly opens a required eCRF follow-up scale in a follow-up platform and calls the patient through a cloud video function, the patient opens the APP after receiving an APP video call request, the two parties can establish video call connection, and follow-up is started;
after the video call is connected, the APP calls a positioning function every 1 minute, sends the coordinate information of the patient to the rear end of the follow-up platform and converts the coordinate information into a specific address for caching;
after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient;
during the period, the APP and the background acquire audio and video in real time through a streaming media transmission technology and transmit the audio and video to a server for recording;
the audio and video are converted into text contents through a real-time speech recognition technology (ASR) during the recording process and are delivered to a natural language processing technology (NLP) shown in a flow 304 for analysis, when the analysis result can be matched with an eCRF question answer, the answer is automatically filled, otherwise, a follow-up staff judges whether the patient needs to repeat once, and if the patient does not need to repeat, the follow-up staff automatically fills the answer; the automatically filled answers need to be checked for accuracy by a follow-up staff, if the answers are incorrect, the answers are corrected manually, and the final audio content and the eCRT scale result are also handed to ASR and NLP for learning and training;
extracting facial features in each frame and recognizing expressions through a Convolutional Neural Network (CNN) in the recording process of the video, calculating the emotional state in a window period (1 second) and caching;
after all follow-up questions are answered, the follow-up staff click to finish follow-up, at the moment, an eCRF data set is generated by the follow-up platform, automatically structured and standardized, and then stored in a database (as shown in a 306 flow);
after the eCRF data set is generated and stored, the evidence chain information of the eCRF data set needs to be continuously updated (as shown in the 307 flow), and the evidence is divided into a plurality of types:
the follow-up visit satisfaction survey data comprise follow-up visit satisfaction survey data, wherein after a follow-up visit answer is finished, a follow-up visit platform actively pushes a satisfaction survey questionnaire to an APP of a patient, and the follow-up visit survey questionnaire is automatically filled, submitted and updated to an evidence chain of an eCRF data set by the patient;
after the follow-up answer is finished, the audio and video original file recorded by the streaming media is stored in a specified storage server, and a storage path is updated to an evidence chain of an eCRF data set;
the emotional state data of the patient during the follow-up visit is updated to an evidence chain of an eCRF data set after the follow-up visit is answered;
and the follow-up positioning coordinate and specific address data of the patient are updated into an evidence chain of the eCRF data set after the follow-up answer is finished.
After the above procedures are completed, a complete follow-up is finished, and the eCRF data set and the evidence chain are complete and effective
According to the multi-modality-based intelligent follow-up method and system, the clinical scientific research projects in the real world have strict requirements on authenticity, integrity, accuracy and traceability of follow-up data, and also have strict requirements on compliance, environment and follow-up time of patients, a large amount of manual intervention is required in the traditional follow-up mode, the real environment of the patients cannot be accurately known, and the data is difficult to be widely recognized. The multi-mode intelligent follow-up system used by the invention utilizes the combination of the mobile internet technology and the artificial intelligence, thereby greatly reducing the workload of follow-up personnel and patients and reducing the probability of occurrence of human errors. The method comprises the following steps of obtaining position information of a patient by using GPS positioning at the beginning and the end of follow-up visit, comparing to obtain a real follow-up visit position, converting answers of the patient into text contents by using a speech recognition technology (ASR) in the follow-up visit answering process, automatically filling or selecting answers of questions by using Natural Language Processing (NLP), and simultaneously decomposing a video stream into frame pictures to recognize six basic expressions of the face by using a Convolutional Neural Network (CNN): happiness, anger, surprise, fear, disgust and sadness, forming mood change data. The follow-up platform automatically pushes a satisfaction questionnaire to be filled out and submitted by the patient after the follow-up answer is finished. Finally, the audio and video original file, the positioning coordinate and address data, the emotion change data, the follow-up visit satisfaction and the eCRF data set are associated and then stored to form a complete data evidence chain, so that the authenticity, integrity, accuracy and traceability of the data are guaranteed, and the follow-up visit requirement of the real world is met.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. An intelligent follow-up method based on multiple modes is characterized in that: the method comprises the following steps:
opening a follow-up client, establishing a video call connection, and starting follow-up;
after the video call is connected, positioning the patient according to a preset time interval and sending the coordinate information of the patient to a follow-up platform for caching;
after the follow-up visit is started, the follow-up staff reads questions in sequence and answers by the patient;
during the follow-up visit, the follow-up visit client acquires audio and video in real time and transmits the audio and video to the follow-up visit platform for recording;
the method comprises the following steps of (1) converting text contents in real time and analyzing the text contents while recording the audio and video;
extracting facial features and recognizing expressions in the video recording process, calculating emotional states and caching;
and after all follow-up questions are answered, the follow-up platform generates an eCRF data set, automatically performs structuring and standardization processing, and then stores the eCRF data set in a database.
2. The multimodal based intelligent follow-up method according to claim 1, wherein: further comprising:
after the eCRF data set is generated and saved, the evidence chain information of the eCRF data set is continuously updated.
3. The multimodal based intelligent follow-up method according to claim 2, wherein: the method comprises the following steps of converting text contents in real time and analyzing the text contents while recording audio and video, and specifically comprises the following steps: automatically filling the answer when the analyzed result can be matched with the answer of the eCRF question, otherwise, judging whether the patient needs to repeat once by the follow-up staff, and if the patient does not need to repeat, automatically filling the answer by the follow-up staff; the automatically filled answers require the accuracy to be checked by the visitors, and if incorrect, manually corrected, the final audio content and eCRF scale results are also handed to ASR and NLP for learning and training.
4. The multimodal based intelligent follow-up method according to claim 3, wherein: evidence chain information is divided into a variety of categories: the method comprises the following steps:
after the follow-up answer is finished, the follow-up platform actively pushes a satisfaction survey questionnaire to the APP of the patient, and the patient autonomously fills, submits and updates the satisfaction survey questionnaire to an evidence chain of an eCRF data set;
after the follow-up answer is finished, the audio and video original file recorded by the streaming media is stored in a specified storage server, and the storage path is updated to an evidence chain of an eCRF data set;
the emotional state number of the patient during the follow-up period is updated to an evidence chain of an eCRF data set after the follow-up answer is finished;
and the follow-up positioning coordinate and specific address data of the patient are updated into an evidence chain of the eCRF data set after the follow-up answer is finished.
5. An intelligent follow-up system based on multiple modes is characterized in that: the method comprises the following steps:
the follow-up client is used for acquiring follow-up data of the patient;
the follow-up platform is used for follow-up data management, intelligent data analysis and data structured standardization.
6. The multimodal based intelligent follow-up system according to claim 5, wherein: the follow-up platform comprises:
the audio and video intelligent analysis subsystem is used for extracting the facial features in each frame and identifying the expressions in the recording process of the video, calculating the emotional state in a window period and caching the emotional state;
and the follow-up table eCRF recording subsystem is used for converting and analyzing the text content in real time during the recording process of the audio and video.
7. The multimodal based intelligent follow-up system according to claim 6, wherein: the follow-up platform further comprises:
and the structuring and standardizing subsystem is used for generating an eCRF data set after all follow-up questions are answered, automatically performing structuring and standardizing treatment and storing the eCRF data set in a database.
8. The multimodal based intelligent follow-up system according to claim 7, wherein: the follow-up platform further comprises:
a data storage subsystem for storing the generated eCRF data set;
and the data integration subsystem is used for continuously updating the evidence chain information of the eCRF data set after the eCRF data set is generated and stored.
9. The multimodal based intelligent follow-up system according to claim 8, wherein: the follow-up platform further comprises:
the service routing gateway is used for realizing wireless communication between the follow-up client and the follow-up platform;
and the audio and video stream transmission subsystem is used for transmitting the audio and video acquired by the follow-up client in real time to the follow-up platform for recording during the follow-up period.
10. The multimodal based intelligent follow-up system according to claim 9, wherein: the follow-up client, the service routing gateway, the audio and video stream transmission subsystem, the audio and video intelligent analysis subsystem, the follow-up table eCRF recording subsystem, the structuring and standardization subsystem, the data storage subsystem and the data integration subsystem are electrically connected in sequence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910897892.2A CN110675922A (en) | 2019-09-23 | 2019-09-23 | Multi-mode-based intelligent follow-up method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910897892.2A CN110675922A (en) | 2019-09-23 | 2019-09-23 | Multi-mode-based intelligent follow-up method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110675922A true CN110675922A (en) | 2020-01-10 |
Family
ID=69077466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910897892.2A Pending CN110675922A (en) | 2019-09-23 | 2019-09-23 | Multi-mode-based intelligent follow-up method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110675922A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541336A (en) * | 2020-12-01 | 2021-03-23 | 华迪计算机集团有限公司 | Data reporting method and management system |
| CN113421630A (en) * | 2021-06-02 | 2021-09-21 | 沃民高新科技（北京）股份有限公司 | Intelligent management method and system for physical and mental health |
| CN114510491A (en) * | 2022-04-19 | 2022-05-17 | 杭州华卓信息科技有限公司 | Dynamic follow-up quantity table design method and system |
| CN115292458A (en) * | 2022-06-29 | 2022-11-04 | 北京梦天门科技股份有限公司 | Investigation information input method and system and electronic equipment |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102299946A (en) * | 2011-04-28 | 2011-12-28 | 大连亿创天地科技发展有限公司 | Network-based comprehensive video diagnosis and treatment supervisory system |
| US20120253848A1 (en) * | 2011-04-04 | 2012-10-04 | Ihas Inc. | Novel approach to integrate and present disparate healthcare applications in single computer screen |
| CN107845422A (en) * | 2017-11-23 | 2018-03-27 | 郑州大学第附属医院 | A kind of remote medical consultation with specialists session understanding and method of abstracting based on the fusion of multi-modal clue |
| CN108388649A (en) * | 2018-02-28 | 2018-08-10 | 深圳市科迈爱康科技有限公司 | Handle method, system, equipment and the storage medium of audio and video |
| CN109478425A (en) * | 2016-05-05 | 2019-03-15 | 迈克尔·江 | The system and method for full point Telemedicine Consultation on demand |
| CN109767845A (en) * | 2018-12-18 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Medical follow up method, apparatus, computer equipment and storage medium |
| CN110164564A (en) * | 2019-05-31 | 2019-08-23 | 贵州精准健康数据有限公司 | The multidisciplinary consultation of doctors platform of MDT |
| CN110189826A (en) * | 2019-05-16 | 2019-08-30 | 苏州好博医疗器械有限公司 | A kind of Measuring scale assessing system and method based on interactive software operating platform |
| CN110211704A (en) * | 2019-05-05 | 2019-09-06 | 平安科技（深圳）有限公司 | The engine method and server of matter of opening |
-
2019
- 2019-09-23 CN CN201910897892.2A patent/CN110675922A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120253848A1 (en) * | 2011-04-04 | 2012-10-04 | Ihas Inc. | Novel approach to integrate and present disparate healthcare applications in single computer screen |
| CN102299946A (en) * | 2011-04-28 | 2011-12-28 | 大连亿创天地科技发展有限公司 | Network-based comprehensive video diagnosis and treatment supervisory system |
| CN109478425A (en) * | 2016-05-05 | 2019-03-15 | 迈克尔·江 | The system and method for full point Telemedicine Consultation on demand |
| CN107845422A (en) * | 2017-11-23 | 2018-03-27 | 郑州大学第附属医院 | A kind of remote medical consultation with specialists session understanding and method of abstracting based on the fusion of multi-modal clue |
| CN108388649A (en) * | 2018-02-28 | 2018-08-10 | 深圳市科迈爱康科技有限公司 | Handle method, system, equipment and the storage medium of audio and video |
| CN109767845A (en) * | 2018-12-18 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Medical follow up method, apparatus, computer equipment and storage medium |
| CN110211704A (en) * | 2019-05-05 | 2019-09-06 | 平安科技（深圳）有限公司 | The engine method and server of matter of opening |
| CN110189826A (en) * | 2019-05-16 | 2019-08-30 | 苏州好博医疗器械有限公司 | A kind of Measuring scale assessing system and method based on interactive software operating platform |
| CN110164564A (en) * | 2019-05-31 | 2019-08-23 | 贵州精准健康数据有限公司 | The multidisciplinary consultation of doctors platform of MDT |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541336A (en) * | 2020-12-01 | 2021-03-23 | 华迪计算机集团有限公司 | Data reporting method and management system |
| CN113421630A (en) * | 2021-06-02 | 2021-09-21 | 沃民高新科技（北京）股份有限公司 | Intelligent management method and system for physical and mental health |
| CN114510491A (en) * | 2022-04-19 | 2022-05-17 | 杭州华卓信息科技有限公司 | Dynamic follow-up quantity table design method and system |
| CN115292458A (en) * | 2022-06-29 | 2022-11-04 | 北京梦天门科技股份有限公司 | Investigation information input method and system and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110675922A (en) | Multi-mode-based intelligent follow-up method and system | |
| US10354051B2 (en) | Computer assisted patient navigation and information systems and methods | |
| US11033216B2 (en) | Augmenting questionnaires | |
| KR102327669B1 (en) | Mental care system using artificial intelligence | |
| CN104598729A (en) | Online doctor-patient interaction system and interaction method based on health management | |
| CN110890161A (en) | Ultrasonic remote real-time consultation system | |
| US20210287800A1 (en) | Ai supported personalized, natural language-based patient interface for medical-bot | |
| CN111863252A (en) | Health monitoring method, system, computer equipment and storage medium | |
| CN108648787A (en) | Medical follow up method and apparatus | |
| CN116028603A (en) | Intelligent pre-consultation method, device and system based on big data, and storage medium | |
| CN108564993A (en) | Electronic health record based on block chain technology integrates translation system automatically | |
| CN111710396A (en) | System and method for controlling intra-medical cataract patient | |
| US11322139B2 (en) | Presentation of indications with respect to questions of a communication session | |
| CN110867257A (en) | Remote consultation system | |
| CN111063455A (en) | Human-computer interaction method and device for telemedicine | |
| CN113362942A (en) | Pediatric critical care medical cloud platform based on regional hospital model | |
| CN103167030B (en) | A kind of relation in communication system detects and relation sets up system and method | |
| CN111815905A (en) | Wearable terminal device based on intelligent endowment system and control method thereof | |
| US20200125647A1 (en) | Determination of biorhythms through video journal services | |
| CN110993122A (en) | Medical health information management system based on cloud computing | |
| US20190332899A1 (en) | Analysis of image media corresponding to a communication session | |
| CN114882983A (en) | Smart ward interaction method, system and storage medium | |
| CN114822738A (en) | Patient intestinal tract preparation condition evaluation system and method based on mobile phone terminal | |
| CN115204127B (en) | Form filling method, device, equipment and medium based on remote flow adjustment | |
| US20190332657A1 (en) | Automated linking of media data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200110 |
|
| RJ01 | Rejection of invention patent application after publication |