WO2023005353A1

WO2023005353A1 - Configuration information acquisition apparatus based on multi-modal data, and related device

Info

Publication number: WO2023005353A1
Application number: PCT/CN2022/092818
Authority: WO
Inventors: 王倩; 唐建东
Original assignee: 苏州景昱医疗器械有限公司
Priority date: 2021-07-30
Filing date: 2022-05-13
Publication date: 2023-02-02
Also published as: CN113577555A

Abstract

A configuration information acquisition apparatus and method based on multi-modal data, and an electronic device, a program control system and a computer-readable storage medium. The apparatus comprises: a bioelectricity acquisition module, which is used for acquiring real-time bioelectricity data of a patient; an image acquisition module, which is used for photographing the patient by using a camera, so as to obtain real-time image data of the patient; a state acquisition module, which is used for acquiring real-time state information of the patient on the basis of the real-time bioelectricity data and the real-time image data of the patient; and a configuration acquisition module, which is used for acquiring configuration information of the patient on the basis of the real-time state information of the patient, wherein the configuration information of the patient is used for controlling a stimulator in the body of the patient to apply an electrical stimulus.

Description

Configuration information acquisition device and related equipment based on multimodal data

This application claims the priority of the Chinese patent with application number 202110874804.4 filed on July 30, 2021, which is incorporated by reference in its entirety.

technical field

The present application relates to the technical field of implantable medical equipment, and in particular relates to a configuration information acquisition device, method, electronic equipment, program control system and computer-readable storage medium based on multimodal data.

Background technique

In recent years, implantable neurostimulators have become more and more widely used in clinical practice. During the treatment process with implanted neurostimulators, doctors can remotely control related electronic equipment, and then control the stimulators to exert pressure on patients. Corresponding electrical stimulation achieves the purpose of treating diseases.

However, if the doctor only observes the patient's state through video and adjusts the parameters when performing remote program control on the patient, it is inevitable that the patient's condition will be inaccurate and electrical stimulation cannot be correctly applied to the patient, especially for implanted neurons. As far as the stimulator is concerned, it directly acts on the nerves in the user's body by electrical stimulation. Once it fails, it may cause irreversible damage to the human body and bring risks to the patient. Therefore, it provides a way to obtain the bioelectric data of the patient. It is very important to grasp the real-time status information of the patient and to accurately apply electrical stimulation to the patient.

Contents of the invention

The purpose of this application is to provide configuration information acquisition device, method, electronic equipment, program control system and computer-readable storage medium based on multi-modal data, based on the real-time bioelectric data and real-time image data of the patient, to obtain the real-time state information of the patient, Further, the configuration information of the patient is obtained to control the stimulator in the patient to apply electrical stimulation.

In the first aspect, the present application provides a configuration information acquisition method based on multimodal data, the method comprising: acquiring real-time bioelectric data of the patient, the bioelectric data including real-time EEG data, real-time ECG data and One or more of the real-time myoelectric data; using a camera to photograph the patient to obtain real-time image data of the patient; based on the real-time bioelectric data and real-time image data of the patient, to obtain real-time status information of the patient ; Based on the real-time state information of the patient, acquire configuration information of the patient, where the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation. In some optional embodiments, the acquiring real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient includes: displaying the real-time bioelectric data and real-time image data; using the doctor equipment to receive a status input operation; in response to the status input operation, determine the real-time status information of the patient.

In some optional embodiments, the acquiring the real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient may include: combining the real-time bioelectric data and real-time image data of the patient Input the state classification model to obtain the real-time state information of the patient.

In some optional embodiments, the method for obtaining the state classification model may include: obtaining training data of a plurality of state sample objects, the training data of each state sample object includes real-time bioelectrical data of the state sample object and real-time image data and corresponding real-time state information, the real-time state information corresponding to real-time bioelectric data and real-time image data; using the training data of the plurality of state sample objects to train the first deep learning model to obtain the state Classification model: input the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain the real-time state information of the patient.

In some optional embodiments, the acquiring configuration information of the patient based on the real-time status information of the patient includes: using a doctor device to display the real-time status information of the patient; using the doctor device to receive configuration input Operation: determining configuration information for the patient in response to the configuration input operation.

In some optional embodiments, the acquiring configuration information of the patient based on the real-time status information of the patient includes: inputting the real-time status information of the patient into a configuration acquisition model to obtain the configuration information of the patient .

In some optional embodiments, the method for acquiring the configuration acquisition model includes: acquiring training data of multiple configuration sample objects, the training data of each configuration sample object includes the real-time status information of the configuration sample object and its corresponding configuration information; using the training data of the plurality of configuration sample objects to train a second deep learning model to obtain the configuration acquisition model; inputting the real-time state information of the patient into the configuration acquisition model to obtain the patient's configuration information.

In the second aspect, the present application provides a device for obtaining configuration information based on multimodal data, the device comprising: a bioelectricity acquisition module for acquiring real-time bioelectricity data of a patient, the bioelectricity data including real-time EEG One or more of data, real-time electrocardiogram data and real-time myoelectric data; image acquisition module, for using camera to shoot described patient, obtains the real-time image data of described patient; State acquisition module, is used for based on described The real-time bioelectric data and real-time image data of the patient are used to acquire the real-time status information of the patient; the configuration acquisition module is used to acquire the configuration information of the patient based on the real-time status information of the patient, and the configuration information of the patient is used Electrical stimulation is applied by controlling a stimulator inside the patient.

In some optional embodiments, the state acquisition module includes: a first display unit, which uses the doctor equipment to display the real-time bioelectric data and real-time image data of the patient; a first receiving unit, which uses the doctor equipment to receive the status input operation; a state determination unit, in response to the state input operation, to determine the real-time state information of the patient.

In some optional embodiments, the state acquisition module is configured to input the real-time bioelectrical data and real-time image data of the patient into a state classification model to obtain real-time state information of the patient.

In some optional embodiments, the state acquisition module includes: a state sample unit, configured to obtain training data of a plurality of state sample objects, the training data of each state sample object includes the real-time bioelectricity of the state sample object data and real-time image data and corresponding real-time state information; the first training unit is used to use the training data of the plurality of state sample objects to train the first deep learning model to obtain the state classification model; the acquisition state unit is used to Inputting the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain real-time state information of the patient.

In some optional embodiments, the configuration acquisition module includes: a second display unit, configured to use the doctor device to display the real-time status information of the patient; a second receiving unit, configured to use the doctor device to receive configuration input Operation: a configuration determining unit configured to determine configuration information of the patient in response to the configuration input operation.

In some optional embodiments, the configuration acquisition module is configured to input the real-time status information of the patient into a configuration acquisition model to obtain the configuration information of the patient.

In some optional embodiments, the configuration obtaining module includes: a configuration sample unit, configured to obtain training data of a plurality of configuration sample objects, the training data of each configuration sample object includes real-time state information of the configuration sample object and the corresponding configuration information; the second training unit is used to train the second deep learning model by using the training data of the plurality of configuration sample objects to obtain the configuration acquisition model; the acquisition configuration unit is used to convert the patient’s The real-time status information is input into the configuration acquisition model to obtain the configuration information of the patient.

In a third aspect, the present application provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor includes:

The bioelectricity acquisition module is used to acquire the real-time bioelectricity data of the patient, and the bioelectricity data includes one or more of real-time EEG data, real-time ECG data and real-time EMG data;

An image acquisition module, configured to use a camera to photograph the patient to obtain real-time image data of the patient;

A status acquisition module, configured to acquire real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient;

The configuration obtaining module is configured to obtain configuration information of the patient based on the real-time status information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.

In some optional embodiments, the electronic device is also provided with a display screen.

In a fourth aspect, the present application provides a program-controlled system, which includes doctor equipment and the electronic equipment described in the third aspect.

In a fifth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Acquiring real-time bioelectric data of the patient, said bioelectric data including one or more of real-time EEG data, real-time ECG data and real-time myoelectric data;

Using a camera to photograph the patient to obtain real-time image data of the patient;

acquiring real-time state information of the patient based on the real-time bioelectric data and real-time image data of the patient;

Based on the real-time status information of the patient, configuration information of the patient is acquired, and the configuration information of the patient is used to control a stimulator inside the patient to apply electrical stimulation.

Using the configuration information acquisition device, method, electronic equipment, program control system and computer-readable storage medium based on multi-modal data provided by this application has at least the following advantages:

By obtaining one or more of the patient's bioelectric data, real-time EEG data, real-time ECG data, and real-time EMG data, combined with the real-time image data captured by the camera, the real-time status information of the patient is obtained, and then real-time The state information is further obtained with configuration information. Since the obtained patient information is real-time, this method has timeliness; at the same time, the obtained patient information can be various, so it is comprehensive; therefore, using this configuration information to control The stimulator inside the patient applies electrical stimulation, which is more reliable.

Description of drawings

The application will be further described below in conjunction with the accompanying drawings and embodiments.

FIG. 1 is a schematic flowchart of a method for acquiring configuration information based on multimodal data provided in an embodiment of the present application;

Fig. 2 is a schematic flow chart of determining the real-time state information of a patient provided by the embodiment of the present application;

Fig. 3 is another schematic flow chart of determining the real-time state information of the patient provided by the embodiment of the present application;

Fig. 4 is a schematic flow chart of determining configuration information of a patient provided by an embodiment of the present application;

Fig. 5 is a schematic flow diagram of another method for determining configuration information of a patient provided by the embodiment of the present application;

FIG. 6 is a schematic structural diagram of an apparatus for acquiring configuration information based on multimodal data provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a status acquisition module provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of another state acquisition module provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a configuration acquisition module provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of another configuration acquisition module provided by an embodiment of the present application;

Fig. 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Fig. 12 is a schematic structural diagram of a program product for implementing a method for acquiring configuration information based on multimodal data provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In this application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b or c can represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be It can be single or multiple. It should be noted that "at least one item (item)" can also be interpreted as "one item (item) or multiple items (item)".

It should be noted that, on the premise of no conflict, the various embodiments or technical features described below can be combined arbitrarily to form new embodiments.

In this application, words such as "exemplary" or "for example" are used to mean an example, illustration or description. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

Referring to FIG. 1 , an embodiment of the present application provides a method for acquiring configuration information based on multimodal data, and the method includes steps S101-S104.

The method for acquiring configuration information based on multi-modal data provided by the embodiment of the present application can be applied to an online remote program control scenario, and can also be applied to an offline program control scenario.

Step S101: Acquire real-time bioelectric data of the patient, the bioelectric data includes one or more of real-time EEG data, real-time ECG data and real-time EMG data. Wherein, most of the information inside the human body is transmitted in the form of bioelectricity, and the bioelectricity data may also include one or more of real-time electrogastric data and real-time retinal electricity data.

Step S102: Using a camera to photograph the patient to obtain real-time image data of the patient. The camera is used to shoot the patient to obtain real-time image data, which complements the bioelectrical data and can more comprehensively reflect the patient's state.

Step S103: Acquire real-time status information of the patient based on the real-time bioelectrical data and real-time image data of the patient.

Step S104: Based on the real-time status information of the patient, acquire the configuration information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.

In the above steps of this application, by obtaining one or more of the patient's bioelectric data, real-time EEG data, real-time ECG data, and real-time EMG data, combined with the real-time image data captured by the camera, the real-time status information of the patient is obtained , and further obtain the configuration information from the real-time status information. Since the obtained patient information is real-time, this method has timeliness; at the same time, the obtained patient information can be various, so it is comprehensive; therefore Using this configuration information to control the stimulator in the patient's body to apply electrical stimulation has reliability.

The camera may include, for example, an optical camera and/or an infrared camera. The real-time image information of the patient may include video and/or images, for example. Through manual recognition or machine vision, based on real-time image information, the degree of completion of the patient's preset actions, the number of shaking of the patient's limbs (single hand or foot or multiple hands and feet) per minute, etc. can be obtained. Wherein, the preset motion may include, for example, one or more of finger pointing motions, arm stretching motions, arm raising motions, straight-line walking motions, and preset curve-walking motions.

In some possible implementations, step S103 may include: acquiring the patient's action information based on the patient's real-time image data; and acquiring the patient's real-time status based on the patient's real-time bioelectrical data and action information information. Wherein, the movement information of the patient can be used, for example, to indicate the degree of completion of the preset movement by the patient and the number of shakes per minute of the patient's limbs (single hand or feet or multiple hands and feet).

In some other possible implementation manners, step S103 may include: obtaining the first state information of the patient based on the real-time bioelectric data of the patient; obtaining the second state information of the patient based on the real-time image data of the patient. Status information: when it is detected that the first status information matches the second status information, use the first status information as the real-time status information of the patient.

The implantable neurostimulation system mainly includes a stimulator implanted in the body and a program-controlled device outside the body. The existing neuromodulation technology mainly uses stereotaxic surgery to implant electrodes in specific structures (i.e., targets) in the body, and the stimulator implanted in the patient sends electrical pulses to the targets through the electrodes to regulate the corresponding neural structures and networks. Electrical activity and its function, thereby improving symptoms and relieving pain.

Among them, the stimulator can be an implantable electrical nerve stimulation device, an implantable cardiac electrical stimulation system (also known as a cardiac pacemaker), an implantable drug infusion device (Implantable Drug Delivery System, referred to as IDDS) and a wire switch. any one of the connected devices. Implantable electrical nerve stimulation devices are, for example, Deep Brain Stimulation (DBS), Implantable Cortical Nerve Stimulation (CNS), Implantable Spinal Cord Stimulation , referred to as SCS), implanted sacral nerve stimulation system (Sacral Nerve Stimulation, referred to as SNS), implanted vagus nerve stimulation system (Vagus Nerve Stimulation, referred to as VNS), etc.

The stimulator can include IPG, extension wires and electrode wires. The IPG (implantable pulse generator, implantable pulse generator) is set in the patient's body, and relies on sealed batteries and circuits to provide controllable electrical stimulation to biological tissues. Extended lead wires and electrode leads provide one or two controllable specific electrical stimulations to specific areas of biological tissue. The extension lead is used in conjunction with the IPG as a transmission medium for the electrical stimulation signal, and transmits the electrical stimulation signal generated by the IPG to the electrode lead. The electrode lead releases the electrical stimulation signal generated by the IPG to a specific area of the biological tissue through multiple electrode contacts; the implantable medical device has one or more electrode leads on one or both sides, so The electrode wires are provided with a plurality of electrode contacts, and the electrode contacts can be arranged uniformly or non-uniformly in the circumferential direction of the electrode wires. As an example, the electrode contacts are arranged in an array of 4 rows and 3 columns (a total of 12 electrode contacts) in the circumferential direction of the electrode wire.

In some possible implementations, the stimulated biological tissue may be the patient's brain tissue, and the stimulated part may be a specific part of the brain tissue. When the patient's disease type is different, the stimulated site is generally different, the number of stimulation contacts used (single source or multi-source), one or more channels (single-channel or multi-channel) specific electrical stimulation signals The application and stimulus parameter data are also different. This application does not limit the applicable disease types, which may be the applicable disease types for deep brain stimulation (DBS), spinal cord stimulation (SCS), pelvic stimulation, gastric stimulation, peripheral nerve stimulation, and functional electrical stimulation. Among the types of disorders that DBS can be used to treat or manage include, but are not limited to: spasticity disorders (e.g., epilepsy), pain, migraine, psychiatric disorders (e.g., major depressive disorder (MDD)), bipolar disorder, anxiety disorders, Post-traumatic stress disorder, hypodepression, obsessive-compulsive disorder (OCD), conduct disorder, mood disorder, memory disorder, mental status disorder, mobility disorder (eg, essential tremor or Parkinson's disease), Huntington's disease, Al Alzheimer's disease, drug addiction disorder, autism, or other neurological or psychiatric conditions and impairments. When DBS is used to treat patients with drug addiction, it can help drug addicts detoxify and improve their well-being and quality of life.

The stimulator in this application is described by taking the deep brain stimulator (DBS) as an example. When the program-controlled device and the stimulator establish a program-controlled connection, the program-controlled device can be used to adjust the stimulation parameters of the electrical stimulation signal of the stimulator, or the stimulator can sense The bioelectric activity in the deep brain of the patient can be measured, and the stimulation parameters of the electrical stimulation signal of the stimulator can be adjusted continuously through the sensed bioelectric activity. The stimulation parameters of the electrical stimulation signal can include frequency (for example, the number of electrical stimulation pulse signals per unit time 1s, the unit is Hz), pulse width (the duration of each pulse, the unit is μs) and amplitude (generally, voltage Expression, that is, the intensity of each pulse, the unit is any one or more of V). In specific applications, various stimulation parameters of the stimulator can be adjusted in current mode or voltage mode.

The embodiment of the present application does not limit the disease type of the patient, and the patient can be any of the following: patients with Parkinson's disease; patients with depression; patients with addictive diseases; patients with obsessive-compulsive disorder; patients with bipolar disorder. Among them, the addictive disease patients include drug addictive disease patients and/or drug addictive disease patients (ie drug addicts). In particular, Parkinson's patients are mostly inconvenient to move, and there is a great demand for remote program control. It should be noted that, in addition to the diseases listed above, other diseases that can be treated are within the scope of protection of the embodiments of this application, including patients with mental diseases and other addictive diseases.

Real-time bioelectric data can be collected by bioelectric data acquisition equipment, which can integrate one or more of EEG acquisition modules, ECG acquisition modules, oculoelectric acquisition modules and myoelectric acquisition modules, and collect them separately One or more of the patient's real-time EEG data, real-time ECG data, and real-time EMG data are preprocessed to obtain corresponding result data. The above-mentioned bioelectric data acquisition device can also integrate one or more of the gastroelectric acquisition module and the retinal electrical data acquisition module to collect one or more of the real-time gastric electrical data and the real-time retinal electrical data respectively, and perform correlation preprocessing to obtain the corresponding result data. In practical applications, the bioelectricity data acquisition device can integrate one or more of the EEG acquisition module, ECG acquisition module, EMG acquisition module, gastric electricity acquisition module and retinal electricity data acquisition module.

In some possible implementation manners, the bioelectricity data acquisition device can acquire real-time bioelectricity data through electrode wires implanted in the patient's body, or through electrode pads arranged outside the patient's body. This application does not limit the EEG acquisition module, ECG acquisition module, oculoelectricity acquisition module and myoelectricity acquisition module integrated in the bioelectricity data acquisition device. The EEG acquisition module is, for example, the EEG acquisition device disclosed in patent CN103519807B, or the EEG signal collector disclosed in patent CN109497998B; The wearable device of the electrical signal acquisition system; the electrooculogram acquisition module is, for example, the electrooculogram signal extraction device disclosed in patent CN103070682B, or the wireless oculoelectric acquisition system disclosed in patent CN103211594A; Type myoelectric acquisition system, or the myoelectric signal acquisition device disclosed in patent CN106102575B.

Generally speaking, in a specific application, in order to obtain more comprehensive patient information, historical bioelectric data of the patient may also be obtained, including text information and/or image information.

This application does not limit the type and quantity of the real-time status information of the patient. In a possible implementation manner, the real-time status information of the patient includes sleeping, eating, and exercising. In another possible implementation, the real-time status information of the patient includes happiness, depression, and pain. In yet another possible implementation manner, the real-time status information of the patient includes normal and sickness. In yet another possible implementation manner, the real-time status information of the patient includes fatigue and non-fatigue.

Referring to Fig. 2, in some implementation manners, the step S103 may include steps S201-S203.

Step S201: displaying the real-time bioelectrical data and real-time image data of the patient by using the doctor's equipment.

Step S202: Utilize the doctor equipment to receive a status input operation.

Step S203: Determine the real-time status information of the patient in response to the status input operation.

Thus, the doctor can obtain the real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient, combined with his own experience, and the doctor can conveniently operate the doctor's equipment to determine the real-time status information of the patient.

This application does not limit the selection of doctor equipment. Doctor equipment is, for example, a mobile phone, a tablet computer, a notebook computer, a desktop computer, or a smart wearable device. The doctor equipment may also be an in vitro program controller or a program controller. When the doctor's equipment is an in vitro program controller, the in vitro program controller is, for example, the in vitro program controller disclosed in patent CN105709336B, or the implantable nerve stimulator program controller disclosed in patent CN207412517U. When the doctor's equipment is a programmer, the programmer is, for example, a programmer in an implantable nerve electrical pulse stimulation system disclosed in patent CN100469401C, or a doctor programmer with security and confidentiality functions disclosed in patent CN201894778U.

This application does not limit the data interaction between the doctor's equipment and the stimulator. When the doctor is remotely programmed, the doctor's equipment can exchange data with the stimulator through the server and the patient's programmer. When the doctor performs program control with the patient face-to-face offline, the doctor's device can exchange data with the stimulator through the patient's programmer, and the doctor's device can also directly exchange data with the stimulator.

The patient programmer may include a host (in communication with the server) and a slave (in communication with the stimulator), the host and slave being communicably connected. Among them, the doctor equipment can exchange data with the server through the 3G/4G/5G network, the server can exchange data with the host through the 3G/4G/5G network, and the host can exchange data with the slave through the Bluetooth protocol/WIFI protocol/USB protocol , the sub-machine can exchange data with the stimulator through the 401MHz-406MHz working frequency band/2.4GHz-2.48GHz working frequency band, and the doctor equipment can directly perform data interaction with the stimulator through the 401MHz-406MHz working frequency band/2.4GHz-2.48GHz working frequency band.

In some embodiments, the step S103 may include: inputting the real-time bioelectrical data and real-time image data of the patient into a state classification model to obtain real-time state information of the patient.

Thus, the real-time bioelectric data and real-time image data of the patient are input into the state classification model to obtain the real-time state information of the patient, which does not require manual setting by the doctor, saves the doctor's operation steps, and has a high degree of intelligence; in addition, the doctor's judgment exists Strong subjectivity, while the state classification model can achieve high accuracy through training with a large amount of data, and the accuracy of judging the patient's state is higher.

Referring to FIG. 3 , in some implementations, the method for obtaining the state classification model may include steps S301-S303.

Step S301: Obtain training data of a plurality of state sample objects, the training data of each state sample object includes real-time bioelectric data and real-time image data of the state sample object and corresponding real-time state information. The real-time status information corresponds to both real-time bioelectrical data and real-time image data.

Step S302: Using the training data of the plurality of state sample objects to train a first deep learning model to obtain the state classification model.

Step S303: Input the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain the real-time state information of the patient.

Thus, on the one hand, the first deep learning model can be trained using the training data of multiple sample objects to obtain a state classification model, as long as the real-time bioelectric data and real-time image data of the target object are input into the state classification model, the target object can be obtained The real-time status information of patients, especially when the number of samples used for training is large enough, the accuracy can reach a very high level. Compared with the way of detecting real-time status information of patients with conventional detection equipment, the degree of intelligence is higher and the efficiency is higher. On the other hand, the real-time state information of the target object can be sent to the user equipment, so that the user of the user equipment knows the real-time state information of the target object, so that the health status of the target object can be monitored at any time.

Among them, by designing, establishing an appropriate amount of neuron computing nodes and a multi-layer computing hierarchy, and selecting an appropriate input layer and output layer, a preset first deep learning model can be obtained. Through the preset first deep learning model The learning and tuning of the system establishes the functional relationship from input to output. Although the functional relationship between input and output cannot be found 100%, it can approach the actual relationship as much as possible. The state classification model obtained from this training and the calculation results High accuracy and high reliability.

In some implementation manners, the present application may use the above training process to train the state classification model, and in other implementation manners, the present application may use a pre-trained state classification model.

The present application does not limit the training process of the state classification model. For example, a supervised learning training method, a semi-supervised learning training method, or an unsupervised learning training method may be used.

The present application does not limit the training end conditions of the state classification model, which can be, for example, that the number of training times reaches a preset number of times (the preset number of times is, for example, 1 time, 3 times, 10 times, 100 times, 1000 times, 10000 times, etc.), or It may be that the training data of multiple state sample objects have completed one or more trainings, or it may be that the total loss value obtained in this training is not greater than the preset loss value.

Referring to Fig. 4, in some implementation manners, the step S104 may include steps S401-S403.

Step S401: Display the real-time status information of the patient by using the doctor's equipment.

Step S402: using the doctor equipment to receive configuration input operations;

Step S403: Determine the configuration information of the patient in response to the configuration input operation.

Therefore, the doctor's equipment is used to display the real-time status information of the patient, combined with the doctor's experience, the doctor manually determines the configuration information of the patient, and the operation is convenient.

In some implementations, the step S104 may include: inputting the real-time state information of the patient into a configuration acquisition model to obtain the configuration information of the patient.

Therefore, the real-time state information of the patient is input into the configuration acquisition model to obtain the concerned configuration information, which does not require manual setting by the doctor, and has a high degree of intelligence; generally speaking, when the doctor manually configures the stimulation parameters of the stimulator, there is a strong subjective The configuration acquisition model can achieve higher accuracy through training with a large amount of data, and the accuracy of the configured stimulus parameters is higher.

Referring to Fig. 5, in some implementation manners, the method for acquiring the configuration acquisition model may include steps S501-S503.

Step S501: Obtain training data of multiple configuration sample objects, the training data of each configuration sample object includes the real-time status information of the configuration sample object and its corresponding configuration information.

Step S502: Using the training data of the plurality of configuration sample objects to train a second deep learning model to obtain the configuration acquisition model.

Step S503: Input the real-time status information of the patient into the configuration acquisition model to obtain the configuration information of the patient.

Thus, the training data of multiple configuration sample objects is obtained, the training data of each configuration sample object includes the real-time state information of the configuration sample object and its corresponding configuration information, and then the training data of multiple configuration sample objects is used to train the first 2. Deep learning model to obtain a configuration acquisition model. The configuration acquisition model has high accuracy, and the configuration acquisition model is used to predict the patient's configuration information, and the result is more accurate; and once the configuration acquisition model is formed, it can be applied to different patients and different conditions. state, a wide range of applications, easy to use, and a high degree of intelligence.

Among them, by designing, establishing an appropriate amount of neuron computing nodes and a multi-layer computing hierarchy, and selecting an appropriate input layer and output layer, a preset second deep learning model can be obtained. Through the preset second deep learning model The learning and tuning of the system establishes the functional relationship from input to output. Although the functional relationship between input and output cannot be found 100%, it can approach the actual relationship as much as possible. The configuration obtained from this training obtains the model and calculates the results. High accuracy and high reliability.

The present application does not limit the training process of the configuration acquisition model. For example, a supervised learning training method, a semi-supervised learning training method, or an unsupervised learning training method may be used.

The present application does not limit the training end conditions of the configuration acquisition model, which can be, for example, that the number of training times reaches a preset number of times (the preset number of times is, for example, 1 time, 3 times, 10 times, 100 times, 1000 times, 10000 times, etc.), or It may be that the training data of multiple configuration sample objects has completed one or more trainings, or it may be that the total loss value obtained in this training is not greater than the preset loss value.

Referring to Fig. 6, the embodiment of the present application also provides a device for acquiring configuration information based on multimodal data. Let me repeat. The devices include:

The bioelectricity acquisition module 101 is configured to acquire real-time bioelectricity data of the patient, the bioelectricity data including one or more of real-time EEG data, real-time ECG data and real-time EMG data;

An image acquisition module 102, configured to photograph the patient with a camera to obtain real-time image data of the patient;

A state acquiring module 103, configured to acquire real-time state information of the patient based on the real-time bioelectrical data and real-time image data of the patient;

The configuration acquiring module 104 is configured to acquire configuration information of the patient based on the real-time state information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.

Referring to FIG. 7, in some implementations, the status acquisition module 103 may include:

The first display unit 201 is used to display the real-time bioelectric data and real-time image data of the patient by using the doctor's equipment;

The first receiving unit 202 is configured to use the doctor equipment to receive a status input operation;

The state determination unit 203 determines the real-time state information of the patient in response to the state input operation.

In some implementations, the state acquisition module 103 can be configured to input the real-time bioelectrical data and real-time image data of the patient into a state classification model to obtain real-time state information of the patient.

Referring to FIG. 8, in some implementations, the status acquisition module 103 may include:

The state sample unit 301 is used to obtain training data of a plurality of state sample objects, the training data of each state sample object includes real-time bioelectric data and real-time image data of the state sample object and corresponding real-time state information;

The first training unit 302 is configured to use the training data of the plurality of state sample objects to train a first deep learning model to obtain the state classification model;

Obtaining a state unit 303, configured to input the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain real-time state information of the patient.

Referring to FIG. 9, in some implementations, the configuration acquisition module 104 may include:

The second display unit 401 is configured to display the real-time status information of the patient by using the doctor equipment;

The second receiving unit 402 is configured to use the doctor equipment to receive a configuration input operation;

The configuration determining unit 403 is configured to determine configuration information of the patient in response to the configuration input operation.

In some implementations, the configuration acquisition module 104 can be configured to input the real-time status information of the patient into a configuration acquisition model to obtain the configuration information of the patient.

Referring to FIG. 10, in some implementations, the configuration acquisition module 104 may include:

The configuration sample unit 501 is configured to acquire training data of a plurality of configuration sample objects, the training data of each configuration sample object includes the real-time state information of the configuration sample object and its corresponding configuration information;

The second training unit 502 is configured to use the training data of the plurality of configuration sample objects to train a second deep learning model to obtain the configuration acquisition model;

The acquisition configuration unit 503 is configured to input the real-time status information of the patient into the configuration acquisition model to obtain the configuration information of the patient.

Referring to FIG. 11 , the embodiment of the present application also provides an electronic device 200, which includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

Memory 210 may include readable media in the form of volatile memory, such as random access memory (RAM) 211 and/or cache memory 212 , and may further include read only memory (ROM) 213 .

Wherein, the memory 210 also stores a computer program, and the computer program can be executed by the processor 220, so that the processor 220 executes the steps of the method for obtaining configuration information based on multi-modal data in the embodiment of the present application. The implementation manners and the achieved technical effects described in the embodiment of the method for obtaining configuration information of the modal data are consistent, and part of the content will not be repeated here.

Memory 210 may also include utility 214 having at least one program module 215 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, examples of each or Implementations of network environments may be included in some combination.

Correspondingly, the processor 220 can execute the above-mentioned computer program, and can execute the utility tool 214 .

Bus 230 may be representative of one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus structures.

The electronic device 200 can also communicate with one or more external devices 240 such as keyboards, pointing devices, Bluetooth devices, etc., and can also communicate with one or more devices capable of interacting with the electronic device 200, and/or communicate with the electronic device 200 200 is capable of communicating with any device (eg, router, modem, etc.) that communicates with one or more other computing devices. Such communication may occur through input-output interface 250 . Moreover, the electronic device 200 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and/or a public network such as the Internet) through the network adapter 260 . The network adapter 260 can communicate with other modules of the electronic device 200 through the bus 230 . It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives And data backup storage platform, etc.

Wherein, the processor includes: a bioelectricity acquisition module for acquiring real-time bioelectricity data of the patient, the bioelectricity data including one or more of real-time EEG data, real-time ECG data and real-time myoelectric data; An acquisition module, configured to use a camera to photograph the patient to obtain real-time image data of the patient; a status acquisition module, configured to acquire real-time status information of the patient based on the patient's real-time bioelectric data and real-time image data; The configuration obtaining module is configured to obtain configuration information of the patient based on the real-time status information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.

In some embodiments, the electronic device may also be provided with a display screen. The display is, for example, a touch display.

An embodiment of the present application also provides a program-controlled system, the program-controlled system includes doctor equipment and the above-mentioned electronic equipment. In some embodiments, the physician device and electronic device may be integrated.

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium is used to store a computer program, and when the computer program is executed, the multimodal data-based configuration information acquisition method in the embodiment of the present application is implemented The specific implementation of the steps is consistent with the implementation and achieved technical effects described in the above-mentioned embodiment of the multimodal data-based configuration information acquisition method, and part of the content will not be repeated.

Fig. 12 shows a program product 300 provided by this embodiment for realizing the above-mentioned configuration information acquisition method based on multimodal data, which may adopt a portable compact disk read-only memory (CD-ROM) and include program codes, and may Run on end devices, such as personal computers. However, the program product 300 of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, device or device. Program product 300 may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

A computer readable storage medium may include a data signal carrying readable program code in baseband or as part of a carrier wave traveling as part of a data signal. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any appropriate medium, including but not limited to wireless, cable, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operation of the present application can be written in any combination of one or more programming languages, and the programming language includes object-oriented programming languages such as Java, C++, etc., and also includes conventional procedural programming languages A programming language such as C or similar. The program code may execute entirely on the user's computing device, partly on an associated device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server to execute. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider). business to connect via the Internet).

This application is elaborated from the perspectives of purpose of use, performance, progress and novelty. The above description and accompanying drawings of this application are only preferred embodiments of this application, and are not intended to limit this application. Therefore, all All structures, devices, features, etc. that are similar or identical to those of the present application, that is, all equivalent replacements or modifications made according to the scope of the patent application of the present application, shall fall within the scope of protection of the patent application of the present application.

Claims

A device for obtaining configuration information based on multimodal data, the device comprising:

The bioelectricity acquisition module is used to acquire the real-time bioelectricity data of the patient, and the bioelectricity data includes one or more of real-time EEG data, real-time ECG data and real-time EMG data;

An image acquisition module, configured to use a camera to photograph the patient to obtain real-time image data of the patient;

A status acquisition module, configured to acquire real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient;

The configuration obtaining module is configured to obtain configuration information of the patient based on the real-time status information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.
The device according to claim 1, wherein the state acquiring module comprises:

The first display unit uses the doctor's equipment to display the real-time bioelectric data and real-time image data of the patient;

a first receiving unit, using the doctor equipment to receive a status input operation;

A state determination unit determines real-time state information of the patient in response to the state input operation.
The device according to claim 1, wherein the state acquisition module is configured to input the real-time bioelectrical data and real-time image data of the patient into a state classification model to obtain real-time state information of the patient.
The device according to claim 3, wherein the state acquiring module comprises:

The state sample unit is used to obtain training data of a plurality of state sample objects, the training data of each state sample object includes real-time bioelectric data and real-time image data and corresponding real-time state information of the state sample object;

A first training unit, configured to use the training data of the plurality of state sample objects to train a first deep learning model to obtain the state classification model;

The obtaining state unit is used for inputting the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain the real-time state information of the patient.
The device according to claim 1, wherein the configuration acquisition module comprises:

The second display unit is used to display the real-time status information of the patient by using the doctor equipment;

A second receiving unit, configured to use the doctor equipment to receive a configuration input operation;

A configuration determining unit configured to determine configuration information of the patient in response to the configuration input operation.
The device according to claim 1, wherein the configuration acquisition module is configured to input the real-time state information of the patient into a configuration acquisition model to obtain the configuration information of the patient.
The device according to claim 6, wherein the configuration obtaining module comprises:

The configuration sample unit is configured to obtain training data of a plurality of configuration sample objects, the training data of each configuration sample object includes real-time state information of the configuration sample object and corresponding configuration information thereof;

A second training unit, configured to use the training data of the plurality of configuration sample objects to train a second deep learning model to obtain the configuration acquisition model;

An acquisition configuration unit, configured to input the real-time status information of the patient into the configuration acquisition model to obtain configuration information of the patient.
A method for acquiring configuration information based on multimodal data, the method comprising:

Acquiring real-time bioelectric data of the patient, said bioelectric data including one or more of real-time EEG data, real-time ECG data and real-time myoelectric data;

Using a camera to photograph the patient to obtain real-time image data of the patient; obtaining real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient;

Based on the real-time status information of the patient, configuration information of the patient is acquired, and the configuration information of the patient is used to control a stimulator inside the patient to apply electrical stimulation.
The method according to claim 8, wherein said obtaining real-time status information of the patient based on the real-time bioelectrical data and real-time image data of the patient comprises:

displaying the real-time bioelectric data and real-time image data of the patient by using the doctor equipment;

receiving a status input operation with the doctor device;

Real-time status information for the patient is determined in response to the status input operation.
The method according to claim 8, wherein said obtaining real-time status information of the patient based on the real-time bioelectrical data and real-time image data of the patient comprises:

The real-time bioelectric data and real-time image data of the patient are input into the state classification model to obtain the real-time state information of the patient.
The method according to claim 10, wherein the method for obtaining the state classification model comprises:

Obtain training data of a plurality of state sample objects, the training data of each state sample object includes real-time bioelectric data and real-time image data of the state sample object and corresponding real-time state information, and the real-time state information corresponds to the real-time biological Electrical data and real-time image data;

Using the training data of the plurality of state sample objects to train a first deep learning model to obtain the state classification model;

Inputting the real-time bioelectrical data and real-time image data of the patient into the state classification model to obtain real-time state information of the patient.
The method according to claim 8, wherein said obtaining the configuration information of the patient based on the real-time status information of the patient comprises:

Using the doctor equipment to display the real-time status information of the patient; using the doctor equipment to receive configuration input operations;

In response to the configuration input operation, configuration information for the patient is determined.
The method according to claim 8, wherein said obtaining the configuration information of the patient based on the real-time status information of the patient comprises:

Inputting the real-time state information of the patient into the configuration acquisition model to obtain the configuration information of the patient.
The method according to claim 13, wherein the method for acquiring the configuration acquisition model comprises:

Acquiring training data of a plurality of configuration sample objects, the training data of each configuration sample object includes the real-time state information of the configuration sample object and its corresponding configuration information;

Using the training data of the multiple configuration sample objects to train a second deep learning model to obtain the configuration acquisition model;

Inputting the real-time state information of the patient into the configuration acquisition model to obtain the configuration information of the patient.
An electronic device, the electronic device also includes a memory and a processor, the memory stores a computer program, and the processor includes:

The bioelectricity acquisition module is used to acquire the real-time bioelectricity data of the patient, and the bioelectricity data includes one or more of real-time EEG data, real-time ECG data and real-time EMG data;

An image acquisition module, configured to use a camera to photograph the patient to obtain real-time image data of the patient;

A status acquisition module, configured to acquire real-time status information of the patient based on the real-time bioelectric data and real-time image data of the patient;

The configuration obtaining module is configured to obtain configuration information of the patient based on the real-time status information of the patient, and the configuration information of the patient is used to control the stimulator in the patient to apply electrical stimulation.
A program-controlled system, comprising doctor equipment and the electronic equipment according to claim 15.
A computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Acquiring real-time bioelectric data of the patient, said bioelectric data including one or more of real-time EEG data, real-time ECG data and real-time myoelectric data;

Using a camera to photograph the patient to obtain real-time image data of the patient;

acquiring real-time state information of the patient based on the real-time bioelectric data and real-time image data of the patient;

Based on the real-time state information of the patient, configuration information of the patient is obtained, and the configuration information of the patient is used to control a stimulator in the patient to apply electrical stimulation.