JP7452855B2 - Swallowing movement prediction method and system using time-series data prediction - Google Patents

Description

The present invention detects biosignals related to the swallowing motion in the front neck and its surroundings during swallowing, extracts feature quantities from the detected biosignals, and extracts features from the swallowing motion in the oral cavity, pharynx/larynx, esophagus, etc. The present invention relates to an eating/swallowing function evaluation/training method and an eating/swallowing function evaluation/training system for evaluating/training the eating/swallowing function by predicting the movement of organs and the movement of the swallowed object (bolus).

If swallowing function deteriorates due to cerebrovascular disease, neuromuscular disease, muscle weakness due to aging, etc., the risk of bolus remaining in the pharynx, entering the larynx, aspiration, and suffocation increases, as shown in Figure 4. This includes drooping of the hyoid and larynx position, associated decrease in the amount of elevation and forward movement of the hyoid and larynx, delay in larynx elevation due to decreased laryngeal elevation speed, delay in triggering the swallowing reflex, and laryngeal closure. This is caused by a decline in the motor and sensory functions of the various swallowing organs, such as a difference in the timing of swallowing. Therefore, in order to evaluate swallowing function, medical institutions evaluate the movement of various organs related to swallowing, such as the tongue, hyoid bone, larynx, epiglottis, and esophageal entrance, as well as the movement of the food bolus. .

For example, the gold standard method for detailed examination of swallowing function is videofluoroscopic examination of swallowing (VF). As shown in Figures 5(a) to (f), VF consists of swallowing a bolus containing a contrast agent under X-ray fluoroscopy, and observing the movements of the swallowing organs during preparation, oral cavity, pharynx, and esophagus, as well as the bolus. It evaluates the movement of In the preparation/oral phase, it is possible to evaluate the process and function of bolus formation through mastication, delivery to the pharynx through tongue movement, and retention in the oral cavity. During the pharyngeal stage, the swallowing reflex is triggered as the bolus is fed, the hyoid bone and larynx are elevated, the nasopharynx is closed, the larynx is closed due to inversion of the epiglottis, the esophageal entrance opens, and the bolus remains in the pharynx. It is possible to evaluate in detail the movement of swallowing organs such as the hyoid bone, epiglottis, and entrance to the esophagus, as well as the movement of the bolus. On the other hand, this VF poses risks such as radiation exposure and aspiration of contrast media, which limits the number, time and frequency of examinations, examination locations, examination conditions, etc.

Recently, videoendoscopic evaluation of swallowing (VE) has also been widely used as a method for observing the movements of swallowing organs and bolus at the bedside or at home. VE has the advantage of being able to assess the condition of the food bolus and the amount of food remaining in the pharynx without the risk of radiation exposure, but because the endoscope is inserted through the nasal cavity, there is a risk of mucous membrane damage and pain, and natural swallowing is not always possible. There are also aspects that cannot be said to be observing the situation. In addition, there is the problem that the moment of swallowing during the pharyngeal stage and movements during the preparation, oral cavity, and esophageal stages cannot be observed.

Image processing is used to quantitatively evaluate the movements of the swallowing organs and bolus as numerical values, and the hyoid bone is one of the important swallowing organs that often receives attention. The hyoid bone is the only bone in the human body that does not form a joint with adjacent bones or cartilage, and is extremely unique because it is suspended in midair. As shown in Figure 1, the hyoid bone is located between the tongue and larynx, and many muscles involved in swallowing movements are attached to it. Examples include the digastric muscle, stylohyoid muscle, mylohyoid muscle, geniohyoid muscle, sternohyoid muscle, thyrohyoid muscle, omohyoid muscle, pharyngohyoid muscle, and oropharyngeal constrictor muscle. . The coordinated activity of these muscle groups enables skillful movements such as mastication, swallowing, and vocalization.

As shown in FIG. 2, the hyoid bone is known to draw a movement trajectory that roughly resembles a triangle during swallowing. First, a lifting movement is started relatively slowly, but at this time, a slight backward movement is often accompanied (1: lifting backward movement). Second, the hyoid bone elevates greatly and at the same time rapidly moves forward (2: forward movement of elevation). After stopping at the maximum raised position and the maximum forward movement position, a third movement, that is, a backward and downward movement is performed to restore the original position (3: descending backward movement). The suprahyoid and infrahyoid muscles are largely involved in these movements. As shown in FIG. 3, during this series of movements, the suprahyoid muscle group contracts together with the pharyngeal muscles and tongue muscles, and the hyoid moves upward and forward. Following the hyoid bone, the larynx is raised by the contraction of the subhyoid muscles, and the cricopharyngeal muscles are continuously relaxed and contracted, allowing the bolus to pass through the entrance to the esophagus. In addition, the timing and duration of elevation of the hyoid bone are closely related to the timing and closure duration of the larynx, which prevents food bolus from entering the larynx.

As a technique for evaluating the eating and swallowing function focusing on the muscle activities of the suprahyoid muscle group and the infrahyoid muscle group, the eating and swallowing function evaluation technique disclosed in Patent Document 1 is known. The feeding and swallowing function evaluation technology of Patent Document 1 detects biological signals from the start of feeding and swallowing to the end of feeding and swallowing, extracts feature amounts from the detected biological signals, and uses machine learning to evaluate feeding and swallowing functions from the feature amounts. This is an eating and swallowing function evaluation method that evaluates eating and swallowing functions by identifying swallowing movements. As biosignals, we use the suprahyoid muscle group biosignal due to the muscle activity of the suprahyoid muscle group and the infrahyoid muscle group biosignal due to the muscle activity of the infrahyoid muscle group. Features are extracted from the infrahyoid muscle group biological signals. However, Patent Document 1 describes differences in swallowing conditions such as differences in the strength of voluntary swallowing, differences in the amount swallowed at a time, and differences in physical property values (hardness, viscosity, temperature, liquid, solidity, etc.) of food and bolus, and differences in aspiration. Swallowing function evaluation method and swallowing function evaluation that can determine the presence/absence and type (overt aspiration, covert aspiration, pre-swallow aspiration, aspiration during swallowing, post-swallow aspiration, etc.) and risk (laryngeal inflow, etc.) It is not possible to directly predict the movements of the swallowing organs and bolus as time-series data, which can be observed with VF and VE. If these can be predicted, non-invasive and simple evaluation of swallowing function and feeding and swallowing training that can be used at the bedside or at home can be realized.

JP2019-208629A

In view of the above points, the present invention provides a non-invasive and low-risk eating and swallowing function evaluation/training technique that can easily predict the movement of swallowing organs and bolus at the bedside or in home medical care. The task is to do so.

[1] A swallowing photographing step of photographing the movement of the bolus ingested by the subject and the movements of various swallowing organs;
a preprocessing step of acquiring the positions of the swallowing organs and the bolus from the image of the swallowing photographing step and digitizing them as coordinates to create a teacher signal of the movement of the swallowing organs and the bolus;
a biosignal detection step of detecting biosignals during ingestion and swallowing with a sensor unit placed on a predetermined skin surface of the subject in synchronization with the swallowing imaging step;
a feature amount extraction step of extracting a feature amount from the biological signal in an analysis unit;
RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory) derived from the RNN, GRU (Gated Recurrent Unit), LSTNet (Long- and Short-term Time-series Network), AR (Autoregressive) model, and the Based on the teacher signal and the feature amount using a time series data prediction method including ARMA (Autoregressive Moving Average), ARIMA (Autoregressive Integrated Moving Average), and SARIMA (Seasonal AutoRegressive Integrated Moving Average) models derived from the AR model. a learning step of learning the movements of the swallowing organs and the bolus to generate a model capable of predicting the movement of at least one of the swallowing organs and the bolus from the feature values;
a prediction step of predicting the movement of at least one of the swallowing organs and the bolus from the feature amounts using the prediction model generated in the learning step ;
It is characterized by having the following.

According to this configuration, the swallowing imaging process, the preprocessing process, the biosignal detection process synchronized with the swallowing imaging process, the feature extraction process, the learning process, the prediction process, and the evaluation/training process are provided. In the swallowing imaging process, if the swallowed object mixed with a contrast agent or coated on the surface is spherical, it becomes easier to quantify the movement of the bolus through image processing. The type of image does not matter, as long as continuous changes in the swallowing process can be confirmed, such as moving images or still images. Furthermore, in the learning process and prediction process, for example, when using LSTM as a prediction method for time series data, long and short-term memory, which is a recurrent neural network architecture that learns long-term time dependence and short-term time dependence, is used. This method is used to learn the movements of the swallowing organs, including the hyoid bone, and the bolus based on teacher signals and feature quantities. LSTM is a recurrent neural network architecture used in the field of deep learning, which solves problems that cannot be learned with long-term time dependence when training with traditional RNNs, and can be used to solve problems that cannot be learned with long-term time dependence or short-term time dependence. You can also learn about sex. When new inputs and outputs are received during the learning process, new patterns are applied, making it possible to deal with the problems of input weight collisions and output weight collisions that occur in RNNs. For this reason, the test subject must first perform at least one swallowing morphological test (VF test) or swallowing endoscopy (VE test), etc., and at the same time as the test, the biological signals can be obtained using the sensor unit. The features related to the movement of at least one of the swallowing organs and the food bolus are learned, and from the second time onwards, at least the features related to the movement of the swallowing organs and the food bolus are learned from only the feature values of the biological signals newly detected after the learning process, without any VF examination etc. The movement of one side of the bolus can be predicted. As a result, the X-ray fluoroscopy equipment required for VF examinations is no longer required, making it possible to easily evaluate and train swallowing functions to predict movements of the swallowing organs and bolus at the bedside or at home, which is non-invasive and has little risk. It can be carried out. In addition, if the same amount and physical properties are swallowed in the same way, the learning data will be appropriate data in one time, but to make the learning data more suitable, it is necessary to change the amount and physical properties. With respect to swallowing when the patient is swallowed, the swallowing data under that condition may be added to the learning and used as the learning data.

[2] Preferably, in the biological signal detection step, a suprahyoid muscle group myoelectric sensor placed in the suprahyoid muscle group portion detects the suprahyoid muscle group biological signal, and the suprahyoid muscle group biological signal is detected in the infrahyoid muscle group portion. The placed myoelectric sensor for the subhyoid muscle group detects the subhyoid muscle group biosignal, the laryngeal behavior sensor placed in the larynx detects the laryngeal behavior signal,
In the feature amount extraction step, feature amounts are extracted from the suprahyoid muscle group biosignal, the infrahyoid muscle group biosignal, and the laryngeal behavior signal as the biosignals.

According to this configuration, in the biological signal detection step, the suprahyoid muscle group biological signal, the infrahyoid muscle group biological signal, and the laryngeal behavior signal are detected, so that at least one of the swallowing organs and the bolus is detected with higher accuracy. It is possible to predict the movement of

[3] Preferably, in the learning step, the coordinate data of the swallowing organs and the food bolus are used as learning data, and the learning data is one source data that does not contain the same coordinate data at a predetermined period. It is shifted and amplified to multiple levels.

According to this configuration, in the learning process, one source data is shifted and amplified into multiple pieces at a predetermined period so that the same coordinate data is not included, so that the predicted value and the predicted value can be calculated in the first learning step. Errors in actual measurements can be reduced, and the movement of at least one of the swallowing organs and the bolus can be predicted with higher accuracy.

[4] Preferably, in the preprocessing step, a coordinate system and its origin are determined for determining the coordinate data of the swallowing organs and the food bolus, and the lower end of the anterior border of the fifth cervical vertebrae of the subject is set as the origin, and By setting a coordinate system with the upper end of the anterior edge of the third cervical vertebra as a point on one axis, the positions of the swallowing organs and bolus in the sagittal or frontal plane of the subject are obtained using the coordinates of the coordinate system. do.

According to this configuration, in the preprocessing step, a coordinate system in the sagittal plane or frontal plane of the subject is set, and the coordinates of the swallowing organs and each position of the bolus in the XY coordinate system are obtained, so that the subject's It is possible to easily predict the movement of at least one of the swallowing organs and the bolus in each direction such as back and forth and up and down.

[5] Preferably, the positions of the swallowing organs and the bolus are obtained from an image of the movement of the bolus ingested by the subject and the movements of the swallowing organs, and are digitized as coordinates. a preprocessing unit that creates a teacher signal for the movement of the food bolus;
a sensor unit that is placed on a predetermined skin surface of the subject and detects biological signals during ingestion and swallowing in synchronization with the imaging of the image;
In addition to extracting feature quantities from the biological signals, RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory) derived from the RNN, GRU (Gated Recurrent Unit), and LSTNet (Long- and Short-term Time-series Using time-series data prediction methods including AR (Autoregressive) models, ARMA (Autoregressive Moving Average), ARIMA (Autoregressive Integrated Moving Average), and SARIMA (Seasonal AutoRegressive Integrated Moving Average) models derived from the AR model. learn the movement of at least one of the swallowing organs and the bolus based on the teacher signal and the feature amount, and generate a model capable of predicting the movement of at least one of the swallowing organs and the bolus from the feature amount. The apparatus further includes an analysis section that uses the generated prediction model to predict the movement of at least one of the swallowing organs and the bolus from the feature amounts.

According to such a configuration, it is possible to conduct a non-invasive and low-risk time-series evaluation and training of the swallowing function that predicts the movement of at least one of the swallowing organs and the bolus, easily at the bedside or in home medical care. A swallowing function evaluation/training system using data prediction can be provided.

Eating and swallowing function evaluation and training that predicts the movement of at least one of the swallowing organs and the bolus can be easily performed at the bedside or in home medical care, which is non-invasive and has little risk.

It is an explanatory view showing a suprahyoid muscle group and an infrahyoid muscle group. It is an explanatory view showing movement of a hyoid bone. FIG. 2 is an explanatory diagram showing the mechanism of swallowing consisting of voluntary movements and swallowing reflexes. It is an explanatory diagram showing aspiration risk. FIG. 3 is an explanatory diagram showing information obtained by VF. FIG. 1 is a configuration diagram of a swallowing function evaluation/training method system using time-series data prediction according to the present invention. It is an explanatory view showing a sensor part. It is an explanatory view showing a laryngeal behavior sensor and an electrode jig. FIG. 3 is an explanatory diagram showing output voltages corresponding to expansion/contraction ratios. FIG. 2 is an explanatory diagram showing a circuit configuration. FIG. 2 is a schematic diagram of Σ-Δ AD conversion and an explanatory diagram showing parallelization and synchronization of each AD conversion module. FIG. 2 is an explanatory diagram showing a data processing/transfer circuit. FIG. 3 is an explanatory diagram showing an insulation configuration. It is an explanatory view showing a synchronization microphone and a portable multi-mixer. It is a flow diagram showing a swallowing function evaluation/training method using time-series data prediction. FIG. 3 is an explanatory diagram showing a state of frame shifting. FIG. 3 is an explanatory diagram showing settings of an object in an X-ray image. It is an explanatory view showing the distance (X-axis and Y-axis) of the starting point of the hyoid bone. FIG. 7 is an explanatory diagram showing the hyoid bone position (white circle) at times A to F on the VF image. FIG. 2 is an explanatory diagram showing motion section determination using a moving image. FIG. 3 is an explanatory diagram showing determination of a motion section based on biological signals. It is a basic diagram of RNN. FIG. 2 is a simplified diagram of the internal configuration of an LSTM block. FIG. 2 is an explanatory diagram showing the internal configuration of an LSTM block. FIG. 3 is an explanatory diagram showing a forgetting gate layer, an input gate layer, cell updating, and an output gate layer. FIG. 2 is an explanatory diagram showing a swallowing function evaluation/training system using an X-ray fluoroscope and time-series data prediction. FIG. 3 is an explanatory diagram showing the arrangement of electrodes. It is an explanatory view showing a sensor part and a state where the sensor part is attached to a subject and the image is transmitted through the subject. It is an explanatory view showing numericalization of hyoid bone movement. It is an explanatory diagram showing learning conditions. FIG. 2 is an explanatory diagram showing creation of learning data and test data. FIG. 2 is an explanatory diagram showing data amplification. FIG. 3 is an explanatory diagram showing the relationship between data amplification and RMSE. 32 is an explanatory diagram showing an example of learning (corresponding to condition 6 in FIG. 31). FIG. FIG. 2 is an explanatory diagram showing an example of the sEMG signal and laryngeal movement during the first swallowing. FIG. 3 is an explanatory diagram showing time-series data of sEMG signals, RMS, CC of each muscle group, laryngeal movement, and hyoid bone movement. FIG. 3 is an explanatory diagram showing the results of learning A-prediction C as an example. FIG. 7 is an explanatory diagram showing the trajectory of the hyoid bone in learning A-prediction C as an example. FIG. 3 is an explanatory diagram showing the RMSE of the actual measured value and the predicted value on the X axis, and the RMSE of the actual measured value and the predicted value on the Y axis. FIG. 3 is an explanatory diagram showing a correlation coefficient between actual measured values and predicted values in the X-axis direction, and a correlation coefficient between actual measured values and predicted values in the Y-axis direction. FIG. 3 is an explanatory diagram showing the results of learning A-prediction C as an example. FIG. 7 is an explanatory diagram showing the trajectory of the tip of the bolus in learning A-prediction C as an example.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the accompanying drawings, taking as an example the prediction of the movement of a hyoid bone. Note that the drawings conceptually (schematically) show the schematic configuration of the eating and swallowing function evaluation system.

First, the overall configuration of an eating and swallowing function evaluation system 10 according to an embodiment of the present invention will be described.
As shown in FIGS. 6 to 8, FIG. 26, and FIG. 27, the eating and swallowing function evaluation system 10 includes a sensor unit 20 that detects biological signals during eating and swallowing, and a sensor unit 20 that amplifies the detected biological signals and sends them to a PC. A multifunctional myoelectric potential measurement device 30 for transmitting data; an analysis unit 40 for predicting the movement of at least one of the swallowing organs including the hyoid bone and the bolus from the biological signals, and using the prediction for evaluating and training the swallowing function; It includes a recording section (not shown) for recording the results of evaluation and training, a display section 41 for displaying the results of evaluation and training, and a battery (not shown) for supplying power to these.

The eating and swallowing function evaluation system 10 also includes an X-ray fluoroscopy device 50 that obtains a contrast video of eating and swallowing by visualizing the movement of the food bolus ingested by the subject 60 and the movements of various swallowing organs, and a contrast video of eating and swallowing. The preprocessing unit 51 acquires the positions of the swallowing organs and the bolus, converts the movements of at least one of the swallowing organs and the bolus into numerical values as coordinates, and creates a teacher signal for each movement. Note that the X-ray fluoroscopy device 50 is used only when the subject 60 first detects biological signals with the sensor section 20 and obtains learning data simultaneously with the VF examination.

Next, the sensor section 20 will be explained.
The sensor unit 20 includes a myoelectric sensor 21 for the suprahyoid muscle group, which is placed in the suprahyoid muscle group and detects biosignals of the suprahyoid muscle group due to muscle activity of the suprahyoid muscle group, and a myoelectric sensor 21 for the suprahyoid muscle group, which is arranged in the suprahyoid muscle group. A myoelectric sensor 22 for the infrahyoid muscle group, which is placed in the larynx area, detects biological signals of the infrahyoid muscle group due to muscle activity of the infrahyoid muscle group, and a myoelectric sensor 22 is placed in the larynx area, which detects laryngeal behavior signals due to the elevation of the larynx. A laryngeal behavior sensor 25 is provided. The sensor unit 20 is placed on a predetermined skin surface of the 60 subjects, and detects biological signals during eating and swallowing in synchronization with acquisition of the eating and swallowing contrast video.

The myoelectric sensor 21 for the suprahyoid muscle group uses an array electrode in which multi-channel electrodes 21a are arranged. The myoelectric sensor 22 for the subhyoid muscle group uses an array electrode in which multi-channel electrodes 22a are arranged.

The multi-channel electrodes 21a and 22a are used by being connected to a multifunctional myoelectric potential measuring device 30. The myoelectric sensor 21 for the suprahyoid muscle group is shaped so as not to interfere with the back of the head, and to also be able to measure the stylohyoid muscle located deep in the bottom of the mandible. The myoelectric sensor 22 for the subhyoid muscle group has a shape that allows measurement without interfering with the movement of the laryngeal protuberance.

The thickness of the substrate itself is 0.3 mm, and the entire substrate is covered with silicon to protect it, and muscle surface electromyography (hereinafter referred to as sEMG signal) signals are extracted via silver electrodes embedded on the silicon. do.

The silver electrodes had a diameter of 2 mm and a height of 2.5 mm, and the electrodes 21a for the suprahyoid muscle group were embedded at intervals of 8 mm in length and 11.5 mm in width, and 22 electrodes were arranged to cover the entire bottom of the mandible. Electrodes 22a for the infrahyoid muscle group were implanted at intervals of 8 mm vertically and 8 mm horizontally, and 22 electrodes were placed so as to cover the front surface of the neck. In addition, a GND electrode 23a and a bipolar reference electrode 23b were placed on the left and right earlobes, and an RLD electrode 24 was placed on the spinous process of the 7th cervical vertebra. During measurement, multi-channel electrodes 21a and 22a whose electrodes are coated with paste (Elefix, Nihon Kohden) are attached to the subject in order to suppress contact resistance. The obtained signal is sent to the multifunctional myoelectric potential measuring device 30. In the present invention, the frequency band is 20 to 4000 Hz and the gain is 125 times.

Furthermore, in the present invention, when the hyoid bone lifts, the larynx also lifts to follow it. Therefore, in order to record changes in the position of the laryngeal prominence, a laryngeal behavior sensor (a stretchable A strain sensor) 25 was used. The laryngeal behavior sensor C-STRETCH (registered trademark) (F51FS01, Bando Chemical Co., Ltd.) was used in the present invention. This sensor is a dielectric capacitance strain sensor composed of an elastomer film and a protective film, and by inputting a power supply voltage, it outputs an analog voltage corresponding to the elongation of the sensor. The sensor extendable part has a length of 50 mm and a width of 5 mm. The expansion/contraction range of the sensor expansion/contraction section is 0 to 100%, and the output voltage corresponding to the expansion/contraction displacement has a value shown in FIG.

In addition, when attaching the 22ch flexible electrode 21a and the 22ch flexible electrode 22a, after taping, the jig 27 (cap and band) for the myoelectric sensor for the suprahyoid muscle group shown in FIG. 8(b) is used. and fixed with a myoelectric sensor jig 28 (band) for the subhyoid muscle group shown in FIG. 8(c).

Next, the multifunctional myoelectric potential measuring device 30 will be explained.
The multifunctional myoelectric potential measurement device 30 is a measurement device for monitoring biological activity that is designed on the premise that a plurality of different sensors are used simultaneously. It is possible to sample up to 64 channels of sensors simultaneously. It is connected to a PC for importing measurement data via a USB 2.0 (High Speed) interface. It is also possible to control the device from any application software. Operates from DC12 (V) AC adapter or external battery input power source. The circuit configuration of the multifunctional myoelectric potential measurement device is divided into four parts: a signal conditioning section, an AD conversion section, a data transfer section, and an insulation section, as shown below.

As shown in FIG. 10, the signal conditioning section can input up to two multichannel electrodes, four general-purpose myoelectric potential sensors, and 16 arbitrary analog sensors. First, a differential amplifier circuit removes in-phase noise between the signal obtained from the reference electrode attached to the earlobe and the signal obtained from each electrode of the multi-channel electrode, and only the difference in signal components is amplified. Also called unipolar lead measurement.

Further, from the obtained differential signal, a low frequency band signal of 1 (Hz) or less is detected and removed by a DC servo circuit. Next, the signal is amplified by either 125 times or 1000 times in a signal amplification circuit PGA (Programmable Gain Amplifier). A three-pole anti-aliasing filter circuit removes unnecessary high-frequency noise. This also has the effect of preventing band folding during AD conversion. Finally, it is input to a high-speed amplifier to drive AD conversion to obtain an output signal.

In addition, in order to additionally remove low frequency signals, it is also possible to perform first-order low-cut filter processing using a digital filter. The cutoff frequency is any one of disabled, 0.01, 0.1, 1.0, 10.0, and 20.0 (Hz).

The signal processing circuit of a general-purpose myoelectric potential sensor has almost the same configuration as that of a multi-channel electrode, but differs in that it differentially amplifies signals obtained from two arbitrary electrodes attached to the body surface. Also called bipolar induction measurement.

The signal processing circuit of the general-purpose analog sensor is designed to accept analog signals of up to ±15 (V) so that various sensors can be connected as desired. When inputting a signal with a large amplitude, gain adjustment is performed using PGA to adjust it to the measurement range (±2.5 (V)) of the multifunctional myoelectric potential measuring device. The value of PGA is one of disable, 1/4, 1/2, and 1 times. The output signal is obtained from a high speed amplifier to drive the AD conversion.

As shown in FIG. 11, the AD conversion function built into the multifunctional myoelectric potential measuring device 30 uses the Σ-Δ conversion method, and can simultaneously sample all channels at a maximum of 10 (kHz) with a resolution of 16 (bit). be. A schematic diagram of Σ-Δ AD conversion is shown. By applying oversampling of the sampling frequency fs (Hz) × n and Σ-Δ modulation to the analog signal Vin, the frequency spectrum of unnecessary noise is shifted to the high frequency band outside the band, and this is removed by a digital filter. do. Finally, by down-rating to fs (Hz), a digitized output signal is obtained. Compared to the widely used successive approximation AD conversion, it is possible to obtain a higher signal-to-noise ratio and to simplify the anti-aliasing filter.

The digital filter of the multifunctional myoelectric potential measuring device 30 has flat amplitude and linear phase characteristics (the effective band is 1/2 of the sampling frequency). The sampling frequency is selected from 1k, 1.25k, 2k, 2.5k, 4k, 5k, 8k, and 10k (Hz).

Furthermore, since the (64) Σ-Δ AD conversion modules corresponding to each channel are driven by the same clock source with wires of equal length, they each perform AD conversion operations in synchronization.

As shown in FIG. 12, in the data transfer section, the measurement data digitized by AD conversion is taken into the PC via the USB 2.0 (High Speed) interface. These processes are realized by firmware written in a DSP (Digital Signal Processor). Measurement data transferred from each AD conversion module at each sampling frequency is transferred to the memory in the DSP by DMA (Direct Memory Access). The DSP performs additional signal processing, such as digital filtering, and stores measurement data in FIFO (First In First Out) memory, which consists of SDRAM. When the USB transmission buffer becomes empty, the target measurement data is sequentially read from the FIFO memory and sent to the PC (USB host). In this way, by inserting FIFO memory between data processing and USB transfer processing, we have made it possible to transfer all measurement data to the PC without any omissions.

As shown in FIG. 13, in the insulating section, the multifunctional myoelectric potential measuring device 30 is a measuring device for monitoring biological activity, so safety must also be considered. The analog part (signal conditioning part and AD conversion part mentioned above) where the electrodes come in contact with the living body, and the digital part (data transfer part mentioned above) that enables connection to the power supply and PC are electrically isolated. And so. Drive power for the analog section is generated by an isolated power supply circuit from a 12 (V) input. Data communication between the digital section and the analog section is performed via a digital isolator.

Next, the X-ray fluoroscope 50 will be explained.
As shown in FIG. 26, the VF inspection device used in the present invention is an X-ray fluoroscopy device 50 (SHIMADZU Corp, Safire II ZS-100), and the voltage output state during inspection is 79 kVp and the current is 250 mA.

Next, the synchronization microphone 52 and the portable multi-mixer 53 will be explained.
As shown in FIG. 14, the synchronization microphone is used for the purpose of synchronizing the moving image obtained by the X-ray fluoroscope 50 and the sEMG signal obtained by the multifunctional myoelectric potential measurement device 30 during an examination. 52 was used. A stereo microphone (AT9941, audio-technica (registered trademark)) is used in the main body of the synchronization microphone 52 shown in (a) of FIG. 14, and a portable multi-mixer 53 (AT-PMX5P, synchronization was achieved by connecting an audio-technica (registered trademark) to the main body of the synchronization microphone 52, the X-ray fluoroscope 50, and the multifunctional myoelectric potential measuring device 30.

Next, the analysis section 40 will be explained.
The analysis unit 40 (see FIGS. 6 and 26) extracts feature quantities from biological signals and uses long/short-term memory, which is a recurrent neural network architecture that learns long-term time dependence and short-term time dependence. (LSTM) is used to learn and predict the movement of swallowing organs such as the hyoid bone and the bolus based on teacher signals and feature amounts (details will be described later).

Next, a method for evaluating eating and swallowing function according to an example of the present invention will be explained.
As shown in FIG. 15, the eating and swallowing function evaluation method includes a swallowing imaging process (VF video process), a preprocessing process, a biological signal detection process (signal measurement process), a feature extraction process, a learning process, and It includes a prediction process (learning/prediction process) and a prediction result evaluation process (motion prediction process).

In the swallowing imaging process (VF video process), a swallowing contrast video is obtained using a swallowing contrast test that allows the movement of the bolus ingested by the subject and the movements of swallowing-related organs to be observed under X-ray fluoroscopy. In the preprocessing process, the positions of the movements of the swallowing organs such as the hyoid bone and the bolus are obtained from the contrast-enhanced video of swallowing, and each movement is digitized as coordinates, and a teacher signal of the movement of the swallowing organs and the bolus is obtained. Create.

In the biosignal detection process (signal measurement process), biosignals during ingestion and swallowing are detected by a sensor section placed on a predetermined skin surface of the subject in synchronization with the contrast video process. In the feature amount extraction step, the analysis unit extracts feature amounts from the biosignal.

In the learning process, LSTM (long short-term memory), which is a recurrent neural network architecture that learns long-term time dependence and short-term time dependence, is used to learn swallowing, including the hyoid bone, based on teacher signals and feature quantities. A model (prediction model) that can predict the movements of the swallowing organs and the bolus from the feature values is generated by learning the movements of the various organs and the bolus. In the prediction process, the motion prediction model of the subject generated in the learning process is used to calculate features of the subject's biological signals newly detected after the learning process or not used in the learning process, including the hyoid bone. Predict movements of swallowing organs and bolus. Furthermore, an evaluation/training step is provided for evaluating and training the eating and swallowing function using the results of the prediction step.

Furthermore, in the biological signal detection process, the suprahyoid muscle group biosignal is detected by the myoelectric sensor for the suprahyoid muscle group placed in the suprahyoid muscle group, and A myoelectric sensor for the muscle group detects the infrahyoid muscle group biosignal, a laryngeal behavior sensor placed in the larynx detects the laryngeal behavior signal, and in the feature extraction process, the suprahyoid muscle group is detected as the biosignal. Features are extracted from biological signals, subhyoid muscle group biological signals, and laryngeal behavior signals.

Furthermore, in the learning process, the coordinate data of swallowing organs such as the hyoid bone and the movement of the bolus are used as learning data, and the learning data is divided into one source data at a predetermined period so that it does not contain the same coordinate data. (See FIG. 32). In addition, when amplifying data, each point (for example, data 1, data 2, data 3 in FIG. An average value (moving average) including the following may be used.

Furthermore, in the preprocessing step, in a side view of the subject, an XY coordinate system is set with the lower end of the anterior edge of the subject's 5th cervical vertebrae as the origin and the upper end of the anterior edge of the subject's 3rd cervical vertebrae as a point on the Y axis. The coordinates of the XY coordinate system are obtained with the lower end of the hyoid bone as the position of the hyoid bone.

Next, an algorithm for predicting the motion of the hyoid bone using time-series data in the method of evaluating eating and swallowing function will be explained. A schematic diagram of an algorithm for predicting the movement of the hyoid bone using time-series data is shown in FIG. 15.

Next, the characteristic part extraction unit for signal measurement will be explained.
The inventors removed noise by applying a band pass filter (250-700 Hz) to 22 channels of the suprahyoid muscle group and 22 channels of the infrahyoid muscle group obtained by signal measurement. Then, characteristic signal components (feature amounts) related to the motion are extracted in each channel of the suprahyoid muscle group and the infrahyoid muscle group. In this study, features were extracted and created for each sEMG signal of each channel of the suprahyoid muscle group and the infrahyoid muscle group while shifting frames of 256 samples in length at a cycle of 16 samples. . The following features were used for this feature.

RMS (Root Mean Square) is expressed by Equation 1, and provides characteristics regarding the amplitude of the EMG signal.

CC (Cepstrum coefficient) is expressed by Formula 2. It is a feature extracted from the frequency domain, and has the characteristic of being able to separate the envelope shape and fine structure of the power spectrum. When the order is low, the features of the envelope shape appear, and when the order is high, the features of the fine structure appear.

Here, n represents the total number of samples, and sEMG represents the sEMG signal.

As shown in FIG. 16, the past n samples of EMG are used for RMS calculation. At this time, a frame shift method is used in which calculations are performed by cutting out n samples as one frame, and the range to be cut out is shifted at a constant cycle.

Next, preprocessing of VF inspection videos will be explained.
As shown in Figure 17, the video obtained by the VF test was captured using the video analysis software DIPP-Motion V (Detect Co., Ltd.) at a sampling speed of 30 fps, and the movement of the hyoid bone was quantified. In the coordinate system, the upper end of the anterior edge of the third cervical vertebrae is P1, the lower end of the anterior edge of the fifth cervical vertebrae is P2, the lower end of the hyoid body is P3, and the straight line passing through P1 and P2 is the Y axis. A straight line that is perpendicular to the Y axis and passes through P2 is set as the X axis. When setting the scale on the screen, the diameter of one of the 22 pure silver rods of the multichannel electrode was set at 2 mm.

As shown in Fig. 18, in analyzing the movement of the lower end of the hyoid body, the lower end of the hyoid body in a resting state is used as the origin of coordinates, and the moving distance (mm) of the X-axis and Y-axis during swallowing is calculated. After that, moving average was performed and smoothing was performed.

As shown in Figure 19, from the image results, A is the start of the movement of the tongue tip with the start of voluntary swallowing, B is the start of hyoid lifting movement, C is the start of rapid hyoid lifting with the start of the swallowing reflex, and D is the tongue tip movement. The bone reached its most anterior superior position, E was determined to be the start of rapid descent of the hyoid bone, and F was determined to be the resting position of the hyoid bone after swallowing, and the interval from A to F was defined as the interval for predicting the movement of the hyoid bone.

As shown in Figure 20, from the movement of the hyoid bone that serves as a teacher signal, the start point is set as the start point of tongue tip movement accompanying the start of voluntary swallowing (see Figure 19), and the end point is determined after the end of swallowing, as shown in Figure 20. The hyoid bone was placed in resting position F.

As shown in Figure 21, the sEMG signal of the suprahyoid muscle group, the sEMG signal of the infrahyoid muscle group, and the laryngeal movement detected by the laryngeal behavior sensor (stretching strain sensor) are combined with respect to the movement of the hyoid bone. , the operating range was determined.

Next, the learning device will be explained.
Here, we used LSTM, which is suitable for predicting time-series data, to predict the motion of the hyoid bone from sEMG signals. LSTM (Long short-term memory) is a recurrent neural network (RNN) architecture used in the field of deep learning. This method solves problems that cannot be learned using dependencies, and can learn both long-term and short-term time dependencies.

RNN is a type of deep learning that is an extension of neural networks, and is a method with excellent performance in the field of time-series data. Currently, it is often used in the fields of machine translation and speech recognition. As shown in FIG. 22, in order to handle variable length data in a neural network, the network structure is such that values obtained in the intermediate layer are input again to the intermediate layer. The intermediate layer h _t looks at the input x _t and outputs the value h _t . Loops allowed information to be passed from one step of the network to the next. However, the longer the prediction is made, the more difficult it becomes for RNNs to associate and learn information if the information related to the predicted value is located at the beginning.

As shown in FIG. 23, LSTM has different parts from RNN: CEC (Constant Error Carousel, also called memory cell, cell), input gate, output gate, and forget gate. CEC is a unit for storing past data and is also called a memory cell. By introducing this, it becomes possible to detect long-period regularity. Input gates and output gates are made to adapt to new patterns when new inputs and outputs arrive during the learning process, making it possible to deal with the problems of input weight collisions and output weight collisions that occurred in RNNs. . In the case of a model without a forgetting gate, even if an input with a large change comes, the influence of that input will be relatively small, and the output will only be the same as before. By introducing a forgetting gate to deal with this problem, it became possible to update the cell state all at once when the input pattern changed significantly.

As shown in FIG. 24, the internal details of the LSTM block are as shown in FIG. 24(a). Each line carries an entire vector from the output of one node to the input of another node, as shown in FIG. 25(b). The small circles represent single point operations, such as vector addition, and the rectangular boxes are the layers of the neural network being trained. A line that joins means a connection, and a line that diverges means that the content is copied and the copy goes somewhere else.

FIG. 25 shows a forgetting gate, and as shown in FIG. 25(a), in the LSTM block, information to be discarded is determined in the first step. This determination is made by a sigmoid layer called the "forgetting gate layer." It looks at the input h _t-1 and x _t and outputs a number between 0 and 1 for each number in the cell state C _t-1 . 1 represents "completely retained" and 0 represents "completely removed". Further, the equation at that time is shown in equation 3, and the equation for σ, which is the gate activation function, is shown in equation 4.

Here, W represents the input weight and b represents the bias.

FIG. 25(b) shows the input gate, and the next step is to determine the new state to save in the cell state. There are two parts to this. First, a sigmoid layer called the "input gate layer" determines which values to update. The tanh layer then creates a new candidate location vector C _t that is added to the cell state. The next step is to combine these two to update the state. The equations at that time are expressed in Equations 5 and 6. Here, tanh represents the hyperbolic tangent function.

FIG. 25(c) shows cell updating, in which the old cell state C _t−1 is updated to the new cell state C _t . Multiply the old cell state by f _t and forget what was previously determined to be forgotten. Then, add i _t ×vector C _t . This is the new candidate value scaled by the percentage determined to update each state value. The equation at that time is expressed in Equation 7.

FIG. 25(d) shows an output gate, and finally, it is necessary to determine what to output. This output is done based on the cell state. First, run the sigmoid layer. This layer determines which part of the cell state to output, and applies tanh to the cell state (to compress the value between -1 and 1) to output only the determined part, Multiply it by the output from the sigmoid layer. The equations at that time are shown in Equation 8 and Equation 9.

Next, prediction of hyoid bone motion using sEMG signals will be explained.
The test subject was a woman in her 70s who had doubts about her oral function, visited Iwate Medical University Hospital, and consented to a VF test. This test was approved by the Iwate Medical University School of Dentistry Ethics Review Committee (No. 01304) and the Iwate University Research Ethics Review Committee (No. 201905), and was conducted within the scope of normal tests.

Regarding the measurement method and measurement operation, as shown in Figure 26, during the VF examination, an X-ray fluoroscope (SafireII ZS-100, Shimadzu Corporation) was used to photograph the movement of the hyoid bone in a chair sitting position from the side of the neck. The test food was 3 ml of a 98 w/w% barium sulfate solution with a 1% thickening, and the examiner injected it into the sublingual region of the subject using a syringe, and the subject swallowed it according to the examiner's instructions. The number of tests was 3 trials, and the shooting speed was 30 fps. In addition, as shown in Figures 27 and 28, at the same time as the VF test, a 22-channel flexible electrode for the suprahyoid muscle group was placed on the mandible, a 22-channel flexible electrode for the infrahyoid muscle group on the neck, and an ear electrode was placed on the earlobe. I installed it. The suprahyoid muscle group was attached at a position where the distance from electrodes 1 and 2 to the mentalis as shown in FIG. 7(c) was between 25 mm and 30 mm, and the electrodes did not touch the jawbone. The infrahyoid muscle group was attached so that the electrodes No. 5 and 6 in FIG. 7(d) were located at the part that most protruded in front of the thyroid cartilage (Adam's apple).

To measure laryngeal movement, a laryngeal behavior sensor (stretch strain sensor) was attached to the thyroid cartilage and connected to the same multifunctional myoelectric potential measurement device used for myoelectric measurement. Note that the amplification factor of the sEMG signal was 125 times and the sampling frequency of the sEMG signal and the laryngeal behavior sensor was 2000 Hz. In order to synchronize the myoelectric potential and laryngeal movement measurement system with the VF examination, a synchronization microphone was connected to the multifunctional myoelectric potential measurement device and the X-ray fluoroscopy device, and a sound trigger input was performed. Figure 29 shows the process of quantifying the movement of the hyoid bone, and the movement of the hyoid bone in the X-axis direction (anterior-posterior direction) and the Y-axis direction (vertical direction) of the hyoid bone is calculated according to the VF video. The movement is graphed. Note that the main purpose of this measurement was the VF test, and the measurement of myoelectric potential was performed incidentally.

Regarding the prediction of hyoid bone motion using LSTM, the learning conditions as analysis conditions are: What happens when the number of muscle groups used for input increases in the learning input value, and what effect does adding laryngeal motion have on prediction accuracy? In order to verify whether the given value is given, verification was performed using six patterns shown in FIG.

The training and prediction datasets used as analysis conditions were tested three times, so they were set as A, B, and C, respectively. Then, six types of learning and test data were created as shown in FIG. Data amplification was performed for each training data to improve prediction accuracy. As shown in FIG. 32, in the present invention, points are taken at every 30 samples so as not to include the same points, and the data is amplified up to 30 samples. Since each data amplified to 30 pieces is 1/30 of the number of samples of the original data, primary data interpolation is performed to match the number of samples of the original data. according to the number of samples.

Regarding the effectiveness of data amplification, the present invention amplifies up to 30 pieces of data, but FIG. 33 shows a comparison with the case where data is not amplified. Figure 33 shows the relationship between data amplification and RMSE, and it can be seen from the error verification using RMSE that the error between the predicted value and the measured value is reduced by data amplification, which shows the effectiveness of data amplification.

FIG. 34 shows an example of learning, and when the input values for learning are the suprahyoid muscle group, the infrahyoid muscle group, and laryngeal movement, the total number of input dimensions is 265. The output value is two-dimensional: the anteroposterior and vertical movements of the hyoid bone. Furthermore, when making predictions using test data, the prediction results are smoothed.

Next, evaluation indicators will be explained.
RMSE (Root Mean Square Error) and Pearson's product moment correlation coefficient were used to verify the difference between the actual measured values and predicted values obtained by VF video. The results of RMSE and correlation coefficient are the average values of six learning results. RMSE is expressed by Equation 10 and is the most common performance index for regression models, and it can be said that the smaller the error, the better the accuracy.

Pearson's product moment correlation coefficient (hereinafter referred to as correlation coefficient) is expressed by Equation 11, and it can be said that the larger the value, the higher the waveform similarity and the higher the prediction accuracy.

Here, y _obs represents the actual measured value of the output, and y _pred represents the predicted value of the output.

Next, the results will be explained.
Regarding the collected data, FIG. 35 shows the sEMG signal and laryngeal movement for one of the three swallowings obtained by the present invention.

Moreover, FIG. 36 shows time series data of the sEMG signal, RMS, CC of each muscle group, laryngeal movement, and hyoid bone movement. Note that qr used here refers to the quefrency of cepstral coefficients (CC).

Regarding the prediction results in the X-axis direction (anterior-posterior direction) and Y-axis direction (vertical direction), the X-axis direction and Y-axis direction of the hyoid bone are set as shown in Figure 17, and the actual measured values of the movement of the hyoid bone in each direction are calculated. The result of one of the six predicted values is shown in FIG.

The trajectory of the movement of the hyoid bone in the sagittal plane (XY coordinate plane) corresponding to the result in FIG. 37 is as shown in FIG. 38. Note that in FIG. 38, the values are shown to include actual measured values and predicted values.

Next, the accuracy of predicting the movement of the hyoid bone in combinations of muscle groups will be explained.
FIG. 39 shows the results of RMSE, which are the results in the X-axis direction and Y-axis direction for the combination of muscle groups.

FIG. 40 shows the results of the correlation coefficients, which are the results in the X-axis direction and Y-axis direction for the combination of muscle groups.

These results suggest that the suprahyoid muscles, including the geniohyoid muscle, act strongly as the muscles that move the hyoid bone in the X-axis direction. Furthermore, it is thought that the muscles that move the hyoid bone in the Y-axis direction are not moved by each muscle group alone, but by the coordinated movement of both muscle groups. Since the larynx is pulled up by the subhyoid muscles as the hyoid moves upward during swallowing, it is thought that adding information on laryngeal movement from the elastic strain sensor improved accuracy.

Among the prediction results of the sagittal plane of the hyoid bone, there were many prediction results that drew a triangular trajectory similar to the actual measurement value. This suggests that accurate predictions could be made from the movement of the hyoid bone during swallowing.

Next, another example will be shown in which the movement of the bolus rather than the movement of the hyoid bone is estimated. The training signal is the position change of the tip of the food bolus (3 ml of 98w/w% barium sulfate solution with 1% thickening) photographed by VF, and the learning conditions such as learning data including features are the estimation of the motion of the hyoid bone. The same as in the case of As shown in Figures 41 and 42, the actual measured values and estimated values (Learning A - Prediction C) of the XY coordinates (origin is the lower end of the anterior border of the fifth cervical vertebra) when the bolus tip passes through the esophageal entrance are almost the same. It can be confirmed that the movement of the bolus can be predicted with high accuracy.

The functions and effects of the eating and swallowing function evaluation method and the eating and swallowing function evaluation system described above will be explained.
For example, in addition to the VF test, there is an echo test using an ultrasonic diagnostic device that can evaluate swallowing function and the movement of the hyoid bone. However, previous research has shown that sticky gruel is easy to detect using echo imaging, but when swallowing liquid fluids such as water or jelly, it is difficult to observe the instantaneous dynamics from the swallowing reflex to aspiration; Therefore, it is difficult to identify aspiration. Furthermore, the image observed differs depending on how the probe is applied, making it difficult to observe with high reproducibility. In this regard, according to the present invention, it is possible to predict not only the hyoid bone but also the movement of at least one of the swallowing organs and the bolus that can be observed in VF and VE, such as the timing of laryngeal closure and the opening and closing of the esophageal entrance. , it can be expected to be an evaluation method without risks such as radiation exposure. In the present invention, if only one swallowing data can be acquired, the accuracy is RMSE of 1.03 and correlation coefficient of 0.64 in the X-axis direction, and RMSE of 1.20 and correlation coefficient of 0.96 in the Y-axis direction. It was confirmed that swallowing movements under the same conditions could be predicted.

According to the embodiment of the present invention, a contrast video process (VF video process), a preprocessing process, a biological signal detection process synchronized with the VF video process, a feature amount extraction process, a learning process, and a prediction process are provided. In the learning process, the motion interval is determined by matching the feature values to the teacher signal, and the teacher uses LSTM (long and short-term memory), a recurrent neural network architecture that learns long-term time dependence and short-term time dependence. The motion of the hyoid bone is learned based on signals and feature amounts. LSTM solves problems that cannot be learned using long-term time dependencies when trained with traditional RNNs, and can learn both long-term and short-term time dependencies. When new inputs and outputs are received during the learning process, new patterns are applied, making it possible to deal with the problems of input weight collisions and output weight collisions that occur in RNNs. For this reason, the subject learns the characteristics related to the hyoid bone movement during swallowing by collecting biological signals with the sensor unit at the same time as the VF test only once, and from the second time onwards (prediction process), the VF test is performed. It is possible to predict the movement of the hyoid bone without any. As a result, there is no need for an X-ray fluoroscope, and it is possible to perform a non-invasive and low-risk assessment of eating and swallowing function that predicts hyoid bone movement easily at the bedside or in home medical care. In addition, if the same amount and physical properties are swallowed in the same way, the learning data will be appropriate data in one time, but to make the learning data more suitable, it is necessary to change the amount and physical properties. Regarding swallowing under certain conditions, swallowing data under that condition may be added to learning and used as learning data.

Furthermore, if the subject learns the motion of the subject's hyoid bone at least once at a hospital with an X-ray fluoroscope, from then on, they can predict the motion of the hyoid bone from any location without VF examination. Swallowing function evaluation and training can be performed. In addition, if the same amount and physical properties are swallowed in the same way, the learning data will be appropriate data in one time, but to make the learning data more suitable, it is necessary to change the amount and physical properties. Swallowing under that condition can be estimated more appropriately by adding the swallowing data under that condition to learning and using it as learning data.

Furthermore, in the biosignal detection step, the suprahyoid muscle biosignal, the infrahyoid muscle biosignal, and the laryngeal behavior signal are detected, so the movement of the hyoid bone can be predicted with higher accuracy.

Furthermore, in the learning process and prediction process, one source data is shifted and amplified into multiple pieces at a predetermined period so that the same coordinate data is not included, so the predicted value and actual measurement data are combined in the first learning process. It reduces the error in values and enables more accurate prediction of hyoid bone movement.

Furthermore, in the preprocessing step, the XY coordinate system in the subject's side view is set, and the coordinates of the XY coordinate system are acquired with the lower end of the hyoid bone as the position of the hyoid bone. Prediction of motion can be made easier to understand.

In the embodiment, the laryngeal behavior sensor is a stretchable strain sensor, but the laryngeal movement sensor is not limited to this, and a pressure sensor, an acceleration sensor, a non-contact type sensor, etc. may be used to measure the movement of the larynx. In addition, in the embodiment, the myoelectric sensor for the suprahyoid muscle group and the myoelectric sensor for the infrahyoid muscle group are arranged in an array, but it is possible to use a plurality of electrodes (at least two channels) even if the electrodes are not arranged in an array at equal intervals. The above electrodes may be provided. Swallowing sounds may also be added. In addition, in the above description and figures, there are parts described as "swallowing" even if the part means eating and swallowing.

In addition, in the example, as a prediction method for time series data, LSTM (Long Short-Term Memory) derived from RNN (Recurrent Neural Network) was used, but the present invention is not limited to this. ARMA (Autoregressive Moving Average), ARIMA (Autoregressive Integrated Moving Average), SARIMA ( A time-series data prediction method such as a Seasonal AutoRegressive Integrated Moving Average) model may also be used.

In addition, in the example, the swallowing imaging process was a swallowing morphological test (VF test) that can be observed under X-ray fluoroscopy, but it is not limited to this, and the swallowing imaging process is a swallowing endoscopic test (VE test) using an endoscope. It may be done by observation, or furthermore, the movement of the hyoid bone may be observed by echo and expressed in coordinates. The images taken in the swallowing photographing process are not limited to moving images, but may be still images as long as continuous changes can be seen, or the still images may be frame-by-frame images.

In addition, in the prediction process, the movements of the swallowing organs and the bolus are predicted from the biological signals during ingestion and swallowing, but this is not limited to the prediction process. It may also be used to predict movements of swallowing organs and food intake from biological signals during indirect training (so-called biofeedback training). Ingestion that trains the swallowing function, such as direct training using food, which is a training method for the swallowing function, evaluation of the movement of swallowing organs in indirect training (basic training) that does not use food, and use for biofeedback training. Suitable for eating and swallowing function training techniques.

That is, the present invention is not limited to the examples as long as the functions and effects of the present invention are achieved.

The present invention detects biosignals during ingestion and swallowing, extracts features from the detected biosignals, predicts the movement of at least one of the swallowing organs and the bolus, and evaluates the ingestion and swallowing function. Suitable for swallowing function evaluation technology. In addition, we train the eating and swallowing function, such as direct training using food, which is a method of training the swallowing function, and indirect training without food (basic training) to evaluate the movement of the swallowing organs and use it for biofeedback training. It is suitable for eating and swallowing function training techniques.

DESCRIPTION OF SYMBOLS 10... Eating and swallowing function evaluation system, 20... Sensor section, 21... Myoelectric sensor for suprahyoid muscle group, 21a... Electrode, 22... Myoelectric sensor for infrahyoid muscle group, 22a... Electrode, 25... Laryngeal behavior Sensor (stretchable strain sensor), 30... Multifunctional myoelectric potential measurement device, 40... Analysis section, 50... X-ray fluoroscopy device, 51... Preprocessing section, 60... Subject.

Claims (5)

a swallowing photographing step of photographing the movement of the food bolus ingested by the subject and the movements of various swallowing organs;
a preprocessing step of acquiring the positions of the swallowing organs and the bolus from the image of the swallowing photographing step and digitizing them as coordinates to create a teacher signal of the movement of the swallowing organs and the bolus;
a biosignal detection step of detecting biosignals during ingestion and swallowing with a sensor unit placed on a predetermined skin surface of the subject in synchronization with the swallowing imaging step;
a feature quantity extraction step of extracting a feature quantity from the biological signal in an analysis unit;
The movements of the swallowing organs and the bolus are learned based on the teacher signal and the feature quantities using a time-series data prediction method, and the movements of at least one of the swallowing organs and the bolus are learned from the feature quantities. a learning process that generates a model that can predict
a prediction step of predicting the movement of at least one of the swallowing organs and the bolus from the feature amounts using the prediction model generated in the learning step ;
A swallowing movement prediction method using time-series data prediction, characterized by comprising: A swallowing movement prediction method using time series data prediction according to claim 1,
In the biological signal detection step, a suprahyoid muscle group biosignal is detected by a myoelectric sensor for the suprahyoid muscle group placed in the suprahyoid muscle group portion, and a biological signal of the suprahyoid muscle group is detected in the infrahyoid muscle group portion placed in the infrahyoid muscle group portion. The group myoelectric sensor detects the subhyoid muscle group biological signals, the laryngeal behavior sensor placed in the larynx detects the laryngeal behavior signals,
In the feature amount extraction step, feature amounts are extracted from the suprahyoid muscle group biosignal, the infrahyoid muscle group biosignal, and the laryngeal behavior signal as the biosignal. Swallowing movement prediction method using data prediction. A swallowing movement prediction method using time series data prediction according to claim 1 or claim 2,
In the learning step, the coordinate data of the swallowing organs and the food bolus are used as learning data, and the learning data is divided into multiple pieces by shifting one source data at a predetermined period so that the same coordinate data is not included. A swallowing movement prediction method using time-series data prediction characterized by amplifying. A swallowing movement prediction method using time series data prediction according to any one of claims 1 to 3,
In the preprocessing step, a coordinate system and its origin are determined for obtaining the coordinate data of the swallowing organs and the food bolus, with the lower end of the anterior edge of the fifth cervical vertebra of the subject as the origin, and the upper end of the anterior edge of the third cervical vertebra of the subject as the origin. By setting a coordinate system with , as a point on one axis, the positions of the swallowing organs and bolus in the sagittal plane or the frontal plane of the subject are obtained using the coordinates of the coordinate system. A swallowing movement prediction method using time-series data prediction. The positions of the swallowing organs and the bolus are obtained from images of the movements of the bolus ingested by the subject and the movements of the swallowing organs, and are digitized as coordinates, and the movements of the swallowing organs and the bolus are calculated. a preprocessing unit that creates a teacher signal;
a sensor unit that is placed on a predetermined skin surface of the subject and detects biological signals during ingestion and swallowing in synchronization with the imaging of the image;
A feature quantity is extracted from the biological signal, and the movement of at least one of the swallowing organs and the food bolus is learned based on the teacher signal and the feature quantity using a time-series data prediction method, and the feature quantity is extracted from the biological signal. A model is generated that can predict the movement of at least one of the swallowing organs and the bolus, and the generated prediction model is used to predict the movement of at least one of the swallowing organs and the bolus from the feature values. A swallowing movement prediction system using time-series data prediction, comprising: an analysis unit that performs prediction;

Images