JP2020042598A - State prediction method and device by individual characteristic separation from biological signal data - Google Patents

Abstract

To provide a state prediction method and device that, without performing additional learning or adjustment for each subjected body, extract and separate a signal component relevant to an individual difference from biological signal data on the subjected body, and enable upgrading of disease diagnosis accuracy of the subjected body and visualization of a disease relevant signal.SOLUTION: A state prediction method is configured to: predict a state of a subjected body using a deep layer generation model by a deep layer neural network, with time-series biological data obtained by measuring biological signals of a plurality of subjected bodies as an input; establish a deep layer generation model by associating a state label staying unchanged in a one measurement, an amount of characteristic representing an individual difference, and a signal component not relying upon the state label and the individual difference but changeable in timewise with the biological signal data; then input the biological signal data on the plurality of subjected bodies, and learn the deep layer generation model; and extract the amount of characteristic representing the individual difference and the signal component from the input biological signal data, and perform a state prediction of the subjected body using the post-extracted signal component.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for predicting various disease states including mental disorders from biological signal data such as brain function image data, brain waves, heart rate and sweating data.

  In recent years, the population with various diseases including mental illness is increasing, and early detection and early treatment are urgent issues. If the degree of disease progression can be quantified from biological signal data, various applications such as evaluation of treatment methods as well as early detection are possible. Therefore, discovery of biomarkers related to diseases by machine learning using medical data of various biological signal data is expected. The discovery of such biomarkers not only elucidates the mechanism of the disease, but also enables early diagnosis and quantitative evaluation of the disease, contributes to the quality of life of patients, and the efficacy of treatment Evaluation is also possible. However, although medical data is being stored, the size of the data set is much smaller than that of general image recognition, and the analysis requires complicated preprocessing and procedures such as feature selection and dimension reduction. is there.

  In addition, information such as the age, sex, physical characteristics, and living environment of the subject exists as signal components in the biological signal data. It is difficult to extract and separate only the data derived from the disease from the biological signal data. Further, since the resolution and sensitivity (S / N ratio) are different for each measuring device that measures the biological signal data, the characteristics of these measuring devices also exist as unnecessary signal components in the biological signal data. Taking brain function image data as an example of biological signal data, there are individual differences in the shape of the head and the functional brain map.

  Under such circumstances, research results using a generative model technique for analyzing brain function image data have been reported (for example, see Non-Patent Documents 1 to 3). For example, using a hidden Markov model, the research results of modeling the dynamics of a large-scale nervous system of the brain behind brain function image data (Non-Patent Document 1), a response generation to a stimulus and a linear generation model Research results (Non-Patent Document 2) and the like, in which a stimulus is estimated from brain function image data by modeling using the above method, have been reported. In these modelings, modeling is performed using a combination of a model common to all subjects and a model expressing individual differences of each subject. Parts need to be adjusted anew.

  On the other hand, high performance results have been reported for various tasks by automatically learning higher-order features required for the purpose from given data using a deep neural network called deep learning. Deep learning mainly consists of a multi-layered perceptron that performs “supervised” classification and a self-encoder (auto-encoder) that can reduce “unsupervised” dimensions, which has been used for analyzing brain functional image data for some time. I have. Deep learning has a structure that can implement a generation model called a deep generation model.

  The present inventors have already shown that using this deep-layer generation model, in addition to brain function image data, a model of the state of a subject (presence or absence of a disease) can be used to diagnose a subject's disease with high accuracy (non- See Patent Document 3). Compared to the classification model generally used for classification, the generated model has a high constraint even if the assumed structure is close to the true structure, even for datasets with a small number of samples, because its structure acts as a kind of constraint. It is known that classification can be performed with accuracy (see Non-Patent Document 4). An approach is also known in which the related signal component is removed by giving the age and gender of the subject as known information (see Non-Patent Document 5). Therefore, by realizing the structure behind the brain function image data on the deep generation model, there is a possibility of improving diagnostic accuracy and further analyzing.

H. Suk et al., “State-space model with deep learning for functional dynamics estimation in resting-state fMRI”, NeuroImage, vol.129, pp.292-307, 2016. P. Chen et al., “A Reduced-Dimension fMRI Shared Response Model”, Advances in Neural Information Processing Systems 28, 2015. T. Matsubara et al., “Deep Neural Generative Model for fMRI Image Based Diagnosis of Mental Disorder Diagnosis”, arXiv: 1712.06260, 2017. A. Prasad et al., “On Separability of Loss Functions, and Revisiting Discriminative Vs Generative Models”, in Advances in Neural Information Processing Systems (NIPS), pp. 7053-7062, 2017. N. Yahata et al., “A small number of abnormal brain connections predicts adult autism spectrum disorder”, Nature Communications, vol.7, no.7, pp.11254, 2016.

As described above, the present inventors are conducting research to improve the accuracy of disease diagnosis using a structured deep generation model in analyzing brain function image data (Non-Patent Document 3).
However, the method disclosed in Non-Patent Document 3 has a problem that signal components of individual differences cannot be separated, and the accuracy of disease diagnosis is limited. Although an approach of separating individual differences using machine learning for each subject is known (Non-Patent Document 2), it is necessary to perform additional learning and adjustment of a huge amount of measurement data for each individual subject. There's a problem. An approach is known in which the related signal components are removed by giving the age and gender of the subject as known information (Non-patent Document 5). There is a problem that it is difficult to remove difficult signal components.

  In view of such circumstances, the present invention relates to individual differences such as the age, gender, physical characteristics, and living environment of a subject from biological signal data of the subject without performing additional learning or adjustment for each subject. It is an object of the present invention to provide a state prediction method and apparatus that extract and separate feature amounts and enable improvement of disease diagnosis accuracy of a subject and visualization of a disease-related signal.

In order to solve the above-described problems, the state prediction method of the present invention uses a deep neural network to generate a test using a deep neural network and a time series biological signal data obtained by measuring biological signals of a plurality of subjects. A method for predicting a body condition, comprising the following steps 1) to 3).
1) Deep generation by associating state labels that do not change in a single measurement, feature quantities representing individual differences, and signal components that do not depend on the state labels and individual differences but can change over time with biological signal data Steps to build the model.
2) inputting the biological signal data of a plurality of subjects and learning a deep generation model;
3) A feature quantity representing individual differences and a signal component which is not dependent on the state label and individual differences but can vary with time are extracted from the input biosignal data, and the extracted signal components are used for testing using the separated signal components. Performing a body state prediction;

According to the present invention, for analysis of biological signal data such as brain function image data, a deep generation model in which individual differences are modeled is constructed, and from the input biological signal data, a feature quantity expressing individual differences, Extract signal components that do not depend on the state label and individual differences but can change over time, and predict the state of the subject using the separated signal components to improve the accuracy of disease diagnosis of the subject and the disease-related Signal visualization can be realized.
The state prediction method of the present invention is based on the assumption that a plurality of biological signal data shares a latent variable that cannot be directly observed (a feature quantity representing an individual difference) and constructs a deep generation model. That is, it is assumed that a plurality of pieces of biosignal data of the same subject share a feature amount expressing individual differences. In this specification, a subject is used to mean a living body such as a human or an animal for measuring biological signal data.

The state label is a label indicating the presence or absence of a state, and does not change in one measurement. When the status label indicates a disease state, it is a diagnostic label for the presence or absence of a disease. The status label may indicate not only the presence or absence of the disease but also the severity other than the presence or absence (a grade corresponds to chronic pain or the like).
Individual differences are modeled as latent variables shared by all biosignal data (for example, brain function image data) obtained from a single subject, and by inferring from these biosignal data, unknown It is possible to generalize to the body. Here, the feature quantity expressing the individual difference is, as described above, a latent variable that cannot be directly observed and expresses the individual difference in the biological signal data.

  The deep generation model is a generation model capable of expressing input data by a multidimensional simultaneous probability distribution and efficiently expressing a characteristic signal component appearing in the input data. In the present invention, a state label that does not change in a single measurement, a feature quantity expressing individual differences, and a signal component that does not depend on the state labels and individual differences but can change over time are associated with biological signal data. Build a deep generation model. Then, the biological signal data of a plurality of subjects are input to learn a deep generation model, and from the input biological signal data, a feature quantity expressing an individual difference, a state label and an individual difference are used, but temporally. Then, a signal component that can be changed to a predetermined value is extracted, and the state of the subject is predicted using the separated signal component.

In the deep generation model, a plurality of pieces of biosignal data of the same subject share a feature amount representing an individual difference. That is, one piece of biosignal data shares a feature amount expressing individual difference with another piece of biosignal data obtained from the same subject.
By assuming this, when decomposing the biological signal data into signal components and reconstructing the original biological signal data again, the constraint condition that the feature value expressing the same individual difference takes the same value Can be given.

  In the step of constructing the deep generation model in the state prediction method of the present invention, the feature amount representing the measurement environment difference is further associated with the biological signal data, and the state label, the feature amount representing the individual difference, and the measurement environment difference are represented. It is preferable that a depth generation model is constructed by associating the characteristic amount and the signal component that does not depend on the state label, the individual difference, and the measurement environment difference but can change over time with the biological signal data.

Here, the characteristic amount expressing the measurement environment difference is a characteristic amount of a measuring device that measures biological signal data, for example, a signal component that fluctuates depending on characteristics such as resolution and sensitivity (S / N ratio) of the device.
In biological signal data, the shared range differs between individual differences and environmental differences. Therefore, biosignal data obtained from the same subject shares a single feature (= individual difference), and biosignal data obtained from a plurality of subjects measured in a single measurement environment is A single feature (= measurement environment difference) is shared. That is, one biological signal data shares a characteristic amount expressing individual difference with another biological signal data obtained from the same subject, and one biological signal data is another biological signal data obtained from the same measurement environment. The signal data and the feature quantity expressing the measurement environment difference are shared.
By assuming this, when the biological signal data is decomposed into signal components and the original biological signal data is reconstructed again, the feature amount expressing the same individual difference and the feature amount expressing the measurement environment difference Have the same value.

In the above-described deep generation model, a plurality of pieces of biosignal data of the same subject share a feature amount expressing an individual difference. In particular, when the deep generation model is configured by a VAE (Variational Auto-Encoder), the feature quantity representing the individual difference is modeled as a latent variable following a prior distribution. VAE is a typical generation model using deep learning, and it is known that latent variables (features) existing in data can be obtained with high accuracy.
Further, in the above-described deep generation model, a plurality of pieces of biosignal data measured in the same measurement environment share a feature amount representing a measurement environment difference. In particular, when the deep generation model is configured by VAE, the feature quantity expressing the measurement environment difference is modeled as a latent variable following a prior distribution.

In the above-described deep learning step, a first neural network that outputs an activity value with one biosignal data and a state label associated with the biometric signal data being input, And a second neural network that outputs a feature value expressing an individual difference by using an average of the activity values obtained from the set as an input.
Using a group encoder in which two neural networks, a first neural network and a second neural network, are overlapped (not hierarchized), a feature amount dependent on individual differences is estimated. Estimate the weight parameters for the latent variables following the distribution. The first neural network outputs an activation value by using a state label as an input. Then, in the second neural network, the posterior distribution of the feature quantity expressing the individual difference is output using re-parameterization with the average of the activity values as input.
Similarly, in the step of deep learning, for estimating a feature amount representing a measurement environment difference, an activity value is output using one or more biosignal data measured in the same environment and a state label associated therewith. A collective encoder configured by a first neural network and a second neural network that outputs a feature quantity expressing a measurement environment difference with an average of activity values obtained from a set of biosignal data as an input. Used.
In the case of estimating the feature value expressing the measurement environment difference, since the biometric signal data of one subject cannot be distinguished from the feature value expressing the individual difference, the data of multiple subjects measured in the same environment It is preferable to use

The state prediction method of the present invention further includes a step of displaying a result of the state prediction of the subject.
The biological signal data in the present invention includes fMRI (functional Magnetic Resonance Imaging) image data, MRI (Magnetic Resonance Imaging) image data, PET (Positron Emission Tomography) image data, ultrasonic image data, brain wave data, electrocardiogram data, and heart rate data. , Perspiration amount data, blood oxygen concentration data, or blood sugar level data.
Here, while the fMRI image data is time-series data, the MRI image data is generally not time-series, but even if it is not time-series data, MRI images taken a plurality of times for follow-up observation, etc. The data can be processed according to time-series biosignal data.

  The deep generation model in the present invention includes AE (Auto-Encoder) including VAE (Variational Auto-Encoder), GAN (Generative Adversarial Nets), GMM (Generative Moment Matching), EP (Energy-based Probabilistic Model), and FG (Flow -based Generative Model) or AR (Auto-regressive Model). Only when VAE is used as the deep generation model, re-parameterization is performed, that is, the posterior distribution of the feature amount depending on individual differences is expressed by using the average of the activation values in the second neural network of the above-described collective encoder as an input. It is possible to output parameters to be performed.

Next, the state prediction device of the present invention will be described.
The state prediction apparatus of the present invention is an apparatus that predicts the state of a subject using a deep generation model based on a deep neural network, with time-series biosignal data obtained by measuring biosignals of a plurality of subjects as inputs. And includes the following 1) to 3).
1) Deep generation by associating state labels that do not change in a single measurement, feature quantities representing individual differences, and signal components that do not depend on the state labels and individual differences but can change over time with biological signal data A generation model construction unit that builds a model.
2) A learning unit that inputs biological signal data of a plurality of subjects and learns a deep generation model.
3) A feature quantity representing individual differences and a signal component which is not dependent on the state label and individual differences but can vary with time are extracted from the input biosignal data, and the extracted signal components are used for testing using the separated signal components. A state prediction unit that predicts the state of the body.

  In the generation model constructing unit of the state prediction apparatus of the present invention, the feature quantity representing the measurement environment difference is further associated with the biological signal data, and the state label, the feature quantity representing the individual difference, and the feature quantity representing the measurement environment difference. Preferably, a deep generation model is constructed by associating the time-independent signal components which do not depend on the state label, the individual difference and the measurement environment with the biological signal data.

In the deep generation model of the state prediction device of the present invention, a plurality of biosignal data of the same subject share a feature amount expressing individual difference, especially when the deep generation model is configured by VAE. , The feature quantity expressing the individual difference is modeled as a latent variable following a prior distribution.
Similarly, in the deep generation model, a plurality of biological signal data measured in the same measurement environment share a feature amount expressing the measurement environment difference, especially when the deep generation model is configured by VAE. The feature quantity representing the measurement environment difference is modeled as a latent variable following a prior distribution.

In the learning unit of the state prediction device of the present invention, a first neural network that outputs one biosignal data and a state label as an input and outputs an activity value is used for estimating a feature amount expressing an individual difference; And a second neural network that outputs an amount of feature representing an individual difference, using an average of the activity values obtained from the set of...
Similarly, in the learning unit, a first neural network that outputs one biosignal data and a state label as inputs and outputs an activity value, and a set of biosignal data are used for estimating a feature amount representing a measurement environment difference. A collective encoder composed of a second neural network that outputs a feature quantity representing a measurement environment difference with the average of the activity values obtained as an input is used.
The state prediction device of the present invention further includes a display unit that displays a result of the state prediction of the subject.

  The state prediction program of the present invention is a program for causing a computer to execute each step in the above-described state prediction method of the present invention. Further, the state prediction program of the present invention is a program for causing a computer to function as the generation model construction unit, the learning unit, and the state prediction unit in the above-described state prediction device of the present invention.

  According to the present invention, without performing additional learning or adjustment for each subject, extract and separate features related to individual differences from the biological signal data of the subject, to improve disease diagnosis accuracy and disease-related signals This has the effect of visualizing. Further, it is possible to extract and separate feature amounts related to the measurement environment difference, and to further improve accuracy.

Flow diagram of state prediction method Functional block diagram of state prediction device Configuration schematic diagram of the deep generation model of the first embodiment Conceptual diagram of deep neural network Schematic diagram of the configuration of the group encoder Illustration of collective encoder Configuration schematic diagram of encoder and decoder Conceptual diagram of encoding / decoding Example of data structure of time-series brain function image data Configuration schematic diagram of the deep generation model of the second embodiment Conceptual diagram of encoding / decoding according to the second embodiment

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. It should be noted that the scope of the present invention is not limited to the following embodiments and illustrated examples, and various changes and modifications are possible.

The state prediction method and the state prediction apparatus of the present invention will be described by taking as an example the brain function image data of magnetic resonance functional imaging (fMRI) as biological signal data. fMRI uses MRI to visualize the hemodynamic response related to brain activity in humans and animals.Based on the structural information provided by MRI, an image of the site where brain functional activity occurred I do.
In the state prediction method according to the present embodiment, the state of the subject is determined by using a time-series brain function image data obtained by measuring brain function images of a plurality of subjects and using a deep generation model by a deep neural network. Predict. First, a state label that does not change in a single measurement, a feature amount that represents individual differences, and a signal component that does not depend on the state label and the individual difference but can change over time are associated with brain function image data, Build a deep generation model. Next, time-series brain function image data of a plurality of subjects is input to the constructed deep generation model to learn the deep generation model. Then, from the input brain function image data, a feature representing the individual difference and a signal component which is not dependent on the state label and the individual difference but can change over time are extracted, and the extracted feature and the signal component are separated. The state of the subject is predicted using the signal component after the above.

FIG. 1 shows an embodiment of the flow of the state prediction method of the present invention. In the flow of FIG. 1, the flow is shown focusing on the feature amount expressing the individual difference. However, in addition to the feature amount expressing the individual difference, the flow does not depend on the state label and the individual difference but may change over time. The signal components are also processed in the same manner as the feature values expressing individual differences.
As shown in FIG. 1, a feature quantity representing an individual difference, a state label, and a signal component which is not dependent on the state label and the individual difference but can change with time are associated with brain function image data, A generation model is constructed (step S01). In order to optimize the constructed deep generation model, brain function image data of a plurality of subjects for learning is input (step S02), and the deep generation model is repeatedly learned (step S03). When the iterative learning is completed (step S04), a feature component expressing the individual difference, a signal component which does not depend on the state label and the individual difference but can change with time is extracted and separated (step S05). The brain function image data of the subject to be predicted is input (step S06), and from the input brain function image data, a temporal amount that does not depend on the feature quantity expressing the individual difference, the state label, and the individual difference, but changes. The state of the subject is predicted using the signal components obtained by extracting and separating the obtained signal components (step S07) (step S08).

FIG. 2 shows a functional block diagram of an embodiment of the state prediction device of the present invention.
The state prediction apparatus 1 is an apparatus that receives time-series brain function image data obtained by measuring brain function images of a plurality of subjects, and predicts the state of the subject using a deep generation model based on a deep neural network. It is. The generation model constructing unit 2 adds, to the input time-series brain function image data, a state label, a feature amount expressing an individual difference, and a signal component which is not dependent on the state label and the individual difference but can change with time. Build a deep generation model by associating. The learning unit 3 inputs brain function image data of a plurality of subjects and repeatedly learns a deep generation model. Then, using the trained deep generation model, from the input time-series brain function image data, a feature quantity representing an individual difference and a signal component which is not dependent on the state label and the individual difference but can change over time are obtained. The state of the subject is predicted using the extracted and separated signal components, and the result of the state prediction is displayed on the display panel.

(About the deep generation model)
Next, a deep generation model of the present embodiment will be described with reference to a schematic configuration diagram shown in FIG.
In the following, a deep generation model will be described using brain function image data as an example of biosignal data. Representing the data set of the brain functional image data _{D = {x i, y i} } and ^N _{i = 1.} Here, N number of subjects, i is an index number, x _i is a set of brain functional image data obtained from a subject i, y _i is the state label of a disease in a subject i (diagnosis result) . In the state label _yi , y = 0 means no disease, and y = 1 means a diseased state. And _{T i} pieces of brain functional image is obtained from a subject _i, represented by _{x i = {x i, t} } Ti t = 1.

  In the deep generation model, it is assumed that each subject i has an individual difference independently of the presence or absence of a disease. The individual differences in the brain function image data correspond to the fact that the size and shape of the brain differ depending on the subject, the usage and habits of the brain in an abstract sense, and the like. The individual differences in the size and shape of the brain are elements that can be removed by preprocessing brain function image data, but it is actually difficult to completely remove them. Here, the pre-processing of the brain function image data is to exclude from the data several brain function images to be measured to secure the stability of the magnetic field, to correct the timing for each imaging plane, and to correct the body movement. And converting the coordinates of the observation point into a standard brain coordinate system, for example, an MNI (Montreal. Neurological Institute) coordinate system (performing spatial normalization).

Deep generation model, the individual differences, is expressed by the feature s _i according to the prior distribution p (s), is modeled as a latent variable direct unobservable to generate a brain functional image data x _i. As shown in FIG. 3, each brain function image data x _{i, t} does not depend on a state label y _i , a feature quantity (hereinafter, referred to as an individual difference) s _i representing an individual difference, and a state label and an individual difference. Is linked to a signal component (hereinafter, referred to as a latent variable) z _{i, t} that can change with time. The latent variable z _{i, t} expresses a variation for each brain function image according to the prior distribution p (z). For example, it corresponds to what the subject thought with his or her head while collecting brain function image data, or body motion that could not be completely removed by preprocessing.

FIG. 4 is a conceptual diagram of a deep neural network using a deep generation model. For example, enter the brain functional image data 5 of the T _i Like the subject i, taking into account the state labels and individual differences, reduces the number of dimensions by coding (Encoder) process, the posterior distribution of the latent variable z _{i, t} Is obtained, and the brain function image data 5 ′ is decoded from the input brain function image data 5 by a decoding (Decorder) process. The brain function image data 5 of the learning data other than the subject and the input of the state are repeated, and the weight parameter is set so as to reduce the difference between the input brain function image data 5 and the output brain function image data 5 ′ in the backward calculation. To change. The weight parameter is improved by repeatedly inputting the brain function image data and its state label.

State generator model p _theta label y _i brain function was subject to image data x _i, is constructed as shown in the schematic diagram of FIG. 3, solid arrows a~c indicates the decoding by generating model, the generation model p _theta constructed from the state label y _i and individual differences s _i and latent variable z _{i, t,} brain functional image data x _{i and t} are generated (solid arrows a to c). If comprised of VAE, the generative model p _theta, expressed by the following equation.

Since it is very difficult to learn a deep generation model by itself, for example, another deep neural network called an inference model or an encoder is introduced, and learning is performed using a variational method. However, if it does not consist of VAE, an inference model is not always necessary.
In FIG. 3, dotted arrows d to h indicate encoding by an inference model. Brain function using an inference model image data x _{i, t} and a state label y _i respectively from individual differences s _i and latent variable z _{i, t} is deduced (dotted arrows D-G), furthermore, potentially from individual differences s _i The variable z _{i, t} is inferred (dotted arrow h). The variational inference with inference model q _phi, model evidence logp the above product model p _theta | lower bound (x i _{y i)} is expressed as following equation.

_{Here, D KL} (· || ·) is the _{Kullback-Leibler divergence, L g (} x i, y i) is the model Evidence lower bound: a (ELBO Evidence Lower BOund). ELBO corresponds to the objective function of the inference model q _phi and generative model p _theta. Model Evidence logp | as an approximation of _{(x i y i), ELBO} L g (x i, y i) by using the posterior probability _p status label _{y i} of the subject i | a _(y i _{x i)} Using Bayes' theorem, the following equation can be used.

Here, it is assumed that the prior probability p (y) of the state label y is, for example, p (y = 0) = p (y = 1) = 0.5. _{That, ELBO L g (x i,} y = 1) The larger the subject i can be said to likely a state of interest. The logarithm logp of state label _{y i} prior probability of _{(y i} | _x i) the approximate value _{L d (x i, y i} ) also can be an objective function. Thereby, the model can be trained so as to clearly identify the two state labels. In order to prevent the collapse into a mere identification model, using the weight parameters ω∈ [0,1], to adjust the two objective functions as following equation, the final the L (x _i, y _i) It may be an objective function (see Non-Patent Document 5 for details regarding this objective function).

(About construction of deep generation model)
Here, using a deep neural networks, by implementing the product model p _theta inference model q _phi, we describe a method of constructing a deep generation model.
For example, after the brain function image data is pre-processed, the obtained brain function image data x _{i, t} , individual difference s _i , and latent variable z _{i, t} are converted into _nx dimension, _ns dimension, and _nz dimension, respectively. Vector.
For example, in the time-series brain function image data, when a 116-dimensional vector is obtained by dividing into 116 regions (Regions-Of-Interest) and averaging the signal intensity in each region, The number of dimensions of the image data x _{i, t} is 116 ( _nx = 116). Individual differences may include age, gender, size and shape of the brain, brain usage, drug use history and other psychiatric illnesses. They When expressed in 10-dimensional vector, the number of dimensions of individual differences _{s i} is 10 _(n s = 10). In addition, in the feature amount of the individual difference, one feature amount is not always one-dimensional. For example, in a complicated feature amount such as a past history, one feature amount is expressed in several dimensions.

When the deep generation model is composed of VAE, a prior distribution is assumed for the latent variables in order to estimate latent variables z _{i, t} , state labels y _i , and individual differences s _i which are parameters of the multidimensional simultaneous probability distribution. Then, learning is performed by the deep neural network so that the posterior distribution appears in the output of the deep neural network.
As an example, generation model consists of a single deep neural network called a decoder (decoder), brain functional image data x _i, the posterior distribution of _{t p θ (x i, t} | z i, t, y _i , s _i ) is an _nz- order multivariate Gaussian distribution N (μx _{i, t} , diag (σx _{i, t} )) having a diagonal matrix as a variance-covariance matrix using a reparameterization technique. Is expressed. Here, μx _{i, t} is an average vector of the posterior distribution of the brain function image data x _{i, t} , and σx _{i, t} is a variance vector of the posterior distribution of the brain function image data x _{i, t} . The re-parameterization method is a method that outputs the data of the posterior distribution of the data instead of outputting the data by point estimation, and by using the re-parameterization method, it includes the ambiguity and variation of the data. Can be estimated.

Inference model, as an example, have a 2 × n _x number of output units, of which n _x number of units using the identity function as activation function, representing the mean vector μx _{i, t} of the posterior distribution. N _x number of units remaining using an exponential function as the activation function, representing the distributed vector sigma] x _{i, t} of the posterior distribution.
The part of the inference model that infers a signal component (latent variable) z _{i, t} that does not depend on the state label and the individual difference but can change with time is a single deep layer also called an encoder. is implemented by the neural network, like the decoder, posterior distributions q _theta by reparameterized _{(z i, t | x i} , t, y i, s i) to express.

On the other hand, post infer individual differences _{s i} of the inference model distribution q _{theta |} adjustment is necessary for the _{(s i x i, y i} ). In this inference model, a set of brain function image data _xi = {xi _{, t} } ^Tit _{= 1} of non-constant length needs to be received as an input for each subject i. Perform deep learning using collection-encoder). As shown in FIG. 5, the collective encoder includes a first neural network 11 that outputs an activity value by inputting individual brain function image data x _{i, t} and a state label y _i associated therewith, A second neural network 12 that outputs an a posteriori distribution q _θ (s _i | x _i , y _i ) for inferring the individual difference s _i with an average value of the activity values obtained from the set of image data as an input; Consists of In FIG. 5, the flow of signal processing is from bottom to top.

The first neural network 11 of the population encoder one brain functional image data x _i, enter the _t and state label y _i, and outputs the activation value h _{i, t.} Second neural network 12 of the population encoder set of brain functional image data _{x i = {x i, t} } Ti t = activity obtained from ₁ value _{h i = {h i, t} } Ti t = mean values obtained from _{1 (1 / T i × Σ} Ti t = 1 {h i, t}) with an input, the posterior distribution q of individual differences _{s i} by using a re-parameterization _{theta (s} i | _{x i} , Y _i ).
Specifically, as shown in FIG. 6, in the first neural network 11, 116-dimensional signal data based on the signal intensities of the 116 regions (ROIs) divided in the brain function image 1 and a state associated therewith. to enter a label _{y 1.} Similarly, brain function image 2, brain function image 3,..., 116-dimensional signal data based on the signal strength of the ROI of brain function image Ti and state label y _Ti associated therewith are input. In the second neural network 12, as inputs averaged value of the active values obtained from a set of brain functional image data, the posterior distribution q to infer individual differences _{s i θ (s i | x} i, y i) Is output.

Then, a plurality of functional brain imaging data of the subject i is posterior distribution q _theta individual differences s _i can not be observed directly _| a (s _i x i, _{y i)} are sharing, to build a deep generation model. That is, when the plurality of functional brain imaging data of the same subject, the posterior distribution of individual differences _{s i q θ (s i |} x i, y i) and share the to. The structure of the deep generation model imitates the structure of the brain function image data, and acts as an appropriate constraint.

In this embodiment, an artificial neural network having three layers is used for each of the encoder and the decoder. FIG. 7A shows the configuration of the encoder, and FIG. 7B shows the configuration of the decoder. Further, as shown in the schematic configuration diagram of FIG. 5, the collective encoder used a two-layer artificial neural network for the first neural network 11 and a one-layer artificial neural network for the second neural network 12.
To describe this in a specific example, as shown in FIG. 8, the encoder associates 116-dimensional signal data based on the signal strength of the 116 regions (ROI) of the time-series brain function image data (input data) and associates them with the signal data. to enter a state label _{y i} and individual differences _{s i} was. In each intermediate layer (hidden layer), normalization is performed, and ReLU is used as an activation function. The artificial neural network of the encoder calculates the mean vector μ and the variance vector σ of the posterior distribution of the brain function image data xi _{, t} , and calculates the posterior distribution q _θ (zi _{, t} | x of the latent variable zi _{, t. i, t} , y _i , s _i ).

As described above, the logarithmic logp of the prior probability of the state label _{y i | ELBO L g (x} i, y i) of the approximate value of _{(y i x} i), it is necessary to calculate the expected value becomes the objective function. Therefore, at the time of deep learning training, the individual difference s _i and the latent variable z _{i, t} are sampled and approximated once every training iteration, and at the time of verification, each MAP estimation (average vector μ _zi representing the posterior distribution) is performed. _{, t} ) was used.
The artificial neural network was trained using an optimization algorithm such as the Adam optimization algorithm, and the imbalance in the number of subjects between states was adjusted by the oversampling method.

FIG. 9 shows an example of the data structure of time-series brain function image data. The table in FIG. 9A shows 116-dimensional signal data based on the signal intensities of 116 regions 1 to 116 (ROI-1 to ROI-116) of the time-series brain function image data (input data x _{i, t} ). Is shown. These signal data are obtained from time-series brain function images, and are data that changes with time.
On the other hand, the state label y _i and the individual difference s _i associated with these 116-dimensional signal data do not temporally change (time-invariant) in a time-series brain function image. The state label y _i is such that the state does not change in one measurement, and the individual difference s _i does not change in the measurement as long as the subject does not change during the measurement and remains the same. As shown in the table of FIG. 9 (2), for example, the individual differences s _i is expressed by 10-dimensional. As described above, in the feature amount of the individual difference, one feature amount is not always one-dimensional, and therefore, the number of features does not always match the number of dimensions. Further, a signal component (latent variable) z _{i, t} that does not depend on the state label and the individual difference but can change with time is also associated with the 116-dimensional signal data. The latent variable z _{i, t} is different from the state label y _i and the individual difference s _i and changes with time. As shown in the lower table of FIG. 9, for example, there are 1 to 32 signal components (latent variables) zi _{, t} that do not depend on the state label and the individual difference, and the number of dimensions is 32.
In this case, when the brain functional image data (input data x _{i, t)} of the series signal data 116 D is includes a status label y _i, and individual differences s _i, the sum of the latent variable z _i, the number of dimensions of _t 43 (= 1 + 10 + 32), and the inferred individual difference s _i and the latent variable z _{i, t} are expressed by a posterior distribution.

(About brain function image data used in the experiment)
The brain function image data was obtained by obtaining a database of schizophrenia and bipolar disorder from an open database (Open fMRI) and performing the preprocessing described below. In the preprocessing, the first 10 brain function images of each subject are discarded in order to secure the stability of the magnetic field, and the time direction adjustment, body motion correction, and spatial normalization using the MNI coordinate system are performed. Was. An AAL (Automated Anatomical Labeling) template was used to divide into 116 regions (ROIs), and the signal intensities were averaged for each of them to obtain a 116-dimensional vector xi _{, t} . As the data washing, from the moment when the body movement of 1.5 mm or 1.5 ° or more was detected, all subsequent brain function images were discarded.
As a result of discarding, a subject in which only less than 100 brain function images remained discarded all of the subject data. In addition, data of subjects that could not be converted to the MNI coordinate system due to a shift in the imaging position were also discarded. As a result, the number of subjects in the control group was 113, the number of subjects with schizophrenia was 44, and the number of subjects with bipolar disorder was 45.

(About prediction accuracy)
The prediction accuracy according to the state prediction method of the embodiment will be described. As the evaluation of the prediction accuracy, the accuracy was compared with the conventional method (Comparative Examples 1 to 5) and evaluated.
As Comparative Example 1, the prediction accuracy was obtained by a method of functionally combining and using the Pearson correlation coefficient (PCC) between the 116 regions (ROIs) divided in the pre-processed brain functional image, and the state prediction of the embodiment was performed. It was compared with the prediction accuracy by the method. Specifically, in Comparative Example 1, m features are selected from the functional coupling of PCC between ROIs using Kendall's tau coefficient (Kendall τ coefficient), and locally linear embedding (LLE: Locally Linear Embedding) is performed. Is embedded in a d-dimensional manifold under the neighborhood parameter k, and clustered by the c-means method.

As Comparative Example 2, using a pre-processed brain function image, the prediction accuracy when state prediction was performed using a support vector machine (SVM) and the prediction precision by the state prediction method of the example were compared. Here, SVM receives brain functional image one by one, which returns a result of binary classification for each, the majority of the classification results for T _i pieces of brain functional image is the diagnosis of a subject i Using.

As Comparative Example 3, using a preprocessed brain function image, the prediction accuracy in the case of performing state prediction using Long Short Term Memory (LSTM) and the prediction accuracy by the state prediction method of the example were compared. Here, the LSTM is a neural network having a recursive structure, and sequentially receives a set of brain function image data x _i = {xi _{, t} } ^Tit _{= 1} obtained from one subject, and posterior probability p | a (y x _i), and outputs using logistic function.

As a comparative example 4, using the pre-processed brain function image, the prediction accuracy when the state is predicted using a combination of an auto-encoder (AE; Auto-encoder) and a hidden Markov model, and the state prediction method of the embodiment The prediction accuracy was compared. In Comparative Example 4, the brain function image was embedded in the d dimension using AE, and two hidden Markov models were trained on the embedded image. One hidden Markov model is for the subject ( _pθ (xi _{, t} | y = 1)) and the other is for the control group ( _pθ (xi _{, t} | y = 0)). Each hidden Markov model is modeled by a Gaussian distribution, and calculates the posterior probability p (y | x _i ) of the state by Bayes' law as in the deep generation model of the embodiment.

As Comparative Example 5, using a preprocessed brain function image, a known deep generation model proposed by the inventors, that is, a brain function image x _i , a state label y _i , and a difference z _{i, t} is implemented using an encoder q (zi _{, t} | xi _{, t} , _yi ) and a decoder p (xi _{, t} | zi _{, t} , _yi ), individual differences s _i compared the prediction accuracy by the prediction accuracy and the state prediction method embodiment in which the state prediction using the generated model does not modeled.

The data set is imbalanced, and the comparison items of the prediction accuracy include: Sensitivity = TP / (TP + FN), Specificity = TN / (TN + FP), Balanced accuracy = 0. 5 × (sensitivity + specificity) is used. Here, TP, TN, FP, and FN represent True Positive, True Negative, False Positive, and False Negative, respectively.
The measurement results of the prediction accuracy were obtained by averaging five times of 10-fold cross validation, and are shown in Table 1 below. From Table 1, in the case of the deep generation model of the example, the data set of schizophrenia shows clearly superior results to all the comparison methods of Comparative Examples 1 to 5, and the data set of bipolar disorder It was confirmed that almost the same or excellent results were obtained. In particular, that higher accuracy than the known deep generative model inventors of Comparative Example 5 has proposed is obtained, an appropriate structure of individual differences s _i gave the deep generation model of Example constraining It can be said that it is a condition.

A deep generation model according to the second embodiment will be described with reference to a schematic configuration diagram illustrated in FIG. Similar to the first embodiment, a deep generation model will be described using time-series brain function image data as an example. 10, similarly to FIG. 3 of the schematic structure of the first embodiment, the data set of brain functional image data _{D = {x i, y i} } N i = 1, N is number of subjects, i is an index number , _Xi is a set of brain function image data obtained from the subject i, and y _i is a disease state label (diagnosis result) of the subject i. In the state label y _i, no y = 0 is a disease, it means a state of y = 1 is there disease, and the T _i pieces of brain functional image is from a subject i is obtained. In the deep generation model of Example 2, it is assumed that each subject i has an individual difference and a measurement environment difference independently of the presence or absence of a disease. The measurement environment difference in the brain function image data corresponds to a difference in resolution, sensitivity, and the like of the measuring device used for measuring the brain function image. In addition, a difference in ambient noise at the time of measurement also corresponds. The measurement environment difference is an element that can be removed by preprocessing the brain function image data, similarly to the individual difference, but it is actually difficult to completely remove it.

Deep generation model of the present embodiment, in addition to individual differences, difference measurement environment is expressed by the feature amount f _i according to the prior distribution p (f), directly unobservable latent variables to generate a brain functional image data x _i Is modeled as As shown in FIG. 10, each brain function image data x _{i, t} includes a state label y _i , an individual difference s _i , a feature quantity (hereinafter, a measurement environment difference) f _i representing a measurement environment difference, and a state label. This is a configuration that is linked to a signal component (hereinafter, a latent variable) z _{i, t} that does not depend on an individual difference and a measurement environment difference but can change over time.

The deep neural network using the deep generation model of the present embodiment, by entering the brain functional image data from a subject, as shown in FIG. 11, considering the state label y _i and individual differences s _i and the measured environmental difference f _i Then, the number of dimensions is reduced by the encoding process _, the posterior distribution of the latent variables z _{i, t} is obtained, and the brain function image data is decoded from the input brain function image data by the decoding process. The brain function image data of learning data other than the subject and the input of the state are repeated, and in the backward calculation, the weight parameter is changed so as to reduce the difference between the input brain function image data and the output brain function image data. The weight parameter is improved by repeatedly inputting the brain function image data and its state label.

In FIG. 10, solid arrows indicate decoding by the generation model, and the brain function image data x _{i, t} is obtained from the state label y _i , the individual difference s _i , the measurement environment difference f _i, and the latent variable z _{i, t.} Generated. Dotted arrows indicate encoding by an inference model _, and individual differences s _i , measurement environment differences f _i, and latent variables z _{i, t} are inferred from brain function image data xi _{, t} and state labels y _i , respectively. A latent variable z _{i, t} is inferred from the difference s _i and the measurement environment difference f _i .
Using the above deep neural network, by implementing a generation model p _theta inference model q _phi, explanation of how to build a deep generation model is omitted the same as in Example 1.

(Other Examples)
In the first embodiment, the brain function image data of fMRI is described as an example of the biological signal data. However, the state prediction method and the state prediction apparatus of the present invention can be used at other times such as sweat measurement, electrocardiogram measurement, and electroencephalogram measurement. The present invention can also be applied to a series of biological signal data.
(1) In the case of sweat measurement, for example, if there are measurement points (for example, 25 measurement points) of the sweating site (palm, soles, forehead, armpits, etc.) of a human body, a 25-dimensional time series Of the perspiration signal x _{i, t} can be obtained. This takes into account the characteristics of the subject, such as age, gender, lifestyle, daily urine frequency, daily hydration, nervousness, antiperspirant use, hair depth, medical history, etc. and, the number of dimensions of feature quantity s _i representing the individual difference is determined. As with the brain function image data, the number of features and the number of dimensions do not always match. If the number of dimensions of feature quantity s _i representing the individual difference is 8, the number of dimensions of the state label of diseases related to perspiration and 1, and 5 the number of dimensions of the signal components that may be time varying. From the 25-dimensional sweat signal data obtained by performing a single sweat measurement with one subject, a 5-dimensional time-varying parameter and a 9-dimensional (= 8-dimensional + 1-dimensional) time-invariant parameter are obtained. Can be Feature value s _i representing the individual difference is unchanged during measurement, the data obtained for the characteristic quantity s _i representing the individual difference of each subject is fixed rather than time varying data. The latent variable z _{i, t,} which is a signal component that does not depend on the state label and the feature amount expressing the individual difference but can change over time, is time-series data that changes over time.
In the same manner as in the first embodiment, the sweat generation signal data is input, the deep generation model is learned, and from the sweat signal data input from the subject, the feature amount representing the individual difference, and the state label and the individual difference are independent. A signal component that can change over time is extracted, and the presence or absence of a disease related to sweating of the subject can be predicted using the separated signal component.

(2) In the case of electrocardiogram measurement, for example, in the case of a 12-lead electrocardiogram, twelve kinds of time-series waveform data, that is, twelve-dimensional time-series electrocardiogram signal data x _{i, t} are obtained. , age, gender, etc. to the consideration history, the number of dimensions of feature quantity s _i representing the individual difference is determined.
(3) If the heart rate measurement, time series of heart rate data is obtained, as a feature of the subject, age, gender, etc. to the consideration of the degree involved in sports, the number of dimensions of feature quantity s _i representing the individual differences It is determined. "Always the heart rate is lower if the athlete such as athletes" information such as the subject is, for example, is treated as a feature value s _i to represent the individual differences. If the information is common regardless of individual differences, for example, temporary information such as "Heart rate is high because you are currently exercising," a signal component (potential Variable) z _{i, t} .
(4) In the case of EEG, when obtained brain wave data series, as a feature of the subject, age, gender, etc. to the consideration history, the number of dimensions of feature quantity s _i representing the individual difference is determined.
(5) In the case of blood oxygen concentration measurement and blood glucose measurement, time-series blood oxygen concentration data and blood glucose level data are obtained, and the characteristics of the subject include age, gender, body fat percentage, obesity degree, and dietary habits. taking into account the number of dimensions of feature quantity s _i representing the individual difference is determined.

(6) In the case of Example 2 is determined number of dimensions of the feature f _i representing the measurement environment difference, the number of dimensions of the biosignal data obtained, the number of dimensions of feature quantity representing the individual difference, condition It is added in the same way as the number of dimensions of the label. Feature value f _i representing the measured environmental difference is shared by a plurality of bio-signal data measured by the same measurement environment.

  INDUSTRIAL APPLICABILITY The present invention is useful for a biological signal data measuring device and a disease diagnostic apparatus for measuring various biological signal data such as brain function image data, brain waves, heartbeat and perspiration data.

DESCRIPTION OF SYMBOLS 1 State prediction apparatus 2 Generating model construction part 3 Learning part 4 State prediction part 5, 5 'Brain function image data 11 1st neural network of a group encoder 12 2nd neural network of a group encoder

Claims (18)

A method for predicting the state of a subject using a deep generation model by a deep neural network, with time-series biosignal data obtained by measuring biosignals of a plurality of subjects as input,
1) A state label that does not change in a single measurement, a feature amount that represents an individual difference, and a signal component that does not depend on the state label and the individual difference but can change over time are associated with the biological signal data, Building a deep generation model;
2) inputting the biological signal data of a plurality of subjects and learning a deep generation model;
3) extracting a feature quantity representing the individual difference and the signal component from the input biological signal data, and performing a state prediction of the subject using the separated signal component;
A state prediction method comprising:   In the step of constructing the deep generation model, further, a feature quantity representing a measurement environment difference is associated with the biosignal data, and a state label, a feature quantity representing an individual difference, a feature quantity representing a measurement environment difference, and 2. The deep generation model is constructed by associating a signal component which does not depend on a state label, an individual difference, and a measurement environment difference but can change with time with the biological signal data. State prediction method described. In the deep generation model,
A plurality of biosignal data of the same subject share a feature amount expressing the individual difference,
When the deep generation model is configured by VAE (Variational Auto-Encoder),
The method according to claim 1, wherein the feature quantity representing the individual difference is modeled as a latent variable following a prior distribution. In the deep generation model,
A plurality of biological signal data measured in the same measurement environment, share a feature amount expressing the measurement environment difference,
When the deep generation model is configured by VAE (Variational Auto-Encoder),
3. The state prediction method according to claim 2, wherein the feature quantity representing the measurement environment difference is modeled as a latent variable following a prior distribution. In the step of deep learning,
The estimation of the feature amount expressing the individual difference includes:
A first neural network that outputs an activity value by using one of the biological signal data and a state label associated therewith as an input;
A second neural network that receives an average of the activity values obtained from the set of biosignal data and outputs a feature representing the individual difference;
The state prediction method according to claim 1, wherein a group coder configured from the following is used. In the step of deep learning,
The estimation of the feature quantity expressing the measurement environment difference includes:
A first neural network that outputs an activity value by using one or more of the biological signal data measured in the same environment and a state label associated therewith as an input;
A second neural network that outputs an amount of characteristic representing the measurement environment difference, with an average of the activity values obtained from the set of the biological signal data as input,
5. The state prediction method according to claim 2, wherein a group encoder composed of the following is used.   7. The state prediction method according to claim 1, further comprising a step of displaying a result of the prediction of the state of the subject.   The biological signal data includes fMRI (functional Magnetic Resonance Imaging) image data, MRI (Magnetic Resonance Imaging) image data, PET (Positron Emission Tomography) image data, ultrasonic image data, brain wave data, electrocardiogram data, heart rate data, sweating The state prediction method according to any one of claims 1 to 7, wherein the state prediction method is one of quantity data, blood oxygen concentration data, and blood sugar level data.   The deep generation model includes AE (Auto-Encoder) including VAE (Variational Auto-Encoder), GAN (Generative Adversarial Nets), GMM (Generative Moment Matching), EP (Energy-based Probabilistic Model), FG (Flow-based 9. The state prediction method according to claim 1, wherein one of a Generative Model) and an AR (Auto-regressive Model) is used. Apparatus for predicting the state of a subject using a deep generation model by a deep neural network, using as input a time series of biological signal data obtained by measuring biological signals of a plurality of subjects,
1) A state label that does not change in a single measurement, a feature amount that represents an individual difference, and a signal component that does not depend on the state label and the individual difference but can change over time are associated with the biological signal data, A generation model builder for building a deep generation model,
2) a learning unit that receives the biological signal data of a plurality of subjects and learns a deep generation model;
3) a state predicting unit that extracts a feature quantity representing the individual difference and the signal component from the input biosignal data, and predicts a state of the subject using the separated signal component;
A state prediction device comprising:   In the generation model construction unit, further, a feature quantity representing the measurement environment difference is associated with the biological signal data, and the state label, the feature quantity representing the individual difference, the feature quantity representing the measurement environment difference, and 11. The deep generation model is constructed by associating a signal component, which does not depend on a state label, an individual difference, and a measurement environment but can change over time, with the biological signal data. State prediction device. In the deep generation model,
A plurality of biosignal data of the same subject share a feature amount expressing the individual difference,
When the deep generation model is configured by VAE (Variational Auto-Encoder),
The state prediction device according to claim 10, wherein the feature quantity representing the individual difference is modeled as a latent variable according to a prior distribution. In the deep generation model,
A plurality of biological signal data measured in the same measurement environment, share a feature amount expressing the measurement environment difference,
When the deep generation model is configured by VAE (Variational Auto-Encoder),
The state prediction device according to claim 11, wherein the feature quantity representing the measurement environment difference is modeled as a latent variable according to a prior distribution. In the learning section,
The estimation of the feature amount expressing the individual difference includes:
A first neural network that outputs an activity value by using one of the biological signal data and a state label as input,
A second neural network that receives an average of the activity values obtained from the set of biosignal data and outputs a feature representing the individual difference;
The state predicting apparatus according to claim 10, wherein a group coder configured from the following is used. In the learning section,
The estimation of the feature quantity expressing the measurement environment difference includes:
A first neural network that outputs an activity value by using one or more of the biological signal data measured in the same environment and a state label associated therewith as an input;
A second neural network that outputs an amount of characteristic representing the measurement environment difference, with an average of the activity values obtained from the set of the biological signal data as input,
14. The state predicting apparatus according to claim 11, wherein a group encoder composed of   The state prediction device according to any one of claims 10 to 15, further comprising a display unit that displays a result of the state prediction of the subject.   A state prediction program for causing a computer to execute each step in the state prediction method according to claim 1. A state prediction program for causing a computer to function as the generation model construction unit, the learning unit, and the state prediction unit in the state prediction device according to claim 10.