KR101919907B1 - Apparatus and method for monitoring interpersonal interaction based on multiple neurophysiological signals - Google Patents

Abstract

The present invention discloses an apparatus and method for measuring multiple physiological signals of different users and observing the degree of interaction between users through correlation between them. The interaction monitoring apparatus according to the present invention includes a signal acquiring unit receiving multiple physiological signals of a plurality of users; A signal processing unit having a preprocessor for preprocessing the signal output from the signal acquiring unit and an interest interval extracting unit for receiving an information signal about an event occurrence time or an interval; A signal analyzer for calculating a correlation between multiple physiological signals of a plurality of users which are preprocessed and output from the signal processor; And a result generating unit for storing the correlation analysis between the signals of the two persons in the data matrix using the signal output to the signal analyzing unit.

Description

[0001] APPARATUS AND METHOD FOR MONITORING INTERPERSONAL INTERACTION BASED ON MULTIPLE NEUROPHYSIOLOGICAL SIGNALS [0002]

The present invention relates to a plurality of bio-signal measurement and interaction analyzing apparatuses. Unlike the prior art in which a person's bio-signal is detected, the activity of the brain and autonomic nervous system of a plurality of users is measured to observe the degree of interaction between bio- And to a method of analyzing the same.

Humans or animals exhibit the biological responses of the central nervous system and the autonomic nervous system to external stimuli. Techniques for observing the vital response include brain waves recording electrical signals generated by central nervous system brain activity, functional magnetic resonance imaging (MRI) measuring blood flow activity, positron emission tomography, functional near-infrared spectroscopy (fNIRS) (PPG), photo-plethysmograph, skin temperature (SKT), eye movement, and pupil size And measurement methods.

Since 2001, hyper-scanning research has been conducted on functional connectivity between EEGs in human-human social interaction. However, conventional hyper-scanning research is limited to studying brain-brain functional connectivity in human interaction. In addition, there are no studies to observe the interaction between different individuals, such as the study of interaction between a specific person and a group of two or more persons, and EEG - autonomic nervous system signals.

Therefore, it is necessary to analyze the interaction between new brain - autonomic nervous system signals beyond analysis of existing brain - brain physiological interactions in physiological signal - based interaction analysis methods.

Korean Patent Publication No. 10-2013-0052280

The present invention provides an apparatus and method for measuring multiple physiological signals of different users and observing the degree of interaction between users through correlation between them.

The solutions described herein are not limited to those mentioned above, and other solutions not mentioned may be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided an interaction monitoring apparatus including: a signal acquisition unit receiving a plurality of physiological signals of a plurality of users; A signal processing unit having a preprocessor for preprocessing the signal output from the signal acquiring unit and an interest interval extracting unit for receiving an information signal about an event occurrence time or an interval; A signal analyzer for calculating a correlation between multiple physiological signals of a plurality of users which are preprocessed and output from the signal processor; And a result generating unit for storing the correlation analysis between the signals of the two persons in the data matrix using the signal output to the signal analyzing unit.

The signal acquisition unit according to an embodiment of the present invention includes a brain activity signal acquisition unit receiving a brain signal of a first user; And an autonomic nervous system signal acquisition unit that receives the autonomic nervous system signal of the second user.

The signal analyzer according to an embodiment of the present invention may include a signal combination selector for selecting one pair of an EEG signal and another autonomic nervous system signal; A functional connectivity calculator for calculating a correlation between two signals selected by the signal combination selector; And a functional connectivity significance determiner for determining whether the value output from the functional connectivity calculator indicates a significant correlation.

The functional connectivity calculator according to an embodiment of the present invention may include a feature value calculator that extracts a feature value from each signal of two signals selected by the signal combination selection unit and extracts a correlation between the feature values; A time-series-to-signal correlation calculator for analyzing a correlation between the signal output from the characteristic value calculator and the time-series signal; A characteristic-value correlation calculator for calculating a linear correlation coefficient between each characteristic value through a Pearson correlation analysis of the signal output from the characteristic-value calculator; And a signal selector for selecting one of the two signals selected by the signal combination selector as a reference, inputting a signal of the other person, extracting the interest interval from the reference signal as the event, and quantifying the degree of the event- And an event-related characteristic calculation unit.

According to an embodiment, the time series correlation calculator may perform a time delay stability analysis considering a time delay between a plurality of users.

According to another embodiment of the present invention, the time series correlation calculator may perform partial directionality consistency analysis between a plurality of users.

According to another embodiment of the present invention, the time-series correlation calculator may perform phase-amplitude synchronization analysis between a plurality of users.

According to an aspect of the present invention, there is provided an interaction monitoring method comprising: (a) receiving a plurality of user's multiple physiological signals by a signal acquisition unit; (b) preprocessing the signal output from the signal acquisition unit by the preprocessing unit; (c) receiving an information signal of an event occurrence time or an interval of the preprocessed signal; (d) calculating a correlation between multiple physiological signals of a plurality of users, which are pre-processed by the signal analysis unit and output in the signal processing unit; And (e) storing the correlation analysis between the signals of the two persons in the data matrix using the signal output by the result generating unit to the signal analyzing unit.

The above-described interaction monitoring method may be implemented by a computer program written in a computer-readable recording medium so as to perform each step.

The interaction monitoring apparatus and method according to the present invention has the following effects.

First, the autonomic nervous system signal is a signal that measures the physiological activity of the autonomic nervous system, which is influenced by the central nervous system brain, and includes not only brain activity information but also physiological activity that can not be extracted from the brain. Thus, by observing the interaction between brain-autonomic nervous system signals beyond the brain-brain signal interactions, we can quantify interactions between users that reflect information that can not be explained solely by the functional connectivity of existing brain-brain signals .

Second, when measuring the degree of interaction between a large number of users, a small number of users can measure the autonomic nervous system signals and a plurality of users can analyze the interaction and thus it is economically advantageous.

Third, if there is no universal montage or procedure for EEG measurement, and if it is judged that it is difficult to collect good quality EEG signals due to excessive motion or artifacts without anesthesia, autonomic nervous system signal measurement The degree of interaction with a person can be observed.

The effects described in the present specification are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a basic idea of an interaction monitoring apparatus and method according to the present invention.
2 is a block diagram schematically illustrating a configuration of an interaction monitoring apparatus according to the present invention;
3 is a block diagram schematically showing a configuration of a preprocessing unit according to the present invention.
4 is a block diagram schematically illustrating the configuration of a signal analysis unit according to an embodiment of the present invention.
5 is a block diagram schematically showing the configuration of the functional connectivity calculation unit included in the signal analysis unit.
6 is a block diagram schematically showing a configuration of a time-series-signal correlation calculator according to the present invention.
7 is a block diagram schematically showing the configuration of an event-related characteristic calculation unit according to an embodiment of the present invention.
8 is a block diagram schematically showing the configuration of an event-related characteristic calculation unit according to another embodiment of the present invention.
9 is an exemplary view of the analysis process of the example shown in FIG.
10 is an illustration of an example of the analysis process shown in FIG.
11 is an exemplary diagram of a data matrix generated by a result matrix generator according to the present specification.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Wherever possible, the same or similar parts are denoted using the same reference numerals in the drawings.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms as used herein include plural forms as long as the phrases do not expressly express the opposite meaning thereto.

Means that a particular feature, region, integer, step, operation, element and / or component is specified and that other specific features, regions, integers, steps, operations, elements, components, and / It does not exclude the existence or addition of a group.

All terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Predefined terms are further interpreted as having a meaning consistent with the relevant technical literature and the present disclosure, and are not to be construed as ideal or very formal meanings unless defined otherwise.

Hereinafter, an interaction monitoring apparatus and method according to the present specification will be described with reference to the drawings.

This specification provides an apparatus and method for monitoring interactions by analyzing the correlation between one person's brain signal and another person's autonomic nervous system signal.

The interaction between brain-brain signals in existing subjects is a function of the inter-domain functional connectivity of signals measured in the cerebrum in the central nervous system, and the dorsolateral prefrontal cortex is known to be highly related to the interaction . However, the mechanisms of interaction between the central nervous system and the autonomic nervous system are unknown when they cause interaction between subjects. Studies on the brain-autonomic nervous system interactions within individuals have shown that EEG synchronization (see E Kroupi et al., 2013; S Shao et al. al., 2011), association between fMRI blood-oxygen level (BOLD) signal and dermal electrical activity according to the amplitude value of an individual's dermal electrical activity signal when laser stimulation is applied (Mobascher A et al. 2008) have been reported. It is important to observe the interaction between the brain and the autonomic nervous system through the above research results. In this way, when observing the interaction between human and human, I will provide you with.

1 is a conceptual diagram of a basic idea of an interaction monitoring apparatus and method according to the present invention.

Referring to FIG. 1, for example, an interaction can be monitored by measuring an EEG in one person, measuring an autonomic nervous system signal in another person, and analyzing a correlation between an EEG signal and an autonomic nervous system signal. At this time, both EEG and autonomic nervous system can be measured. In this case, the interaction between the brain and the autonomic nervous system signal and the interaction between the autonomic nervous system and the brain signal can be analyzed.

2 is a block diagram schematically illustrating a configuration of an interaction monitoring apparatus according to the present invention;

Referring to FIG. 2, an interaction monitoring apparatus according to the present invention may include a signal acquiring unit, a signal processing unit, a signal analyzing unit, and a result generating unit.

The signal acquisition unit may receive multiple physiological signals of a plurality of users. The multiple physiological signals include a brain activity signal and an autonomic nervous system activity signal. A brain activity signal includes an EEG signal. In addition, a functional magnetic resonance imaging signal, a functional near infrared ray spectroscopy signal, and the like can be input. The autonomic nervous system activity signals include eye movement, pupil size, photoplethysmogram, PPG, galvanic skin response (GSR), skin temperature , SKT). Equipment and techniques capable of measuring the multiple physiological signals are well known to those skilled in the art (hereinafter referred to as " a person skilled in the art "), and a detailed description thereof will be omitted.

More specifically, the signal acquisition unit may include a brain activity signal acquisition unit for receiving the brain signal of the first user and an autonomic nervous system signal acquisition unit for receiving the autonomic nervous system signal of the second user.

The signal processor may receive the signal output from the signal acquisition unit and process the signal to facilitate analysis. For this, the signal processing unit may include a pre-processing unit and an interest interval extracting unit.

3 is a block diagram schematically showing a configuration of a preprocessing unit according to the present invention.

Referring to FIG. 3, it can be seen that the EEG signal pre-processing and the autonomic nervous system signal preprocessing are distinguished.

First, in the case of EEG signals, we can observe the response of stimulus in each region of cerebrum with high temporal resolution. The cerebrum is an organ that handles high-level cognitive functions that receive signals from each sensory and judge and issue commands. Various cognitive functions can be observed. First, amplify to increase the signal-to-noise ratio. Then, the artifacts caused by eye flicker, electrocardiogram, etc. are removed by independent component analysis (ICA). Then, EEG is filtered for each frequency band. The frequency band can be divided into five bands by Delta (1-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz) and Gamma . It is possible to obtain a signal by performing low-frequency band filtering (LPF) of 50 Hz or less without discriminating the frequency band.

As another embodiment for pre-processing the brain image signal, a preprocessing method disclosed in Korean Patent Laid-Open Publication No. 10-2016-0016357 (name: Pattern Classification Apparatus and Method for fMRI) The collection of brain image signals may be used.

For the next autonomic nervous system signal, amplify to increase the signal-to-noise ratio. Since the autonomic nervous system signal is a slowly changing signal, the sampling frequency is downsampled to about 4 Hz and preprocessing is performed for each signal type.

The skin electrical activity response is the degree of conductivity of the current applied to the skin in a trace amount. This indicates the level of secretion of sweat on the surface of the skin. By measuring the minute electrical changes on the surface of the skin, the activity of the sympathetic nervous system can be observed. Thus, the dermal electrical activity signal can be used to measure the degree of arousal of the sympathetic nervous system and to obtain an indication of stress, work, or interest. Emotional awakening causes sweating in the palms, fingers and toes, which are mainly distributed in the outer glands of the glands.

The dermal electrical activity signal consists of two main components, a slowly changing basal level referred to as skin conductance level (SCL) and a phase change level referred to as skin conductance responses (SCR). The direct current (DC) component of the dermal electrical activity signal, referred to as SCL, reflects the general activity of sweat glands affected by body temperature or external temperature. SCR is a short waveform that appears in a unique form in the dermal electrical activity signal, which can be obtained from the dermal electrical activity signal and observed by the specific stimulus.

Photoplethysmography (PPG) is a measure of the volume of an organ as a photodynamic pulse wave. PPG emits two beams of light with different wavelengths to the skin, and then measures the amount of light (oxygen concentration) by measuring the transmitted or reflected light and obtaining the signal with an oxygen saturation meter. Light is absorbed by various tissues, and absorption by tissue, bones, venous blood, and fixed arterial blood, which do not change in the PPG waveform, appears as a direct current (DC) component. appear. PPG can be used to obtain the blood volume pulse (BVP) signal, which is mainly used to identify the heart rate of the user attaching the sensor. The heart rate can be obtained by detecting the peak from the PPG, calculating the number of vertices, and calculating the interval length between adjacent vertices. The peak-to-peak interval (PPI) can be used to obtain a temporary heart rate in addition to the average heart rate obtained through a plurality of vertices. Heart rate is controlled by various mechanisms, among which autonomic nervous system and hormones are influenced. Heart rate variability (HRV) is influenced by both the sympathetic and parasympathetic nervous system of the autonomic nervous system. The sympathetic nervous system activity increases the heart rate, while the activity of the parasympathetic nervous system lowers the heart rate. Parasympathetic nervous system is observed in the entire frequency band, and sympathetic modulation is observed in the low frequency band below 0.15 Hz.

Skin surface temperature is a physiological indicator that reflects human temperature response. The skin changes the skin temperature according to the external temperature environmental factors to keep the living body temperature constant. Skin surface temperature is also the result of autonomic nervous system activity, which changes not only by the external environment but also by internal factors. When negative photographic stimuli are presented to people, they may reflect a higher emotional state such as a higher skin surface temperature when positive (Vos P et al., 2012).

The pupil reaction is a physiological response that changes the size of the pupil through the optic nerve and cranial nerves. The dilation of the pupil is caused by contraction of radial muscle cells under the control of the sympathetic nervous system. Conversely, pupil contraction occurs when the circular muscle contracts under the control of the parasympathetic nervous system. Pupil size varies with various factors and reflects interest in attention. Eye movements or pupil size signals can cause missing or distorted signals if the eye blinks or goes out of sight. Thus, the signal distortion caused by blinking or winding eyes is corrected using spline interpolation. In addition, various interpolation methods can be applied.

Referring back to FIG. 2, the signal processing unit may have an interest interval extracting unit. The interest section extracting section may extract a section in which a specific event occurs in the signal section. To this end, the interest interval extracting unit may receive a signal indicating an event occurrence point or an interval from the external equipment.

The interest section extractor may record an event marker (input event information at a specific time in the physiological signal measuring device) from the external equipment in the measured physiological signal data by synchronizing the external equipment and the signal measuring equipment. For example, if you are in school, you can take an image with a camera, record an event with class time and break time, or record an event marker when you recognize the bell at the micro school. Based on the recorded event markers, we can monitor the interaction of class time and the interaction at break time.

In addition to the above example, a method of extracting a region of interest by an external device may include selecting a time when an actual interaction occurs through camera equipment, selecting a voice time when an interaction occurs through a microphone device, An event marker can be input to the measuring device when the button is pressed. That is, if the external equipment signal and the signal measuring device are synchronized with each other, the interest section extractor can extract the measured signal of the time when the external equipment selects a specific time.

 Referring again to FIG. 2, the signal analyzer may calculate the correlation between multiple physiological signals of a plurality of users, which are preprocessed and output in the signal processor.

4 is a block diagram schematically illustrating the configuration of a signal analysis unit according to an embodiment of the present invention.

Referring to FIG. 4, the signal analyzer may include a signal combination selector, a functional connectivity calculator, and a functional connectivity significance determiner.

The example shown in FIG. 4 will be described on the basis of a case in which an EEG signal of 64 channels in a gamma frequency band and an autonomic nervous system signal in another person are input. The signal combination selection unit may select one pair of the electrocardiographic signals of the gamma band of the left prefrontal FP1 channel and the autonomic nervous system among the EEG signals. The functional connectivity calculator then calculates the correlation between the two signals. The above procedure is repeated for every possible combination of signals between two people. The functional connectivity significance determiner determines whether the value output from the functional connectivity calculator is a result indicating a significant correlation. For this, the functional connectivity significance determiner may use a surrogation technique. For example, an artificial signal can be generated by randomly mixing the time sequence of the gamma band EEG signal of the FP1 channel and the time sequence of the skin electrokinetic time series signal. Artificial signals can be created by various methods. Statistical methods, surrogation, permutation, bootstrapping, etc. can be applied. The correlation between randomly mixed signals is calculated to obtain a null distribution of correlation values. It is judged that there is a significant connection between the two signals if the correlation between the two signals is higher than the predetermined value in the distribution of the result of the correlation between the artificial signals. 95% or 99% can be applied on a preferred significance basis.

The image shown on the right side of FIG. 4 is an image showing the result of the functional connectivity pattern. The correlation results from the artificial signals are calculated to provide only the connectivity results with statistical significance.

5 is a block diagram schematically showing the configuration of the functional connectivity calculation unit included in the signal analysis unit.

Referring to FIG. 5, the functional connectivity calculator may include a feature value calculator, a time-series signal correlation calculator, a feature value correlation calculator, and an event-related feature calculator.

The functional connectivity calculator may use a method of calculating a correlation value by extracting a characteristic value from a signal combination pair and a method of calculating an event-related characteristic.

And a characteristic value calculation unit for extracting a characteristic value and analyzing the correlation. The characteristic value calculation unit can not only observe the similarity of the two signals, but also can extract a characteristic value from each signal and observe the correlation between the characteristic values. For example, the asymmetry value calculated from the asymmetry of EEG extracted from the symmetric region of right brain and left brain is known to be related to emotion. In this example, the selected FP1 and FP2 alpha-EEG brainwave signals are subjected to Fourier transform while shifting the sliding window for 30 seconds every second, thereby extracting a change in the alpha wave power value in the form of a time series signal. Here, the natural logarithm of the right and left power values is taken and a difference is obtained to obtain a frontal asymmetry signal. In addition, it is possible to extract the change of the frequency band power value, the change of the frequency band power value of the entire region, and the like as characteristic values. The peak number of the SCR signal is detected in the skin electrical activity signal of the autonomic nervous system. We can extract the change of SCR peak number for 30 seconds as a time series signal while moving the sliding window every 30 seconds for 30 seconds. In addition, heart rate variability (HRV) is obtained by performing Fourier transform on the heart rate signal. Here, the ratio of the power value of the low frequency band, the power ratio of the low frequency band to the high frequency, and the fixed time And so on.

The time-series correlation calculator may analyze the correlation between the time-series signal and the signal output from the characteristic-value calculator.

The correlation calculator may calculate a linear correlation coefficient between each characteristic value through a Pearson correlation analysis.

This analysis has the advantage of deriving a result that is closer to the meaning of the functional connectivity pattern in terms of observing the correlation between two characteristic values having meaning.

The event-related characteristic calculation unit receives a signal of a person from a signal of one of the two signals selected by the signal combination selection unit as a reference, extracts an interest interval from the reference signal by using the characteristic point as an event, Can be quantified. This will be described in more detail below.

6 is a block diagram schematically showing a configuration of a time-series-signal correlation calculator according to the present invention.

Referring to FIG. 6, there are three methods for analyzing the correlation between time-series signals in the time-series signal correlation calculator. In order to analyze the correlation between the autonomic nervous system signals of one person's brain and others, information is transmitted from the brain to the autonomic nervous system and appears as an autonomic nervous system signal, or information is transmitted from the autonomic nervous system to the brain, Time delay between the nervous system should be considered.

Time-delay stability (TDS) is an analytical method that seeks to see a certain degree of delay in the signals of these two people. For example, if the length of the input signal is 30 minutes in total, the time delay of two signals in every 10 seconds can be obtained while moving the time of 1 second every 10 seconds. First, when analyzing a person's EEG signal for 10 seconds and the other person's autonomic nervous system dermal electrical activity signal, the time delay value is first obtained by cross-correlation of two time series signals. If this process is repeated for the whole time, the time delay value is obtained every second from the whole time. When the difference between the time delay value of the specific time and the time delay value of the next time is maintained for only one second for the three time delay values, it is determined to be the stable state (1), otherwise, the unstable state (0) (t) -delay (t-tau) < 1s). Then, the ratio of the number of samples determined to be stable among the number of delayed samples obtained during a specific time of 30 seconds is calculated as a stabilization ratio. Finally, the average of the stabilization ratio values obtained from the whole time is obtained as the stabilization ratio value. (In the conventional time delay stability analysis, one stabilization ratio value is obtained by calculating the ratio of the number of stabilization samples to the total time (30 minutes), but in this analysis, every 30 seconds The change of the stabilization ratio can be observed by obtaining the stabilization ratio value every time. As a result, the correlation value is obtained by averaging the values of the stabilization ratio values obtained by removing the outliers.

Other analytical methods require a step to compensate for delay between the brain signal and the autonomic nervous system signal, not by observing the delay between signals, but by observing the correlation from signal similarity. Assuming that the total time is 30 minutes, the time delay value can be obtained by cross-correlation analysis between the two signals every 30 seconds. An average of the obtained time delay values is obtained to obtain an average time delay value tau (tau). Various analyzes as well as cross-correlation analysis can be used to obtain time delay values. Then, the skin conductivity time series signal of the autonomic nervous system is corrected from y (t) to y (t + τ).

Partial di- eected coherence (PDC) analysis is a concept proposed by Baccal et al. (Baccal and Sameshima, 2001), which analyzes the interaction and information transfer in the brain, based on signals detected from multiple channels . PDC is based on the concept of spectral granger-causality and multivariate autoregressive modeling, and is a frequency-domain approach to calculating functional connectivity. For example, if there is an EEG time series signal x (t) and an autonomic nervous system SCR signal y (t), the current y (t) of the channel y from the information of x (ti) If we can predict the value well, we can say that there is a causality between x (t) and y (t), which can be quantified as increasing the PDC value from x (t) to y (t) . That is, it is predicted that there is information transfer from x (t) to y (t), and a value obtained by quantifying this information transfer flow is obtained as a PDC value. In addition to the PDC analysis, the flow of information transfer can be quantified by using the Grandeur causality on the frequency band, the DTF (Directed Transfer Function), and the like. The PDC result has a value between 0 and 1.

). 각 시간에서 뇌파 신호의 순간 진폭 값은 복소 벡터의 길이로 표현할 수 있는 반면, 같은 시간 자율신경계 피부전기활동 신호의 위상 값은 복소 벡터의 각도로 표현된다. 전체 시간에 대해 복소 벡터 값들의 분포를 관찰하면, 위상-진폭 결합이 없을 때는 벡터 값이 대략적으로 균일한 원형 분포를 가지므로 총합은 0에 근사하게 된다. 두 신호간의 결합이 있을 경우, 뇌파 신호의 진폭은 자율신경계 피부전기활동 신호의 특정 위상에서 높게 된다. 즉, 특정 각도에서 복소 벡터의 길이가 크게 나타나고 이는 비대칭적 분포를 갖게 되어 복소 평면 상에서의 전체 벡터 값의 합으로부터 벡터 길이를 얻을 수 있다. 이후 전체 시간에서 얻은 벡터 길이 값들의 평균값을 구하여 평균 벡터길이를 두 신호 간 조절 지수 (modulation index)로 얻는다.Phase-amplitude coupling (PAC) is a method of observing the degree of synchronization between the low-frequency component signal and the high-frequency component signal. The phase of the low frequency band is synchronized with the power of the high frequency band associated therewith, resulting in the synchronization between the amplitude envelope of the fast frequency band signal and the phase of the slow frequency band signal. It is known that from one person's EEG signal, it is mainly used to see the relationship between signals in different frequency bands in the brain region, and it is known that the phase-amplitude coupling can change when a behavioral task is given (Voytek et al ., 2010). It is also known to potentially involve sensory integration, memory processing, and attention selection (Lisman and Idiart et al., 1995). This method can be applied to observe the correlation between the rapidly changing EEG signal and the rather slowly changing autonomic nervous system signal. First, one can input EEG signals in the high frequency band, and the other can input the low frequency band signals in the autonomic nervous system dermatophyte activity. From this, the EEG signal extracts the envelope and extracts the amplitude value through the Hilbert transform, and the autonomic nervous system electric activity signal extracts the phase. For example, a time of 10 seconds can be defined as an analysis interval, and the analysis can be performed while observing the coupling between two signals and increasing the time every 1 second. The complex signal can be computed from the exponent product of the amplitude value and the phase value (

). At each time, the instantaneous amplitude value of the EEG signal can be expressed as the length of the complex vector, while the phase value of the EEG signal at the same time is expressed as the angle of the complex vector. Observing the distribution of the complex vector values over the whole time, the sum is approximated to zero because the vector values have a roughly uniform circular distribution in the absence of phase-amplitude coupling. When there is a coupling between the two signals, the amplitude of the EEG signal becomes high at a specific phase of the autonomic nervous system dermal electrical activity signal. That is, the length of the complex vector is large at a certain angle, and it has an asymmetric distribution, so that the vector length can be obtained from the sum of all the vector values on the complex plane. Then, the mean value of the vector length values obtained from the whole time is obtained, and the average vector length is obtained as a modulation index between two signals.

The event-related characteristic calculation unit shown in FIG. 5 will be described.

7 is a block diagram schematically showing the configuration of an event-related characteristic calculation unit according to an embodiment of the present invention.

Referring to FIG. 7, a signal measured by the user A is used as a reference signal. If A measures both the brain and autonomic nervous system signals, both signals go into the input, and it is possible to select which signal is the event. In the case of the brain signal, one of the signals filtered in a specific frequency band is selected, and the power value is obtained from the time-frequency power extracting unit according to time, and then the frequency power peak values are extracted to determine the time position of the peak values of the upper I% . Alternatively, you can select one of the filtered signals and determine the time position of the point where the maximum amplitude value of the signal is above the upper J% (eg, about 15%) of the standard deviation of the signal amplitude values. For the autonomic nervous system signal, you can select a specific autonomic nervous system signal and set the time position of the values above the upper K% of the maximum amplitude / power value as the brain signal. In the case of the eye movement signal among the autonomic nervous system signals, the eye is detected from the eye tracker and the eye is located in the user B but there is no movement, and the length of the state in which the fixation is performed is higher than P% An intermediate time position can be defined as an event. In the signal of user B, the corresponding time is taken as an event, and the section before and after each event time is extracted. The waveforms obtained from the extracted sections are then averaged. If you want to observe before the event origin from the average waveform, make the base value correction after the base point of the event, and if you want to observe after the event point, correct the base value by setting the baseline level before the event point : The mean value of the waveform at the base level is obtained and the base average value is obtained from the whole waveform value. The degree of change can be outputted as it is, and the rate of change can be outputted by quantifying the ratio of the time when the amount of change is higher than a certain level.

8 is a block diagram schematically showing the configuration of an event-related characteristic calculation unit according to another embodiment of the present invention.

Referring to FIG. 8, it can be seen that the reference signal is almost the same as the embodiment shown in FIG. 7, and the reference signal is changed to the autonomic nervous system signal measured by the user B in the signal measured by the user A. As a reference signal, signals such as skin electrical activity, heart rate, skin surface temperature, and pupil size can be used as the autonomic nervous system signal of the user B. For example, in the case of a heart rate signal, an event detector may be an event with a high heart rate. The time corresponding to the peak of the heart rate higher than the threshold value can be detected as an event with a threshold of 15% of the maximum heart rate as a threshold value. The process of extracting the analysis interval from the EEG signal of the user A with the event as a reference point and obtaining the average waveform and obtaining the change rate value from the change to be observed is the same as that of FIG. Therefore, the remainder of the description is the same except for the difference, so that repetitive description will be omitted.

9 is an exemplary view of the analysis process of the example shown in FIG.

Referring to FIG. 9, a skin electrical activity signal of user A and a brain wave signal of FP1 of user B are input as a pair. In order to detect the vertex as an event in the skin electrical activity signal of the user A, the signal is differentiated to detect the peak, and the amplitude is obtained. 10% of the highest vertex is set as the threshold value and the vertex higher than the threshold value is set as the event. And converts the EEG signal of the user B into a time-frequency spectrum. Then, the time-frequency spectrum is used to extract the 5 second interval before and after the event as a reference point. The average of the spectra of the extracted regions is obtained. In the figure, only [-800 ms to 1800 ms] sections are displayed. After the event, we set the period of interest as the period of interest and corrected the event as the baseline. And detects a section in which the amount of power change in the entire frequency band is larger than a certain level. In this figure, since the interval detected during the entire 5-second interval after the start of the event is 0.5 second, the change rate is 10%.

10 is an illustration of an example of the analysis process shown in FIG.

Referring to FIG. 10, the user A shows a case in which the alpha-band signal of P3 and the skin electrical activity signal of the user B are input as a pair. A time series signal of the alpha wave power is obtained from the signal filtered by the alpha frequency band of the user A. Here, in order to detect the time when the alpha power value is lowered, the alpha wave power signal is smoothed and then differentiated to detect a time point at which the alpha power is greatly reduced. Then, a 10 second interval is extracted from the skin electrical activity signal of the user B based on the event as a reference point. The average of the signals of the extracted intervals is obtained. The pre - event point was defined as the interval of interest and the post - event point was set as the baseline. And detects a section where the amount of change is larger than a certain level by the signal amplitude value. In this figure, since the detected interval of the entire 5-second interval before the start of the event is 3 seconds, the change rate is 30%.

The result matrix generator shown in FIG. 2 will be described.

The result matrix generator may store the correlation analysis between the signals of the two persons in the data matrix using the signal output to the signal analyzer.

11 is an exemplary diagram of a data matrix generated by a result matrix generator according to the present specification.

Referring to FIG. 11, rows and columns are calculated for each user's number and stored in a matrix. The diagonal matrix components represent the results of the brain and autonomic nervous system interaction in the subject. (1, 2) and matrix (2, 1) components have different meanings from each other, because PDC analysis is performed in correlation analysis between time series signals. For example, the matrix (1, 2) component represents the amount of information flow from user 1 to user 2. When the event-related characteristics analysis is performed, it is seen that the characteristics of the user 2 in the case of the user 1 are shown from the user 1 to the user 2 and stored in the matrix (1, 2) component. (2, 1) component and the matrix (2, 1) are stored in the matrix (2, 1) 1) The components have different meanings from each other. For other analyzes that do not have directionality, the meanings of the matrix (1,2) and matrix (2,1) components are the same.

Conventional brain-brain hyper-scanning systems have limitations in that artifacts and high costs are required due to difficulty in controlling brain waves of a large number of people for interaction analysis in the field in actual industrial aspects. The apparatus and method for monitoring an interaction according to the present invention are capable of performing an interaction analysis even when the autonomic nervous system is measured by a plurality of people.

The embodiments and the accompanying drawings described in the present specification are merely illustrative of some of the technical ideas included in the present invention. Accordingly, the embodiments disclosed herein are for the purpose of describing rather than limiting the technical spirit of the present invention, and it is apparent that the scope of the technical idea of the present invention is not limited by these embodiments. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (17)

A signal acquiring unit receiving multiple physiological signals of a plurality of users; A signal processing unit having a preprocessor for preprocessing the signal output from the signal acquiring unit and an interest interval extracting unit for receiving an information signal about an event occurrence time or an interval; A signal analyzer for calculating a correlation between multiple physiological signals of a plurality of users which are preprocessed and output from the signal processor; And a result generation unit for storing correlation analysis between signals of two persons in a data matrix using signals output to the signal analysis unit,
Wherein the signal analyzer comprises: a signal combination selector for selecting one pair of an EEG signal and another autonomic nervous system signal; A functional connectivity calculator for calculating a correlation between two signals selected by the signal combination selector; And a functional connectivity significance determiner for determining whether the value output from the functional connectivity calculator indicates a significant correlation,
A signal of one person is inputted to each signal of the two signals selected by the signal combination selection unit as a reference, the interest signal is extracted from the reference signal by using the feature point as an event, and the degree of occurrence of the event related change is quantified And an event-related characteristic calculation unit. The method according to claim 1,
Wherein the signal obtaining unit comprises:
A brain activity signal acquisition unit receiving a brain signal of a first user; And
And an autonomic nervous system signal acquisition unit that receives the autonomic nervous system signal of the second user. delete The method according to claim 1,
Wherein the functional connectivity calculation unit comprises:
A characteristic value calculation unit for extracting a characteristic value from each signal of the two signals selected by the signal combination selection unit and extracting a correlation between the characteristic values;
A time-series-to-signal correlation calculator for analyzing a correlation between the signal output from the characteristic value calculator and the time-series signal; And
And a characteristic-value correlation calculator for calculating a linear correlation coefficient between each characteristic value through a Pearson correlation analysis of the signal output from the characteristic value calculator. The method of claim 4,
Wherein the time series correlation calculator performs a time delay stability analysis considering a time delay between a plurality of users. The method of claim 4,
Wherein the time series signal correlation calculation unit performs a partial directionality consistency analysis between a plurality of users. The method of claim 4,
Wherein the time-series correlation calculator performs phase-amplitude synchronization analysis between a plurality of users. (a) receiving a plurality of user's multiple physiological signals by a signal acquisition unit; (b) preprocessing the signal output from the signal acquisition unit by the preprocessing unit of the signal processing unit; (c) receiving an information signal of an event occurrence time or an interval of the preprocessed signal; (d) calculating a correlation between multiple physiological signals of a plurality of users, which are pre-processed by the signal analysis unit and output in the signal processing unit; And (e) a result generating unit, using the signal output to the signal analyzing unit, storing correlation analysis between signals of two persons in a data matrix,
The step (d) includes the steps of: (d-1) selecting the signal combination selecting unit as a pair of an EEG signal of one person and an autonomic nervous system signal of another person; (d-2) calculating a correlation between the two signals selected by the signal combination selection unit; And (d-3) determining whether the value of the functional connectivity determiner is indicative of a significant correlation between the value output from the functional connectivity calculator,
In the step (d-2), the event-related characteristic calculation unit inputs a signal of one person in each signal of the two signals selected by the signal combination selection unit as a reference, and a feature point in the reference signal as an event Extracting interest intervals and quantifying the degree of occurrence of event related changes. The method of claim 8,
The step (a)
(a-1) a brain activity signal acquisition unit receiving a brain signal of a first user; And
(a-2) the autonomic nervous system signal acquisition unit receiving the autonomic nervous system signal of the second user. delete The method of claim 8,
The step (d-2)
Extracting a characteristic value from each signal of the two signals selected by the signal combination selection unit and extracting a correlation between the characteristic values; And
And analyzing the correlation between the time-series signals and the signals output from the characteristic-value calculating unit. The method of claim 11,
The step (d-2)
Extracting a characteristic value from each signal of the two signals selected by the signal combination selection unit and extracting a correlation between the characteristic values; And
And calculating a linear correlation coefficient between each characteristic value through a Pearson correlation analysis of the signal output from the characteristic value calculation unit. delete The method of claim 11,
Wherein the step of analyzing the correlation between the signals output from the characteristic value calculating unit and the time series signals is a step of performing a time delay stability analysis considering a time delay appearing among a plurality of users. Method of Monitoring Activity. The method of claim 11,
Wherein the step of analyzing the correlation between the signals output from the characteristic value calculation unit and the time series signal is a step of performing a partial directionality consistency analysis among a plurality of users. The method of claim 11,
Wherein the step of analyzing the correlation between the signals output from the characteristic value calculation unit and the time series signals is a step of performing phase-amplitude synchronization analysis between the plurality of users. . A computer program recorded on a computer-readable recording medium, the computer program being written in the computer to perform the steps of the interaction monitoring method according to any one of claims 8, 9, 11, 12 and 14 to 16.

Images