KR20190045041A - Method for recogniging user intention by estimating brain signals, and brain-computer interface apparatus based on head mounted display implementing the method - Google Patents

Abstract

The present invention relates to a method for enabling a controller to recognize the intention of a user wearing a head-mounted display attached with brain conduction electrodes. The method comprises the steps of: obtaining learning brain signals measured at each of the head-mounted display and an additional electrode device coupled to the head-mounted display in a state in which a learning screen inducing brain signals is displayed on the head-mounted display; learning a relationship model of the electrodes of the head-mounted display and the electrodes of the additional electrode device from the learning brain signals; acquiring the brain signals measured in the head-mounted display, from which the additional electrode device is detached, in a state in which a specific screen is displayed on the head-mounted display; estimating brain signals of the additional electrode device not measured from the brain signals measured in the head-mounted display based on the relationship model; and detecting user intention corresponding to the measured brain signals and the estimated brain signals.

Description

TECHNICAL FIELD [0001] The present invention relates to a method of recognizing a user's intention by estimating a brain signal, and a head-mounted display-based brain-computer interface device implementing the method. }

The present invention relates to a Brain-Computer Interface (BCI).

The brain-computer interface is a technology for recognizing user's intention by analyzing brain signals generated during various brain activities and controlling the external device through this. The user can control the computer or machines without using the muscle through the brain-computer interface. The brain-computer interface has been used mainly for the development of devices to assist the movement of patients with motor neuropathy due to accidents or diseases. Recently, studies are being conducted to apply brain-computer interfaces in various fields.

A wearable device such as a head mount display provides augmented reality or virtual reality by outputting an image at a position facing a user's eyes. The head mount display can detect a user's operation using a gyro sensor or the like, but the user must wear a head mount display and control the screen using a joystick, a keyboard, or the like. If a brain-computer interface is used for head-mounted display control, it will be possible to control without using a tool. However, in order to recognize the intention of the user through the brain-computer interface, brain signals generated in various brain regions must be analyzed. Therefore, many electrodes must be attached to the scalp. If a small number of electrodes are attached, only the brain signal of the area where the electrodes are attached can be obtained, which lowers the recognition accuracy. It is also possible to attach electrodes to a head-mounted display in the form of a hat, but it is difficult to apply a brain-computer interface to the head-mounted display due to the difficult and costly manufacture of small and lightweight displays.

 Therefore, a brain-computer interface technology capable of maintaining a high recognition performance while using a small number of electrodes is required for a user to use the device conveniently in a real life environment. There is also a need for a method for efficiently controlling the head-mounted display using the brain signal measurement paradigm of various brain-computer interfaces.

A problem to be solved by the present invention is to provide a method for recognizing a user's intention by estimating a brain signal and a head-mounted display-based brain-computer interface device implementing the same. Specifically, the present invention provides a head-mounted display having electrodes attached to a removable additional electrode device to acquire brain signals, and generates an inter-electrode relationship model using brain signals acquired from the electrodes , And brain signals required for user's intention recognition using the brain signals measured in the head mount display and the relationship model between the electrodes.

Another object of the present invention is to provide a method of controlling a head-mounted display using an electro-oculography (EOG) signal.

There is provided a method of recognizing a user's intention of wearing a head mounted display to which brain conduction electrodes are attached in a state in which a learning screen for causing brain signals is displayed on the head mounted display, Acquiring learning brain signals measured at each of a display and an additional electrode device coupled to the head-mounted display, learning relationship models of the electrodes of the head-mounted display and the electrodes of the additional electrode device from the learning brain signals The method comprising the steps of: obtaining brain signals measured in the head-mounted display from which the additional electrode device has been separated, with a specific screen displayed on the head-mounted display; Signals from the unmeasured phase Estimating brain signals of the additional electrode device, and detecting user intent corresponding to the measured brain signals and estimated brain signals.

The user's intention recognition method may further include learning a user's intention classification model from the learning brain signals. The step of detecting the user intention may detect a user intention corresponding to a signal obtained by combining the measured brain signals with the estimated brain signals based on the learned user's intention classification model.

The learning screen may be an interface screen designed to generate brain signals corresponding to a specific brain-computer interface paradigm.

The brain-computer interface paradigm includes at least one of a motor imagery potential, a steady state visual / auditory / somatosensory evoked potential, and an event related potential. .

When the learning screen is an interface screen for generating brain signals corresponding to an operation image, the brain signals for learning may be imaginary signals that move a body part designated on the learning screen.

When the learning screen is an interface screen for generating brain signals corresponding to an event related potential, the learning screen separates the left screen and the right screen corresponding to the respective eyes, and provides the left screen and the right screen with the same object Can be provided with a visual stimulus that blinks at different frequencies. The learning brain signals may be signals induced by visual stimulation included in the learning screen.

When the learning screen is an interface screen for generating brain signals corresponding to the steady state visual evoked potential, the learning screen may provide a plurality of selectable objects as different visual stimuli. The learning brain signals may be a signal induced when a specific object among a plurality of objects is gazed on the learning screen.

Wherein the learning of the relationship model includes generating brain signal feature vectors including a predetermined number of brain signals measured in past cycles on a brain signal measured in a current cycle for each electrode, The model can be learned.

The user's intention recognizing method may further include controlling the specific screen based on the detected user intention.

The step of controlling the specific screen may include detecting eye movements based on the eye conduction signals obtained from the eye conduction electrodes attached to the head mount display, changing the view of the specific screen according to the detected eye movement, The specific screen can be controlled according to the user's intention detected by the measured brain signals and the estimated brain signals.

Wherein the user's intention recognizing method includes a step of displaying a relationship model of the cursor velocity with the learning eye conduction signals obtained from the eye conduction electrodes in a state in which a cursor moving along the eye direction is displayed on the cursor screen prior to the step of controlling the specific screen And may further include learning steps. The step of controlling the specific screen may detect eye movements corresponding to the obtained eye conduction signals based on a relationship model of the eye conduction signals and the cursor speed.

A brain-computer interface apparatus according to one embodiment, comprising: a head-mounted display having a first channel group of brain conduction electrodes attached thereto; an additional electrode unit detachably attached to the head-mounted display, Storing a relationship model of brain signals obtained at the brain conduction electrodes of the first channel group and the brain conduction electrodes of the second channel group, and based on the relationship model, And a controller for controlling the display. Wherein the controller estimates brain signals of brain conduction electrodes of the second channel group based on the relationship model when the brain signals measured in the first channel group are inputted while displaying a specific screen on the head mount display Detects user intention using brain signals measured in the first channel group and brain signals estimated in the second channel group, and controls the specific screen based on the detected user intention.

Wherein the controller displays on the head mounted display a learning screen that causes brain signals in a learning mode in which the additional electrode device is coupled to the head mounted display, Acquiring brain signals, learning the relational model using the learning brain signals, and storing the learned relational model.

Wherein the learning screen is an interface screen produced to induce brain signals corresponding to a specific brain-computer interface paradigm, wherein the brain-computer interface paradigm includes a motor imagery potential, a steady state visual / auditory / tactile evoked potential Steady-State Visual / Auditory / Somatosensory Evoked Potential, and Event Related Potential.

The controller may learn the relational model by regression analysis of the learning brain signals.

The controller detects the user intention corresponding to the feature of the signal obtained by combining the brain signals measured in the first channel group and the brain signals estimated in the second channel group based on the learned user's intention classification model . The user's intention classification model may be a model learned to classify into one of a plurality of intents specified in a specific brain-computer interface paradigm based on characteristics of brain signals acquired in the first channel group and the second channel group .

A brain-computer interface apparatus according to another embodiment, comprising: a main body including at least one processor, a memory and a display, wherein at least a portion of the inner surface is in contact with the user's face, And an additional electrode device coupled to or separated from the head mount display and having an inner surface adjacent to the user's scalp and having brain conduction electrodes attached thereto do. The head-mounted display includes a plurality of brain conduction electrodes arranged at an arbitrary interval along a circumference of a user on the inner surface of the fixing unit, a plurality of brain conduction electrodes arranged in a plurality of directions from a position where the user & A plurality of eye conduction electrodes arranged at intervals of a predetermined distance, a reference electrode disposed in a region where a part of the user's face is in contact with the main body, and at least one of the main body and the fixing portion, And a separate fastening portion.

According to embodiments, after attaching the removable additional electrode device to the head-mounted display to create the inter-electrode relationship model, the user's intention can be accurately identified from the brain signals measured at the electrodes of the head- have.

According to the embodiment, a head-mounted display having a brain conduction electrode and an ocular conduction electrode can provide a user with a convenient, realistic and natural control environment. Therefore, a user who wears a head-mounted display-based brain-computer interface device can control the object to be controlled by thinking or eye movement instead of using the body of the hand or arm directly to control the object.

According to the embodiments, it is possible to provide augmented reality and virtual reality similar to the real life according to the user's eye motion by providing a screen change provided to the user in accordance with the change of the user's view.

According to embodiments, a brain-computer interface can be applied to various fields such as game, entertainment, healthcare, and mobile display utilizing a head mount display, thereby providing an optimal control environment suitable for contents.

1 is a view for explaining a wearing state of a head-mounted display-based brain-computer interface device according to an embodiment.
FIG. 2 is a conceptual illustration of a relationship model between electrodes of a head-mounted display-based brain-computer interface device according to an embodiment.
Each of Figures 3 and 4 is an example of a head-mounted display based brain-computer interface device according to one embodiment.
5 is a conceptual block diagram of a head-mounted display-based brain-computer interface device according to an embodiment.
6 is a functional block diagram of a controller according to one embodiment.
7 is a flowchart of a method of learning the inter-electrode relationship model of a controller according to an embodiment.
8 is a flowchart of a classification model learning method for user's intention recognition of a controller according to an embodiment.
9 is a flowchart of a brain signal estimation method of a controller according to an embodiment.
FIG. 10 is an illustration of a learning interface screen for operation imagination in a brain-computer interface paradigm.
Figs. 11 and 12 show an example of screen control according to the operation image.
13 is an illustration of a learning interface screen for steady state visual evoked potentials in the brain-computer interface paradigm.
14 is an illustration of a learning interface screen for event related potentials in the brain-computer interface paradigm.
15 is an illustration of a character input interface using event-related potential-based visual stimulation.
Figure 16 is an illustration of a learning interface screen for steady state auditory evoked potentials in a brain-computer interface paradigm.
17 is an illustration of screen control by eye movement of a user according to an embodiment.
18 is an illustration of screen control over the head and eye movements of a user according to one embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as " comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, " " module, " and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

FIG. 1 is a view for explaining a wear state of a head-mounted display-based brain-computer interface device according to an embodiment, and FIG. 2 is a conceptual Fig.

1, a head-mounted display-based brain-computer interface device 10 includes a head-mounted display 100 to which electrodes are attached and an electrode device (hereinafter referred to as an "additional electrode device" (200). The positions of the brain conduction electrodes disposed in each of the head mount display 100 and the additional electrode device 200 can be preset or adjusted by the user to measure the user's brain signal. The head mount display 100 may further include eye conduction electrodes capable of measuring an ocular conduction signal.

As shown in Fig. 1 (a), the head-mounted display 100 and the additional electrode device 200 can be combined, which is referred to as a coupled state. As shown in FIG. 1 (b), the additional electrode device 200 can be disconnected from the head mounted display 100, which is referred to as a disconnected state.

Although only one additional electrode device 200 is described as being coupled to or separated from the head-mounted display 100, a plurality of additional electrode devices may be coupled to the head-mounted display 100 in the description. In this case, the separated state of at least one additional electrode device among the plurality of additional electrode devices may be referred to as a disconnection state.

Referring to FIG. 2 (a), brain signals are measured with the user wearing the head-mounted display 100 with the additional electrode device 200 coupled thereto. The brain signals measured in the coupled state are used as learning data for brain signal estimation in the separated state. Using the measured brain signals, a relationship model W (simply referred to as an inter-electrode relationship model) between electrodes of the head mount display 100 and the additional electrode device 200 is learned. That is, the brain signals x measured at the electrodes X ₁ , X ₂ , ..., X _{nx of the} head mount display 100 and the electrodes Y ₁ , Y ₂ , ..., Y _ny ), the inter-electrode relationship model (W) is learned. The interelectrode relationship model (W) can be a regression model derived from regression analysis such as linear regression analysis and nonlinear regression analysis.

을 추정할 수 있다. 2 (b), the electrodes X ₁ , X ₂ ,... Of the head-mounted display 100 are connected to the head-mounted display 100 while the additional electrode device 200 is worn by the user. ., X _nx ). At this time, the electrodes (Y ₁ , Y ₂ , ..., Y _ny ) of the additional electrode device 200 are not measured with the inter-electrode relationship model W learned from the brain signals measured in the coupled state, ) Brain signals

Can be estimated.

According to the present invention, even if only the brain signals of some points (electrode channels) are measured while wearing only the head mount display 100, brain signals of other points can be estimated through the learned inter-electrode relationship model. That is, brain signals of a level required to accurately recognize the user's intention through the brain signal estimation are additionally obtained. Therefore, even if the user wears only the head mounted display 100 after learning, the user does not always have to wear the head mounted display 100 to which the additional electrode device 200 is attached, the brain signals are sufficiently acquired, The intention can be recognized easily and accurately.

Each of Figures 3 and 4 is an example of a head-mounted display based brain-computer interface device according to one embodiment.

3 and 4, the head mount display 100 includes a main body 110 that is worn by a user and provides augmented reality and virtual reality-based contents, and a fixing unit 120 that fixes and supports the main body 110 to the head ). When the user wears the head mount display 100 on the head, the main body 110 is tightly supported so that the inner surface thereof contacts at least a part of the face of the user, and the fixing portion 12 has the inner surface, So as to contact at least a part of the scalp along the head circumference.

The head mount display 100 may include various components necessary to provide augmented reality and virtual reality, and may include components of a typical head mount display. For example, the main body 110 may include at least one processor, a memory, a display for displaying an augmented reality image or a virtual reality image, and may be configured to photograph outside in real time in order to provide an augmented reality outside the main body 110 The camera 111 can be mounted. A lens 112 may be mounted on the face of the main body 110 at a position facing the user's eyes. The fixing unit 120 may include an earphone 121 or a speaker (not shown) for providing sound (audio signal) generated in the head mount display control. Meanwhile, the insertion portion of the earphone 121 (i.e., the portion directly inserted into the user's ear) may further include a brain conduction electrode capable of measuring a brain signal in the ear.

At least one of the main body 110 and the fixing portion 120 is formed with coupling portions 130 and 131 for coupling / separating the additional electrode device 200. Depending on the shape of the coupling, the position of the coupling part can be variously formed. 2, the front coupling part 230 and the rear coupling part 231 of the additional electrode device 200 are inserted into the coupling parts 130 and 131, respectively, .

The head mount display 100 includes a plurality of electrodes. The positions of the electrodes can be variously arranged according to brain signal measurement points / eye conduction signal measurement points. A plurality of brain conduction electrodes 141, 142, ..., 149 may be attached along the head circumference to the inner surface of the fixation portion 120 (i.e., a portion of the surface directly facing the user's head). A plurality of ocular conductive electrodes 151, 152, ..., 158 may be attached to the inner surface directly facing the face of the user of the body 11. [ Each eye conduction electrode is disposed at a position close to the user's eye in the face portion, and can be attached to the upper / lower / left / right of the eyes. The main body 11 includes a reference electrode (or a ground electrode) 160 that serves as a reference for a brain signal and an ocular conduction signal, and may be attached to, for example, an area facing a user's nose position. At this time, the reference electrode and the ground electrode can be used as one electrode.

The additional electrode device 200 is attached with a plurality of brain conduction electrodes (not shown) on the inner surface (lower surface) contacting the wearer's head. For example, a plurality of brain conduction electrodes may be arranged in the lateral direction and the longitudinal direction on the lower surface of the additional electrode device 200.

The shape of the additional electrode device 200 can be variously designed. For example, the additional electrode device 200 is composed of a first body having a predetermined length and area in the forward and backward directions and a second body having a predetermined length and area in the left and right direction, The first body and the second body may be physically connected to each other through the connection part 210 and may be electrically connected to each other.

In order to align the additional electrode device 200 with the user's head and adjust the gap between the brain conduction electrodes, the additional electrode device 200 includes the front and rear electrode interval extension part 220 and the left and right electrode interval extension part 222 do. The front and rear electrode interval extension portions 220 extend in the front-rear direction from the connection portion 210 and the left and right electrode interval extension portions 222 can extend in the left-right direction from the connection portion 210. Each of the electrode interval extension portions 220 and 222 has a plurality of bar shapes and each bar is inserted into a previous bar toward the bar 210 and can be sequentially extended outwardly as the bar is extended. It may be a curved bar shape that closely adheres to the head curvature of the floor user. The brain conduction electrode may be attached to each node of the connection portion 210, the front and rear electrode interval extension portion 220, and the left and right electrode interval extension portion 222. Through this additional electrode device 200, brain signals of various brain areas of the user, which can not be measured in the head-mounted display 100, can be measured.

Each of the front and rear electrode spacing extension portions 220 and the left and right electrode spacing extension portions 222 may extend leg portions 240 and 242 for supporting and fixing on the user's head. The legs 240 and 242 are in contact with the user's head and may be constructed of a strap or a flexible material. At least one end of the legs 240 and 242 may be formed with a front engaging portion 230 and a rear engaging portion 231 that are inserted into the engaging portions 130 and 131 of the head mount display 100.

3 and 4, the brain-computer interface device 10 includes a processor (not shown), a memory (not shown), a communication module (not shown), and the like (E.g., a head mount display control program) that performs the method described in the present invention through hardware. The programs are stored in at least one memory, and at least one processor executes the programs to process the methods described in the present invention. Each of the head mount display 100 and the additional electrode device 200 may be operated by hardware such as at least one memory, at least one processor, and at least one communication module, or the head mounted display 100 may be coupled The brain signal measurement of the additional electrode device 200 can be controlled.

The brain signals measured at the additional electrode device 200 are transmitted to a device for learning an inter-electrode relationship model based on the measured brain signals. The device for learning the inter-electrode relationship model may be a head mount display 100, May be an external control device. At this time, the brain signals measured at the additional electrode device 200 may be transmitted to the combined head-mounted display 100, or the additional electrode device 200 may transmit the measured brain signals directly to a designated server. The head mount display 100 and the additional electrode device 200 can communicate through a communication module. Or the additional electrode device 200 coupled to the head mounted display 100 is electrically connected to the head mounted display 100 and the head mounted display 100 measures the brain signals through the combined additional electrode device 200 can do.

The head mount display 100 may communicate with an external device (not shown), e.g., a mobile terminal, via a communication module. The head mount display 100 can communicate with an external control server via a communication module.

The following describes how a head-mounted display-based brain-computer interface device estimates the brain signal and recognizes the user's intention. The method described in the present invention can be performed by a processor built in the head mount display 100 or by the processors incorporated in the head mount display 100, the additional electrode device 200, can do. In the description, it is assumed that the controller 300, which is operated by at least one processor, processes the method described in the present invention, and that the controller 300 is included in the head mounted display 100.

FIG. 5 is a conceptual block diagram of a head-mounted display-based brain-computer interface apparatus according to an embodiment, and FIG. 6 is a functional block diagram of a controller according to an embodiment.

Referring to FIG. 5, the brain-computer interface device 10 includes a head-mounted display 100 and an additional electrode device 200, and may include a controller 300 for controlling them. Additionally, the head mount display 100 may communicate with an external control device including a user terminal or an external control server.

On the other hand, the controller 300 may be designed and modified to communicate with the head mounted display 100 outside the head mounted display 100 to control the head mounted display 100. In addition, some of the functions of the controller 300 can be modified by the processor of the head mount display 100, and the remaining functions can be processed by the external control device.

Referring to FIG. 6, the controller 300 includes a brain signal measuring unit 310, a brain signal learning unit 330, a brain signal estimating unit 350, and a user's intention recognizing unit 370.

The brain signal measuring unit 310 measures the user's brain signals when the head-mounted display 100 and the additional electrode device 200 are in a coupled state. The brain signals measured in the combined state are the learning data used to learn the relationship model between the brain conduction electrodes. At this time, the brain signal measuring unit 310 outputs a screen, vibration, and sound produced for each brain-computer interface paradigm to a display of the head mount display 100, a vibration sensor, a speaker, The induced brain signals can be measured. The brain-computer interface paradigm, for example, can be classified into three categories: motor Imagery (MI) potential induced by imagining body movements, steady state visual / auditory / somatosensory evoked potential, An Event Related Potential (ERP), and the like.

The brain signal measuring unit 310 receives the estimated brain signals from the brain signal estimating unit 330 along with the brain signals measured in the head mount display. The input data can be used as learning data.

In addition, the brain signal measuring unit 310 measures brain signals of the user wearing the head-mounted display 100, with the additional electrode device 200 separated. The brain signals measured in the detached state are used to estimate the brain signals of the unmeasured points.

The brain signal measuring unit 310 may set a brain signal measuring mode before measurement to distinguish the measured brain signals. The brain signal measurement mode can be classified into a learning mode or a combination mode for learning brain signal measurement, a user intention recognition mode, or a separation mode. In addition, the brain signal measuring unit 310 can set a brain-computer interface paradigm. The brain signal measuring unit 310 separately stores the brain signals measured according to the setting.

The brain signal learning unit 330 learns a relationship model between the head mounted display 100 and the brain conduction electrodes of the additional electrode device 200 based on the learning brain signals measured by the brain signal measuring unit 310.

In addition, the brain signal learning unit 330 learns a classification model for user's intention recognition suitable for the brain-computer interface paradigm based on the brain signals measured by the brain signal measuring unit 310.

The brain signal learning unit 330 can increase the learning data by performing data augmentation on the acquired learning data according to the characteristics of the deep learning that can provide higher performance as more data to learn. Data reinforcement methods can vary.

The brain signal estimating unit 330 estimates brain signals of the additional electrode device 200 not measured from brain signals measured by the user wearing only the head mount display 100 based on the learned inter- .

The user's intention recognition unit 250 determines a user's intention corresponding to the inputted brain signals based on the classification model for the learned user's intention recognition. The input brain signals may be brain signals measured at the head mount display 100 and estimated brain signals. Or the inputted brain signals may be brain signals measured in the state in which the head mounted display 100 and the additional electrode device 200 are coupled.

7 is a flowchart of a method of learning the inter-electrode relationship model of a controller according to an embodiment.

Referring to FIG. 7, in a state in which the user wears the head-mounted display 100 to which the additional electrode device 200 is coupled, the controller 300 specifies a brain-computer interface paradigm (S110). The brain-computer interface paradigm can be selected by the user, or a paradigm for learning in the learning mode can be specified.

The controller 300 measures the user's brain signals while displaying an interface screen for generating brain signals corresponding to the designated brain-computer interface paradigm (S120). (X ₁ , x ₂ , ..., x _nx ) measured at the electrodes of the head mount display 100 and the brain signals y ₁ , y ₂ , ..., y _ny are obtained. At this time, brain signals measured at the head mount display 100 and the additional electrode device 200 have the same sample rate. Each electrode can be identified by a channel identifier. On the other hand, the electrodes (channels) of the head mount display 100 may be referred to as a first channel group, and the electrodes (channels) of the additional electrode device 200 may be referred to as a second channel group.

The controller 300 stores brain signals measured for a predetermined time for the designated brain-computer interface paradigm (S130). The measured brain signal is subjected to a band-pass filter and a normalized pre-processing. The bandpass filter can be set to a frequency range that is related to the specified brain signal measurement paradigm. For example, the normalization method can be normalized such that the mean value of the brain signals of each electrode (channel) is 0 and the variance value is 1.

The controller 300 generates learning data for the brain-computer interface paradigm from the measured brain signals (S140). The learning brain signal can be obtained by repeatedly attempting to measure a certain time at each electrode, or it can be a signal measured for successive times at each electrode. Or a learning brain signal can be generated by accumulating brain signals (samples) measured at each electrode in the past (e. G., Previous measurement periods). Meanwhile, the controller 300 separates the measured brain signals for learning, verification, and evaluation, and uses some of the measured brain signals as learning data.

The controller 300 initially learns the relationship model W between the electrodes of the head-mounted display 100 and the additional electrode device 200 using the learning data (S150). The inter-electrode relationship model W includes the relationship between the brain signals of the head-mounted display 100 and the brain signals of the additional electrode device 200, and then the relationship between the brain signals of the head- It is used to estimate brain signals of measurement points. The unmeasured points are electrode positions of the electrodes of the additional electrode device 200 and may be referred to as a brain signal estimation point.

The controller 300 optimizes the parameters of the initial learned inter-electrode relationship model W using the verification brain signals (S160).

의 상관 계수(Correlation coefficient), 또는 평균 제곱 오차를 사용할 수 있다. 상관 계수가 1에 가까울수록 또는 평균 제곱 오차가 0에 가까울수록 관계 모델의 정확도가 증가한다.The controller 300 evaluates the optimized inter-electrode relationship model W using the brain signals for evaluation (S170). The relational model evaluation index is calculated from the brain signal (y) measured and the brain signal

Correlation coefficient, or mean square error, can be used. The closer the correlation coefficient is to 1 or the closer to the mean square error is, the more accurate the relationship model is.

If the accuracy is equal to or greater than the reference value, the controller 300 ends the learning of the inter-electrode relationship model (S180). If the accuracy is less than the reference value, the controller 300 can repeat the learning, optimization, and evaluation of the inter-electrode relationship model. If desired, the controller 300 may request additional brain signal measurements. The controller 300 may learn the inter-electrode relationship model by brain-computer interface paradigm.

The controller 300 can improve the accuracy of the inter-electrode relation model through an error correction method of optimizing the inter-electrode relation model by feeding back the error of the estimated brain signal. The controller 300 can correct the error by a combination of various regression analysis techniques. For example, the error generated in the brain signal estimated by the linear regression analysis is subjected to kernel ridge regression, The error can be reduced.

The relationship model between electrodes can be variously generated according to the applied regression analysis technique. The relationship model between the electrodes can be generated by, for example, linear regression analysis, ridge regression analysis, kernel ridge regression analysis, ensemble regression analysis, and the like. The number of variables in the inter-electrode relationship model depends on the regression model. For example, the relation model generated by the ridge regression analysis is the ridge size, and the relationship model generated by the kernel ridge analysis is the ridge size and the kernel size It can be a variable.

는 리지의 크기를 나타내는 변수이다. k는 커널을 의미하고, 커널은 선형(Linear) 커널, 다항(Polynomial) 커널, 가우시안 커널 등이 선택적으로 사용될 수 있다. m은 앙상블에 사용된 모델의 개수이며, 3개의 모델뿐만 아니라 다양한 모델의 조합으로 형성될 수 있다.The relationship model between the electrodes according to the regression analysis can be defined as shown in Table 1. In Table 1, x is the brain signal measured at the head-mounted display 100 and y is the brain signal measured at the additional electrode device 200.

Is a variable indicating the size of the ridge. k means the kernel, and the kernel can optionally be a linear kernel, a polynomial kernel, or a Gaussian kernel. m is the number of models used in the ensemble, and can be formed of a combination of various models as well as three models.

Regression analysis technique Inter-electrode relationship model Linear regression analysis

Ridge regression analysis

Kernel ridge regression analysis

Ensemble regression analysis

The controller 300 may select measurement methods and process the measured data to generate training data of various dimensions. For example, in a particular brain-computer interface paradigm, the controller 300 may repeatedly attempt measurements over a period of time to obtain brain signals in that paradigm. The brain signals obtained according to the repeated measurement attempts can be stored in the same data dimension (number of data) as the type 1 in Table 2. The brain signals acquired over successive times are the same as Type 2 in Table 2, and the data dimension in this case is equal to the number of electrodes. The brain signals obtained by the repeated attempts can be transformed into data combining trial times as in Type 3 of Table 2 and stored.

Kinds The brain signals of the head mount display
(N _cx : number of electrodes) The brain signals of the additional electrode device
(N _cy : number of electrodes) One

T_trial: 단일 시도(trial)의 시간
N_trial: 시도 수

T _trial : the time of a single trial
N _trial : number of attempts

T _trial : Time of a single attempt
N _trial : number of attempts 2

T: Time

T: Time 3

T _{trial x nt} : Total trial time

T _{trial x nt} : Total trial time

On the other hand, the controller 300 can generate learning data by accumulating a predetermined number of brain signals (samples) measured in the past (past period) at each electrode. The controller 300 may use the time delayed samples to convert brain signal characteristics at a particular time t (current period).

는 전극 수와 시간 지연된 샘플 수의 곱이고,

는 전체 시도 시간이다. The feature vector x [tau] (t) of the brain signal converted into the time delay samples at a specific time t may be expressed by a vector as shown in Equation (1) 2 < / RTI > In Equation (1), ch is the corresponding electrode channel and k is the number of samples delayed in time. In Equation (2)

Is the product of the number of electrodes and the number of samples delayed by time,

Is the total attempt time.

Since the time-delayed samples are accumulated for each electrode and used for learning the interelectrode relation at a specific time t, there is an effect that the number (dimension) of brain signals used at a specific time is increased. That is, when generating the learning data indicating the relationship between the electrodes at the specific time point t, not only the brain signals measured at the specific time point t but also the brain signals measured in the past are also used at a specific time point t In the learning data indicating the interelectrode relationship in the electrodes. For example, using 10 time-delayed samples for brain signals measured at three electrodes on the head-mounted display, 30 feature values are used to estimate the brain signal at a given time. Comparing Table 3 through Table 5 shows that the sample delay method can increase the accuracy of the relational model.

의 상관 계수이다. 표 4는 10초의 학습 데이터에 대해 15개의 샘플 지연을 적용한 경우의 두 뇌 신호

의 상관 계수이다. 표 5는 25초의 학습 데이터에 대해 15개의 샘플 지연을 적용한 경우의 두 신호

의 상관 계수이다. 표 3부터 표 5를 비교하면, 시간 지연된 샘플들을 이용하여 학습한 경우, 관계 모델의 정확도를 평가하는 지표인 상관 계수가 증가하는 것을 보이고, 학습 데이터를 증가시킬 때에도 상관 계수가 증가되는 것을 확인할 수 있다.Table 3 shows the measured brain signals in the absence of sample delay for 10 seconds of training data and the estimated brain signals

. Table 4 shows the results of two brain signals with 15 sample delays applied to 10 seconds of training data

. Table 5 shows two signals with 15 sample delays applied for 25 seconds of training data

. Comparing Table 3 to Table 5 shows that correlation coefficient, which is an index for evaluating the accuracy of relationship model, increases when learning with time delayed samples, and that correlation coefficient increases even when learning data is increased have.

On the other hand, according to the characteristics of the deep learning that can provide a higher classification performance as the learning data is larger, the data of each learning data can be augmented. For example, the data enhancement method can be used for steady state visual / visual / tactile evoked potentials that utilize frequency information of the brain signal. For example, by applying a small-sized window to a brain signal collected at a single trial time, frequency characteristics can be extracted by moving one sample in the time axis and used as learning data.

8 is a flowchart of a classification model learning method for user's intention recognition of a controller according to an embodiment.

Referring to FIG. 8, the controller 300 generates training data for classifying model learning using brain signals measured for a certain period of time with respect to the brain-computer interface paradigm (S210). The controller 300 may generate training data for classification model learning using brain signals measured for learning of the inter-electrode relationship model. Meanwhile, the controller 300 may use brain signals estimated from the measured brain signals and the inter-electrode relationship model as learning data for classifying model learning. The training data may be an EEG _combination of the brain signal x of the head mounted display 100 and the brain signal y of the additional electrode device 200. EEG _combination can be variously processed according to the trial time, number of attempts, and so on, as shown in Table 2, which is used for learning the inter-electrode relation model.

The controller 300 generates a classification model for user's intention recognition based on learning data for each brain-computer interface paradigm (S220). At this time, the controller 300 may generate a common spatial pattern, a linear discriminant analysis, a support vector machine, a canonical correlation analysis, a Fourier transform, And classifying models can be generated using various pattern recognition and machine learning techniques such as deep learning. The classification model can be generated through learning, verification, and evaluation, similar to the inter-electrode relationship model learning.

The classification model is generated differently according to the brain-computer interface paradigm.

In the case of a brain signal measured in the Motor Imagery paradigm, a spatial filter that maximizes dispersion between two different classes (ie, maximizes dispersion for one class while minimizing dispersion of other classes) A common spatial pattern method can be used. The variance of the brain signal transformed through the filter is calculated and normalized and used as a feature vector for user's intention classification. At this time, classifiers such as Linear Discriminant Analysis, Support Vector Machine, and Deep Learning may be used.

For brain signals measured at steady state visual / auditory / tactile evoked potentials, Canonical Correlation Analysis and Fourier Transform can be used to consider frequency characteristics. When the canonical correlation analysis is performed, reference signals having frequencies such as visual / auditory / tactile stimuli are generated. Then, a weight vector that maximizes the correlation between the reference signals and the trial brain signal is obtained through canonical correlation analysis, and the user intention is classified using the correlation with each reference signal. Classifiers such as maximum value comparison, k-nearest neighbor, Support Vector Machine, and Deep Learning may be used.

In the case of a brain signal measured at the event evoked potential, the user's intention is classified using the magnitude change of the brain signal occurring after a certain time (several hundred ms) after the event (event or stimulus) is given. First, the brain signal features related to the event evoked potentials are extracted from the brain signal through Linear Discriminant Analysis, Principal Component Analysis, and Independent Component Analysis. Classifiers such as linear discriminant analysis, support vector machines, and deep learning may be used.

9 is a flowchart of a brain signal estimation method of a controller according to an embodiment.

Referring to FIG. 9, the controller 300 stores the learned inter-electrode relationship model and the learned user's degree classification model (S310).

When the user wears the separate head-mounted display 100 with the additional electrode device 200, the controller 300 acquires the brain signals x ^* measured at the head-mounted display 100 (S320) . The controller 300 preprocesses the brain signals measured in the same manner as the preprocessing performed during learning.

The controller 300 generates the feature vector of the brain signals measured to correspond to the input vector of the learned inter-electrode relationship model (S330). That is, the controller 300 measures and processes brain signals in accordance with the data dimension and data format used in the learning of the inter-electrode relationship model. For example, as shown in Equation (1), it is possible to accumulate time-delayed samples for each electrode and generate it as a feature vector at a specific time t.

The controller 300 estimates brain signals of the unmeasured points from the feature vectors of the measured brain signals based on the learned inter-electrode relationship model (S340). The non-measured point corresponds to the electrode position of the additional electrode device 200.

의 조합 신호(EEG_combination)를 생성한다(S350). 이때, 조합 신호는 수학식 3과 같이, 헤드 마운트 디스플레이 전극들과 추정된 추가 전극 장치의 전극들에서 시간(T) 동안 획득된 신호 벡터로서 표현될 수 있다.The controller 300 compares the measured brain signals (x ^* ) and estimated brain signals

The generation of the combined signal (EEG _combination) (S350). At this time, the combination signal can be expressed as a signal vector obtained for the time (T) at the head-mounted display electrodes and the electrodes of the estimated additional electrode device, as shown in Equation (3).

The controller 300 determines a user intention corresponding to the characteristic of the combination signal based on the classification model for learning the user intention (S360). For example, a classification model that learns dynamic imaginary potentials can classify a user imagined part from brain signal characteristics. The classification model is learned to be classified into one of a plurality of intents according to the brain-computer interface paradigm.

As described above, even if a large number of electrodes are not attached to the user's head, improved user's intention recognition and corresponding virtual / augmented reality can be conveniently provided by wearing the head-mounted display.

FIG. 10 shows an example of a learning interface screen for an operation image of a brain-computer interface paradigm, and FIGS. 11 and 12 show an example of a screen control according to an operation image.

10, the controller 300 may be provided for a user who wears the head-mounted display 100 to which the additional electrode device 200 is coupled for operation imaginative learning of the brain-computer interface paradigm, Tongue, and other parts of the body. For example, the controller 300 displays a scene in which a user repeatedly performs a blood clenching operation, a scene involving a toe, or a scene in which a tongue is moved, by grabbing the right and left hands of a user, Signal can be derived. In addition, the controller 300 may induce imagination of the body movements by displaying arrows or arrows of an arbitrary shape, or alternate the body parts of the imaginary body to be used.

4 (a) is an example of an image presented in a learning stage for deriving a right-hand motion imaginary brain signal, and FIG. 4 (b) is an example of a right- Which is an example of the image presented in the learning step for deriving the image.

The controller 300 can recognize the movement of the specific region on the virtual reality screen / augmented reality screen presented to the user based on the inter-electrode relationship model and the classification model learned through the brain signals induced by the operation imagery. According to the present invention, a user can perform three-dimensional space movement using an operation image.

For example, if the head mount display 100 provides the user with a left screen and a right screen as shown in FIG. 11 (a), the user can display the screen shown in FIG. 11 (b) . At this time, the controller 300 maps the motion of the foot to the straight line, the right-hand motion to the right, the left-hand motion to the left, and the tongue to the backward motion, It is possible to recognize the region and to control the screen mapped to the recognized region. The controller 300 can stop the screen movement when the user does not imagine the operation.

Referring to FIG. 12, the controller 300 may change the color / shape / size of the direction indicating arrow corresponding to a specific body movement on the interface screen when the specific body motion imagination is detected.

The controller 300 may feedback the duration or intensity of the user intention detected using the operationalimage. For example, the controller 300 may change the arrow / shape / size of the forward direction when the forward motion imagination mapped to the forward is persistent or intensified. This allows the controller 300 to provide feedback to the user as to whether a correct operational image is being generated, thereby providing a convenient and accurate interface.

13 is an illustration of a learning interface screen for steady state visual evoked potentials in the brain-computer interface paradigm.

13, the controller 300 may provide a user wearing the head-mounted display 100 with an additional electrode device 200 for steady-state visual evoked potential learning to a user wearing at least one visual stimulus Present the screen. For example, the visual stimulus may be flickered at a specific frequency while the stimulus is presented, and may include shapes, letters, object shapes, and the like.

Based on the inter-electrode relationship model and the classification model learned through the brain signals induced by the steady-state visual evoked potential, the controller 300 generates a virtual stimulus pattern on the virtual reality screen / . ≪ / RTI > To this end, the virtual environment and the augmented reality image are produced so as to measure brain signals induced by visual stimulation by assigning visual stimulus patterns corresponding to specific objects. In this case, the form of the visual stimulus can be various forms such as a figure, a letter, and the like, as well as a specific object, thereby detecting various user intentions.

13 (a), when the brain-computer interface paradigm is selected as the steady state visual evoked potential, the head mount display 100 provides the user with a left screen and a right screen, It is possible to provide different visual stimulation frequencies. For example, the visual stimulus for the first object may be given F ₁ Hz with the left eye and F ₃ Hz with the right eye. The visual stimulus for the second object may also be given F ₂ Hz with the left eye and F ₄ Hz with the right eye.

The user recognizes the screen as shown in FIG. 13 (b) through the head mount display device. At this time, the user can recognize two frequency components, thereby enhancing the user's intention recognition performance. That is, the user inputs the first object as F ₁ Hz F ₃ Hz, and recognizes the second object as F ₂ Hz and F ₄ Hz. ≪ / RTI >

FIG. 14 is an illustration of a learning interface screen for event related potentials in a brain-computer interface paradigm, and FIG. 15 is an illustration of a character input interface using visual stimulation based on event related potentials.

14, the controller 300 displays a screen including at least one visual stimulus to the user wearing the head-mounted display 100 coupled with the additional electrode device 200 for event-related dislocation learning present. For example, one or more visual stimuli may be presented, and visual stimuli may be presented alternately in an order that the user can not predict. Visual stimuli can include shapes, letters, pictures, faces, sounds, and so on. Event - related potentials can be conveniently used when various stimuli must be presented in a three - dimensional space.

Based on the inter-electrode relationship model and the classification model learned through brain signals induced by the event-related potential, the controller 300 displays the objects selected by the user through visual stimulation on the virtual reality screen / Can be recognized. To this end, the virtual environment and the augmented reality image are produced so as to measure brain signals induced by visual stimulation by assigning visual stimulus patterns corresponding to specific objects.

Referring to FIG. 14A, when the brain-computer interface paradigm is selected as the event-related potential, the head mount display 100 provides visual stimulation using an event-related potential to an object existing in a three-dimensional space . The user recognizes the screen as shown in FIG. 14 (b) through the head mount display.

There are a variety of ways to provide visual stimuli using event related potentials. For example, objects at different distances are alternately blinked, and blinking repetitions are provided in unpredictable or unpredictable order.

The controller 300 learns a brain signal generated when a user looks at an object desired on the interface screen, and recognizes the user's intention based on the learned classification model.

Referring to FIG. 15, the controller 300 may provide a character input function using an input interface that causes an event related potential. When the head mount display 100 provides an input interface to the user as shown in FIG. 15A, the user recognizes the display as shown in FIG. 15B. For example, the input interface may be a QWERTY keyboard in which characters alternate and flash.

Figure 16 is an illustration of a learning interface screen for steady state auditory evoked potentials in a brain-computer interface paradigm.

Referring to Figure 16, the controller 300 may provide a user wearing the head-mounted display 100 with an additional electrode device 200 for steady-state auditory evoked potential learning to a user wearing at least one auditory stimulus Present the screen. For example, one or more auditory stimuli may be presented, and auditory stimuli may be presented alternately in a sequence that the user can not predict. The auditory stimulus can be composed of a single frequency or a combination of multiple frequencies. The auditory stimulation may be provided through a speaker or earphone included in the head mount display 100. [

The controller 300 can recognize the user's intention on the virtual reality screen / augmented reality screen presented to the user based on the inter-electrode relationship model and the classification model learned through the brain signals induced by the steady state auditory evoked potential .

The methods of presenting auditory stimuli can vary, for example, providing auditory stimuli that sound like they occur on the left and right sides, respectively, relative to the user. At this time, it is possible to provide the sound that appears to be generated from the left side without time delay, and the sound that appears to occur on the right side can be provided with a delay time compared to the left side, and vice versa.

The sound heard from both sides is composed of sounds of different frequencies, and the user can detect the left or right intention by analyzing the brain signal generated when the user concentrates on one sound. It may also provide the direction of sound stimulation by providing a different amount of sound as well as time delay.

Additionally, at least one vibratory stimulus wired / wirelessly connected to the head-mounted display 100 may be attached to the user's body, and the controller 300 may present a stimulus via a vibrational stimulus for steady state tactile evoked potential learning .

As described above, the controller 300 recognizes the user's intention corresponding to the brain signal measured and estimated in the virtual reality screen / augmented reality screen presented to the user by using a classification model for learning the user's intention learned based on the brain signal can do. The controller 300 can control (move, object select, keyboard input, etc.) the virtual reality screen / augmented reality screen according to the recognized user's intention.

FIG. 17 shows an example of screen control by the user's eye movement according to an embodiment, and FIG. 18 shows an example of a screen control on a user's head and eye movement according to an embodiment.

Referring to FIGS. 17 and 18, the controller 300 displays user intention (for example, direction change, eye movement direction) according to the movement of the user's gaze on the basis of the eye conduction signal measured from the head mount display 100, Can be recognized. That is, the controller 300 can control the head mount display by recognizing the eye conduction signal as a user's intention. The controller 300 can control the head mount display based on the user's intention detected from the brain signal and the ocular conduction signal.

At this time, similar to the method of recognizing the intention of the user using the classification model for learning the intention of the user based on the brain signal, the controller 300 may classify the intention recognition (eye movement) based on the eye conduction signal The user can learn the model and detect the user's gaze movement based on the learned classification model.

The controller 300 may provide a cursor screen for the user to gaze on the head mount display 100 for user's gaze motion detection according to eye conduction. According to the movement of the cursor, the user moves the eye, and an ocular conduction signal generated at this time is obtained through the ocular conductive electrodes.

The cursor screen forms a horizontal axis (x-axis) and a vertical axis (y-axis) from the center, and the velocity (v) in the horizontal direction and the vertical direction through the change of the cursor position can be calculated as shown in Equation (4).

,

,

,

)과 오른쪽 눈 주변에 부착된 전극들(

,

,

,

)을 이용하여 수직(

,

) 성분과 수평(

,

) 성분을 추출한다. 왼쪽 눈에서 측정된 수직 성분과 수평 성분은 수학식 5와 같이 표현될 수 있다.The controller 300 includes electrodes (e.g.,

,

,

,

) And electrodes attached around the right eye (

,

,

,

),

,

) And horizontal (

,

) Components. The vertical component and the horizontal component measured in the left eye can be expressed by Equation (5).

The controller 300 can generate the dependent variable (i.e., the velocity of the cursor) and the independent variable (i.e., the ocular conduction) of the regression model as shown in Equation (6) to establish a model of the relationship between the velocity of the cursor and the ocular conduction.

The controller 300 generates a relational model by applying various regression analysis techniques similar to the learning method in brain signal processing and detects eye movements from eye conduction signals generated when the user moves his eyes based on the relational model can do. Accordingly, the controller 300 outputs a screen that is changed in viewpoint according to the user's gaze motion, thereby providing a natural screen.

Referring to FIG. 17 (a), the user moves the eyeball in the vertical direction while wearing the head mount display 100. Then, the controller 300 photographs by controlling the lens angle of the camera as shown in FIG. 17 (b). The controller 300 can provide a photographing screen matched with the field of view viewed by the user as shown in (c) of FIG.

In the case of the virtual reality image, the controller 300 analyzes the eye conduction signal generated when the user views the up / down / left / right / remote / close range and scrolls to the corresponding point in the virtual reality image.

Referring to FIG. 18, the user can move the eyes and head together while wearing the head mount display 100. Head movement can be measured with a gyro sensor or the like.

If the user's head and eyes move as shown in (a) of FIG. 18, the controller 300 determines that the current user's gazing screen is the same as (a-1) to (a-3) Provides screen changes. The controller 300 provides screen changes such as (c-1) to (c-3) when the user's head and eyes move as shown in (c) of FIG.

In this way, the controller 300 learns the brain signals induced according to the brain-computer interface paradigm to generate a user's intention classification model, and also generates a relationship model between the head mount display 100 and the additional electrode device 200 Can be generated. Accordingly, the user can acquire brain signals required for controlling the virtual reality / augmented reality image even if only the head mount display 100 is worn.

In addition, the controller 300 can learn the eye conduction signal and detect eye movement (change in the visual field direction). Accordingly, the user can control the viewpoint of the virtual reality / augmented reality image by moving his / her eyes while wearing only the head mount display 100.

At this time, brain signal measurement through the brain conduction electrode of the head mount display 100 and ocular conduction signal measurement through the ocular conduction electrode proceed in parallel and can be applied to the head mount display control in a complex manner.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded. Furthermore, the described method can be implemented as a recording medium including instructions readable by a computer.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (17)

CLAIMS What is claimed is: 1. A method for a controller to recognize a user's intention to wear a head-mounted display to which brain conduction electrodes are attached,
Obtaining learning brain signals measured at each of the head-mounted display and the additional electrode device coupled to the head-mounted display, with the learning screen displaying brain signals being displayed on the head-mounted display,
Learning relationship models of the electrodes of the head-mounted display and the electrodes of the additional electrode device from the learning brain signals,
Acquiring brain signals measured in the head-mounted display from which the additional electrode device is detached, with a specific screen displayed on the head-mounted display,
Estimating brain signals of the further electrode device not measured from brain signals measured at the head-mounted display based on the relational model, and
And detecting user intention corresponding to the measured brain signals and estimated brain signals. The method of claim 1,
Further comprising learning a user's intention classification model from the learning brain signals,
The step of detecting the user intention
And a user intention corresponding to a signal obtained by combining the measured brain signals with the estimated brain signals based on the learned user's intention classification model. The method of claim 1,
Wherein the learning screen is an interface screen produced to induce brain signals corresponding to a specific brain-computer interface paradigm. 4. The method of claim 3,
The brain-computer interface paradigm
Wherein the method is at least one of a Motor Imagery Potential, a Steady State Visual / Auditory / Somatosensory Evoked Potential, and an Event Related Potential. 4. The method of claim 3,
If the learning screen is an interface screen for generating brain signals corresponding to an operation image,
Wherein the learning brain signals are images generated by imagining moving body parts designated on the learning screen. 4. The method of claim 3,
When the learning screen is an interface screen for generating brain signals corresponding to an event related potential, the learning screen separates the left screen and the right screen corresponding to the respective eyes, and provides the left screen and the right screen with the same object To a different frequency,
Wherein the learning brain signals are signals caused by visual stimulation included in the learning screen. 4. The method of claim 3,
When the learning screen is an interface screen for generating brain signals corresponding to a steady state visual evoked potential, the learning screen provides a plurality of selectable objects as different visual stimuli,
Wherein the learning brain signals are signals generated when a specific object among a plurality of objects is gazed at the learning screen. The method of claim 1,
The step of learning the relational model
A brain signal feature vector including a predetermined number of brain signals measured in past cycles on a brain signal measured at a current cycle for each electrode and learning the relational model using the brain signal feature vector, Way. The method of claim 1,
And controlling the specific screen based on the detected user intention. The method of claim 9,
The step of controlling the specific screen
The method includes detecting eye movements based on eye conduction signals obtained from the eye conduction electrodes attached to the head mount display, changing the view of the specific screen according to the detected eye movement, And controlling the specific screen according to a user's intention detected with brain signals. 11. The method of claim 10,
Before the step of controlling the specific screen,
Further comprising the step of learning a relationship model of cursor velocity and learning ocular conduction signals obtained from the ocular conduction electrodes while displaying a cursor moving along a line of sight on a cursor screen,
The step of controlling the specific screen
And a gaze movement corresponding to the acquired ocular conduction signals is detected based on a relationship model between the ocular conduction signals and the cursor velocity. A brain-computer interface device,
A head-mounted display to which brain conduction electrodes of the first channel group are attached,
An additional electrode device attached to the head mounted display and attached to the brain conduction electrodes of the second channel group,
Storing a relationship model of brain signals obtained at the brain conduction electrodes of the first channel group and the brain conduction electrodes of the second channel group and based on the relationship model, And a control unit
The controller
When the brain signals measured in the first channel group are inputted while displaying a specific screen on the head mount display, the brain signals of the brain conduction electrodes of the second channel group are estimated based on the relationship model, Wherein the user's intention is detected using brain signals measured in one channel group and brain signals estimated in the second channel group, and the specific screen is controlled based on the detected user intention. The method of claim 12,
The controller
In the learning mode in which the additional electrode device is coupled to the head mounted display, a learning screen for causing brain signals is displayed on the head mounted display, and the learning brain signals measured in the first channel group and the second channel group Acquiring learning models of the brain models, and learning the relationship models using the learning brain signals, and then storing the learned relationship models. The method of claim 13,
The learning screen is an interface screen generated to generate brain signals corresponding to a specific brain-computer interface paradigm,
The brain-computer interface paradigm
A brain-computer interface device that is at least one of a Motor Imagery Potential, a Steady State Visual / Auditory / Somatosensory Evoked Potential, and an Event Related Potential. The method of claim 13,
The controller
And the learning brain signals are regression analyzed to learn the relational model. The method of claim 12,
The controller
Detecting a user intention corresponding to a feature of a signal obtained by combining brain signals measured in the first channel group and brain signals estimated in the second channel group based on the learned user's intention classification model,
Wherein the user's intention classification model is a model learned to classify into one of a plurality of intents designated in a specific brain-computer interface paradigm based on characteristics of brain signals obtained in the first channel group and the second channel group, - Computer interface device. A brain-computer interface device,
A main body including at least one processor, a memory and a display; at least a portion of an inner surface of the main body being worn to abut the user ' s face, the main body being connected to the main body and supporting the main body in a state of being worn on the user ' A head-mounted display including a head, and
An additional electrode device coupled to or separated from the head mount display and having brain conduction electrodes attached to an inner surface of the user's scalp,
Wherein the head mount display comprises:
A plurality of brain conduction electrodes arranged at an arbitrary interval along the user's head circumference on an inner surface of the fixing portion,
A plurality of eye conduction electrodes arranged at an arbitrary interval in a plurality of directions from a position where the user ' s eyeball contacts the inner surface of the body,
A reference electrode disposed in an area where a part of the user's face is in contact with the main body, and
And a fixing unit (20) disposed on at least one of the main body and the fixing unit,
Wherein the computer-readable medium further comprises a computer-readable medium.

Images