KR102584544B1 - Brain-computer interface method and device using video to stimulate movement image - Google Patents

Abstract

The disclosed technology relates to a brain-computer interface method and device using motor imagery stimulation images, comprising: an analysis device outputting an image for a first device to the central area of the screen; outputting a first image showing that the analysis device controls a first operation of the first device to a first region among a plurality of regions formed around the central region; detecting, by the analysis device, a first brain wave generated in the user's brain at the time the first image is output using a plurality of electrodes connected to the user; and storing, by the analysis device, the first brain wave as a first control signal for performing the first operation.

Description

Brain-computer interface method and device using movement imagery stimulation image {BRAIN-COMPUTER INTERFACE METHOD AND DEVICE USING VIDEO TO STIMULATE MOVEMENT IMAGE}

The disclosed technology relates to a method and device for analyzing brain wave signals for a Brain Computer Interface (BCI) using motor imagery stimulation images.

BCI systems using Event Related Potential (ERP) have been studied in various ways, and most existing ERP-based BCI systems utilize an ERP characteristic called P300. P300-based BCI is mainly used to recognize the user's intention to select one of several options. For example, it is possible to recognize whether the brain wave signal generated from the user's brain is for selecting one of the keyboard keys, one of several functions of the device, or one of several directions when controlling a wheelchair.

On the other hand, when using the P300, an important issue is the distinction between the signal when the subject recognizes the target object and the signal when the subject recognizes the non-target object. To this end, signal characteristics are usually obtained through the process of repeatedly presenting the object several times. You get However, in terms of practical use and commercialization of the technology, it is not appropriate to perform many repetitive tasks to perform the function, so there is a need to improve the number of repetitions essential for obtaining signal characteristics.

The BCI system that applies existing movement imagery interprets the movement intention and uses it to control robots according to the movement intention, or selects one of several options based on differences in brain wave characteristics that appear during the process of imagining the movement of different body parts. It is used in a way. For the purpose of recognizing the intention of choosing one of several options, existing motor imagery had the limitation that it was difficult to increase the number of options due to the limitations of the body parts whose characteristics could be obtained through imagination.

Korean Patent Publication No. 10-0943396

The disclosed technology provides a method and device for analyzing brain wave signals for BCI (Brain Computer Interface) using motor imagery stimulation images.

The first aspect of the technology disclosed to achieve the above technical problem is the step of the analysis device outputting an image for the first device to the central area of the screen, the analysis device outputting an image for the first device to the central area of the screen, the analysis device Outputting a first image showing controlling the first operation of the first device in area 1, the analysis device uses a plurality of electrodes connected to the user to detect the user's brain at the time the first image is output. A brain-computer interface method using a motor imagery stimulation image comprising detecting a first brain wave generated in and storing the first brain wave as a first control signal for performing the first operation by the analysis device. is to provide.

The second aspect of the technology disclosed to achieve the above technical problem is to output an image of the first device in the central area of the screen and display the image of the first device in a first area among a plurality of areas formed around the central area. 1 An output device that outputs a first image showing controlling an operation, an electrode that detects a first brain wave generated in the user's brain at the time the first image is output to the output device, and the first brain wave The object is to provide a brain-computer interface device using a motor imagery stimulation image that includes a processor that stores the first control signal for performing a first movement.

Embodiments of the disclosed technology may have effects including the following advantages. However, since this does not mean that the embodiments of the disclosed technology must include all of them, the scope of rights of the disclosed technology should not be understood as being limited thereby.

According to one embodiment of the disclosed technology, a brain-computer interface method and device using motor imagery stimulation images applies BCI to various environments by distinguishing in detail EEG signals that can input multiple control operations for each of multiple devices. There is an effect that can be done.

In addition, it has the effect of providing the user with images that stimulate motor imagery and thereby inducing the user to generate accurate brain wave signals.

In addition, it has the effect of increasing user convenience by reducing the number of repetitive tasks for brain wave learning.

1 is a diagram illustrating a brain-computer interface system using motor imagery stimulation images according to an embodiment of the disclosed technology.
Figure 2 is a flowchart of a brain-computer interface method using motor imagery stimulation images according to an embodiment of the disclosed technology.
Figure 3 is a block diagram of a brain-computer interface device using motor imagery stimulation images according to an embodiment of the disclosed technology.
Figure 4 is a diagram showing output of a motor imagery stimulation image according to an embodiment of the disclosed technology.
FIG. 5 is a diagram illustrating repetitive output of images showing controlling a device according to an embodiment of the disclosed technology.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

As used herein, singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise. And terms such as "include" mean the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but one or more other features, numbers, steps, operations, components, parts, etc. It should be understood that it does not exclude the existence or addition of combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions.

In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by . Therefore, the presence or absence of each component described throughout this specification should be interpreted functionally.

1 is a diagram illustrating a brain-computer interface system using motor imagery stimulation images according to an embodiment of the disclosed technology. Referring to Figure 1, the system consists of an analysis device and a plurality of devices. The analysis device and a plurality of devices are connected through a network, and each device performs a specific operation according to a control signal transmitted from the analysis device. The analysis device may be a server in the network. And, as shown in FIG. 1, device A may be a TV connected to a network, device B may be a radio, and device C may be a speaker. Of course, the types of devices shown in FIG. 1 are only for illustrative purposes, and other types of devices may also be included. Each device may be an IOT device with a unique address on the Internet. Hereinafter, the description will be made assuming that the first device among a plurality of devices is controlled. The first device may be one of devices A, B, and C.

Meanwhile, unlike general device control methods, the analysis device uses brain waves detected from the user to generate control signals to control the device. In other words, the analysis device corresponds to a device that supports BCI and may be equipped with electrodes to detect the user's brain waves. The electrode is attached to the user's head and can be provided in the form of a non-invasive patch that the user does not feel uncomfortable with.

Meanwhile, the analysis device can output images that stimulate the user's motor imagery and detect brain waves according to specific actions for a specific device. Here, an image that stimulates motor imagery refers to an image that induces the user to imagine a specific movement. For example, it may be an image showing pressing the power button of the first device or an image showing adjusting the volume of the first device. Users using BCI are often unable to directly control the device. For example, a person may not be able to accurately control the device due to a physical disability or mental disability. Therefore, device control using brain waves is performed by having these users imagine performing an action even though they cannot actually perform the action.

Meanwhile, the image provided in this way allows the user to generate brain waves that are different from usual. For example, brain waves may be generated in a situation where a specific operation of the first device is input or controlled. The analysis device can detect the user's brain waves that occur in such situations and store them as a control signal for controlling the device. The brain wave detected when the first image for the first device is output is defined as the first brain wave. The analysis device can store the first brain wave as a first control signal for the first device, and when the first brain wave is detected from the user, it can transmit the first control signal to the first device through the network. The analysis device can repeatedly provide the first image to the user and detect brain waves each time. And the first control signal can be generated by learning the detected brain waves. For example, the first brain wave can be stored as a first control signal, and then when the brain wave of the same pattern as the first brain wave is detected again, the first control signal can be transmitted to the first device.

Meanwhile, in controlling a device, multiple control operations may be required rather than simply performing one control operation on one device. For example, multiple controls are required in one device, such as turning the TV on and off, adjusting the volume, and changing channels. Therefore, the analysis device can provide a plurality of images representing a specific control operation for a specific device. For example, a first image showing controlling a first operation for a first device may be output, and a second image showing controlling a second operation for a first device may be output. The number of control images for one device may vary for each device. The analysis device may divide the screen into a plurality of areas and output an image of the first device in the central area. Additionally, a plurality of images showing control of the first device can be output to surrounding areas.

Meanwhile, the analysis device can repeat the operation of detecting the user's brain waves by providing images that stimulate motor imagery. For example, the first image of the first device can be repeatedly output at a certain period and brain waves can be detected each time it is output. By repeating the brain wave detection operation in this way, the user can be trained to generate accurate brain waves, and it is also possible to compare whether the user's brain waves match the brain waves of the previously stored control signal for the first device. In this way, more accurate device control is possible by repeating the brain wave detection operation according to the image provided a certain number of times, and since images that stimulate motor imagery are provided, there is an advantage in that the operation is not repeated too many times. Of course, there may be some variation depending on the user, but unlike training brain waves using only images, it is possible to obtain similar results even with a significantly lower number of repetitions by using a method that stimulates motor imagery.

Meanwhile, the analysis device may output the first image to each of the plurality of areas sequentially or in a random order according to a preset cycle. Here, sequential means outputting the first images sequentially in continuous areas in a clockwise or counterclockwise direction based on the area at a specific location. The number of a plurality of areas formed based on the central area of the screen of the analysis device can be determined according to the number of control operations of the first device, and the analysis device outputs the first image sequentially or randomly according to a preset cycle to provide the user with Brain waves can be detected. Of course, it is also possible to control the first image to be output to only one of the plurality of areas according to a preset cycle.

Meanwhile, when the analysis device receives the first brain wave signal according to the above-described process, it displays a second image showing controlling the second operation of the first device in a second area different from the first area among the plurality of areas of the screen. can be output. That is, the first image and the second image may be output in different areas. This is to fix the area where the image is output so that there is no confusion when the user perceives each image. For example, if the first image is output in the 1 o'clock direction, the second image may be output only in the 3 o'clock direction. In this way, images showing device control can be output one by one in each of a plurality of surrounding areas, and are outputted a certain number of times in a designated area.

Meanwhile, the analysis device receives the second brain wave generated by the user at the time the second image is output and stores it as a second control signal for performing the second operation. Just as previously receiving the first brain wave and storing it as a first control signal, it can be stored as a value for controlling the first device. According to this process, the analysis device can generate a control signal that allows the user to perform a specific operation on the first device using brain waves. And the user can control the specific operation of the device he or she wants to control using brain waves.

Figure 2 is a flowchart of a brain-computer interface method using motor imagery stimulation images according to an embodiment of the disclosed technology. Referring to FIG. 2, the brain-computer interface method 200 using a motor imagery stimulation image includes steps 210 to 240.

In step 210, the analysis device outputs an image of the first device in the central area of the screen. The image for the first device may be a two-dimensional image or a three-dimensional image representing the external appearance of the first device. The analysis device can output a clear image of the first device in the central area of the screen so that the user can easily identify it with the naked eye.

In step 220, the analysis device outputs a first image showing controlling the first operation of the first device to a first area among a plurality of areas formed around the central area of the screen. The first image may be a video showing the user performing a specific input for the first operation of the first device. For example, it may be a video showing the user's finger pressing the power button of the first device. The analysis device may divide the screen into a plurality of areas, output an image for the first device in the central area, and output the first image in one of the remaining peripheral areas. The default value of the area where the first image is output may be determined according to the settings of the analysis device. Additionally, the analysis device may always output the first image only to a designated area, sequentially output it to a plurality of surrounding areas, or output it to an area at a random location.

In step 230, the analysis device detects the first brain wave generated in the user's brain at the time the first image is output using a plurality of electrodes connected to the user. The first brain wave detected at this time may have a different pattern from the brain wave normally generated by the user due to the first image that stimulates the motor image provided by the analysis device. The analysis device can store the results of detecting the user's brain waves and compare them with the first brain wave. If the first brain wave is the same as the normal brain wave, it can be considered that proper measurement was not made. And if the two brain waves show different patterns, the analysis device can store the first brain wave.

In step 240, the analysis device stores the first brain wave as a first control signal for performing the first operation. After storing the first brain wave, the analysis device may transmit a first control signal to the first device when it detects a brain wave showing the same pattern as the first brain wave from the user. By repeating steps 210 to 240, brain waves for a plurality of control inputs to the first device can be stored.

Figure 3 is a block diagram of a brain-computer interface device using motor imagery stimulation images according to an embodiment of the disclosed technology. Referring to FIG. 3, a brain-computer interface device 300 using a motor imagery stimulation image includes an output device 310, an electrode 320, and a processor 330.

A first image showing that the output device 310 outputs an image of a first device in the central area of the screen and controls the first operation of the first device in a first area among a plurality of areas formed around the central area. Outputs . The output device 310 may be implemented as a monitor or display device. The output device 310 divides the screen into a plurality of areas under the control of the processor 330, outputs the image for the first device in the central area, and controls the first operation in the first area, which is one of the remaining peripheral areas. The first image showing what is happening can be output.

The electrode 320 detects the first brain wave generated in the user's brain at the time the first image is output to the output device. The electrode 320 may use a patch-type electrode attached to the head to accurately detect brain waves generated in the user's brain. These electrodes may be non-invasive electrodes that prevent user resistance, and may be connected to a device such as a head-mounted display so that when the user wears them on their head, the electrodes come into contact with the head.

The processor 330 stores the first brain wave detected by the electrode as a first control signal for performing the first operation. And later, when the first brain wave is detected again through the electrode, the first control signal is transmitted to the first device. The processor 330 may be implemented as a CPU or AP of the brain-computer interface device 300.

Meanwhile, the brain-computer interface device 300 described above may be implemented as a program (or application) including an executable algorithm that can be executed on a device such as a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), and EPROM (Erasable PROM, EPROM). Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Enhanced SDRAM (Enhanced RAM). It refers to various types of RAM, such as SDRAM (ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

Figure 4 is a diagram showing output of a motor imagery stimulation image according to an embodiment of the disclosed technology. Referring to FIG. 4, a plurality of areas are formed on the screen of the analysis device. Among these, images for the first device may be output in the central area 401, and images representing respective control operations for the first device may be output in the remaining areas 402, 403, 404, and 405. Here, the remaining areas may be formed around the central area 401, and the number of areas may be determined according to the number of control inputs to the first device. For example, if there are three control inputs for the first device: power on/off, volume control, and channel change, the number of peripheral areas may be output as three.

FIG. 5 is a diagram illustrating repetitive output of images showing controlling a device according to an embodiment of the disclosed technology. Referring to FIG. 5, the analysis device may output the first image to each of a plurality of areas sequentially or in a random order according to a preset cycle, or may output the first image to only one area among the plurality of areas. In the case of Figure 5, it is an example of outputting the first image in the order of bottom left, top left, bottom right, and top right, and an empty image that does not include the first image to initialize brain wave generation while outputting each image. can be output.

According to the disclosed technology, by combining P300 and movement imagery signals (Movement related cortical potential (MRCP)), signals that appear when a subject moves his or her body or imagines a similar situation can be used together. The P300 signal and the movement imagery signal are It is a stimulus that is triggered in different cognitive situations in the brain, and in order to cause it simultaneously, it is necessary to produce a separate stimulus. Therefore, the first image of the first device is provided through an analysis device to stimulate motor imagery. Accordingly, BCI Because not only the time characteristics but also the frequency characteristics can be used together, the range of signal characteristics that can be utilized is expanded. In other words, multiple simultaneous signals for a single response enhance learning more efficiently and significantly reduce the number of repetitions. Therefore, the practicality and commercialization of the system can be improved by reducing the number of repetitions of object presentation required in existing BCI.

The brain-computer interface method and device using motor imagery stimulation images according to an embodiment of the disclosed technology have been described with reference to the embodiments shown in the drawings to aid understanding, but this is merely illustrative and is based on common knowledge in the field. Those who have will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the disclosed technology should be determined by the appended patent claims.

Claims (15)

A step of the analysis device outputting an image of the first device to the central area of the screen;
outputting a first image showing that the analysis device controls a first operation of the first device to a first region among a plurality of regions formed around the central region;
detecting, by the analysis device, a first brain wave generated in the user's brain at the time the first image is output using a plurality of electrodes connected to the user; and
Including, the analysis device storing the first brain wave as a first control signal for performing the first operation,
The analysis device repeatedly outputs the first image to each of the plurality of areas sequentially or in a random order according to a preset cycle, and detects the user's brain waves each time the first image is output. 1 Compare whether it matches the brain wave,
The first brain wave includes a P300 signal and a movement related cortical potential (MRCP) signal induced from the first image,
The first image is an image of controlling the first device with a finger,
A brain-computer interface method using motor imagery stimulation images. According to claim 1,
When the analysis device receives the first brain wave, the analysis device outputs a second image showing controlling a second operation for the first device to a second region of the plurality of regions that is different from the first region. A brain-computer interface method using stimulus images. According to claim 2,
The analysis device receives the second brain wave generated by the user at the time the second image is output and stores it as a second control signal for performing the second operation,
The second brain wave includes a P300 signal and a movement related cortical potential (MRCP) signal induced from the second image.
A brain-computer interface method using motor imagery stimulation images. According to claim 1,
The first device is one of a plurality of IOT devices located around the user,
The analysis device is a brain-computer interface method that uses a motor imagery stimulation image connected to the plurality of IOT devices through a network. According to claim 1,
The analysis device is connected to the first device through a network, and when the first brain wave is detected from the user, a brain-computer interface using a motor imagery stimulation image transmits a first control signal to the first device through the network. method. delete delete delete Outputting an image for a first device in the central area of the screen and outputting a first image showing controlling the first operation of the first device in a first area among a plurality of areas formed around the central area. Device;
an electrode that detects a first brain wave generated in the user's brain at the time the first image is output to the output device; and
A processor that stores the first brain wave as a first control signal for performing the first operation,
The output device repeatedly outputs the first image to each of the plurality of areas sequentially or in a random order according to a preset cycle,
The processor detects the user's brain waves each time the first image is output and compares whether it matches the first brain wave,
The first brain wave includes a P300 signal and a movement related cortical potential (MRCP) signal induced from the first image,
The first image is an image of controlling the first device with a finger,
Brain-computer interface device using motor imagery stimulation images According to clause 9,
When the processor receives the first brain wave, the output device outputs a second image showing controlling a second operation of the first device to a second region of the plurality of regions that is different from the first region. A brain-computer interface device that uses motor imagery stimulation images to control . According to claim 10,
The processor receives the second brain wave generated by the user at the time the second image is output and stores it as a second control signal for performing the second operation,
The second brain wave includes a P300 signal and a movement related cortical potential (MRCP) signal induced from the second image.
A brain-computer interface device using motor imagery stimulation images. According to clause 9,
The brain-computer interface device further includes a communication device for communicating with the first device,
The communication device is connected to the first device through a network, and when a first brain wave is detected from the user, a motor imagery stimulation image for transmitting a first control signal to the first device through the network under the control of the processor. A brain-computer interface device used. delete delete delete