KR102484444B1 - Apparatus for generating AR contents for recognition of brain waves for improving visibility in AR environment - Google Patents

Abstract

An apparatus for generating a BCI brain wave stimulus for improving visibility in an AR environment according to an embodiment includes: a captured image extracting unit that captures a real world image and extracts it as an image; a pre-processing unit that designates specific areas in the image extracted by the captured image extraction unit to display virtual objects, which are stimuli, and extracts RGB values for the specific areas; a stimulus setting unit for generating a stimulus having the color of the complementary color by calculating an RGB value as a complementary color of the RGB values for each region extracted by the pre-processing unit; and a stimulus output unit outputting the stimulus of the color set by the stimulus setting unit to a specific region of the real world region.

Description

Apparatus for generating AR contents for recognition of brain waves for improving visibility in AR environment}

The present invention relates to an apparatus capable of increasing an examiner's brain wave recognition rate by changing the color of a stimulus generated according to a change in the surrounding environment to the most visible color when brain wave stimulation is acquired using an AR device.

BCI (Brain-Computer Interface) technology refers to an interface that can use a computer using brain waves. Decision-making, which is the result of information processing in the brain, does not go through language or body movements, and specific brain waves thought and decided by the examiner are transmitted to the sensor of the system, and the computer executes the corresponding command.

Since the method of manipulating a computer with brain waves rather than a keyboard or mouse can be useful when the body cannot be used due to a disability, research on brain-computer interfaces is constantly being conducted.

Brain waves can be studied in three ways.

First, it is to identify which part of the brain wave is expressed when performing a certain action, and secondly, it is to analyze when the waveform of the brain wave emitted according to the action is different. Thirdly, research is being conducted through the difference in the time it takes for certain stimuli to be emitted as brain waves.

In the present invention, the above third method, a technique of analyzing the examiner's cognition by using the difference in time from presenting a stimulus to being emitted as an brain wave, is intended to be dealt with.

In order to present the time, the present invention utilizes an AR device that can present the time by being mounted on the examiner's head without using a TV or monitor, which is a general display device. AR provides a sense of immersion that allows real-time interaction between the inspector and the virtual world without the inspector being aware that the real and virtual environments are separated by well harmonizing the real world and the virtual world. In order to use BCI in real life using these characteristics of AR technology, it was necessary to express virtual visual stimuli in the real world, and AR, the most appropriate technology, was applied.

In particular, in order to enable the application of BCI in real life of the present invention, among the various characteristics of visual stimuli generated by the AR device, the color is changed to the optimal color according to the background environment to create a stimulus that can increase the recognition rate and , It is to provide a solution that can be used for real-life device control using BCI, medical care, and leisure through the generation of these stimulants.

An apparatus for generating a BCI brain wave stimulus for improving visibility in an AR environment according to an embodiment includes: a captured image extracting unit that captures a real world image and extracts it as an image; a pre-processing unit that designates specific areas in the image extracted by the captured image extraction unit to display virtual objects, which are stimuli, and extracts RGB values for the specific areas; a stimulus setting unit for generating a stimulus having the color of the complementary color by calculating an RGB value as a complementary color of the RGB values for each region extracted by the pre-processing unit; and a stimulus output unit outputting the stimulus of the color set by the stimulus setting unit to a specific region of the real world region.

By the device of the embodiment as proposed, if a stimulus with a high recognition rate using BCI AR content is applied, it can be used in real life, medical care, leisure, etc., and can be widely used for the elderly and disabled.

1 is a diagram showing the configuration of an AR content stimulus production apparatus capable of generating visual stimulus in the present invention.
2 is a diagram showing that a real world image of an AR device is extracted by a captured image extraction unit according to the present embodiment.
3 to 6 are diagrams for explaining a process of extracting RGB values for four regions of a real world image by a pre-processing unit according to the present embodiment.
7 is a diagram for explaining a process of determining a color of a virtual object, which is a stimulus, as a color complementary to a color of a real world image by the stimulus setting unit of the present embodiment.
8 and 9 are diagrams showing that a stimulus having a color complementary to a color of a real world image is output by the stimulus output unit of the present embodiment.

Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings. However, the scope of the inventive idea of this embodiment can be determined from the matters disclosed in this embodiment, and the inventive idea of this embodiment is the implementation of addition, deletion, change, etc. of components with respect to the proposed embodiment. will include transformation.

In order to detect visual stimulation brain wave signals, a configuration capable of producing AR contents has been proposed by the present applicant. Devices for detecting the P300 and SSVEP (Steady State Visually Evoked Potential) EEG signals are connected to the examiner, and the authoring system plays a role in generating stimuli, which are contents delivered to the AR device worn by the examiner.

In the following embodiments, the stimuli will be mainly described in the case of still images (images), but they may also be moving images depending on the embodiments.

Therefore, the examiner checks the stimuli generated by the system of the present invention through the AR device, the examiner watches the stimuli shown through the AR device worn, and a separate brain wave signal analysis device is used to analyze the stimuli generated in the examiner's brain. Detects P300 and SSVEP EEG signals.

In particular, due to the nature of the AR device, the screen of the real world and the image of the virtual object are shown to the examiner together. It is characterized in that the color of the virtual object, which is a stimulus, is also varied in response to

Meanwhile, four or more stimuli are generated to extract P300 and SSVEP signals that can be recognized using visual stimuli in BCI. Using the position, size, and color of the generated stimulus, set the stimulus most easily perceived by the wearer.

The set stimulus is expressed in the AR device through the stimulus output unit, and the visualization method is classified and output according to the P300 and SSVEP stimulus generation methods. Hereinafter, a configuration in which the color of the stimulus changes in comparison with the background color of the real world will be described with emphasis.

1 is a diagram showing the configuration of an AR content stimulus production apparatus capable of generating visual stimulus in the present invention.

According to the present embodiment, a real world image seen through an AR device worn by an inspector is photographed, and a color of a stimulus for brain wave signal extraction is determined using the captured real world image information.

As shown in FIG. 1, the BCI brain wave stimulus generating device for improving visibility in the AR environment of the present embodiment includes a captured image extraction unit 100 that captures a real world image viewed by an inspector through the AR device, and the captured image extraction A pre-processing unit 200 that divides the real-world image captured by the unit 100 into four areas and performs pre-processing on each of the four areas, and the real-world image extracted by the pre-processing unit 200. A stimulus setting unit 300 that determines a complementary color for the RGB values of each region and generates a stimulus including the complementary color, and outputs the stimulus including the complementary color into each region by the stimulus setting unit 300 and a stimulus output unit 400 to be shown to the examiner.

That is, according to this embodiment, four stimuli are generated to extract EEG signals that can be recognized using visual stimuli in the BCI. The color of the stimulus is set to the color that the wearer is most recognizable by considering the background the inspector is viewing through the AR display.

The set stimulus is expressed in the AR device through the stimulus output unit, and the visualization method is classified and output according to the brain wave stimulus generation method.

Hereinafter, each component constituting the AR content stimulus production apparatus capable of generating the visual stimulus will be described in detail.

2 is a diagram showing that a real world image of an AR device is extracted by a captured image extraction unit according to the present embodiment.

The captured image extraction unit of this embodiment may be formed of a built-in camera configured in the AR device worn by the examiner, and captures an image currently viewed by the examiner through the AR display and extracts it as an image.

The extracted image has a resolution of 563 pixel × 368 pixel, and the time of image extraction is generally based on before EEG stimulation begins.

3 to 6 are diagrams for explaining a process of extracting RGB values for four regions of a real world image by a pre-processing unit according to the present embodiment.

According to the present embodiment, virtual objects, which are stimuli, are displayed in four regions of a real world image through an AR device, and a process of extracting RGB values for regions a to d is described with reference to FIGS. 3 to 6 see.

In order to apply the RGB Histogram to a specific region of the extracted image, four regions where the stimulus will be located are specified as coordinates, and the lower left corner of the image is used as the origin.

Each area defines the name of the area as ⓐ, ⓑ, ⓒ, ⓓ from the top left, and the coordinates of each area are as follows.

ⓐ{(0, 4.8), (0, 9.6), (5, 9.6), (5, 4.8)}

ⓑ{(10, 4.8), (10, 9.6), (15, 9.6), (15, 4.8)}

ⓒ{(0, 0), (0, 4.8), (5, 4.8), (5, 0)}

ⓓ{(10, 0), (10, 4.8), (15, 4.8), (15, 0)}

Next, color histograms for each area are individually extracted. The color histogram provides all three components of R, G, and B for one area, and in each histogram, the X-axis represents 256 elements from 0 to 255, and the Y-axis specifies the total pixel amount of the corresponding value.

As shown in FIG. 3, it is confirmed which of the components of each of R, G, and B has the most through the color histogram of the area ⓐ. By looking at the Y-axis value, the element with the most pixels is identified to extract the RGB value representing the ⓐ area.

As shown in FIG. 4 , RGB values representing the region ⓑ are extracted through the image extracted in FIG. 2 .

Look at the color histogram of the ⓑ area and check which of each component of R, G, and B has the most. By looking at the Y-axis value, the element with the most pixels is identified to extract the RGB value representing the ⓑ area.

As shown in FIG. 5 , RGB values representing the ⓒ area are extracted through the image extracted in FIG. 2 .

ⓒ Look at the color histogram of the area and check which of each component of R, G, and B has the most. Check the element with the most pixels by looking at the Y-axis value to extract the RGB value representing the ⓒ area.

As shown in FIG. 6, RGB values representing the region ⓓ are extracted through the image extracted in FIG. 2.

Look at the color histogram of the ⓓ area and check which of each component of R, G, and B has the most. By looking at the Y-axis value, the element with the most pixels is identified to extract the RGB value representing the ⓓ area.

7 is a diagram for explaining a process of determining a color of a virtual object, which is a stimulus, as a color complementary to a color of a real world image by the stimulus setting unit of the present embodiment.

That is, the stimulus setting unit 300 of the embodiment obtains the complementary color of the RGB values representing each area and sets the contrast RGB values.

Due to the nature of AR equipment, stimuli are placed in the real environment that the inspector is viewing through the display, so the surrounding environment has a great influence on the visibility of the stimuli.

As a way to solve this problem, based on the RGB values for each region extracted in FIGS. 3 to 6, a complementary color contrasting thereto is set as a contrast RGB value to create a stimulus with the highest visibility in the environment the inspector is currently viewing.

In order to obtain the contrast RGB value, as shown in FIG. 7 , the RGB value of the complementary color may be obtained by subtracting the existing RGB value from 255. In this way, contrast RGB values are obtained for each of the four areas ⓐ, ⓑ, ⓒ, and ⓓ, and the stimulus is set to the corresponding color.

8 and 9 are diagrams showing that stimuli composed of colors complementary to the color of the real world image are output by the stimulus output unit of the present embodiment. In particular, FIG. 9 shows different colors when the color of the real world image varies. It is a diagram showing that a stimulus having is output.

The color of each stimulus is set to the contrast RGB value set in FIG. 7, and it is a screen that outputs a visual stimulation paradigm to induce EEG stimulation of the examiner on the actual AR screen.

Also, as in the example shown in FIG. 9 , the colors of stimuli in various background environments may be simultaneously modified and applied, and output may be made in real time.

Claims (5)

a captured image extraction unit that captures a real world image and extracts it as an image;
a pre-processing unit that designates specific areas in the image extracted by the captured image extraction unit to display virtual objects, which are stimuli, and extracts RGB values for the specific areas;
a stimulus setting unit for generating a stimulus having the color of the complementary color by calculating an RGB value as a complementary color of the RGB values for each region extracted by the pre-processing unit; and
A stimulus output unit outputting the stimulus of the color set by the stimulus setting unit to a specific region of the real world image;
The pre-processing unit sets four specific areas in the image of the real world image extracted by the captured image extraction unit, extracts RGB values in each of the four specific areas, and RGB values through color histograms for each of the four specific areas. extract the value,
The image extracted by the captured image extraction unit has a resolution of 563 pixels × 368 pixels. According to claim 1,
A BCI brain wave stimulus generating device for improving visibility in an AR environment, characterized in that the virtual object, which is the stimulus set in the complementary color, is displayed in real time together with the real world image by the stimulus output unit.
delete delete delete

Images