KR102548507B1 - Apparatus and method for providing user interface of game content using brain wave data - Google Patents

Abstract

Embodiments provide a method for providing a user interface based on brain wave data by analyzing the brain waves of a user. The method for providing a user interface according to an embodiment of the present invention comprises the steps of: outputting, by an output part of a game device, first multimedia content including at least one user interface; generating, by a wearable module connected to the game device, brain wave data by detecting the brain waves of a user; generating, by a control part, at least one characteristic wave data by applying a band-pass filter to the brain wave data; calculating, by the control part, a concentration score based on the characteristic wave data and generating processed data; and selecting, by the control part, second multimedia content from a memory based on the processed data and outputting the second multimedia content to the output part, wherein the characteristic wave data are generated based on the intensity value of the at least one characteristic wave and the change rate of the intensity value, and the control part may calculate the concentration score based on the intensity value of the characteristic wave, and generate the processed data based on the concentration score. Therefore, multimedia content can be provided to a user based on the brain waves of the user analyzed in real time.

Description

Apparatus and method for providing user interface of game contents using brain wave data

Embodiments of the present invention relate to an apparatus and method for providing a user interface for game content using brain wave data, and more specifically, based on a BCI that analyzes a user's brain wave in real time and actively reflects biosignals based on the analysis result. It relates to an apparatus and method for providing a user interface including content.

Brain-Computer Interface (BCI) means to analyze information such as intention and mental and emotional state of the user by using brain waves. BCI has enormous research value because it can be used to control a computer by using BCI or to be used as a variety of device interfaces. In the past, very expensive equipment was required to measure the user's brain waves, so it was mainly used only for specialized research facilities or medical purposes. However, recently, various miniaturized BCI equipment for healthcare and entertainment purposes have appeared. The scope of application of BCI has rapidly expanded.

However, in general, BCI equipment for health care and entertainment purposes is expensive, but is designed in the form of a passive-BCI that extracts quantitative information such as concentration and emotional state from brain waves and reflects it to applications, thereby increasing market competitiveness. difficult to equip In addition, existing BCI-based contents are monotonous and simple in composition, and user intervention in game and application environments is quite limited.

Embodiments provide an apparatus and method for providing a user interface including BCI-based content that analyzes brain waves of a user playing game content in real time and actively reflects biosignals based on the analysis result.

Technical tasks to be achieved in the embodiments are not limited to those mentioned above, and other technical tasks not mentioned will be considered by those skilled in the art from various embodiments to be described below. can

A user interface providing method according to an embodiment of the present invention includes outputting first multimedia content including at least one user interface by an output unit of a game device; generating brain wave data by sensing brain waves of a user by a wearable module connected to the game device; generating, by a controller, at least one characteristic wave data by applying a band-pass filter to the brain wave data; calculating, by the control unit, a concentration score based on the characteristic wave data and generating processing data; and selecting, by the control unit, second multimedia content from a memory based on the processed data and outputting the second multimedia content to the output unit, wherein the characteristic wave data includes an intensity value and an intensity value of at least one characteristic wave. The control unit may calculate the concentration score based on the intensity value of the characteristic wave, and generate the processing data based on the concentration score.

The first multimedia content includes a first interface displaying a gauge visually representing the concentration score, and a second interface displaying a gauge visually representing the action score, and the action score is determined by the user. 1 means a score required to control one's own character appearing in multimedia content, and the control unit may calculate the action score based on the characteristic wave data.

The characteristic wave data includes first characteristic wave data corresponding to a beta wave and second characteristic wave data corresponding to a theta wave, and the controller determines the first characteristic wave data and the second characteristic wave data based on the characteristic wave data. The concentration score may be calculated.

[mathematical expression]

The control unit calculates the concentration score using the [Equation Equation], and in the [Equation Equation], CP may mean a concentration score, Pbeta may mean a beta wave intensity value, and Ptheta may mean a theta wave intensity value. .

Analyzing the characteristic wave data by a server processor connected to the game device, wherein the server processor includes: a multilayer neural network including an input layer, one or more hidden layers, and an output layer; And a learning engine for performing supervised learning on the multilayer neural network using training data having the characteristic wave data as input values and concentration scores as output values, and calculating optimized hyperparameters based on the training data. . It receives an input and outputs an output vector using an activation function, the multilayer neural network includes a loss function layer connected to the output layer, the output vector is input to the loss function layer, and the loss function layer corresponds to the output vector and A loss value may be output using a loss function that compares correct answer vectors for each feedback data, and parameters of the multilayer neural network may be learned in a direction in which the loss value decreases.

A method for providing a user interface according to another embodiment of the present invention includes analyzing first characteristic wave data by a controller; When the characteristic wave corresponding to the first characteristic wave data satisfies a preset first brain wave condition, an output unit outputs content reflecting a first game effect corresponding to the first brain wave condition, and the output unit By, outputting a user input interface; When a user input signal is received through the user input interface, the control unit selects one of at least one multimedia content stored in a memory based on the user input signal, outputting; analyzing second characteristic wave data generated based on the brain waves of the user at a second time point subsequent to the first time point; and when the characteristic wave corresponding to the second characteristic wave data satisfies a preset second brain wave condition, outputting, by the output unit, content reflecting a second game effect corresponding to the second brain wave condition; can include

The method for providing a user interface according to embodiments of the present invention may analyze a user's brain wave in real time and provide multimedia content to the user based on the user's brain wave. In addition, the user's interest can be improved by measuring the user's brain waves, which change according to the provision of new content, in real time.

Effects obtainable from the embodiments are not limited to the effects mentioned above, and other effects not mentioned are clearly derived and understood by those skilled in the art based on the detailed description below. It can be.

BRIEF DESCRIPTION OF THE DRAWINGS Included as part of the detailed description to aid understanding of the embodiments, the accompanying drawings provide various embodiments and, together with the detailed description, describe technical features of the various embodiments.
1 is a diagram for explaining the configuration of a game device according to an embodiment of the present invention.
2 is a flowchart illustrating a method of providing a user interface according to an embodiment of the present invention.
3 is a diagram illustrating a first game screen displayed on an output unit.
4 is a diagram schematically showing the waveforms of various brain waves.
5 is a diagram illustrating a second game screen displayed on an output unit.
6 is a flowchart illustrating a user interface providing method according to another embodiment of the present invention.
7 is a block diagram for explaining a control unit of a game device.
8 is a view for explaining a wearing module according to an embodiment of the present invention.
9 is a diagram illustrating a game device and a server communicating with the game device according to an embodiment of the present invention.
10 is a block diagram showing a server according to an embodiment of the present invention.
11 is a diagram for explaining a multilayer neural network structure of a server processor.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

In the description of the drawings, procedures or steps that may obscure the gist of various embodiments are not described, and procedures or steps that can be understood by those skilled in the art are not described. did

Throughout the specification, when a part is said to "comprising" or "including" a certain element, it means that it may further include other elements, not excluding other elements, unless otherwise stated. do. In addition, terms such as “… unit”, “… unit”, and “module” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. can be implemented as Also, “a or an”, “one”, “the” and like terms are used herein in the context of describing various embodiments (particularly in the context of the claims below). Unless otherwise indicated or clearly contradicted by context, both the singular and the plural can be used.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of various embodiments, and is not intended to represent a single embodiment.

In addition, specific terms used in various embodiments are provided to help understanding of various embodiments, and the use of these specific terms may be changed into other forms without departing from the technical spirit of various embodiments. .

1 is a diagram for explaining the configuration of a game device 100 according to an embodiment of the present invention.

Referring to FIG. 1 , a game device 100 according to an embodiment of the present invention may include a control unit 110, an output unit 120, a holding unit 130, and a support unit 140.

The controller 110 may control the overall operation of the game device 100 . The controller 110 may determine at least one of activity of the user's brain, degree of concentration, and emotional state of the user, based on the EEG data received from the wearing module 200 . The controller 110 may generate biometric information based on at least one of activity of the user's brain, degree of concentration, and emotional state of the user. Here, the user's emotional state may be classified into, for example, a positive emotional state and a negative emotional state. In another example, the user's emotional state may be classified into a plurality of states. The plurality of states may include at least one of a first state of feeling pleasure, a second state of feeling annoyed, a third state of feeling bored, and a fourth state of feeling anger.

The controller 110 may analyze biometric information, select multimedia content based on the biometric information, and output the selected multimedia content to the output unit 120 .

The controller 110 may compare biometric information generated at a plurality of viewpoints, generate feedback data based on the comparison result, and store the generated feedback data in the memory 112 . Each of the plurality of points of view may refer to a point in time when different multimedia contents are being output (played) from the output unit 120 . For example, it may be defined as a first time point when the first content is being output, and a second time point when a second content different from the first content is being output. Here, the feedback data refers to the user's biometric information newly acquired at the time when the multimedia content selected by the control unit 110 based on the user's biometric information is displayed through the output unit 120, by comparing the user's biometric information with the existing biometric information, It may be data reflecting the evaluation of how much the activity and concentration increased.

The output unit 120 may output multimedia content. The controller 110 may select one of a plurality of multimedia contents outputtable by the processor 111 based on a user's input. For example, the controller 110 may select multimedia content corresponding to a user's touch input from among icons for each of a plurality of multimedia content output on the screen of the output unit 120 .

The output unit 120 may include a display module. The display module may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. Alternatively, the display module may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch. In this case, the display module may function as an input device generating an input signal by a user's manipulation.

The holding unit 130 may be formed outside the game device 100 to hold the wearing module 200 when the wearing module 200 is not in use. The shape of the mounting portion 130 may vary. In one embodiment, the mounting unit 130 may have a cylindrical shape attached to the bezel portion of the output unit 120 of the game device 100 as shown in FIG. 2 .

The support unit 140 may support the game device 100 so that the game device 100 can be stably fixed to the ground while a user enjoys multimedia content or a game using the game device 100 .

The wearing module 200 may detect the user's brain waves using only one or some of the plurality of sensors or all of the plurality of sensors. In this case, each of the plurality of sensors may be closely attached to each part of the user's brain (eg, frontal lobe, parietal lobe, temporal lobe, occipital lobe, etc.). Accordingly, each of the plurality of sensors may individually detect brain waves generated in a specific region of the brain.

The wearing module 200 may be formed in the form of a headband so that at least a part of it comes into contact with the user's head.

The wearing module 200 may be connected to the game device 100 through a cable 230 . The wearing module 200 may receive electrical energy from the game device 100 through the cable 230 . In addition, the cable 230 may include a communication cable capable of transmitting data when the wearing module 200 exchanges data with the game device 100 in a wired communication method. That is, the wearable module 200 may transmit a signal or power to an external electronic device or receive it from the outside through the cable 230 .

2 is a flowchart illustrating a method of providing a user interface according to an embodiment of the present invention. FIG. 3 is a diagram showing a first game screen displayed on the output unit 120 . 4 is a diagram schematically showing the waveforms of various brain waves. 5 is a diagram showing a second game screen displayed on the output unit 120 .

Referring to FIG. 2 , in step S210 , when the user wears the wearing module 200 , power is supplied to the wearing module 200 and the game device 100 and the wearing module 200 are connected.

In step S220, the first multimedia content is output from the output unit 120 of the game device 100. In this case, the first multimedia content may include a game screen including at least one user interface.

Referring to FIG. 3, the first multimedia content output from the output unit 120 includes a first interface IF1, a second interface IF2, a third interface IF3, and a fourth interface IF4. 1 The game screen D1 can be output.

The first interface IF1 of the first game screen D1 may include a user EEG analysis monitoring connection button. The control unit 110 of the game device 100 responds to the user's first interface IF1 touch operation (including a click operation using a pointer such as a mouse, the same below), and displays the user's A screen showing the result of analyzing the brain wave may be output.

The second interface IF2 of the first game screen D1 may include a mini game connection button for the user. The controller 110 of the game device 100 may display a mini game list screen on the first game screen D1 in response to a user's touch operation on the second interface IF2 so that the user can select a mini game. . Here, the mini-games may include various types of games such as four arithmetic operations, quizzes, and finding the wrong picture so that the user can train his/her quickness, judgment, memory, and the like.

The third interface IF3 of the first game screen D1 may include an interface indicating the user's concentration point CP. The concentration score (CP) may be calculated by the controller 110 based on a result of analyzing the brain wave data of the user. The control unit 110 of the game device 100 may output a result of analyzing the user's brain wave data as a concentration score (CP) on the screen through the third interface IF3. Concentration points (CP) may be output in the form of a gauge.

The fourth interface IF4 of the first game screen D1 may include an interface indicating the user's action score AP. Action points (AP) refer to points required for the user to issue various commands to his or her character. The user manipulates his character through various input devices (eg, touch display module, mouse, joystick, etc.) of the game device 100 and executes action commands including at least one of running, jumping, attacking, and collecting items. can be entered. In this case, a specific action command may require a unique action score for the character to perform the corresponding action. If the user's action points are less than the action points required for issuing the corresponding command, the character does not respond even if the user inputs the corresponding command.

The controller 110 of the game device 100 may output the user's current action score (AP) to the screen through the third interface IF3. Action points (AP) may be output in a gauge form.

In step S230, brain wave data of the user is generated by detecting the user's brain wave by the wearing module 200.

The wearing module 200 may detect the user's brain waves using only one or some of the plurality of sensors or all of the plurality of sensors. When the wearing module 200 detects brain waves using a plurality of sensors, each sensor may individually detect brain waves flowing from the user's brain. For example, the first sensor may detect a first brain wave flowing from the frontal lobe, and the second sensor may detect a second brain wave flowing from the occipital lobe.

Here, brain wave data may be expressed as a waveform whose amplitude changes with time. Also, brain wave data may include at least one characteristic wave data. As shown in FIG. 4 , the characteristic wave data may mean data for each of individual brain waves (delta wave, theta wave, alpha wave, beta wave, gamma wave, etc.) that can be obtained by measuring the user's brain wave.

4 is a graph in which the user's brain waves are measured using the sensor of the wearing module 200 and the measured brain waves are displayed according to frequencies through a power spectrum analysis method. Referring to FIG. 4 , brain waves may be classified into a plurality of individual brain waves according to frequency and amplitude. EEG is an electrical wave in which the activities of tens of thousands of brain cells are combined, and is divided into EEG generated by external stimulation and EEG generated by brain activity without external stimulation. In this case, brain waves generated by brain activity without external stimuli can be further classified into delta waves, theta waves, alpha waves, beta waves, and gamma waves according to natural frequencies.

Delta waves vibrate 1 to 4 times per second at a frequency of 1 to 4 Hz, vibrating the least among brain waves. Delta waves appear during very deep sleep, deep meditation, or unconsciousness.

Theta waves vibrate about 4 to 7 times per second with a frequency of 4 to 7 Hz. It usually appears at the moment when creative thoughts arise or in the process of emotional stability or falling asleep. It is more common in children than adults and also appears during meditation.

Alpha waves, with a frequency of 8 to 12 Hz, are brain waves that appear when the mind is in a comfortable and stable state. In particular, stable alpha waves appear when you are in a relaxed state with your eyes closed. Since alpha waves weaken when the eyes are open and become stronger when the eyes are closed, it is speculated that they are connected to the visual area of the brain.

Beta waves have a frequency of 18 to 25 Hz, and are brain waves that appear while thinking and working with eyes open. In particular, it often appears when dealing with anxiety or tension, or complex calculations. It appears when you have a lot of thoughts or worry.

Gamma waves have a frequency of 30 to 100 Hz) and appear well in a state of nervousness or when deep attention is focused.

Hereinafter, in the present specification, a characteristic wave may mean any one of the aforementioned delta waves, theta waves, alpha waves, beta waves, and gamma waves.

Next, in step S240, the control unit 110 generates at least one characteristic wave data by applying a band-pass filter to the EEG data. A bandpass filter is a filter that passes only signals between specific frequencies. A bandpass filter may be implemented using an RLC circuit.

The controller 110 may generate data for delta waves, theta waves, alpha waves, beta waves, and gamma waves, respectively, based on brain wave data. The controller 110 may generate characteristic wave data based on intensity values of delta waves, theta waves, alpha waves, beta waves, and gamma waves, and a rate of change of the intensity values. That is, the characteristic wave data may be generated based on the intensity value of each individual brain wave and the rate of change of the intensity value.

In an embodiment, the characteristic wave data may include first characteristic wave data and second characteristic wave data. In an embodiment, the first characteristic wave data may be generated based on an intensity value of the beta wave and a rate of change of the intensity value. The second characteristic wave data may be generated based on the intensity value of the theta wave and the rate of change of the intensity value.

In step S250, the control unit 110 calculates a concentration score based on at least one characteristic wave data and generates processing data.

The controller 110 may generate processing data based on the characteristic wave data acquired in step S240. The processing data may include a concentration score. The controller 110 may calculate a concentration score based on the characteristic wave data.

In an embodiment, a concentration score may be calculated based on the first characteristic wave data and the second characteristic wave data. The first characteristic wave data may include information about an intensity value of the beta wave and a rate of change of the intensity value. The second characteristic wave data may include information about an intensity value of the theta wave and a rate of change of the intensity value.

In one embodiment, the controller 110 may calculate the concentration score using [Equation 1]. In [Equation 1], CP may mean a concentration score, Pbeta may mean a beta wave intensity value, and Ptheta may mean a theta wave intensity value.

However, this is just an example, and the method by which the control unit 110 calculates the concentration score based on the characteristic wave data may vary.

In step S260, the controller 110 selects the second multimedia content based on the processed data, and the output unit 120 of the game device 100 outputs the second multimedia content.

The second multimedia content may include a second game screen D2 including at least one user interface.

Referring to FIG. 5 , the second game screen D2 may be different from the first game screen D1. Specifically, at least one piece of content output from the second game screen D2 may be determined based on the processed data generated in step S250. Referring to FIG. 5 , the concentration point gauge displayed through the third interface IF3 of the second game screen D2 may be different from the concentration point gauge of the first game screen D2. That is, when the controller 110 determines that the user's concentration has decreased based on the brain wave data in step S250, the controller 110 displays the second game screen D2 displayed on the output unit 120 of the game device 100. ) through the third interface IF3, a gauge reflecting the user's lowered concentration score may be output.

6 is a flowchart illustrating a user interface providing method according to another embodiment of the present invention.

Referring to FIG. 6 , in step S610, the controller 110 generates first characteristic wave data and analyzes the generated first characteristic wave data. The first characteristic wave data may be generated through step S240 described with reference to FIG. 2 .

In step S620, the controller 110 analyzes information on at least one brain wave included in the first characteristic wave data, and determines whether the brain wave meets a preset brain wave condition.

Here, the 'brain wave condition' may be a condition for a value (physical quantity) of any one of several brain waves such as delta waves, theta waves, and beta waves. For example, the brain wave condition may be a condition for 'intensity' of 'beta waves'. In one embodiment, the brain wave condition may include information about a first threshold value for the intensity of beta waves. That is, determining whether the control unit 110 satisfies the brain wave condition in step S620 is to determine whether the intensity of a beta wave among the first characteristic wave data generated by measuring the user's brain wave exceeds a first threshold value. it could be

In another embodiment, the EEG condition may include information about a second threshold value for the intensity of the theta wave. In this case, determining whether the controller 110 satisfies the brain wave condition in step S620 is to determine whether the strength of the theta wave among the first characteristic wave data generated by measuring the user's brain wave is equal to or less than the second threshold value. can

In another embodiment, the EEG condition may include information about a time interval in which the intensity of a beta wave exceeds a first threshold value. In this case, determining whether the controller 110 satisfies the brain wave condition in step S620 determines the time when the intensity of the beta wave exceeds the first threshold among the first characteristic wave data generated by measuring the user's brain wave. It may be to determine whether it is over a critical time.

The memory of the controller 110 may store information on at least one EEG condition. Accordingly, the controller 110 may determine whether various EEG conditions are met in step S620. Also, the memory of the controller 110 may store information about a game effect corresponding to at least one stored brain wave condition. The game effect may refer to an effect that is executed when an brain wave condition is satisfied.

In step S620, the controller 110 analyzes information on at least one brain wave included in the first characteristic wave data, determines whether the brain wave meets a preset brain wave condition, and if the brain wave meets the predetermined brain wave condition , In step S630, the game effect is output through the output unit 120 of the game device 100, and the user input interface is output.

Here, the game effect means a game effect corresponding to a predetermined brain wave condition. In one embodiment, when the brain wave condition is a condition in which the strength of the beta wave exceeds the first threshold value, the corresponding game effect may be an increase in the action point (AP) described with reference to FIG. 3 . In another embodiment, when the brain wave condition is a condition in which the intensity of the theta wave is equal to or less than the second threshold value, the corresponding game effect may be that the action point (AP) decreases. In another embodiment, when the brain wave condition requires that the time interval in which the intensity of the beta wave exceeds the first threshold is equal to or longer than a specific threshold, the corresponding game effect is proportional to the time exceeding the specific threshold. action points may increase.

The user input interface may refer to an interface that recognizes a user's input operation and allows the game device 100 to generate a user input signal. For example, the game device 100 detects that a user clicks a user input interface displayed on a game screen using a mouse or touches a user input interface using a body part such as a finger, and receives a user input signal. can create

In step S640, when a user input signal is received through the input unit of the game device 100, in step S650, third multimedia content is output through the output unit 120 of the game device 100 based on the user input signal. .

The control unit 110 of the game device 100 may select and output the third multimedia content through the output unit 120 based on the user input signal.

In step S640, when a user input signal is not received through the input unit of the game device 100, returning to step S630, the user input interface is output.

In this case, the content displayed on the user input interface may include a 'wait for input' guidance message.

In step S660, the controller 110 measures the user's brain wave for the third multimedia content, generates second characteristic wave data, and analyzes the second characteristic wave data. Step S660 may correspond to step S610 described above.

Subsequently, in step S670, the controller 110 analyzes information on at least one brain wave included in the second characteristic wave data, and determines whether the brain wave meets a preset brain wave condition. In this case, the first time point is when the third multimedia content is displayed through the output unit 120 of the game device 100, and the generation time point of the second characteristic wave data analyzed by the controller 110 is the first time point. It may be a second point in time that follows. That is, the controller 110 may measure and analyze the changed brain waves when the user recognizes the third multimedia content. Step S670 may correspond to step S620 described above.

As described above, the method for providing a user interface according to an embodiment of the present invention may analyze a user's brain wave in real time and provide multimedia content to the user based on the user's brain wave. In addition, the user's interest can be improved by measuring the user's brain waves, which change according to the provision of new content, in real time.

7 is a block diagram for explaining the control unit 110 of the game device 100.

Referring to FIG. 7 , the game device 100 according to an embodiment of the present invention may include a control unit 110, an output unit 120, a holding unit 130, and a support unit 140.

The output unit 120 may output multimedia content. The controller 110 may select one of a plurality of multimedia contents outputtable by the processor 111 based on a user's input. For example, the controller 110 may select multimedia content corresponding to a user's touch input from among icons for each of a plurality of multimedia content output on the screen of the output unit 120 .

The output unit 120 may include a display module. The display module may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. Alternatively, the display module may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch. In this case, the display module may function as an input device generating an input signal by a user's manipulation.

The controller 110 may control the overall operation of the game device 100 . Referring to FIG. 4 , the controller 110 of the game device 100 may include a processor 111, a memory 112, a communication module 113, and an antenna module 114. Alternatively, at least one of these components may be omitted or one or more other components may be added to the controller 110 . In other embodiments, some of these components may be integrated into a single component.

The processor 111 of the controller 110 may determine at least one of activity of the user's brain, degree of concentration, and emotional state of the user, based on brain wave data received from the wearing module 200 to be described later. The processor 111 may generate biometric information based on at least one of activity of the user's brain, degree of concentration, and emotional state of the user.

Here, the user's emotional state may be classified into, for example, a positive emotional state and a negative emotional state. In another example, the user's emotional state may be classified into a plurality of states. The plurality of states may include at least one of a first state of feeling pleasure, a second state of feeling annoyed, a third state of feeling bored, and a fourth state of feeling anger.

The processor 111 may analyze biometric information, select multimedia content from the memory 112 based on the biometric information, and output the selected multimedia content to the output unit 120 .

The processor 111 may compare biometric information generated at a plurality of viewpoints, generate feedback data based on the comparison result, and store the generated feedback data in the memory 112 . Each of the plurality of points of view may refer to a point in time when different multimedia contents are being output (played) from the output unit 120 . For example, it may be defined as a first time point when the first content is being output, and a second time point when a second content different from the first content is being output.

Here, the feedback data means that the processor 111 compares the user's biometric information newly obtained at the time when the multimedia content selected based on the user's biometric information is displayed through the output unit 120 with the existing biometric information, It may be data reflecting the evaluation of how much the activity and concentration increased. For example, if the processor 111 displays new content different from the multimedia content currently being displayed on the output unit 120 based on the user's first biometric information, then the newly acquired user's second biometric information. As a result of the analysis, if the activity or concentration of the user's brain has increased compared to the first biometric information, the processor 111 provides feedback data including a score proportional to the increase (eg, a score having a positive value). can create Conversely, if the user's brain activity or concentration is rather low, the processor 111 may generate feedback data including a score having a negative value.

The processor 111 stores commands or data received from other components (eg, the wearable module 200 or the server 300) in a volatile memory, processes the commands or data stored in the volatile memory, and generates the resulting data can be stored in non-volatile memory. In one embodiment, the processor 111 is a main processor (eg, a central processing unit or application processor) or a secondary processor (eg, a graphics processing unit, a neural network processing unit (NPU) that can operate independently of or in conjunction therewith). processing unit) and sensor hub processor). For example, when the game device 100 includes a main processor and an auxiliary processor, the auxiliary processor may use less power than the main processor or may be set to be specialized for a designated function. A secondary processor may be implemented separately from, or as part of, the main processor.

The auxiliary processor is, for example, in place of the main processor while the main processor is in an inactive state, or together with the main processor while the main processor is in an active state, at least one of the components of the game device 100 At least some of the functions or states related to the component may be controlled. In one embodiment, a coprocessor may be implemented as part of other functionally related components.

In one embodiment, when the auxiliary processor is a neural network processing device, the auxiliary processor may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the game device 100 itself where the artificial intelligence model is performed, or may be performed through a separate server.

The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 112 of the controller 110 may store a plurality of multimedia contents. The game device 100 may select any one of a plurality of multimedia contents stored in the memory 112 based on a user's input and output the selected contents through the output unit 120 .

In addition, the memory 112 may store a first threshold value related to the activity of the user's brain and a second threshold value related to the degree of concentration of the user's brain. Here, the first threshold value may mean any one value within the user's brain activity distribution range. For example, when the user's brain activity is expressed as a value in the range of 0 to 100, the first threshold value may be any one of the values in the range of 0 to 100, for example 50. However, this is exemplary, and embodiments of the present invention are not limited thereto. Similarly, the second threshold value may mean any one value within the concentration distribution range of the user's brain, where the minimum and maximum values of the 'range' may vary.

The processor 111 may change and store the first and/or second threshold values in the memory 112 based on user information, biometric information, and feedback data. For example, the processor 111 analyzes the biometric information, and if the activity of the user's brain is less than or equal to a first threshold value among the biometric information, the multimedia content currently being output from the output unit 120 and other content are stored in the memory 112. ) can be selected and output to the output unit 120. Alternatively, the processor 111 selects, from the memory 112, content different from the multimedia content currently being output from the output unit 120 when the degree of concentration of the user's brain among the biometric information is equal to or less than the second threshold, and outputs the output unit. It can be output to (120).

Also, the memory 112 may store trigger information related to a user's emotional state. Here, the trigger information on the user's emotional state may be information on a threshold value of biometric information necessary to change the user's emotional state from one state to another.

In addition, the memory 112 may store various data used by at least one component of the game device 100 . Data may include, for example, input data or output data for software and related instructions. The memory 112 may include volatile memory or non-volatile memory.

The communication module 113 may support establishment of a wired or wireless communication channel between the game device 100 and an external electronic device, and communication through the established communication channel. The communication module 113 may include one or more communication processors that operate independently of the processor 111 and support wired communication or wireless communication. In one embodiment, the communication module 113 may include a wireless communication module or a wired communication module. Among these communication modules, a corresponding communication module may communicate with the game device 100 through the network 400 . These various types of communication modules may be integrated into one component or implemented as a plurality of separate components.

The wireless communication module may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module may support a high frequency band, for example, to achieve a high data rate. The wireless communication module uses various technologies for securing performance in a high frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiple-output (FD). Technologies such as full dimensional MIMO (MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module may support various requirements stipulated in the game device 100, an external electronic device, or a network system. In one embodiment, the wireless communication module may support peak data rate for realizing eMBB, loss coverage for realizing mMTC, or U-plane latency for realizing URLLC.

The antenna module 114 may transmit a signal or power to an external electronic device or receive it from the outside. In one embodiment, the antenna module 114 may include an antenna including a radiator formed of a conductor or conductive pattern formed on a substrate. In one embodiment, the antenna module 114 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the network 400 may be selected from the plurality of antennas by, for example, the communication module 113 . A signal or power may be transmitted or received between the communication module 113 and an external electronic device through the selected at least one antenna.

In one embodiment, commands or data may be transmitted or received between game devices 100 connected to network 400 . The game device 100 may be the same as the game device 100 or a different type of device. In one embodiment, all or part of the operations executed in the game device 100 may be executed in the game device 100 .

For example, when the game device 100 needs to perform a certain function or service automatically or in response to a request from a user or other device, the game device 100 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the game device 100 . The game device 100 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The game device 100 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing.

The holding portion 130 of the game device 100 may be formed outside the game device 100 to hold the wearing module 200 when the wearing module 200 is not in use. The shape of the mounting portion 130 may vary. In one embodiment, the mounting unit 130 may have a cylindrical shape attached to the bezel portion of the output unit 120 of the game device 100 as shown in FIG. 7 .

The support unit 140 may support the game device 100 so that the game device 100 can be stably fixed to the ground while a user enjoys multimedia content or a game using the game device 100 .

8 is a diagram for explaining the wearing module 200 according to an embodiment of the present invention.

Referring to FIG. 8 , the wearing module 200 may detect the user's brain waves using only one or some of the plurality of sensors or all of the plurality of sensors. In this case, each of the plurality of sensors may be closely attached to each part of the user's brain (eg, frontal lobe, parietal lobe, temporal lobe, occipital lobe, etc.). Accordingly, each of the plurality of sensors may individually detect brain waves generated in a specific region of the brain.

The wearing module 200 may be formed in the form of a headband so that at least a part of it comes into contact with the user's head.

The wearing module 200 may include a plurality of sensors, a communication unit 220, and a cable 230. The plurality of sensors may include a first sensor 210_1 and a second sensor 210_2. The number of sensors included in the plurality of sensors may vary. Although only two sensors are shown in FIG. 8, the embodiment of the present invention is not limited thereto. Each of the sensors 210_1 and 210_2 may include a metal measuring electrode for detecting an electroencephalogram. Each of the sensors 210_1 and 210_2 may individually detect brain waves flowing from the user's brain. When the user wears the wearing module 200 on the head, the measurement electrode may contact the user's scalp or facial skin. Each of the sensors 210_1 and 210_2 may individually detect brain waves flowing from different parts of the brain. For example, the first sensor 210_1 may detect a first brain wave flowing from the frontal lobe, and the second sensor 210_2 may detect a second brain wave flowing from the occipital lobe. Alternatively, the first sensor 210_1 may detect a first brain wave flowing from the left brain, and the second sensor 210_2 may detect a second brain wave flowing from the right brain.

The communication unit 220 may transmit EEG data acquired by the plurality of sensors of the wearable module 200 to an external electronic device (eg, the game device 100) or receive signals or data from the external electronic device. there is. The communication unit 220 may establish a wired communication channel or a wireless communication channel between an external electronic device such as the game device 100 and the wearable module 200 and support communication through the established communication channel.

The communication unit 220 may include a wireless communication module or a wired communication module. Among these communication modules, a corresponding communication module may communicate with the game device 100 through a network. These various types of communication modules may be integrated into one component or implemented as a plurality of separate components.

The cable 230 may be connected to the game device 100 . The wearing module 200 may receive electrical energy from the game device 100 through the cable 230 . In addition, the cable 230 may include a communication cable capable of transmitting data when the wearing module 200 exchanges data with the game device 100 in a wired communication method. That is, the wearable module 200 may transmit a signal or power to an external electronic device or receive it from the outside through the cable 230 .

9 is a diagram showing a game device 100 and a server 300 communicating with the game device 100 according to an embodiment of the present invention. 10 is a block diagram showing a server 300 according to an embodiment of the present invention. 11 is a diagram for explaining a multilayer neural network structure of the server processor 310.

Referring to FIG. 9 , in a network environment, a game device 100 may communicate with a server 300 through a network. The server 300 may manage the game device 100 and user terminals (not shown) connected to the server 300 . The server 300 receives EEG data obtained by measuring the user's EEG from the game device 100, biometric information generated by analyzing the EEG data, and selection information and feedback data according to a plurality of biometric information comparison results, and receives the feedback data. can be saved In addition, the server 300 may receive characteristic wave data, concentration points, action points, and processing data from the game device 100 and store them.

Referring to FIG. 10 , the server 300 may include a server processor 310, a server memory 320, and a user manager 330. The server processor 310 may receive brain wave data, biometric information, and feedback data stored in the memory 112 of the game device 100 . The server memory 320 may store received EEG data, biometric information, and feedback data. In addition, the server memory 320 may store characteristic wave data, concentration points, action points, and processed data received from the game device 100 .

The server processor 310 may analyze the characteristic wave data using the learning model and correct the concentration score generated based on the characteristic wave data.

To this end, the server processor 310 may include a multilayer neural network 311 and a learning engine 312 .

The learning engine 312 may pre-supervise the multilayer neural network 311 using a plurality of training data. A multilayer neural network is a predictive model implemented in software or hardware that mimics the computational power of a biological system by using a large number of artificial neurons (or nodes).

The learning engine 312 uses the learning data having characteristic wave data as an input value and the concentration score as an output value so that the multilayer neural network 311 can accurately correct the concentration score based on the training data, and the multilayer neural network ( 311) can be supervised.

At this time, supervised learning means learning to find an output value according to a given input value by using data with input values and corresponding output values as learning data, and means learning that takes place in a state where the correct answer is known. The set of input values and output values given in supervised learning is called training data.

Referring to FIG. 11 , the multilayer neural network 311 may include an input layer 311_a, one or more hidden layers 311_b, and an output layer 311_c.

In one embodiment, the multilayer neural network 311 receives an input value and multiplies an input layer 311_a having nodes corresponding to the number of components of the first feature vector and an output value of the input layer 311_a by a weight, A first hidden layer 421_b1 that outputs after adding a bias, a second hidden layer (421_b2) that multiplies a weight for each output value of the first hidden layer 421_b1, adds a bias, and outputs the result, and An output layer 311_c that multiplies each output value of the hidden layer 421_b2 by a weight and outputs the result using an activation function may be included. Although only two hidden layers 421_b1 and 421_b2 are shown in FIG. 6 , one or more hidden layers 311_b include a larger number of hidden layers in addition to the first hidden layer 421_b1 and the second hidden layer 421_b2. can do.

For example, the activation function may be a Softmax function, but embodiments of the present invention are not limited thereto, and the activation function may be various other functions such as a LeRU function. Weights and biases can be continuously updated by supervised learning.

Specifically, the output vector may be input to a loss function layer connected to the output layer 311_c. The loss function layer may output a loss value using a loss function that compares an output vector with a correct answer vector for each training data. Parameters of the multilayer neural network 311 may be supervised in a direction in which a loss value decreases.

For example, the loss function layer can calculate a loss value according to [Equation 2]. In [Equation 2], N is the number of a plurality of training data, n is a natural number identifying the learning data, k is a natural number identifying the value of the nth learning data, nk means the kth value of the nth learning data , t may mean correct answer data, y may mean an output vector, and E may mean a loss value.

Alternatively, the loss function layer may calculate a loss value according to [Equation 3]. In [Equation 3], n is the number of learning data for each class, y and j are identifiers representing classes, C is a constant value, M is the number of classes, x_y is the probability that the training data belongs to class y, x_j is learning A probability value, L, that data belongs to class j may mean a loss value.

The server processor 310 may be manufactured in the form of at least one hardware chip and mounted in a device. For example, the server processor 310 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor (eg, CPU or application processor) or a graphics-only processor (eg, It may be manufactured as a part of GPU) and mounted in various devices described above.

In this case, the server processor 310 may be mounted on one device or may be mounted on a separate device. Meanwhile, the learning model of the server processor 310 may be implemented as a software module. When the learning model is implemented as a software module (or a program module including instructions), the software module may be stored in a computer-readable, non-transitory computer readable media. there is. Also, in this case, at least one software module may be provided by an Operating System (OS) or a predetermined application. Alternatively, some of the at least one software module may be provided by an Operating System (OS), and the other part may be provided by a predetermined application.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

outputting first multimedia content including at least one user interface by an output unit of the game device;
generating brain wave data by sensing brain waves of a user by a wearable module connected to the game device;
generating, by a controller, at least one characteristic wave data by applying a band-pass filter to the brain wave data;
calculating, by the control unit, a concentration score based on the characteristic wave data and generating processing data; and
Selecting, by the controller, second multimedia content from a memory based on the processed data and outputting it to the output unit; including,
The characteristic wave data,
It is generated based on an intensity value and a rate of change of the intensity value of at least one characteristic wave,
The control unit calculates the concentration score based on the intensity value of the characteristic wave, and generates the processing data based on the concentration score;
The first multimedia content,
A first interface displaying a gauge visually representing the concentration score; and
A second interface displaying a gauge that visually represents an action score;
The action score means a score required for the user to manipulate his character appearing in the first multimedia content,
The control unit calculates the action score based on the characteristic wave data.
How to present the user interface. delete According to claim 1,
The characteristic wave data includes first characteristic wave data corresponding to a beta wave and second characteristic wave data corresponding to a theta wave,
The control unit calculates the concentration score based on the first characteristic wave data and the second characteristic wave data.
How to present the user interface. According to claim 3,
Further comprising: analyzing the characteristic wave data by a server processor connected to the game device;
The server processor may include a multi-layer neural network including an input layer, one or more hidden layers, and an output layer; And a learning engine for performing supervised learning on the multilayer neural network using training data having the characteristic wave data as input values and concentration scores as output values, and calculating optimized hyperparameters based on the training data. ,
The input layer receives the characteristic wave data, the at least one hidden layer multiplies a weight for each output value of the input layer, adds a bias, and outputs the result, and the output layer inputs an output value of the hidden layer. Take it and use the activation function to output the output vector,
The multilayer neural network includes a loss function layer connected to the output layer, the output vector is input to the loss function layer, and the loss function layer compares the output vector with a correct answer vector for each feedback data. A loss value is output using , and the parameters of the multilayer neural network are learned in a direction in which the loss value decreases.
How to present the user interface. analyzing the first characteristic wave data by a control unit;
When the characteristic wave corresponding to the first characteristic wave data satisfies a preset first brain wave condition, an output unit outputs content reflecting a first game effect corresponding to the first brain wave condition, and the output unit By, outputting a user input interface;
When a user input signal is received through the user input interface, the control unit selects one of at least one multimedia content stored in a memory based on the user input signal, outputting;
analyzing second characteristic wave data generated based on the brain waves of the user at a second time point subsequent to the first time point; and
When the characteristic wave corresponding to the second characteristic wave data satisfies a preset second brain wave condition, outputting, by the output unit, content reflecting a second game effect corresponding to the second brain wave condition; including,
How to present the user interface.

Images