CN117582655A - VR game emotion regulation and control method based on EEG signals - Google Patents

VR game emotion regulation and control method based on EEG signals Download PDF

Info

Publication number
CN117582655A
CN117582655A CN202311646228.3A CN202311646228A CN117582655A CN 117582655 A CN117582655 A CN 117582655A CN 202311646228 A CN202311646228 A CN 202311646228A CN 117582655 A CN117582655 A CN 117582655A
Authority
CN
China
Prior art keywords
degree
subject
game
eeg
wake
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311646228.3A
Other languages
Chinese (zh)
Inventor
赵莎
石江豪
潘纲
李石坚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202311646228.3A priority Critical patent/CN117582655A/en
Publication of CN117582655A publication Critical patent/CN117582655A/en
Pending legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/212—Input arrangements for video game devices characterised by their sensors, purposes or types using sensors worn by the player, e.g. for measuring heart beat or leg activity
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/53—Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
    • A63F13/537—Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game using indicators, e.g. showing the condition of a game character on screen
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/54—Controlling the output signals based on the game progress involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2131—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on a transform domain processing, e.g. wavelet transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2431—Multiple classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1012—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/30—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by output arrangements for receiving control signals generated by the game device
    • A63F2300/303—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by output arrangements for receiving control signals generated by the game device for displaying additional data, e.g. simulating a Head Up Display
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60—Methods for processing data by generating or executing the game program
    • A63F2300/6063—Methods for processing data by generating or executing the game program for sound processing
    • A63F2300/6081—Methods for processing data by generating or executing the game program for sound processing generating an output signal, e.g. under timing constraints, for spatialization
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8082—Virtual reality
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Cardiology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Acoustics & Sound (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)

Abstract

The invention discloses an EEG signal-based VR game emotion regulation and control method, which comprises the steps of exciting the arousal degree of a subject through a game, detecting the current arousal degree of the subject in real time through an electroencephalogram signal, and helping the subject to regulate the arousal degree of the subject through a proper auxiliary strategy when detecting that the current arousal degree of the subject is too high or too low. According to the invention, the arousal level of the current subject is evaluated and regulated by the electroencephalogram signals, so that a machine can more timely and accurately understand the current intention of the subject and regulate and control the current intention, and the EEG is further popularized for use.

Description

VR game emotion regulation and control method based on EEG signals
Technical Field
The invention belongs to the technical field of EEG signal recognition, and particularly relates to a VR game emotion regulation and control method based on EEG signals.
Background
The emotion ring model considers that emotion can be divided into two dimensions of titer and arousal degree, the arousal degree is the reaction strength of human beings to external stimulus, and the emotion titer is a pleasure degree, and can be either negative (such as sadness) or positive (such as happiness). Arousal is mainly focused on the intensity of emotion, and is reflected by a series of changes from calm to excited, for example, when a person reads a book in a quiet library, the arousal can be low, and calm and concentration are reflected; however, when a person is in an environment full of excitement and motion, the arousal level may rise, exhibiting an excited or excited state. Many studies have revealed that there is a close correlation between the level of arousal and the actual performance or behavior of an individual, such as in social interactions, academic or careers, where moderate arousal allows us to concentrate better and increase efficiency; however, when the arousal level is too high, some adverse reactions such as excessive stress, pressure, etc. may be caused. Thus, by adjusting or controlling the degree of arousal, we can better cope with and control such conditions; in general, understanding and mastering how to adjust arousal is helpful for controlling self emotion and maintaining physical and mental health.
Electroencephalogram is a bioelectric phenomenon derived from the activity of neurons in the cerebral cortex, which generates weak voltages when neurons in the cerebral cortex are active, and which is expressed in the form of complex electric waves when thousands of neurons are active at the same time, and which can be captured and recorded by electrodes placed on the surface of the scalp, and the resulting Electroencephalogram is called an Electroencephalogram (EEG). Electroencephalogram is data which intuitively reflects brain activity states, people can calculate frequency characteristics, space characteristics and the like of EEG in different wave bands through a mathematical method, and distribution of brain electricity can be fitted through a deep learning method by utilizing a brain electricity data set. The brain electricity has close relation with the psychological activities of people, and the current psychological state of a person can be judged through the brain electricity, such as states of attention, relaxation, sleep, excitation or anxiety; as such, EEG has been widely used in the fields of neuroscience, psychology, and medicine for studying brain and psychological behaviors of humans.
Virtual Reality (VR) can make a user put in a more realistic environment by simulating a three-dimensional space in a real environment; when a subject performs a motion of moving or head rotating in an actual environment, the VR device can accurately calculate the current posture angle of the human body through hardware devices such as a gyroscope and an accelerometer built in the VR device, and present a visual picture with reality in a 3D display scene accordingly, so that the subject can feel the virtual world in an immersive manner. The game is an interactive entertainment mode, and compared with single video and music, the immersion feeling obtained by the subject when using the VR game is deeper.
As people continue to research EEG signals, there has been significant progress in using EEG signals to resolve emotion and measure arousal; nevertheless, current approaches still face several challenges and limitations, and existing research is often limited to specific application scenarios when using EEG signals for mood control. In general, it is considered important to maintain consistency of the training environment and the regulatory environment; ideally, however, the mood control method should be able to adapt to a variety of different environments and maintain its effectiveness. In addition, the EEG regulation means used at present are often single and lack innovation and diversity; in order to change the emotional state, widely adopted methods include playing music or video, which is effective in a certain range, but these conventional methods have a disadvantage in interactivity and cannot accurately or individually cope with the needs of people in different situations.
Thus, there is a great need to develop more research to enhance the application of EEG signals in innovative ways that allow for more efficient, flexible control and adaptation to the emotional state of an individual.
Disclosure of Invention
In view of the above, the invention provides an emotion regulation and control method for a VR game based on EEG signals, which detects the arousal degree through the EEG signals, regulates and controls a subject according to the arousal degree, and fuses virtual reality technology to generate more immersive regulation and control experience so as to improve the practicability and effectiveness of EEG in emotion regulation and control.
An EEG signal-based VR game emotion regulation and control method comprises the following steps:
(1) Acquiring electroencephalogram signals of different subjects by utilizing VR-EEG integrated equipment in a training mode, so as to acquire a large number of EEG signals and record wake-up degree labels corresponding to the EEG signals;
(2) Preprocessing the acquired EEG signals to obtain a large amount of sample data, and dividing all the sample data into a training set and a testing set;
(3) Constructing a wake-up degree classification model, which comprises:
an input layer for calculating differential entropy of each frequency band of the EEG signal;
the feature extraction layer is used for extracting frequency domain features and time domain features of the EEG signals and performing feature stitching;
the output layer is used for converting the characteristic vector obtained after the splicing into the prediction probability of the EEG signal corresponding to each class of awakening degree;
(4) Training the wake-up degree classification model by using sample data of a training set;
(5) Inputting sample data of the test set into a trained wake-up degree classification model, and outputting the model to obtain the current wake-up degree (the class with the maximum corresponding prediction probability) of the corresponding subject;
(6) And regulating and controlling the VR game scene of the subject by using the emotion regulating and controlling system according to the current arousal degree of the subject.
Further, the specific implementation manner of the step (1) is as follows: and the VR-EEG integrated equipment is utilized to enable a subject to watch selected videos capable of effectively activating different degrees of arousal degrees in a VR environment, so that EEG signals of the subject are collected, and the subject can evaluate the arousal degree according to the irritation degree of the subject, which is sensed as the video induction, as arousal degree labels of the EEG signals, and the arousal degree labels are classified into three grades of high, medium and low.
Further, the specific implementation manner of the step (2) is as follows: the acquired EEG signals are two-channel signals, and are influenced by factors of myoelectricity, electrooculogram and circuits, so that filtering and denoising treatment is needed for the EEG signals; then converting EEG signals of each channel in a fixed segment length (such as 1.5 s) into frequency domain signals by using Fourier transformation, dividing the frequency domain signals into five frequency bands according to the frequency size, namely delta frequency bands corresponding to 0.1-4 Hz, theta frequency bands corresponding to 4-8 Hz, alpha frequency bands corresponding to 8-14 Hz, beta frequency bands corresponding to 14-30 Hz, gamma frequency bands corresponding to 30-45 Hz, and a group of sample data comprises the EEG signals, wake-up degree labels corresponding to the EEG signals and the frequency domain signals after the dividing.
Further, the input layer calculates differential entropy of each frequency band of the EEG signal according to the following formula to obtain a 2×5 differential entropy matrix (2 represents the channel number), and finally adjusts the matrix shape to obtain a 1×10 differential entropy vector to be provided for the feature extraction layer;
wherein: sigma (sigma) i For variance of energy in ith frequency band of EEG signal, DE i The differential entropy of the ith frequency band is represented, i is any natural number from 1 to 5.
Further, the feature extraction layer comprises a time domain module and a frequency domain module, wherein the time domain module takes an EEG signal as input, extracts time domain features of the signal through multi-layer convolution, and finally passes the extracted time domain features through a gating linear unit to inhibit possible noise; the frequency domain module takes differential entropy vectors of EEG signals as input, extracts frequency domain features of the signals through two full-connection layers, and finally, outputs the frequency domain features in the form of feature vectors through one full-connection layer after splicing the time domain features and the frequency domain features.
Further, the output layer converts the feature vector output by the feature extraction layer into a confidence vector by adopting a Softmax function, wherein the confidence vector comprises the prediction probability of the corresponding arousal degree of each class of EEG signals.
Further, the specific implementation manner of the step (4) is as follows:
4.1 initializing model parameters, including bias vectors and weight matrixes of each layer, learning rate and an optimizer;
4.2, inputting sample data of the training set into the model one by one, and outputting the model forward transmission to obtain a corresponding prediction result, and calculating a loss function between the prediction result and the wake-up degree label;
4.3, using an optimizer to update model parameters through a gradient descent method according to the loss function (such as cross entropy loss) until the loss function converges and training is completed.
Further, the specific implementation manner of the step (6) is as follows: firstly, setting a state bar with a numerical range of 0-100, wherein the numerical value of an initial state bar is 50, and the emotion regulating system can increase or decrease the state bar according to the wake-up degree predicted and output by the current model, specifically: under the condition that the current state bar value is lower than 50, the state bar value is enabled to be +2 if the predicted awakening degree is high, the state bar value is enabled to be +1 if the predicted awakening degree is medium, and the state bar value is enabled to be-1 if the predicted awakening degree is low; if the current state bar value is higher than 50, the state bar value is +1 if the predicted wake-up degree is high, the state bar value is-1 if the predicted wake-up degree is medium, and the state bar value is-2 if the predicted wake-up degree is low; under the condition that the current state bar value is equal to 50, the state bar value is enabled to be +1 if the predicted awakening degree is high, the state bar value is unchanged if the predicted awakening degree is medium, and the state bar value is enabled to be-1 if the predicted awakening degree is low; finally, the emotion regulating system regulates and controls the VR game scene of the subject according to the value of the state bar, if the value of the state bar is lower, the system can increase dynamic elements of the game (such as giving auxiliary props and playing the excited music), so that the game environment becomes more stimulated to improve the arousal degree of the subject; if the state bar value is higher, the system can reduce the excitement of the game (such as triggering a invigoration state and playing relaxed music), so that the game environment becomes more comfortable and relaxed, and the arousal degree of the subject is reduced.
The main core idea of the invention is that the current arousal degree of the subject can be obtained in real time by using EEG, on the other hand, the subject obtains more immersive feeling by using VR game, and then the emotion regulation is carried out in a corresponding mode according to the state of the arousal degree. Firstly, regarding how to acquire the current arousal degree state, a technology based on deep learning is adopted, and the current arousal degree is identified through EEG signals, and the technology can quickly train a model capable of judging the current arousal degree by acquiring the EEG data of a subject, and the model can analyze the EEG signals in real time, so that a machine can directly acquire the current arousal degree state of the subject; then according to the arousal degree analyzed by the model, the system can automatically select a proper regulation and control mode (for example, when the system detects that the arousal degree of the subject is too high, the game arousal degree of the subject can be reduced by adjusting the attribute of the game role, using audio-visual assistance and the like), the dynamic regulation and control based on the actual state of the subject can enhance the attraction of the game, reduce the possibility that the subject is in an unhealthy state such as overfatigue, tension and the like, and simultaneously lay a good foundation for further brain electricity related research and application.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the emotion regulating system for VR game according to the present invention.
Fig. 2 is a schematic flow chart of acquiring an electroencephalogram signal in a training mode.
FIG. 3 is a schematic diagram of the overall architecture of the wake-up classification model according to the present invention.
Fig. 4 is a schematic diagram of a closed-loop mood control flow for a subject using a VR game.
Fig. 5 is a schematic diagram of different modes of control of the subject triggering at different status bar values.
Fig. 6 is a VR game scene view experienced by a subject.
Detailed Description
In order to more particularly describe the present invention, the following detailed description of the technical scheme of the present invention is provided with reference to the accompanying drawings and the specific embodiments.
As shown in fig. 1, the invention mainly uses EEG signals as the basis to realize emotion regulation and control of a user in a VR game, analyzes and processes the signals by noninvasively collecting the EEG signals of the user, and adjusts the game environment in real time, so that the arousal degree of a subject is in an optimal state when the VR game is played, further the game experience is improved, and the self state is controlled. To achieve this goal, we designed and realized the following specific system:
the main duty of this system is to provide various stimulating materials in VR environment, which can excite different arousal degrees of the subject, so as to collect the brain electrical signals of the subject at different arousal degrees. The stimulation system provides video clips or game scenes capable of inducing different arousal degrees (for example, when training a model for identifying arousal degrees, provides some stimulated videos to keep the subject highly concentrated and alert, or provides some relaxed videos to keep the subject relaxed), and when selecting different control modes according to arousal degrees, the stimulation system presents a three-dimensional game scene to induce the arousal degrees of the subject, so that the three-dimensional game scene watched by the subject can be adjusted according to the current arousal degrees, and closed-loop control is realized, as shown in fig. 4.
The acquisition system is mainly used for acquiring the brain electrical signals of the subject, and the brain electrical activity data of the subject are acquired through electrodes arranged on the head of the subject; after the electroencephalogram signal is acquired, the electroencephalogram signal is high in noise and is easily influenced by myoelectricity, electrooculogram, electric wires and other factors, so that the signal needs to be further subjected to filtering noise reduction processing. In addition, the acquired signals are further divided into subjects, training sets, test sets, signal segments and the like, so that the acquired data can be normally input into an analysis system.
And the analysis system is used for analyzing the acquired brain electricity in a training mode and is used for training a wake-up degree classification model. Specifically, the electroencephalogram signals which are well arranged by the acquisition system before being input into the model are subjected to feature extraction such as differential entropy calculation, or feature vectors corresponding to the electroencephalogram signals are obtained after the electroencephalogram signals are subjected to a neural network such as a convolution layer and a full connection layer, further probability distribution of the wake-up degree corresponding to the electroencephalogram signals is obtained by using softmax, then the difference between a predicted result and a real result is calculated, gradient of a model tensor to be updated is calculated through a loss function such as Adam, then the model is updated, and an accurate wake-up degree classification model is trained from data by continuously repeating the process, so that the wake-up degree state of a subject can be obtained through the electroencephalogram signals. The brain electrical signals are identified by using a model trained previously in the regulation mode, and the identified arousal degree is transmitted into the emotion regulation system, and the process is performed in real time when the testee experiences the VR game, which means that the machine can directly know the current arousal degree of the testee, and automatically adjust the VR environment according to the arousal degree of the testee, so that the optimal game experience is provided, and the testee is helped to better understand and control the arousal degree of the testee.
The main responsibility of the regulation system is to select a regulation mode according to the arousal degree of the received subjects in the VR game environment and present corresponding content. For example, if the arousal level of the subject is low, the regulatory system may increase the dynamic elements of the game, making the game environment more irritating to increase the arousal level of the subject; conversely, if the arousal level of the subject is high, the regulation system may reduce the excitement of the game, making the game environment more comfortable and relaxed, so as to reduce the arousal level of the subject. The regulation and control can be realized in a plurality of layers, and when the proper arousal degree of the subject is detected, the regulation and control system is not changed; when the arousal degree is higher, the color of the display is changed, when the arousal degree is further increased, music is played, and the game content is directly changed to perform emotion regulation; conversely, similar but opposite regulatory content is also available as the arousal level decreases.
The specific implementation steps of the system are as follows:
(1) Electroencephalogram (EEG) signal acquisition equipment is used for acquiring EEG signals on a wearer, the equipment is VR-EEG integrated equipment, two electrodes capable of acquiring Fp1 and Fp2 positions are arranged at the front end of the equipment, a tested person can be informed of the experimental purpose and can explain the specific meaning of the arousal degree, and then the tested person can adjust sitting posture to normally watch videos in the VR equipment. VR videos watched by the subject are manually selected videos with different levels of arousal degrees, and during the acquisition period, the subject evaluates the arousal degrees induced by the videos watched before according to the self-perception as the evoked irritation of the videos, as shown in fig. 2, each time a video is watched, the subject evaluates the arousal degrees induced by the videos watched before.
(2) The collected electroencephalogram data is transmitted to a wake-up degree classification system, specifically, after the original electroencephalogram signals are collected, the electroencephalogram signals are filtered, sliced and the like and then input into a wake-up degree classification model, the model can learn the mapping relation between the data and the wake-up degree from the electroencephalogram data based on deep learning, and the input electroencephalogram-related signals can be converted into wake-up degrees corresponding to the signals. In the present invention, we divide the arousal level of a subject into three levels, low, medium and high, and we will introduce in more detail the details of the conversion of an electroencephalogram signal into a corresponding arousal level:
after the acquisition device acquires an original electroencephalogram signal, the shape of which is 2×450, that is, a signal of which the two channels contain 450 data points (because the acquisition frequency is 300Hz, which means that the 450 data points are signals of 1.5 s), differential entropy of different bands of the electroencephalogram signal, that is, frequency domain characteristics corresponding to the electroencephalogram signal, is calculated, specifically, (1) delta frequency band: 0.1-4 Hz, (2) theta band: 4-8 Hz, (3) alpha frequency band: 8-14 Hz, (4 beta frequency band): 14-30 Hz, (5) gamma frequency band: 30-45 Hz. By calculating the differential entropy corresponding to the five wave bands of the two channels, the feature vector of 2 multiplied by 5 is obtained, and then the matrix shape is adjusted to obtain the frequency domain feature vector of 1 multiplied by 10. Regarding differential entropy, first, a signal in one time segment of one channel is converted into a frequency domain signal by using fourier transform, then the frequency domain is divided into five bands, and differential entropy corresponding to different bands is calculated according to a formula:
wherein: i is [1,5 ]]One number in the spectrum represents the ith band (delta, theta, alpha, beta, gamma five bands), sigma i For the variance of the energy of the electroencephalogram signal in the ith frequency band under the Fourier transform, DE i Representing the differential entropy of the ith frequency band.
The original electroencephalogram signal and the frequency domain information are input into a wake-up degree classification model shown in fig. 3, wherein the classification model mainly comprises two parts, namely a time domain module for processing time domain features and a frequency domain module for processing frequency domain features. In the time domain module, inputting an electroencephalogram signal, extracting features through convolution, and finally, inhibiting possible noise through a gating linear unit by the extracted features; and inputting the differential entropy extracted through calculation into the frequency domain module, finally splicing the characteristics extracted by the time domain module and the characteristics extracted by the frequency domain module through the full-connection layer, converting the time domain characteristics and the frequency domain characteristics into characteristic vectors corresponding to final electroencephalogram signals through the full-connection layer, and converting the weight output by the model into different types of probabilities through the softmax layer.
After the probability of the arousal degree of the current electroencephalogram signal is obtained, a real label obtained by scoring a subject is compared with a prediction result, a corresponding cross entropy loss is calculated, then the gradient generated by the loss on tensors is calculated by using an Adam algorithm, then the model weight is modified according to the calculated gradient, the error of model output is reduced, and the process is repeated continuously, so that the arousal degree classification model is obtained.
(3) In the regulation mode, when the subject will play the VR game, the game task is executed, specifically, at this time, the system may provide a virtual reality game, where the game scene is shown in fig. 6, the upper part of the graph shows the current arousal level of the subject, the upper small circle points to the specific degree of the current arousal level, the characters beside the upper small circle indicate the range of the current arousal level, and the three classes are classified into a high class, a medium class and a low class, the red arc bar on the left side is the current blood volume of the game character operated by the subject, the ammunition number of weapons is held by the game character on the right side, the game overall frame is the first person shooting game under the VR environment, and the subject will require to kill a certain number of enemies in the game.
(4) When a subject plays a game task, the acquisition system transmits the brain electricity acquired by the subject during the VR game to the arousal degree analysis system, the arousal degree analysis system calculates the corresponding arousal degree according to the currently received brain electricity signals, and then the arousal degree of the subject is transmitted into the emotion regulation and control system in real time.
(5) In the emotion control system, in order to reduce noise influence caused by factors such as large actions of a subject, the control system performs smoothing processing on the collected awakening degrees, specifically, the system records the awakening degrees received in the past, then classifies and integrates the collected awakening degrees into the awakening degrees in a period of time in the past, and a state bar of the awakening degree is displayed in front of the eyes of the subject, wherein the range of the state bar is 0-100. The regulation and control system comprehensively considers the change direction of the state bar according to the value of the current state bar and the just received wake-up degree from the wake-up degree analysis system; for example, when the wake-up degree collected over a period of time is classified as a high wake-up degree, the current numerical bar is increased; when the wake-up degree collected in the past period of time is classified as a medium wake-up degree, the current numerical bar changes towards the middle direction; when the wake-up degree collected in the past period is low wake-up degree, the current numerical bar is reduced; the numerical value strip is used, and the wake-up degree presented in front of eyes of a subject is smoother through a counting method in the past for a period of time, so that on one hand, the transient influence caused by noise is reduced, and on the other hand, the subject can more emphasize the change of the wake-up degree per se, and the task of regulation is more met.
(6) The subject can notice the current arousal level in real time, so that the subject is helped to notice the current arousal level, the regulation and control can be carried out in a mode of combining various modes of vision, hearing and game contents, specifically, the regulation and control mode is shown in fig. 5, when the state bar data of the arousal level of the subject is too high, the subject is calm down to make the subject enter a non-hostile state currently, and the blood volume is not reduced; when the number bar data of the subject is slightly higher, softer music can be played, and the environment is slightly changed so that the subject enters a cold state; when the numerical value of the subject is in the normal range, no additional regulation and control are performed; when the number of the test subject is slightly lower, more aggressive music can be played, so that the test subject is assisted to increase the arousal degree of the test subject; when the number of the test subjects is too low, the test subjects can temporarily enter an infinite bullet state, and the current arousal degree of the test subjects is improved.
(7) After the subject completes the game session, the game ends, at which point the subject will proceed with a regulatory-related questionnaire answer.
The embodiments described above are described in order to facilitate the understanding and application of the present invention to those skilled in the art, and it will be apparent to those skilled in the art that various modifications may be made to the embodiments described above and that the general principles described herein may be applied to other embodiments without the need for inventive faculty. Therefore, the present invention is not limited to the above-described embodiments, and those skilled in the art, based on the present disclosure, should make improvements and modifications within the scope of the present invention.

Claims (8)

1. An EEG signal-based VR game emotion regulation and control method comprises the following steps:
(1) Acquiring electroencephalogram signals of different subjects by utilizing VR-EEG integrated equipment in a training mode, so as to acquire a large number of EEG signals and record wake-up degree labels corresponding to the EEG signals;
(2) Preprocessing the acquired EEG signals to obtain a large amount of sample data, and dividing all the sample data into a training set and a testing set;
(3) Constructing a wake-up degree classification model, which comprises:
an input layer for calculating differential entropy of each frequency band of the EEG signal;
the feature extraction layer is used for extracting frequency domain features and time domain features of the EEG signals and performing feature stitching;
the output layer is used for converting the characteristic vector obtained after the splicing into the prediction probability of the EEG signal corresponding to each class of awakening degree;
(4) Training the wake-up degree classification model by using sample data of a training set;
(5) Inputting sample data of the test set into a trained wake-up degree classification model, and outputting the model to obtain the current wake-up degree of the corresponding subject;
(6) And regulating and controlling the VR game scene of the subject by using the emotion regulating and controlling system according to the current arousal degree of the subject.
2. The VR game emotion regulating method of claim 1, wherein: the specific implementation mode of the step (1) is as follows: and the VR-EEG integrated equipment is utilized to enable a subject to watch selected videos capable of effectively activating different degrees of arousal degrees in a VR environment, so that EEG signals of the subject are collected, and the subject can evaluate the arousal degree according to the irritation degree of the subject, which is sensed as the video induction, as arousal degree labels of the EEG signals, and the arousal degree labels are classified into three grades of high, medium and low.
3. The VR game emotion regulating method of claim 1, wherein: the specific implementation mode of the step (2) is as follows: the acquired EEG signals are two-channel signals, and are influenced by factors of myoelectricity, electrooculogram and circuits, so that filtering and denoising treatment is needed for the EEG signals; and then converting the EEG signal of each channel in the fixed segment length into a frequency domain signal by using Fourier transformation, and dividing the frequency domain signal into five frequency bands according to the frequency size, namely, a delta frequency band corresponding to 0.1-4 Hz, a theta frequency band corresponding to 4-8 Hz, an alpha frequency band corresponding to 8-14 Hz, a beta frequency band corresponding to 14-30 Hz and a gamma frequency band corresponding to 30-45 Hz, wherein one group of sample data comprises the EEG signal, a corresponding wake-up degree label thereof and the frequency domain signal after the division.
4. The VR game emotion regulating method of claim 1, wherein: the input layer calculates differential entropy of each frequency band of the EEG signal according to the following formula to obtain a 2 multiplied by 5 differential entropy matrix, and finally adjusts the matrix shape to obtain a 1 multiplied by 10 differential entropy vector which is provided for the feature extraction layer;
wherein: sigma (sigma) i For variance of energy in ith frequency band of EEG signal, DE i The differential entropy of the ith frequency band is represented, i is any natural number from 1 to 5.
5. The VR game emotion regulating method of claim 1, wherein: the characteristic extraction layer comprises a time domain module and a frequency domain module, wherein the time domain module takes an EEG signal as input, extracts time domain characteristics of the signal through multi-layer convolution, and finally passes the extracted time domain characteristics through a gating linear unit to inhibit possible noise; the frequency domain module takes differential entropy vectors of EEG signals as input, extracts frequency domain features of the signals through two full-connection layers, and finally, outputs the frequency domain features in the form of feature vectors through one full-connection layer after splicing the time domain features and the frequency domain features.
6. The VR game emotion regulating method of claim 1, wherein: the output layer adopts a Softmax function to convert the feature vector output by the feature extraction layer into a confidence vector, and the confidence vector comprises the prediction probability of the EEG signal corresponding to each class of awakening degree.
7. The VR game emotion regulating method of claim 1, wherein: the specific implementation manner of the step (4) is as follows:
4.1 initializing model parameters, including bias vectors and weight matrixes of each layer, learning rate and an optimizer;
4.2, inputting sample data of the training set into the model one by one, and outputting the model forward transmission to obtain a corresponding prediction result, and calculating a loss function between the prediction result and the wake-up degree label;
and 4.3, continuously and iteratively updating model parameters by using an optimizer through a gradient descent method according to the loss function until the loss function converges and training is completed.
8. The VR game emotion regulating method of claim 1, wherein: the specific implementation manner of the step (6) is as follows: firstly, setting a state bar with a numerical range of 0-100, wherein the numerical value of an initial state bar is 50, and the emotion regulating system can increase or decrease the state bar according to the wake-up degree predicted and output by the current model, specifically: under the condition that the current state bar value is lower than 50, the state bar value is enabled to be +2 if the predicted awakening degree is high, the state bar value is enabled to be +1 if the predicted awakening degree is medium, and the state bar value is enabled to be-1 if the predicted awakening degree is low; if the current state bar value is higher than 50, the state bar value is +1 if the predicted wake-up degree is high, the state bar value is-1 if the predicted wake-up degree is medium, and the state bar value is-2 if the predicted wake-up degree is low; under the condition that the current state bar value is equal to 50, the state bar value is enabled to be +1 if the predicted awakening degree is high, the state bar value is unchanged if the predicted awakening degree is medium, and the state bar value is enabled to be-1 if the predicted awakening degree is low; finally, the emotion regulating system regulates and controls the VR game scene of the subject according to the value of the state bar, if the value of the state bar is lower, the system can increase dynamic elements of the game, so that the game environment becomes more stimulated, and the arousal degree of the subject is improved; if the status bar value is higher, the system can reduce the excitement of the game, so that the game environment becomes more comfortable and relaxed, and the arousal degree of the subject is reduced.
CN202311646228.3A 2023-12-04 2023-12-04 VR game emotion regulation and control method based on EEG signals Pending CN117582655A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311646228.3A CN117582655A (en) 2023-12-04 2023-12-04 VR game emotion regulation and control method based on EEG signals

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311646228.3A CN117582655A (en) 2023-12-04 2023-12-04 VR game emotion regulation and control method based on EEG signals

Publications (1)

Publication Number Publication Date
CN117582655A true CN117582655A (en) 2024-02-23

Family

ID=89914929

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311646228.3A Pending CN117582655A (en) 2023-12-04 2023-12-04 VR game emotion regulation and control method based on EEG signals

Country Status (1)

Country Link
CN (1) CN117582655A (en)

Similar Documents

Publication Publication Date Title
Bhatti et al. Human emotion recognition and analysis in response to audio music using brain signals
Jeong et al. Cybersickness analysis with eeg using deep learning algorithms
CN107530012B (en) system for brain activity resolution
CN107402635B (en) Mental health adjusting method and system combining brain waves and virtual reality
Wang et al. Design of intelligent EEG system for human emotion recognition with convolutional neural network
Bigirimana et al. Emotion-inducing imagery versus motor imagery for a brain-computer interface
Chang et al. Based on support vector regression for emotion recognition using physiological signals
KR20200117089A (en) The virtual reality contents control system and the virtual reality contents control method operating by the same
KR20200049930A (en) The biological signal analysis system and biological signal analysis method for operating by the system
KR20190030611A (en) Method for integrated signal processing of bci system
KR20080107961A (en) User adaptative pattern clinical diagnosis/medical system and method using brain waves and the sense infomation treatment techniques
Zhao et al. Human-computer interaction for augmentative communication using a visual feedback system
Vettivel et al. System for detecting student attention pertaining and alerting
CN117582655A (en) VR game emotion regulation and control method based on EEG signals
Kolivand et al. Emotion interaction with virtual reality using hybrid emotion classification technique toward brain signals
Yang et al. Ground truth dataset for EEG-based emotion recognition with visual indication
Hossain et al. Emotion recognition using brian signals based on time-frequency analysis and supervised learning algorithm
Hassib Mental task classification using single-electrode brain computer interfaces
Bharti et al. An enhanced feature extraction method and classification method of EEG signals using artificial intelligence
Wan et al. Learning immersion assessment model based on multi-dimensional physiological characteristics
Yun et al. A Study on Training Data Selection Method for EEG Emotion Analysis Using Artificial Neural Network
CN117524422B (en) Evaluation system and method for improving stress recovery of human body based on indoor green planting
Alzoubi Automatic affect detection from physiological signals: Practical issues
Wan et al. Using Support Vector Machine on EEG Signals for College Students' Immersive Learning Evaluation
WO2022165832A1 (en) Method, system and brain keyboard for generating feedback in brain

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination