DE102019217447A1 - Device and method for processing and evaluating signals from brain activity - Google Patents

Abstract

Computer-implemented method (100) for processing and evaluating signals (S1) of a brain activity of a user of a technical system, the method comprising the following steps: receiving (130) magnetic signals (S1) of a brain activity of the user; Determining (140) at least one variable (P1) which represents an emotion of the user of the technical system as a function of the magnetic signals (S1).

Description

State of the art

The disclosure relates to a computer-implemented method for processing and evaluating signals of a brain activity of a user of a technical system.

The disclosure also relates to a device for processing and evaluating signals of brain activity of a user of a technical system.

From the prior art it is known to use a so-called brain-computer interface (BCI), also brain-machine interface (BMI), or brain-computer interface in German to connect man and machine. This is a special human-machine interface that can be used without activating the peripheral nervous system, e.g. B. the use of the extremities, a connection between the brain and the machine, such as a computer, enables. For this purpose, electrical signals of an activity of the brain are recorded, for example by means of electroencephalography (EEG) sensors, analyzed with the help of computers and converted into operating parameters, in particular control signals, for controlling the machine.

A brain-computer interface is based on the observation that the very idea of a behavior, referred to below as intention, triggers measurable changes in brain activity based on electrical signals. For example, the idea of moving a hand or foot leads to neural activity in the brain, particularly in the cortex, the outer layer of the cerebrum. In a training process, the brain-computer interface learns which changes in brain activity correlate with which intentions. This information can then be converted into control signals for various applications.

A brain-computer interface can for example be used in a vehicle in a driving system. On the basis of changes in the user's brain activity, it can be recognized in about up to half a second before the actual action is carried out by the user that the user would like to perform a driving action, for example turning or braking. Based on this information, the driving system can support the user of the vehicle with partially automated driving.

With the upcoming highly automated driving (HAD) or fully automated driving, direct interactions of the user with the vehicle, in particular for steering and braking the vehicle, will be rarer. As a result, intentions, ideas of behavior or activity decrease and the measurable changes in electrical brain activity triggered by them decrease.

The disclosure provides an improved method and apparatus over the prior art.

Disclosure of the invention

Preferred embodiments relate to a computer-implemented method for processing and evaluating signals of a brain activity of a user of a technical system, the method comprising the following steps: receiving magnetic signals of a brain activity of the user; Determining at least one first variable that represents an emotion of the user of the technical system as a function of the magnetic signals.

Neural activities related to emotions are triggered in a deeper region of the brain, in the limbic system, particularly in the thalamus and amygdala. The limbic system is approximately 6 cm from the top of the skull. The neural activities of the limbic system can therefore not be detected at all or only with difficulty using sensors for detecting electrical signals of brain activity, in particular EEG sensors. The neural activities from the limbic system can, however, be recorded as magnetic signals, in particular by means of magnetoencephalography (MEG) sensors.

Based on the magnetic neural activities, a variable that represents an emotion, in particular a mental state and / or the well-being of the user, can then be determined. This information can then be converted into control signals for various applications.

For example, the method can be used in a driving system, in particular for highly automated driving. If the user is afraid in a given driving situation, for example, the driving system could adapt the driving based on this information, for example slow it down or, if necessary, output information that no danger is to be expected.

According to a further preferred embodiment, the method comprises receiving electrical signals of a brain activity of the user.

According to a further preferred embodiment, the method comprises determining at least one second variable, which represents an intention of the user of the technical system, as a function of the electrical signals.

Neural activities related to intentions are triggered in the cortex. These activities can therefore be recorded with suitable sensors, in particular electroencephalography (EEG) sensors.

According to a further preferred embodiment, the method comprises: determining a first subset of the magnetic signals that is assigned to a brain activity of a first area of the brain of the user. The received magnetic signals of brain activity of the user include signals of brain activity from different areas of the brain. For example, the received magnetic signals include both signals of brain activity in the limbic system and signals of brain activity in the cortex. In order to be able to evaluate the received signals in a differentiated manner, a first subset is determined from the received signals, which, for example, can be assigned to brain activities in the cortex.

According to a further preferred embodiment, determining the first subset of the magnetic signals includes calculating the first subset of the magnetic signals as a function of the electrical signals. Using the physical relationships between electrical and magnetic fields as well as with electrical charges and electrical current under given boundary conditions, magnetic signals can be calculated based on electrical signals. These relationships are described by the Maxwell equations for electromagnetism. In neuroelectromagnetism, the frequencies are usually below 100 Hz, and the time derivatives of the electric and magnetic fields are typically much smaller than the ohmic currents, so the following quasi-static form of Maxwell's equations can be applied:

und der Ladungsdichte ρ gilt: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{E}}=\frac{\mathrm{\rho }}{\mathrm{\epsilon }\cdot {\mathrm{\epsilon }}_0}.$$

For the relationship between the divergence of the electric field $\overrightarrow{\text{E.}}$

and the charge density ρ applies: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{E.}}=\frac{\mathrm{\rho }}{\mathrm{\epsilon }\cdot {\mathrm{\epsilon }}_0}.$$

$$\overrightarrow{\nabla }\times \overrightarrow{\text{E}}=0$$

gilt, kann das elektrische $\text{Feld }\overrightarrow{\text{E}}$

als Gradient des elektrostatischen Potentials beschrieben werden: $$\overrightarrow{\text{E}}=\overrightarrow{\nabla }\mathrm{\varphi }\mathrm{.}$$

As in electrostatics for the rotation of the electric field $\overrightarrow{\text{E.}}$

$$\overrightarrow{\nabla }\times \overrightarrow{\text{E.}}=0$$

applies, can the electrical $\text{field}\overrightarrow{\text{E.}}$

can be described as the gradient of the electrostatic potential: $$\overrightarrow{\text{E.}}=\overrightarrow{\nabla }\mathrm{\varphi }\mathrm{.}$$

kann die magnetische Flussdichte beschrieben werden als $$\overrightarrow{\text{B}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A}}.$$

mit dem Vektorpotential A, das die Coulomb Eichung $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{A}}=0$$

erfüllt.Since the following applies to the divergence of the magnetic field: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{B.}}=0.$$

the magnetic flux density can be described as $$\overrightarrow{\text{B.}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A.}}.$$

with the vector potential A, which is the Coulomb calibration $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{A.}}=0$$

Fulfills.

The following applies to the relationship between the rotation of the magnetic field and the current density $$\overrightarrow{\nabla }\times \overrightarrow{\text{B.}}={\mathrm{\mu }}_0\cdot \overrightarrow{\text{J}}.$$

The following applies to the divergence of the electrical current density: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{J}}=0.$$

der Gehirnströme setzt sich zusammen aus einem primären Stromfluss $\overrightarrow{{\text{J}}_{\text{p}}}$

und einem sekundären Stromfluss $\overrightarrow{{\text{J}}_{\text{r}}},$

wobei gilt: $$\overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\overrightarrow{{\text{J}}_{\text{r}}}.$$

The current density $\overrightarrow{\text{J}}$

the brain waves are composed of a primary current flow $\overrightarrow{{\text{J}}_{\text{p}}}$

and a secondary current flow $\overrightarrow{{\text{J}}_{\text{r}}},$

where: $$\overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\overrightarrow{{\text{J}}_{\text{r}}}.$$

ist der Stromfluss der durch die Gehirnaktivität im Cortex hervorgerufen wird. Der sekundäre Stromfluss $\overrightarrow{{\text{J}}_{\text{r}}}$

ist eine Reaktion auf den primären Stromfluss und kann über das ohmsche Gesetz $$\overrightarrow{{\text{J}}_{\text{r}}.}=\mathrm{\sigma }\cdot \overrightarrow{\text{E}}$$

beschrieben und berechnet werden.The primary flow of electricity $\overrightarrow{{\text{J}}_{\text{p}}}$

is the flow of electricity caused by brain activity in the cortex. The secondary flow of electricity $\overrightarrow{{\text{J}}_{\text{r}}}$

is a reaction to the primary current flow and can be controlled by Ohm's law $$\overrightarrow{{\text{J}}_{\text{r}}.}=\mathrm{\sigma }\cdot \overrightarrow{\text{E.}}$$

can be described and calculated.

The following equations now result from the quasi-static Maxwell equations mentioned above: $$\begin{array}{c}\overrightarrow{\nabla }\cdot \left(\mathrm{\sigma }\cdot \overrightarrow{\nabla }\mathrm{\varphi }\right)=\overrightarrow{\nabla }\cdot \overrightarrow{{\text{J}}_{\text{p}}}\\ \mathrm{\Delta }\overrightarrow{\text{A.}}=-{\mathrm{\mu }}_0\cdot \overrightarrow{\text{J}}\\ \overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\mathrm{\sigma }\cdot \overrightarrow{\nabla }\mathrm{\varphi }\\ \overrightarrow{\text{B.}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A.}}.\end{array}$$

und den Parametern ε und σ, ergibt sich dann das elektrostatische Potential Φ durch Lösen der Laplace Gleichung für Φ.From the primary flow of electricity $\overrightarrow{{\text{J}}_{\text{p}}}$

and the parameters ε and σ, the electrostatic potential Φ is obtained by solving the Laplace equation for Φ.

und den Parametern ε und σ Vektorpotential A, und daraus die magnetische Flussdichte $\overrightarrow{\text{B}}$

berechnet werden.Analog can be based on the primary current flow $\overrightarrow{{\text{J}}_{\text{p}}}$

and the parameters ε and σ vector potential A, and from this the magnetic flux density $\overrightarrow{\text{B.}}$

be calculated.

As shown above, the magnetic signals and the electrical signals, described via the Maxwell equations for electromagnetism, correlate with one another, so that knowledge of the electrical signals associated with a brain activity in the first area of the user's brain can be used to determine the First determine subset of the magnetic signals of brain activity for that area of the brain. Using the measured electrical signals of the brain activity of the cortex, the magnetic signals of the brain activity of the cortex can be calculated.

According to a further preferred embodiment, the method further comprises: determining a second subset of the magnetic signals that are assigned to a brain activity in a second area of the user's brain. The second subset of the magnetic signals can be determined, for example, from the difference between the total amount of the received magnetic signals minus the first subset, in particular the calculated first subset, of the magnetic signals. The second subset of the magnetic signals, which are assigned to a brain activity of the limbic system of the brain of the user, can therefore be determined from the total amount of magnetic signals minus the calculated first subset, which is assigned to a brain activity of the cortex.

According to a further preferred embodiment, the first variable is determined as a function of the second subset of the magnetic signals that are assigned to a brain activity in the second area of the user's brain. The first variable, which represents an emotion of the user, is then determined based on the second subset of the magnetic signals that are assigned to a brain activity of the second area, that is to say of the limbic system, of the user's brain.

According to a further preferred embodiment, the second variable is determined as a function of the electrical signals of the brain activity and as a function of the first subset of the magnetic signals. The second variable represents the intention of the user of the technical system. Until now, this was calculated as a function of the electrical signals of brain activity in the cortex. As explained above, the first subset of the magnetic signals can be calculated based on the electrical signals. These signals are then advantageously also used to determine the second variable, which represents the intention of the user.

According to a further preferred embodiment, determining the first variable, which represents an emotion of the user of the technical system, includes predicting the first variable by means of a first neural network. The first neural network receives the second subset of the magnetic signals as input data. The second subset of the magnetic signals that are assigned to a brain activity in the limbic system is used to predict an emotion of the user that is related to neural activities in the limbic system.

Further preferred embodiments include a method for training the first neural network. The first neural network is trained, for example, with a suitable training data set comprising a second subset of magnetic signals and known, correlating first quantities that represent an emotion of the user.

According to a further preferred embodiment, determining the second variable, which represents an intention of the user of the technical system, includes predicting the second variable by means of a second neural network. The second neural network receives the electrical signals and / or the first subset of the magnetic signals as input data. A combination of measured electrical signals and the calculated first subset of the magnetic signals is advantageously used in order to predict an intention of the user that is related to neural activities in the cortex.

Further preferred embodiments include a method for training the second neural network. The second neural network is trained, for example, with a suitable training data set comprising electrical signals and / or a first subset of magnetic signals and known, correlating second variables that represent an intention of the user.

According to a further preferred embodiment, the determination of the first subset of the magnetic signals includes predicting the first subset of the magnetic signals by means of a third neural network.

The third neural network receives the electrical signals for predicting the first subset of the magnetic signals as input data.

Further preferred embodiments include a method for training the third neural network. The neural network is trained, for example, with a suitable training data set comprising an electrical signal and a known, correlating first subset of magnetic signals.

Further preferred embodiments relate to a device for processing and evaluating signals of a brain activity of a user of a technical system, the device being designed to carry out the method according to the embodiments.

According to a further preferred embodiment, the device comprises a brain-machine interface, BMI. Brain-machine interface, with at least one computing device and at least one memory device.

According to a further preferred embodiment, the device comprises a first neural network for predicting the first variable and / or a second neural network for predicting the second variable and / or a third neural network for predicting the first subset of the magnetic signals.

According to a further preferred embodiment, the device comprises a sensor device for capturing the electrical signals of the brain activity, in particular an electroencephalography (EEG) sensor device and / or a sensor device for capturing the magnetic signals of the brain activity, in particular a magnetoencephalography (MEG) sensor device. The EEG sensor device is suitable for being able to detect electrical signals from brain activities of the cortex, in particular via EEG electrodes. The MEG sensor device is suitable for being able to detect magnetic signals from brain activities, in particular in the deep limbic system, in particular via MEG sensors / electrodes.

Further preferred embodiments relate to a computer program, the computer program comprising computer-readable instructions which, when executed by a computer, execute a method according to the embodiments.

Further preferred embodiments relate to a computer program product, the computer program product comprising a computer-readable storage medium on which the computer program according to the embodiments is stored.

Further preferred embodiments relate to a use of a method according to the embodiments and / or a device according to the embodiments and / or a computer program according to the embodiments and / or a computer program product according to the embodiments for processing and evaluating signals of a brain activity of a user of a technical system for determining operating variables for controlling an actuator of a technical system, in particular a motor vehicle, in particular for at least partially autonomous driving, based on electrical and magnetic signals of a brain activity of a user of the technical system, or a computer or a smartphone or a robot, based on electrical and / or magnetic signals of a brain activity of a user of the technical system, in particular the motor vehicle or the computer or the smartphone or the robot ..

Further features, possible applications and advantages of the disclosure emerge from the following description of exemplary embodiments of the invention, which are shown in the figures of the drawing. All of the features described or shown form the subject matter of the disclosure by themselves or in any combination, regardless of how they are summarized in the claims or their reference and regardless of their wording or representation in the description or in the drawing.

• 1 eine schematische Darstellung eines Verfahrens zum Verarbeiten und Auswerten von Signalen einer Gehirnaktivität eines Nutzers eines technischen Systems gemäß einer Ausführungsform;
• 2 eine schematische Darstellung einer Vorrichtung zum Verarbeiten und Auswerten von Signalen einer Gehirnaktivität eines Nutzers eines technischen Systems gemäß einer Ausführungsform;
• 3 eine weitere schematische Darstellung einer Vorrichtung zum Verarbeiten und Auswerten von Signalen einer Gehirnaktivität eines Nutzers eines technischen Systems gemäß einer weiteren Ausführungsform;
• 4 eine schematische Darstellung eines ersten neuronalen Netzes zur Verwendung in einer Vorrichtung gemäß 2 oder 3;
• 5 eine schematische Darstellung eines zweiten neuronalen Netzes zur Verwendung in einer Vorrichtung gemäß 2 oder 3, und
• 6 eine schematische Darstellung eines dritten neuronalen Netzes zur Verwendung in einer Vorrichtung gemäß 2 oder 3.

In the drawing shows:

• 1 a schematic representation of a method for processing and evaluating Signals of a brain activity of a user of a technical system according to one embodiment;
• 2 a schematic representation of a device for processing and evaluating signals of a brain activity of a user of a technical system according to one embodiment;
• 3rd a further schematic representation of a device for processing and evaluating signals of a brain activity of a user of a technical system according to a further embodiment;
• 4th a schematic representation of a first neural network for use in a device according to FIG 2 or 3rd ;
• 5 a schematic representation of a second neural network for use in a device according to 2 or 3rd , and
• 6th a schematic representation of a third neural network for use in a device according to FIG 2 or 3rd .

1 show schematically a method for processing and evaluating signals of a brain activity of a user of a technical system according to one embodiment.

The procedure 100 is in its entirety with the reference number 100 designated.

The procedure 100 can with a device 200 for processing and evaluating signals of a brain activity of a user of a technical system, which are further discussed with reference to 2 and 3rd will be explained in more detail.

The procedure 100 includes the following steps.

In one step 130 of the procedure 100 magnetic signals of a brain activity of the user are received. Magnetic signals of brain activity can be recorded with suitable sensors, in particular magnetoencephalography (MEG) sensors.

Neural activities related to emotions are triggered in a deeper region of the brain, in the limbic system, particularly in the thalamus and amygdala. The limbic system is approximately 6 cm from the top of the skull. The neural activities of the limbic system can therefore not be detected at all or only with difficulty with sensors for detecting the electrical signals of the brain activity, in particular EEG sensors. However, the neural activities of the limbic system can be recorded as magnetic signals.

In one step 140 at least one first variable, which represents an emotion of the user of the technical system, is determined as a function of the magnetic signals. This information can then be converted into control signals for various applications.

According to the illustrated embodiment, in one step 110 of the procedure 100 receive electrical signals of a brain activity of the user. Electrical signals of brain activity can be recorded with suitable sensors, in particular electroencephalography (EEG) sensors. With the help of EEG sensors, signals of brain activity in the cortex, the outer layer of the cerebrum, can be detected. The neuronal activities of the cortex are mainly related to intentions.

In one step 120 at least one second variable, which represents an intention of the user of the technical system, is determined as a function of the electrical signals. This information can then be converted into control signals for various applications.

The received magnetic signals of brain activity of the user include signals of brain activity from different areas of the brain. For example, the received magnetic signals include both signals of brain activity in the limbic system and signals of brain activity in the cortex. In order to be able to evaluate the received signals in a differentiated manner, as described below, a first subset is determined from the received signals, which is assigned to brain activities of the cortex, for example, and a second subset is determined, which is assigned to brain activities of the limbic system, for example.

In one step 150 In the method, a first subset of the magnetic signals that is assigned to a brain activity of a first area of the brain of the user is determined. According to one embodiment, the first area of the brain is the cortex.

According to a further preferred embodiment, the step comprises 150 determining the first subset of the magnetic signals, calculating 150a the first subset of the magnetic signals as a function of the electrical signals. Using the physical relationships between electrical and magnetic fields as well as with electrical charges and electrical current under given boundary conditions, magnetic signals can be calculated based on electrical signals.

These relationships are explained by the Maxwell equations for electromagnetism described. In neuroelectromagnetism, the frequencies are usually below 100 Hz, and the time derivatives of the electric and magnetic fields are typically much smaller than the ohmic currents, so the quasi-static form of Maxwell's equations described below can be used.

und der Ladungsdichte ρ gilt: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{E}}=\frac{\mathrm{\rho }}{\mathrm{\epsilon }\cdot {\mathrm{\epsilon }}_0}.$$

For the relationship between the divergence of the electric field $\overrightarrow{\text{E.}}$

and the charge density ρ applies: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{E.}}=\frac{\mathrm{\rho }}{\mathrm{\epsilon }\cdot {\mathrm{\epsilon }}_0}.$$

$$\overrightarrow{\nabla }\times \overrightarrow{\text{E}}=0$$

gilt, kann das elektrische Feld $\overrightarrow{\text{E}}$

als Gradient des elektrostatischen Potentials beschrieben werden: $$\overrightarrow{\text{E}}=\overrightarrow{\nabla }\mathrm{\varphi }\mathrm{.}$$

As in electrostatics for the rotation of the electric field $\overrightarrow{\text{E.}}$

$$\overrightarrow{\nabla }\times \overrightarrow{\text{E.}}=0$$

applies, the electric field can $\overrightarrow{\text{E.}}$

can be described as the gradient of the electrostatic potential: $$\overrightarrow{\text{E.}}=\overrightarrow{\nabla }\mathrm{\varphi }\mathrm{.}$$

kann die magnetische Flussdichte beschrieben werden als $$\overrightarrow{\text{B}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A}},$$

mit dem Vektorpotential A, das die Coulomb Eichung $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{A}}=0$$

erfüllt.Since the following applies to the divergence of the magnetic field: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{B.}}=\mathrm{0,}$$

the magnetic flux density can be described as $$\overrightarrow{\text{B.}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A.}},$$

with the vector potential A, which is the Coulomb calibration $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{A.}}=0$$

Fulfills.

The following applies to the relationship between the rotation of the magnetic field and the current density $$\overrightarrow{\nabla }\times \overrightarrow{\text{B.}}={\mathrm{\mu }}_0\cdot \overrightarrow{\text{J}}.$$

The following applies to the divergence of the electrical current density: $$\overrightarrow{\nabla }\cdot \overrightarrow{\text{J}}=0.$$

der Gehirnströme setzt sich zusammen aus einem primären Stromfluss $\overrightarrow{{\text{J}}_{\text{p}}}$

und einem sekundären Stromfluss $\overrightarrow{{\text{J}}_{\text{r}}}$

$$\overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\overrightarrow{{\text{J}}_{\text{r}}}.$$

The current density $\overrightarrow{\text{J}}$

the brain waves are composed of a primary current flow $\overrightarrow{{\text{J}}_{\text{p}}}$

and a secondary current flow $\overrightarrow{{\text{J}}_{\text{r}}}$

$$\overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\overrightarrow{{\text{J}}_{\text{r}}}.$$

ist der Stromfluss der durch die Gehirnaktivität im Cortex hervorgerufen wird. Der sekundäre Stromfluss $\overrightarrow{{\text{J}}_{\text{r}}}$

ist eine Reaktion auf den primären Stromfluss und kann durch das ohmsche Gesetz $$\overrightarrow{{\text{J}}_{\text{r}}.}=\mathrm{\sigma }\cdot \overrightarrow{\text{E}}$$

beschrieben werden.The primary flow of electricity $\overrightarrow{{\text{J}}_{\text{p}}}$

is the flow of electricity caused by brain activity in the cortex. The secondary flow of electricity $\overrightarrow{{\text{J}}_{\text{r}}}$

is a reaction to the primary current flow and can be determined by Ohm's law $$\overrightarrow{{\text{J}}_{\text{r}}.}=\mathrm{\sigma }\cdot \overrightarrow{\text{E.}}$$

to be discribed.

As a result, the quasi-static Maxwell equations mentioned above result in the following equations, which can be solved using the measured electrical and / or magnetic signals and known parameters: $$\begin{array}{c}\overrightarrow{\nabla }\cdot \left(\mathrm{\sigma }\cdot \overrightarrow{\nabla }\mathrm{\varphi }\right)=\overrightarrow{\nabla }\cdot \overrightarrow{{\text{J}}_{\text{p}}}\\ \mathrm{\Delta }\overrightarrow{\text{A.}}=-{\mathrm{\mu }}_0\cdot \overrightarrow{\text{J}}\\ \overrightarrow{\text{J}}=\overrightarrow{{\text{J}}_{\text{p}}}+\mathrm{\sigma }\cdot \overrightarrow{\nabla }\mathrm{\varphi }\\ \overrightarrow{\text{B.}}=\overrightarrow{\nabla }\times \overrightarrow{\text{A.}}.\end{array}$$

und den Parametern ε und σ, ergibt sich dann das elektrostatische Potential Φ durch Lösen der Laplace Gleichung für Φ.From the primary flow of electricity $\overrightarrow{{\text{J}}_{\text{p}}}$

and the parameters ε and σ, the electrostatic potential Φ is obtained by solving the Laplace equation for Φ.

und den Parametern ε und σ das Vektorpotential $\overrightarrow{\text{A}},$

und daraus die magnetische Flussdichte $\overrightarrow{\text{B}}$

berechnet werden.Analog can be based on the primary current flow $\overrightarrow{{\text{J}}_{\text{p}}}$

and the parameters ε and σ the vector potential $\overrightarrow{\text{A.}},$

and from this the magnetic flux density $\overrightarrow{\text{B.}}$

be calculated.

As shown above, the magnetic signals and the electrical signals, described via the Maxwell equations for electromagnetism, correlate with one another, so that knowledge of the electrical signals associated with a brain activity in the first area of the user's brain can be used to determine the First determine subset of the magnetic signals of brain activity for that area of the brain. Based on the measured electrical signals of the brain activity of the cortex, the magnetic signals of the brain activity of the cortex are calculated.

According to a further preferred embodiment, the method further comprises: determining 160 a second subset of the magnetic signals that are assigned to a brain activity of a second area of the brain of the user. The second subset of the magnetic signals can be determined 160, for example, from the difference between the total amount of received magnetic signals minus the first subset, in particular the calculated first subset, of the magnetic signals Assigned to the brain of the user, can therefore be determined from the total amount of magnetic signals minus the calculated first subset, which are assigned to a brain activity of the cortex.

According to a further preferred embodiment, the determination takes place 140 the first variable as a function of the second subset of the magnetic signals that are assigned to a brain activity of the second area of the brain of the user. The first variable, which represents an emotion of the user, is therefore determined based on the second subset of the magnetic signals that are assigned to a brain activity of the second area, that is to say of the limbic system, of the brain of the user.

According to a further preferred embodiment, the determination takes place 120 the second variable as a function of the electrical signals of the brain activity and as a function of the first subset of the magnetic signals. The second variable represents the intention of the user of the technical system. Up to now, this was calculated as a function of the electrical signals of brain activity in the cortex. As explained above, the first subset of the magnetic signals can be calculated based on the electrical signals. This subset is then also advantageously used to determine the second variable, which represents the intention of the user.

According to a further preferred embodiment, the determination comprises 140 the first size P1 that represents an emotion of the user of the technical system, the prediction 180 the first size P1 by means of a first neural network NN1 , which is referred to below with reference to 4th is explained.

According to a further preferred embodiment, the determination comprises 120 the second size P2 , which represents an intention of the user of the technical system, the prediction 170 the second size P2 by means of a second neural network NN2 , hereinafter and with reference to 5 is explained.

According to a further preferred embodiment, the determination comprises 150 the first subset M1 of magnetic signals S1 the prediction 190 the first subset M1 the magnetic signals by means of a third neural network NN3 , which is referred to below with reference to 6th is explained.

2 shows the device 200 for processing and evaluating signals of brain activity of a user of a technical system, the device 200 is trained, the procedure 100 perform according to the embodiments.

According to the illustrated embodiment, the device comprises 200 a brain-machine interface, BMI, engl. Brain-machine interface, with at least one computing device 210 and at least one storage device 220 . The storage device 220 is for at least temporary storage of at least one computer program PRG1 (and / or of data) for the computing device 210 educated.

For example, the first computer program PRG1 the implementation of the above by way of example with reference to 1 described procedure 100 through the device 200 or the computing device 210 Taxes.

The device further comprises an input 230 for receiving magnetic signals S1 and optionally an entrance 240 for receiving electrical signals S2 .

The signals S2 and or S1 can at least temporarily in the storage device 220 get saved.

According to the embodiment shown, the device comprises at least one output 250 for outputting an operating variable and / or for outputting the first and / or second variable P1 , P2 , based on the first and / or second size P1 , P2 in addition, the operating variable for controlling an actuator of a technical system (not shown) can be calculated.

3rd shows the device 200 in a further schematic representation.

The device 200 optionally includes an EEG sensor device 260 for acquiring the electrical signals S2 brain activity, especially an electroencephalography (EEG) sensor device 260 . The EEG sensor device 260 includes several EEG electrodes 270 that measure the electrical potential on a skull surface of a user of the system. The EEG electrodes 270 come with an EEG detection unit 280 connected, by means of which signals can be collected, amplified, shaped, filtered and temporarily stored.

The device 200 further comprises a MEG sensor device 290 for capturing the magnetic signals S1 brain activity, especially a magnetoencephalography (MEG) sensor device 290 . The MEG sensor device 290 includes multiple MEG sensors / electrodes 300 that measure the magnetic flux density created by the electrical activity of neurons in the brain. The MEG sensors / electrodes 300 are suitable for magnetic signals S1 to be able to detect brain activity in the deep limbic system. The MEG sensors / electrodes 300 are with a MEG detection unit 310 connected, by means of which signals can be collected, amplified, shaped, filtered and temporarily stored.

Next is a 3rd schematically the head of a user of the device 200 The MEG and EEG electrodes are shown in the vicinity of the person's skull, for example directly on the head or integrated into a helmet or a similar structure 270 , 300 are provided.

According to the illustrated embodiment, the device comprises 200 a first neural network NN1 to predict the first size P1 and a second neural network NN2 to predict the second quantity P2 and a third neural network NN3 to predict the first subset M1 of magnetic signals.

In general, there is a neural network NN1 , NN2 , NN3 from a sequence of linear and non-linear layers, such as activation functions, pooling layers, normalization layers. Generally includes a neural network NN1 , NN2 , NN3 a large number of parameters, for example weights of linear layers, which are used during training of the neural network NN1 , NN2 , NN3 adjusted to minimize a prediction error.

It can prove beneficial when it comes to neural networks NN1 , NN2 , NN3 It is about deep neural networks, whereby a deep neural network is understood to mean the use of several such layers in a neural network. Deep neural networks have proven to be efficient, among other things, with regard to minimizing a prediction error, even for highly non-linear functions.

The EEG detection unit 280 is with a device for determining 150 a first subset M1 of magnetic signals S1 associated with a brain activity of a first area of the brain of the user. According to the embodiment shown, the device is the third neural network NN3 to predict the first subset M1 of magnetic signals.

The third neural network NN3 receives the electrical signals as input data S2 from the EEG detection unit 280 to predict the first subset M1 of magnetic signals.

The third neural network is trained to make use of the electrical signals S2 predict magnetic signals. As the received electrical signals S2 originate from the cortex, the predicted first subset refers M1 the magnetic signals to magnetic signals of the cortex.

Further preferred embodiments include a method for training the third neural network NN3 . The third neural network NN3 for example, with a suitable training data set comprising an electrical signal S2 and a known first subset correlating therewith M1 on magnetic signals, trained, cf. 6th .

The output of the third neural network NN3 and the MEG detection unit 310 are with a function block 320 to determine 160 a second subset M2 the magnetic signals that are assigned to brain activity in a second area of the user's brain. The second subset M2 of magnetic signals S1 can be determined, for example, from the difference in the total amount of the magnetic signals received with the MEG detection unit S1 less calculated first partial amount M1 of magnetic signals determine 160. The second subset M2 the magnetic signals, which are assigned to a brain activity in the limbic system of the user's brain, can therefore be derived from the entire set of magnetic signals S1 minus the calculated first partial amount M1 which are assigned to a brain activity of the cortex can be determined. The output of the function block 320 thus provides the magnetic signals that come from the lower limbic system of the user's brain.

An output of the function block 320 is with a device for determining 140 the first size P1 that represents an emotion of the user of the technical system, in particular with a first neural network NN1 to predict 180 the first size P1 connected.

The first neural network NN1 receives the second subset as input data M2 of magnetic signals. So it becomes the second subset M2 uses the magnetic signals associated with brain activity in the limbic system to determine the first size P1 that represent an emotion of the user that is related to neural activity in the limbic system. Based on the first size P1 an operating variable for controlling an actuator of a technical system can then be determined. The actuator is schematically through the block 330 shown.

Further preferred embodiments include a method for training the first neural network NN1 . The first neural network NN1 is, for example, with a suitable training data set comprising a second subset M2 of magnetic signals and known, correlating first quantities P1 , which represent an emotion of the user, trained, cf. 4th .

The device further comprises 200 means for determining 120 the second size P2 that represents an intention of the user of the technical system. The device is, for example, the second neural network NN2 to predict 170 the second size P2 . The device receives both the electrical signals as input data S2 so also the first subset M1 of magnetic signals S1 from the facility, in particular the third neural network NN3 to predict the first subset M1 of magnetic signals.

According to the embodiment shown, a combination of measured electrical signals is therefore used S2 and the calculated first subset M1 the magnetic signals used to make the second size P2 that represents an intention of the user to predict, which is related to neural activity in the cortex. Based on the second size P2 an operating variable for controlling the actuator can then be determined.

Further preferred embodiments include a method for training the second neural network NN2 . The second neural network NN2 is, for example, with a suitable training data set comprising electrical signals S2 and / or a first subset M1 of magnetic signals and known, correlated second quantities P2 , which represent an intention of the user, trained, cf. 5 .

Further preferred embodiments relate to a computer program PRG1 wherein the computer program comprises computer-readable instructions, when executed by a computer, a method according to the embodiments is carried out.

Further preferred embodiments relate to a computer program product, the computer program product comprising a computer-readable storage medium on which the computer program according to the embodiments is stored.

Further preferred embodiments relate to the use of a method 100 according to the embodiments and / or a device 200 according to the embodiments and / or a computer program PRG1 according to the embodiments and / or a computer program product according to the embodiments for processing and evaluating signals S2 , S1 brain activity of a user of a technical system for determining operating variables for controlling an actuator of a technical system, in particular a motor vehicle, in particular for at least partially autonomous driving, based on electrical and magnetic signals S1 , S2 a brain activity of a user of the technical system, or a computer or a smartphone or a robot, based on electrical and / or magnetic signals S2 , S1 brain activity of a user of the technical system, in particular of the motor vehicle or the computer or the smartphone or the robot.

Claims (18)

Computer-implemented method (100) for processing and evaluating signals (S2, S1) of brain activity of a user of a technical system, the method comprising the following steps: Receiving (130) magnetic signals (S1) of a brain activity of the user; Determining (140) at least one variable (P1) which represents an emotion of the user of the technical system as a function of the magnetic signals (S1). Method (100) according to Claim 1 wherein the method further comprises: receiving (110) electrical signals (S2) of a brain activity of the user. Method (100) according to Claim 2 wherein the method further comprises: determining (120) at least one second variable (P2), which represents an intention of the user of the technical system, as a function of the electrical signals (S2). The method (100) according to at least one of the preceding claims, wherein the method (100) further comprises: determining (150) a first subset (M1) of the magnetic signals (S1) which are assigned to a brain activity of a first area of the brain of the user. Method (100) according to Claim 2 wherein determining (150) the first subset (M1) of the magnetic signals (S1) comprises calculating (150a) the first subset (M1) of the magnetic signals (S1) as a function of the electrical signals (S2). The method (100) according to at least one of the preceding claims, wherein the method (100) further comprises: determining (160) a second subset (M2) of the magnetic signals (S1) which are assigned to a brain activity of a second area of the brain of the user. The method (100) according to at least one of the preceding claims, wherein the determination (140) of the first variable (P1) as a function of the second subset (M2) of the magnetic signals (S1) which are assigned to a brain activity of the second area of the brain of the user , he follows. Method (100) according to at least one of the preceding claims, wherein the determination (120) of the second variable (P2) takes place as a function of the electrical signals (S2) of the brain activity and as a function of the first subset (M1) of the magnetic signals (S1). The method (100) according to at least one of the preceding claims, wherein the determination (140) of the variable (P1) which represents an emotion of the user of the technical system, the prediction (180) of the first variable (P1) by means of a first neural network ( NN1) includes. The method (100) according to at least one of the preceding claims, wherein the determination (120) of the second variable (P2), which represents an intention of the user of the technical system, the prediction (170) of the second variable (P2) by means of a second neural network (NN2) includes. The method (100) according to at least one of the preceding claims, wherein the determination (150) of the first subset (M1) of the magnetic signals (S1) is the prediction (190) of the first subset (M1) of the magnetic signals (S1) by means of a third neural Network (NN3). Device (200) for processing and evaluating signals (S2, S1) of brain activity of a user of a technical system, the device being designed to carry out the method (100) according to one of the Claims 1 to 11 to execute. Device (200) according to Claim 10 , the device having a brain-machine interface, BMI, engl. Brain-machine interface, with at least one computing device (210) and at least one memory device (220). Device (200) according to at least one of the Claims 10 or 11 , wherein the device has a first neural network (NN1) for predicting the first variable (P1) and / or a second neural network (NN2) for predicting the second variable (P2) and / or a third neural network (NN3) for predicting the comprises the first subset (M1) of the magnetic signals (S1). Device (200) according to at least one of the Claims 12 to 14th , wherein the device comprises a sensor device (260) for detecting the electrical signals (S2) of the brain activity, in particular an electroencephalography (EEG) sensor device (260), and / or a sensor device (290) for detecting the magnetic signals (S1) of the brain activity , in particular a magnetoencephalography (MEG) sensor device (290). Computer program (PRG1), the computer program comprising computer-readable instructions when they are executed by a computer, a method (200) according to one of the Claims 1 to 11 is performed. Computer program product, the computer program product comprising a computer-readable storage medium on which the computer program (PRG1) is based Claim 16 is stored. Use of a method (100) according to at least one of Claims 1 to 11 and / or a device (200) according to at least one of Claims 12 to 15th and / or a computer program (PRG1) Claim 16 and / or a computer program product Claim 17 for processing and evaluating signals of brain activity of a user of a technical system for determining operating parameters for controlling an actuator of a technical system, in particular a motor vehicle, in particular for at least partially autonomous driving, or a computer or a smartphone or a robot, based on electrical and / or magnetic signals (S2, S1) of a brain activity of a user of the technical system, in particular the motor vehicle or the computer or the smartphone or the robot.

Images