KR20200063337A - Apparatus and method for learning user's operation intention using artificial neural network - Google Patents

Abstract

A method for learning a user′s intention to use an artificial neural network includes the steps of: receiving information on a signal measuring a user′s motion from a motion sensor unit; outputting a user′s motion intention corresponding to information on a signal measuring the user′s movement using a pre-trained artificial neural network; receiving information on the EEG signal measured for the user from the EEG sensor unit; and modifying, when the user′s operating intention corresponding to the information on the EEG signal does not match the output user′s operating intention, the label of the output intention of the user to the motion intention of the user corresponding to the information on the EEG signal and inputting it into the pre-trained artificial neural network.

Description

Apparatus and method for learning user's operation intention using artificial neural network}

The present invention relates to a method and apparatus for learning a user's intention to operate using an artificial neural network.

As the market for artificial intelligence (e.g., self-driving cars) grows, it is expected that the disabled and the elderly who could not drive will be able to use the cars. However, in the case of such a handicapped person, the body is not free, and there is a difficulty in that it is not possible to operate the vehicle internal functions freely. In the case of the disabled or the elderly, noise due to movement is expected to be large due to physical unnaturalness.

It is expected that the performance of the conventional trained model will deteriorate due to such noise. In the present invention, an apparatus for continuously learning the performance of a trained model is proposed to prevent a user who is not free of body from having difficulty in manipulating and using a device (for example, a vehicle) internal functions at will.

The technical problem to be achieved in the present invention is to provide a method for learning a user's motion intention using an artificial neural network.

Another technical object to be achieved in the present invention is to provide a device for learning a user's motion intention using an artificial neural network.

Another technical object to be achieved in the present invention is to provide a computer-readable recording medium that records a program for executing a method for learning a user's operating intention using an artificial neural network.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description. Will be able to.

In order to achieve the above technical problem, a method of learning a user's motion intention using an artificial neural network includes receiving information on a signal measuring a user's motion from a motion sensor unit; Outputting a user's operating intention corresponding to information on a signal measuring the user's movement using a pre-trained artificial neural network; Receiving information on the EEG signal measured for the user from the EEG sensor unit; And if the user's intention to operate corresponding to the information on the EEG signal does not match the user's intention to output, the label for the user's intention to operate corresponds to the information on the EEG signal. It may include the step of inputting the pre-trained artificial neural network by modifying it with an intention to operate.

The method may further include learning to recognize the label of the output intention of the user as the user's intention corresponding to information on the EEG signal.

In the method, the user's intention to operate corresponding to the information on the EEG signal and the information on the EEG signal indicates whether the user's intention to operate corresponding to the information on the EEG signal corresponds to the output intention of the user. The method may further include determining based on a predefined mapping relationship between a user's motion intention corresponding to information on a signal measuring the user's motion and the output user's motion intention. The predefined mapping relationship may include a relationship corresponding to each row in the following table.

Here, the first, second, and third frequencies are characterized not in multiple relations with each other. The measured EEG signal may correspond to a signal caused by a flashing stimulus.

The step of outputting the user's motion intention is performed when the signal measured for the user's movement is determined to be greater than or equal to a predefined feature value, and the predefined feature value may include an average signal strength of the EMG signal. . The artificial neural network may include deep neural networks (DNN). The motion sensor unit and the EEG sensor unit may be characterized in that the user is portable.

In order to achieve the above other technical problems, an apparatus for learning a user's motion intention using an artificial neural network includes a communication unit that receives information on a signal measuring a user's movement from a motion sensor unit; And a processor that outputs a user's intention to operate corresponding to information on a signal measuring the movement of the user using a pre-trained artificial neural network, wherein the communication unit measures the EEG signal measured for the user from the EEG sensor unit. When the user's operating intention corresponding to the information on the EEG signal does not match the output user's operating intention, the processor receives the label of the output user's operating intention in the EEG signal. It is modified to the user's intention to operate corresponding to the information on and input to the previously learned artificial neural network.

The processor may learn to recognize the label of the output intention of the user as the user's intention corresponding to the information on the EEG signal.

The processor determines whether the user's operating intention corresponding to the information on the EEG signal coincides with the output user's operating intention, the information on the EEG signal, and the user's operating intention on the EEG signal. The user's motion may be determined based on a predefined mapping relationship between the user's intentions to operate corresponding to information on the measured signal.

The measured EEG signal may correspond to a signal caused by a flashing stimulus. The processor outputs a user's intention to operate corresponding to the signal measured for the user's movement when the signal measured for the user's movement is determined to be greater than or equal to a predefined feature value. The predefined feature value includes the average signal strength of the EMG signal. The artificial neural network includes Deep Neural Networks (DNN).

The motion sensor unit and the EEG sensor unit are characterized in that the user can wear the portable. The EEG sensor part includes an Electroencephalography (EEG) sensor. The motion sensor unit includes an EMG sensor unit or an acceleration sensor unit. The signal measured for the user's movement includes the user's EMG signal or acceleration signal.

A method of learning a user's motion intention according to an embodiment of the present invention can reduce an error occurring when a device is operated using a machine learning algorithm model for recognizing a user's motion intention.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. will be.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention and describe the technical spirit of the present invention together with the detailed description.
1 is a view showing a 10-20 EEG system as an example of an EEG sensor system.
2 is a diagram illustrating a layer structure of an artificial neural network.
3 is a diagram illustrating a block diagram of a learning device for learning a user's intention to operate using an artificial neural network according to the present invention.
4 is a flowchart briefly illustrating a method of learning a user's operating intention using an artificial neural network according to the present invention.
5 is a diagram illustrating a system including an apparatus 300 for learning a user's intention to operate using an artificial neural network according to the present invention.
6 shows a graph of the EEG detected by the EEG sensor 520 when the user 500 watches a 15 Hz stimulus image.
7 is a flowchart specifically illustrating a method of learning a user's operating intention using an artificial neural network according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION The following detailed description, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention, and is not intended to represent the only embodiments in which the present invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, one skilled in the art knows that the present invention may be practiced without these specific details.

In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted, or block diagrams centered on the core functions of each structure and device may be illustrated. In addition, the same components throughout the specification will be described using the same reference numerals.

The present invention is to propose a method for learning a user's motion intention using an artificial neural network and a learning device therefor. The learning device is an EEG from the EEG sensor unit and the motion sensor unit (for example, an EMG sensor unit or an acceleration sensor unit, etc., which are exemplarily described as EMG sensor units for convenience of explanation). By receiving signals and EMG signals, the user's operating intention is learned using an artificial neural network. Before explaining the present invention, the EEG sensor unit and the motion sensor unit will be briefly described.

The EEG sensor unit (eg, an electroencephalography (EEG) sensor) measures the flow of electricity generated when a signal is transmitted from the nervous system to the brain nerve. A neurophysiological measurement method for electrical activity of the brain is a method of recording through an electrode attached to the scalp.

1 is a view showing a 10-20 EEG system as an example of an EEG sensor system.

In order to measure EEG, the electrode position on the scalp is required. As a system for the measurement method, a 10-20 EEG system can be used. The 10-20 EEG system is the same as that shown in FIG. 1 and is named descriptive. 10 and 20 of "10-20" refer to 10% and 20% of the distance between NASION (anterior) and INION (posterior).

In the channel name shown in FIG. 1, the odd number always refers to the electrode position on the left side of the head, and the even number refers to the electrode position on the right side of the head. The letter "z" is used to indicate a point between the center or the centerline between NASION (Nz) and INION (Iz). F, C, P, O, and T are initials for the brain region, F is the frontal lobe, C is the central nerve, P is the temporal lobe, O is the occipital lobe, and T is the Temporal lobe.

In FIG. 1, NASION refers to a small notch under the forehead and above the nose, and INION refers to a small bump or bump on the base of the neck and occiput. The distance between these two points can be measured using a tape measure, etc. It is accurate to measure in centimeters, and a typical measurement is 36 cm.

In NASION, 10% above this distance is the midpoint between two prefrontal lobe sites called FP1 and FP2. In addition, Fz is the place that went 20% more between NASION and INION. If NASION and INION move 20% more in Fz, it is Cz, the middle point between NASION and INION.

As an example, the brain waves are artificially delta (δ) waves (0.2 to 3.99 Hz), theta (θ) waves (4 to 7.99 Hz), and alpha (α) waves (8 to 12.99 Hz) depending on the range of oscillating frequencies. , Beta β wave (13 ~ 29.99 Hz), gamma g wave (30 ~ 50 Hz). Delta waves are predominant in deep sleep of normal people or in newborns. Theta waves are waves that appear mainly in the process of emotional stability or sleep, and are related to various states such as memory, creativity, and concentration. Alpha waves appear mainly in a relaxed state, such as tension relaxation, and the amplitude increases as they become stable and comfortable. Beta waves are common in the frontal tofu, and appear in all conscious activities, such as when awake and speaking. In particular, it appears predominantly in anxiety, tension, and complicated calculations. Gamma waves vibrate faster than beta waves, and are more emotionally irritated, or are associated with advanced cognitive information processing such as reasoning and judgment.

1 illustrates an EEG measurement channel measured by the EEG sensor of the wearable device according to the present invention. Four Ep1, A1, A2, and Cz can be used as EEG channels, EEG1 among the EEG sensors measures sleepiness and concentration in the Fp1 channel, and EEG 2 measures sleepiness in the Cz channel (EEG2). A1 channel is used as REF (Reference) and A2 channel is used as GND (Ground).

The following are examples that can be characteristic values of EEG signals.

1. Mean: Indicates the average signal strength of the EEG signal.

2. Standard deviation: It shows the standard deviation of EEG signal strength.

3. Zero Crossing: It shows the number of zero crossings when the EEG signal is secondly differentiated.

4. Kurtosis: This is a value obtained by normalizing the fourth-order momentum of the signal to the square of the standard deviation, and represents a value for determining the peakedness of the probability distribution of signal strength.

5. Crest factor: The ratio of the peak value of the signal. It is similar to Kurtosis and is a characteristic value that describes the signal's Impluseness.

6. Power Spectrum Density: Shows the power spectrum density generated through Fourier transform of EEG signal.

7. Correlation: It represents auto-correlation of EEG signal.

8. Threshold Crossing: Shows the number of filtered raw data exceeding a certain threshold in one epoch.

9. Skewness: As the third momentum of the EEG signal, it indicates the degree to which the distribution of signal strength is biased.

10. Entropy: indicates the predictability of the signal.

11. Band Energy: Represents the energy in the frequency domain related to stress.

12. Spectral Flux: means the rate of change of the power spectrum over time when analyzing the frequency of a signal (PSD(t)/PSD(t-1))

Next, the EMG sensor unit will be described.

The EMG sensor unit may be a wearable device and may be worn on a user's wrist or arm. The EMG sensor unit may sense the user's biosignal and extract or obtain data on the EMG signal. The EMG sensor unit may include an electromyography (EMG) sensor. At this time, preferably, in the case of EMG, it is possible to receive and proceed with sampling at a minimum of 400 Hz per second. The EMG sensor unit may transmit the calculated or extracted EMG signal data or information to the learning apparatus of the present invention. At this time, the EMG sensor unit may convert the extracted EMG signal data into digital signal data and transmit the converted EMG signal to the learning device of the present invention.

EMG The extraction value from the signal or Feature value

1. Mean: Indicates the average signal strength of the EMG signal.

2. Standard deviation: The standard deviation of the EMG signal strength.

3. Zero Crossing: This is the number of zero crossings when the EMG signal is secondly differentiated.

4. Kurtosis: This is the value obtained by normalizing the fourth order momentum of the signal to the square of the standard deviation.

5. Crest factor: The ratio of the peak value of the signal. It is similar to Kurtosis and is a characteristic value that describes the signal's Impluseness.

6. Power Spectrum Density: Shows the power spectrum density generated through the Fourier transform of the EMG signal.

7. Correlation: It represents the auto-correlation of the EMG signal.

8. Threshold Crossing: Shows the number of filtered raw data exceeding a certain threshold in one epoch.

9. Skewness: As the third momentum of the EMG signal, it indicates the degree to which the distribution of signal strength is biased.

10. Entropy: indicates the predictability of the signal.

11. Band Energy: It represents the energy in the frequency domain related to concentration.

12. Spectral Flux: means the rate of change of the power spectrum over time when analyzing the signal's frequency (PSD(t)/PSD(t-1))

Hereinafter, a method of learning a user's intention to operate using the artificial neural network according to the present invention will be briefly described in terms of deep learning, artificial neural network, and deep neural network.

Deep learning is a type of artificial neural network (ANN) using the theory of the human neural network (Neural Network).It is composed of a layer structure, and it is one between the input layer and the output layer. It is a set of machine learning models or algorithms that refer to Deep Neural Networks (DNNs) that have the above hidden layer (hereinafter referred to as the middle layer). In short, deep learning is an artificial neural network with deep layers.

It is estimated that the human brain is composed of 25 billion neurons. The brain is composed of neurons, and each neuron (neuron) refers to one neuron constituting a neural network. Neurons contain one cell body and one axon (axon or nurite), which is a process of the cell body, and usually several dendrite or protoplasmic processes. The exchange of information between these neurons is transmitted through a junction between neurons called synapses. It is very simple to separate only one nerve cell, but when these nerve cells are gathered, they can have human intelligence. The dendrites are the parts that receive signals from other neurons (Input), and the axons are the parts that extend very long from the cell body (Output). There is a connection called a synapse that connects between the axons that transmit signals between neurons and the dendrite, but does not transmit signals from nerve cells unconditionally, but only when the signal strength is above a certain value (threshold). Is to do. That is, not only is the connection strength different for each synapse, but it is also determined whether or not to transmit a signal.

An artificial neural network (ANN), an area of artificial intelligence, is a mathematical model modeled after mimicking the brain structure (neural network) of biology (usually human). That is, the artificial neural network is implemented by imitating the information processing and delivery process of these biological neurons. It is implemented similarly to the way the human brain solves problems, and the neural network has excellent parallelism because each neuron operates independently. In addition, because information is distributed over many connecting lines, even if a problem occurs in some neurons, it does not affect the whole, so it is resistant to certain levels of errors and has the ability to learn about a given environment.

Deep neural networks are considered to be descendants of artificial neural networks, and are the latest version of artificial neural networks, with success stories in areas where many artificial intelligence technologies have failed in the past beyond the existing limits. Looking at the modeling of an artificial neural network by imitating a biological neural network, in terms of processing unit, biological neurons are nodes, connections are synapse, and weights are next. Modeled as in Table 1.

Biological neural network Artificial neural network Cell body Node Dendrites Input Axon Output Synapse Weight

2 is a diagram illustrating a layer structure of an artificial neural network.

As in the case of artificial neural networks, individual neurons are connected to each other through synapses so that multiple layers are connected to each other. It can be updated by weight. As such, it is used in fields for learning and cognition due to its multi-layer structure and connection strength.

Each node is connected by weighted links, and the whole model learns by repeatedly adjusting the weights. The weight represents the importance of each node as a basic means for long-term memory. Simply put, the artificial neural network trains the entire model by initializing these weights and updating and adjusting the weights with a set of data to train. After the training is completed, if a new input value comes in, the proper output value is inferred. The learning principle of the artificial neural network can be regarded as a process in which intelligence is formed from the generalization of experience and is done in a bottom-up manner. In FIG. 2, when the number of intermediate layers is 2 or more (ie, 5 to 10), the layer is considered to be deep and is referred to as a deep neural netowkr. have.

Artificial neural networks can play a certain role even if they have one intermediate layer (usually referred to as a'hidden layer') except for input and output, but if the complexity of the problem increases, the number or number of nodes The number should be increased. Among them, it is effective to increase the number of layers to take a multi-layered structure model, but the effective range is limited due to the limitation that efficient learning is impossible and there is a large amount of computation to train the network.

However, as the above limitations were overcome, the artificial neural network was able to take a deep structure. As a result, it is possible to build a complex and highly expressive model, resulting in breakthrough results in various fields such as voice recognition, face recognition, object recognition, and text recognition.

Deep Neural Network (DNN) is an artificial neural network (ANN) composed of several hidden layers between an input layer and an output layer. Deep neural networks can model complex non-linear relationships just like a normal artificial neural network. For example, in a deep neural network structure for an object identification model, each object may be represented by a hierarchical configuration of basic elements of an image. At this time, the additional layers can aggregate features of the lower layers that are gradually collected. This feature of the deep neural network allows modeling of complex data with fewer units (nodes) than a similarly performed artificial neural network.

Previous deep neural networks have usually been designed as forward-feeding neural networks, but recent studies have successfully applied deep learning structures to recurrent neural networks (RNNs). One example is the application of deep neural network structures to the field of language modeling. The Convolutional Neural Network (CNN) has not only been well applied in the field of computer vision, but has also been well documented for each successful application. More recently, the convolutional neural network has been applied to the field of acoustic modeling for Automatic Speech Recognition (ASR), and is evaluated to have been applied more successfully than existing models. Deep neural networks can be trained using standard error back propagation algorithms. At this time, the weights can be updated through stochastic gradient descent using the following equation.

3 is a diagram illustrating a block diagram of a learning device for learning a user's intention to operate using an artificial neural network according to the present invention.

Referring to FIG. 3, a learning device 300 that learns a user's intention using an artificial neural network may include a communication unit 310 and a processor 320. The communication unit 310 and the processor 320 are electrically connected in the learning device 300 for learning the user's intention to operate.

The communication unit 310 receives a signal measuring a user's movement from the motion sensor unit through a wireless communication method. The processor 320 may output (or estimate, determine) a user's operating intention corresponding to a signal measuring a user's movement using a pre-trained artificial neural network. In addition, when the communication unit 310 receives information on the EEG signal measured for the user from the EEG sensor unit, the processor 320 displays a user's intention to operate corresponding to the EEG signal information of the output user. It is determined whether it does not match the intention of operation. If they do not match, the processor 320 corrects the output user's action intention label (or correct answer) to the user's action intention corresponding to the information on the EEG signal and inputs it to the pre-trained artificial neural network.

The processor 320 learns to modify and recognize a label (or a correct answer) of a user's operating intention output from a pre-trained artificial neural network as a user's operating intention corresponding to information on the EEG signal. The processor 320 determines whether the user's intention to operate corresponding to the information on the EEG signal coincides with the output user's intention to operate, the user's operation corresponding to the EEG signal information, and the EEG signal information. It can be determined based on a predefined mapping relationship between an intention, a user's motion intention corresponding to information on a signal measuring the user's motion, and the output user's motion intention. Here, the predefined mapping relationship may include a relationship corresponding to each row in Table 2 below.

Frequency of the EEG signal received from the EEG sensor User's motion intention estimated from EEG signal Information about a signal that measures the movement of the user received from a motion sensor unit (eg, a near-low sensor unit, an acceleration sensor unit, etc.) User's intention to output through the already learned artificial neural network 1st frequency A intended operation First movement reaction A intended operation 2nd frequency Intended to move B 2nd movement reaction Intended to move B 3rd frequency C intended operation Third movement reaction C intended operation

Referring to Table 2, the predefined mapping relationship or rule is that the user's intention to operate is that the frequency of the EEG signal received from the EEG sensor unit corresponds to the first frequency and the first frequency is A operation, and in this case, from the motion sensor unit The information on the signal measuring the motion of the received user is the first motion response, and the motion intention of the user corresponding to the first motion is also the A motion. For example, referring to Table 2, it is assumed that the processor 320 outputs the B intention as the user's motion intention through a pre-trained artificial neural network. In this case, if the frequency of the EEG signal received from the communication unit 310 by the processor 320 is the first frequency, the operation intention of the user corresponding to the first frequency is the A operation, so it coincides with the output B operation intention of the user. You can judge not to. Then, the processor 320 determines that there is an error in the user's motion intention output through the pre-trained artificial neural network, and the user's label (or correct answer) of the motion intention corresponds to the information on the EEG signal. It is corrected to the operation intention (operation B) and input to the pre-trained artificial neural network. As described above, by generating a flashing stimulus, when the user looks at the stimulus, the brainwave generated in the user's brain is monitored to provide an answer to the learning system, and the learning system proposes incremental learning using this answer.

Here, the measured EEG signal may correspond to a signal caused by a flashing stimulus. The processor 320 may determine a user's intention to operate corresponding to the signal measured for the user's movement when the signal measured for the user's movement is determined to be greater than or equal to a predefined feature value. Artificial neural networks may include deep neural networks (DNNs). In addition, the EEG sensor unit may include an Electroencephalography (EEG) sensor. The motion sensor unit may include an EMG sensor unit or an acceleration sensor unit. The signal measured for the user's movement may include the user's EMG signal or acceleration signal. Hereinafter, for convenience of description, the motion sensor unit is assumed to be an EMG sensor unit.

4 is a flowchart briefly illustrating a method of learning a user's operating intention using an artificial neural network according to the present invention.

Referring to FIG. 4, an apparatus 300 for learning a user's intention using an artificial neural network receives information on a signal measuring a user's movement from a motion sensor unit. The apparatus 300 for learning the user's operating intention using the artificial neural network outputs the user's operating intention corresponding to information on the signal measured by the user's movement using the pre-trained artificial neural network. The apparatus 300 for learning a user's intention using an artificial neural network receives information on the EEG signal measured for the user from the EEG sensor unit. In addition, the apparatus 300 for learning the user's operating intention using an artificial neural network determines whether the user's operating intention corresponding to the information on the EEG signal matches the output user's operating intention.

If it is determined that they do not match, the label of the output intention of the user may be corrected to be the user's intention corresponding to the information on the EEG signal and input to the previously learned artificial neural network.

Then, the apparatus 300 for learning the user's motion intention using the artificial neural network may learn to recognize the label of the output user's motion intention as the user's motion intention corresponding to the information on the EEG signal. The apparatus 300 for learning a user's operating intention using an artificial neural network determines whether the user's operating intention corresponding to the information on the EEG signal coincides with the output user's operating intention, Determine based on a predefined mapping relationship between a user's motion intention corresponding to the information on the EEG signal, a user's motion intention corresponding to information on the signal measuring the user's motion, and the output user's motion intention can do. Here, the predefined mapping relationship is shown in Table 2 above.

The measured EEG signal may correspond to a signal caused by a flashing stimulus. The apparatus 300 for learning a user's motion intention using an artificial neural network performs the step of outputting the user's motion intention when a signal measured for the user's motion is determined to be greater than or equal to a predefined feature value. The predefined feature value may include the average signal strength of the EMG signal. The motion sensor unit and the EEG sensor unit are portable to the user.

5 is a diagram illustrating a system including an apparatus 300 for learning a user's intention to operate using an artificial neural network according to the present invention.

As illustrated in FIG. 5, it is assumed that the user 500 (for example, the physically disabled or the elderly) is driving. The learning device 300 may receive a signal for the EMG response from the EMG sensor 510. The learning apparatus 300 estimates or outputs the user's intention to operate based on the EMG classification algorithm using the pre-trained artificial neural network. At this time, the learning device 300 receives information about the EEG signal from the user's EEG sensor 520 (such as information about the EEG frequency measured for the user). The learning device 300 determines whether the user's intention to operate matches the user's intention to correspond to the received EEG frequency information. If they do not match, the learning device 300 performs incremental learning using the received data. That is, the learning device 300 modifies the label of the output intention of the user to the operation intention of the user corresponding to the information on the EEG signal, inputs it to the pre-trained artificial neural network, and outputs the output of the user. It can be learned to recognize the label of the intention to operate as the user's intention to respond to the information on the EEG signal.

The EEG sensor 520 is the brain of the user 500 as the user 500 views any one of the displays (head-up displays) 530, 531, 532, 533 for stimulating multiple flashes provided in the vehicle. The measured EEG frequency can be calculated. Each display is provided at a different position in the vehicle, and the frequencies of the signals flashing on each display are not multiples of each other. In addition, it is assumed that the operation intention of the user corresponding to the corresponding display is matched differently for each location, and the user 500 recognizes this matching relationship.

For example, suppose that the user 500 moves his arm upward to intentional the action A (for example, playing music) among the functions inside the car. The user 500 looks at the blinking stimulus in the display at the corresponding location corresponding to the operation A. However, the user's intention to operate is output from the learning device 300 using the pre-trained artificial neural network (or EMG classification algorithm) after receiving information on the EMG response of the user 500's arm upward. For example, if it is a B operation (air conditioning is ON) that is other operation than playing music, the learning apparatus 300 processes this as an incorrect result due to noise of the movement of the user 500. The learning apparatus 300 incrementally learns by inputting a motion (eg, playing music) A, which is a user's true intention, into a pre-trained artificial neural network based on the received EEG frequency information.

FIG. 6 shows a graph of the EEG detected by the EEG sensor 520 when the user 500 watches a 15 Hz stimulus image.

When the user 500 watches a stimulus image of 15 Hz, a frequency proportional to the frequency of the stimulus image is observed in the brain. For example, the EEG sensor 520 may detect brain waves of 15 Hz, 30 Hz, and 45 Hz, which are multiples of 15 Hz, as shown in FIG. 6. When the user 500 watches a stimulus image of 15 Hz, since a frequency proportional to the frequency of the stimulus image is observed in the brain, a display (head-up display) 530, 531, 532 for a plurality of flashing stimuli provided in the vehicle , 533) The frequency of the flashing signal in each should not be a multiple of each other. For example, the frequency of the flashing signal in each of the display (head-up display) 530, 531, 532, 533 may be 15 Hz, 17 Hz, 18 Hz, 19 Hz, respectively.

7 is a flowchart specifically illustrating a method of learning a user's operating intention using an artificial neural network according to the present invention.

Referring to FIG. 7, it is assumed that the user 500 (eg, a physically disabled person) is driving while wearing an EEG measuring band (or EEG sensor unit) for EEG measurement and an EMG sensor unit for EMG measurement. . The EEG sensor unit and the EMG sensor unit worn by the user perform monitoring and extract feature values to determine whether the feature values exceed a predetermined threshold. When it is determined that the characteristic value for the extracted EMG response exceeds a predetermined threshold, the EMG sensor unit transmits the wireless signal to the communication unit 300 of the learning device 300. Here, the feature value may be the average intensity of the EMG signal.

The processor 320 of the learning device 300 outputs or estimates a user's intention to operate using the EMG classification algorithm of the pre-trained artificial neural network using the EMG information received from the EMG sensor unit.

Then, the communication unit 310 of the learning device 300 receives information on the EEG signal measured for SSVEP stimulation generation for operation verification from the EEG sensor unit. At this time, the processor 320 of the learning device 300 may grasp the user's intention to operate corresponding to the information on the EEG signal received by the EEG classification algorithm or a predetermined rule. The processor 320 of the learning device 300 determines whether the user's operating intention output by the EMG algorithm classifier and the user's operating intention identified by the EEG classification algorithm (or a predetermined rule) match.

If they do not match, the processor 320 of the learning device 300 modifies the label of the information on the EMG response. After correction, the processor 320 of the learning device 300 performs EEG algorithm device control, and stores the observed EMG and SSVEP-responsive EEG in a data set for incremental learning to learn a pre-trained artificial neural network.

As described above, a method and a learning apparatus for learning to reduce an error occurring when a device is manipulated using a machine learning algorithm model are proposed.

The embodiments described above are those in which the components and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to configure embodiments of the present invention by combining some components and/or features. The order of the operations described in the embodiments of the present invention can be changed. Some configurations or features of one embodiment may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is apparent that claims that do not have an explicit citation relationship in the claims can be combined to form an embodiment or included as a new claim by amendment after filing.

It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential features of the present invention. Accordingly, the above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the invention are included in the scope of the invention.

Claims (24)

Receiving information on a signal measuring a user's motion from a motion sensor unit;
Outputting a user's operating intention corresponding to information on a signal measuring the user's movement using a pre-trained artificial neural network;
Receiving information on the EEG signal measured for the user from the EEG sensor unit; And
When the user's intention to operate corresponding to the information on the EEG signal does not match the user's intention to output, a label for the user's intention to operate corresponds to the information on the EEG signal A method of learning a user's operating intention using an artificial neural network, including the step of modifying it with an intention and inputting the previously learned artificial neural network. According to claim 1,
And learning to recognize the label of the output intention of the user as the user's motion intention corresponding to the information on the EEG signal, the method of learning the user's motion intention using an artificial neural network. According to claim 1,
Whether the user's operating intention corresponding to the information on the EEG signal coincides with the output user's operating intention, information about the EEG signal, the user's operating intention corresponding to the information on the EEG signal, and the user's Further comprising the step of determining on the basis of a predefined mapping relationship between the user's motion intention corresponding to the information on the signal for measuring the motion and the motion intention of the output, the user's motion intention using the artificial neural network How to learn. According to claim 3,
The predefined mapping relationship includes a relationship corresponding to each row in the following table, and a method for learning a user's operating intention using an artificial neural network:

. The method of claim 4,
The first, second and third frequencies are characterized in that there is no multiple relationship between each other, a method of learning the user's operating intention using an artificial neural network. According to claim 1,
The measured EEG signal corresponds to a signal caused by a flashing stimulus, a method of learning a user's operating intention using an artificial neural network. According to claim 1,
The step of outputting the user's motion intention is performed when a signal measured for the user's motion is determined to be greater than or equal to a predefined feature value, and a method of learning a user's motion intention using an artificial neural network. The method of claim 7,
The predefined feature value includes an average signal strength of the EMG signal, and a method of learning a user's operating intention using an artificial neural network. According to claim 1,
The artificial neural network includes deep neural networks (DNN), and a method of learning a user's operating intention using an artificial neural network. According to claim 1,
The motion sensor unit and the EEG sensor unit, characterized in that the user can be worn by hand, a method of learning the user's operating intention using an artificial neural network. A communication unit that receives information on a signal measuring a user's movement from the motion sensor unit; And
It includes a processor for outputting the user's intention to operate corresponding to the information on the signal measuring the movement of the user using a pre-trained artificial neural network,
The communication unit receives information on the EEG signal measured for the user from the EEG sensor unit,
When the user's operating intention corresponding to the information on the EEG signal does not coincide with the output user's operating intention, the processor sets a label of the output user's operating intention to the EEG signal information. A device for learning a user's intention by using an artificial neural network, which is modified into an intention of operation and input to the previously learned artificial neural network. The method of claim 11,
The processor learns to recognize the label of the user's motion intention as the user's motion intention corresponding to the information on the EEG signal, and an apparatus for learning the user's motion intention using an artificial neural network. The method of claim 11,
The processor determines whether the user's operating intention corresponding to the information on the EEG signal coincides with the output user's operating intention, the information on the EEG signal, and the user's operating intention on the EEG signal. An apparatus for learning a user's operating intention using an artificial neural network, which is determined based on a predefined mapping relationship between the user's operating intention corresponding to information on a signal measuring the user's motion. The method of claim 13,
The predefined mapping relationship includes a relationship corresponding to each row in the following table, an apparatus for learning a user's operating intention using an artificial neural network:

. The method of claim 14,
The first, second and third frequencies are characterized in that there is no multiple relationship between each other, the apparatus for learning the user's operating intention using an artificial neural network. The method of claim 11,
The measured EEG signal corresponds to a signal caused by a flashing stimulus, a device for learning a user's intention using an artificial neural network. The method of claim 11,
The processor outputs a user's motion intention corresponding to a signal measured for the user's movement when the signal measured for the user's movement is determined to be greater than or equal to a predefined feature value. A device that learns the intent to act. The method of claim 17,
The predefined feature value includes the average signal strength of the EMG signal, and a device for learning a user's operating intention using an artificial neural network. The method of claim 11,
The artificial neural network includes deep neural networks (DNNs), and a device for learning a user's intention using an artificial neural network. The method of claim 11,
The motion sensor unit and the electroencephalography sensor unit, characterized in that the user can be worn by hand, a device for learning the user's operating intention using an artificial neural network. The method of claim 11,
The EEG sensor unit includes an electroencephalography (EEG) sensor, a device for learning a user's operating intention using an artificial neural network. The method of claim 11,
The motion sensor unit includes an EMG sensor unit or an acceleration sensor unit, and a device for learning a user's intention using an artificial neural network. The method of claim 22,
A device for measuring the user's motion intention using an artificial neural network, wherein the signal measured for the user's movement includes the user's EMG signal or acceleration signal. A computer-readable recording medium having a program for executing a method for learning a user's intention to operate on a computer using the artificial neural network according to any one of claims 1 to 10.

Images