WO2024117546A1 - Edge device for detecting somnambulism - Google Patents

Abstract

An edge device for detecting somnambulism according to the present invention may comprise: a wireless communication unit that receives a compressed electroencephalogram (EEG) signal from an external apparatus; and a processor. The processor: decompresses the compressed EEG signal and inputs the decompressed EEG signal to an input neuron layer of a trained recurrent spiking neural network (SNN) model; performs spiking encoding in the input neuron layer, thereby generating spatiotemporal spike features; and applies the generated spatiotemporal spike features to the trained recurrent SNN model to output a result regarding the presence of somnambulism.

Description

Edge device that detects sleepwalking

The present invention relates to an edge device for detecting sleepwalking, and more specifically, to an edge device for detecting sleepwalking in real time by measuring brain waves. This application is being filed as a result of the research and development project of neurochip design technology and neurocomputing platform that mimics the human nervous system as part of the Information and Communications Broadcasting Innovation Talent Development (R&D) project of the Information and Communication Planning and Evaluation Institute of the Ministry of Science and ICT.

Sleep can be divided into five stages. Stages 1 to 4 are NREM sleep stages without rapid eye movements, and REM sleep stages with rapid eye movements. The sleep cycle comprised of the NREM sleep stage and the REM sleep stage lasts about 100 minutes, and the cycle from the NREM sleep stage to the REM sleep stage is repeated while sleeping. Symptoms of sleepwalking are estimated to occur during stages 3 and 4 of sleep, and the actions experienced during sleep due to sleepwalking can be dangerous because they are not remembered. Diagnosing various sleep disorders, including sleepwalking, is possible through polysomnography conducted at a hospital. In polysomnography, a sleep technician comprehensively measures vital signs such as brain waves, electrocardiogram, electromyogram, respiration, and electrocardiogram during sleep, and simultaneously records the sleep state through video. As a result, it is a precise and professional test in which a sleep technician diagnoses sleep disorders while watching recorded video, but it cannot be interpreted in real time.

Previous research on sleep data analysis or sleep stage classification was about a sleep stage classification method based on multi-channel pressure signals. In an existing study, multi-channel ballistic cardiac (BCG) bio-signals were measured from the human body using a pressure sensor, and the user's sleep stage was classified using multi-channel bio-signals and heart rate variability (HRV). Such a non-contact sleep information collection device and a sleep analysis system that uses the same to determine the user's sleep state are being proposed. In particular, after measuring data in the user's sleep environment, the sleep data is used to provide analysis information about the user's sleep state. A system that analyzes the sleep state using measured sleep data using a computing device and then transmits the user's analysis information to the user terminal has also been proposed.

The technical problem to be achieved by the present invention is to provide an edge device that detects sleepwalking.

Another technical problem to be achieved by the present invention is to provide a method for an edge device to determine sleepwalking.

Another technical problem to be achieved by the present invention is to provide a computer-readable recording medium that records a program for executing a method for determining sleepwalking by an edge device on a computer.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

In order to achieve the above technical problem, an edge device for detecting sleepwalking according to the present invention includes a wireless communication unit that receives a compressed EEG (Electroencephalogram) signal from an external device; The compressed EEG signal is restored, the restored EEG signal is input to the input neuron layer of the learned Recurrent SNN (Spiking Neural Network) model, and spiking encoding is performed in the input neuron layer to generate spatiotemporal spike features, It may include a processor that applies the learned Recurrent SNN model to the generated spatiotemporal spike features and outputs a result on whether or not the user is sleepwalking.

The learned Recurrent SNN model is learned by applying a back propagation learning algorithm based on a predefined approximation function.

The approximation function defined in the dictionary can be expressed as the following equation 1,

[Equation 1]

Here, S(t) is the output value from each neuron layer, and x is the input value from each neuron layer.

The edge device may further include a memory that stores the result of sleepwalking. The edge device may further include a display unit that allows the user to check the results of sleepwalking under the control of the processor.

The external device may correspond to a wearable device worn on the user's brain. The wireless communication unit may receive the compressed EEG signal directly from the external device or through a network. The received compressed EEG signal is compressed through a compressed sensing algorithm.

In order to achieve the above other technical problems, a method for determining sleepwalking by an edge device according to the present invention includes receiving a compressed EEG (Electroencephalogram) signal from an external device; restoring the compressed EEG signal; Inputting the restored EEG signal into the input neuron layer of the learned Recurrent SNN (Spiking Neural Network) model; performing spiking encoding in the input neuron layer to generate spatiotemporal spiking features; And it may include applying the learned Recurrent SNN model to the generated spatiotemporal spike characteristics and outputting a result on whether or not the person is sleepwalking.

The method may further include displaying the results on whether sleepwalking occurs on a display unit so that the user can check the results.

The method may further include learning the learned Recurrent SNN model by applying a back propagation learning algorithm based on a predefined approximation function.

According to the sleepwalking detection method according to the present invention, sleepwalking can be detected in real time with high accuracy and low power.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention, and together with the detailed description, explain the technical idea of the present invention.

Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

Figure 2 is a diagram showing an example of a deep neural network.

Figure 3 is a diagram to explain monitoring using the Spiking Neural Network (SNN) algorithm model.

FIG. 4 is a block diagram illustrating the function of the edge device 400 according to the present invention.

Figures 5 and 6 are exemplary diagrams to explain a process for detecting sleepwalking in the edge device 400 and the wearable device 500 according to the present invention.

Figure 7 is a diagram illustrating the Recurrent SNN model structure proposed in the present invention.

Figure 8 is a diagram illustrating a Recurrent SNN model 430 for sleepwalking detection according to the present invention.

Figure 9 is a diagram showing simulation results for power consumption and accuracy between the existing artificial intelligence model (CNN) and the recurrent SNN model used in the present invention.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the invention. However, one skilled in the art will appreciate that the present invention may be practiced without these specific details.

In some cases, in order to avoid ambiguity of the concept of the present invention, well-known structures and devices may be omitted or may be shown in block diagram form focusing on the core functions of each structure and device. In addition, the same components are described using the same reference numerals throughout this specification.

Before explaining the present invention, artificial intelligence (AI), machine learning, and deep learning will be explained. The easiest way to understand the relationship between these three concepts is to imagine three concentric circles. Artificial intelligence is the largest circle, followed by machine learning, and deep learning, which leads the current artificial intelligence boom, is the smallest circle.

The concept of artificial intelligence first appeared at the Dartmouth Conference held by Professor John McCarthy at Dartmouth College in 1956, and has been growing explosively in recent years. In particular, it has been accelerating since 2015 with the introduction of GPUs, which provide fast and powerful parallel processing performance. The advent of the big data era, in which storage capacity is increasing explosively and data in all areas, including images, text, and mapping data, are flooding in, also had a significant impact on this growth.

Artificial Intelligence - Implementing human intelligence into machines

The dream of artificial intelligence pioneers in 1956 was to ultimately create a complex computer with characteristics similar to human intelligence. In this way, artificial intelligence that thinks like a human while possessing human senses and thinking skills is called 'general AI', but artificial intelligence that can be created at the current level of technological development is called 'narrow AI'. Included. Narrow AI is characterized by its ability to perform certain tasks with better than human capabilities, such as social media image classification services or facial recognition functions.

Machine Learning - A Specific Approach to Implementing Artificial Intelligence

Machine learning automatically filters spam from your mailbox. Meanwhile, machine learning basically analyzes data using algorithms, learns through analysis, and makes judgments or predictions based on what has been learned. Therefore, the ultimate goal is to learn how to perform tasks by 'learning' the computer itself through large amounts of data and algorithms, rather than directly coding specific instructions for decision-making criteria into the software. Machine learning comes from concepts directly proposed by early artificial intelligence researchers, and algorithmic methods include decision tree learning, inductive logic programming, clustering, reinforcement learning, and Bayesian networks. However, none of these have achieved general AI, which is the ultimate goal, and it is true that it was often difficult to complete even narrow AI with early machine learning approaches.

Currently, machine learning is making great achievements in fields such as computer vision, but it has encountered a limitation in that a certain amount of coding work is involved throughout the process of implementing artificial intelligence, even if it is not a specific guideline. For example, when recognizing an image of a stop sign based on a machine learning system, developers use a boundary detection filter to programmatically identify the start and end of the object, shape detection to identify the side of the object, and letters such as 'S-T-O-P'. A classifier that recognizes must be produced through direct coding. In this way, machine learning works by recognizing images from a 'coded' classifier and 'learning' stop signs through an algorithm.

The image recognition rate of machine learning achieves sufficient performance for commercialization, but the image recognition rate may drop in certain situations where signs are difficult to see due to fog or trees. The reason computer vision and image recognition have not been able to reach human level until recently is because of these recognition rate issues and frequent errors.

Deep Learning - The technology that makes complete machine learning possible

Another algorithm created by early machine learning researchers, the artificial neural network, was inspired by the biological properties of the human brain, particularly the neuronal connection structure. However, unlike the brain, where any neuron in physical proximity can be interconnected, artificial neural networks have constant layer connections and data propagation directions.

For example, if an image is cut into a number of tiles and fed into the first layer of a neural network, the neurons pass the data to the next layer and repeat the process until the final output is produced in the last layer. Each neuron is then assigned a weight that represents the accuracy of the input based on the task it performs, and then the weights are all added together to determine the final output. In the case of a stop sign, characteristics of that image - octagonal shape, red color, sign letters, size, whether it's moving or not - are chopped up and 'examined' by neurons, and the neural network's job is to identify whether this is a stop sign or not. Here, a 'probability vector' is used, which predicts the result according to weights based on sufficient data.

Deep learning is a form of artificial intelligence that has evolved from artificial neural networks and learns data using information input and output layers similar to neurons in the brain. However, because even basic neural networks require a huge amount of computation, the commercialization of deep learning faced difficulties from the beginning. Nevertheless, the researchers' research continued, and they succeeded in parallelizing the algorithm that proves the concept of deep learning based on a supercomputer. And the emergence of GPUs optimized for parallel computation dramatically accelerated the computation speed of neural networks, ushering in the emergence of true deep learning-based artificial intelligence.

Neural networks are likely to produce numerous wrong answers during the 'learning' process. Going back to the stop sign example, we might need to learn hundreds, thousands, or even millions of images to adjust the weights of neuron inputs precisely enough to always produce the correct answer, regardless of weather conditions or changes between day and night. Only when this level of accuracy is reached can the neural network be considered to have properly learned stop signs. In 2012, Google and Stanford University professor Andrew Ng implemented a 'Deep Neural Network' consisting of more than 1 billion neural networks with 16,000 computers. Through this, they extracted and analyzed 10 million images from YouTube and succeeded in having the computer classify pictures of people and cats. The computer learned the process of recognizing and judging the shape and appearance of the cat shown in the video.

The image recognition capabilities of systems trained with deep learning are already ahead of humans. Other areas of deep learning include the ability to identify cancer cells in the blood and tumors in MRI scans. Google's AlphaGo learned the basics of Go and further strengthened its neural network by repeatedly playing against AI like itself. With the advent of deep learning, the practicality of machine learning has been strengthened and the scope of artificial intelligence has expanded. Deep learning breaks down tasks in any way that can be supported by a computer system. Deep learning-based technologies, such as driverless cars, better preventive medicine, and more accurate movie recommendations, are already being used in our daily lives or are about to be put into practical use. Deep learning is evaluated as the present and future of artificial intelligence with the potential to realize general AI that appeared in science fiction.

Below we look at deep learning in more detail.

Deep learning is a type of artificial neural network (ANN) that uses human neural network theory. It is composed of a layer structure and has one input layer and one output layer. It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) with more than one hidden layer (hereinafter referred to as the middle layer). Simply put, Deep Learning can be said to be an artificial neural network with deep layers.

It is estimated that the human brain consists of 25 billion nerve cells. The brain is made up of nerve cells, and each nerve cell (neuron) refers to one nerve cell that makes up a neural network. A nerve cell contains one cell body, one axon or nurite, which is a projection of the cell body, and usually several dendrites (dendrite or protoplasmic process). Information is exchanged between these nerve cells through junctions between nerve cells called synapses. If you look at a single nerve cell in isolation, it is very simple, but when these nerve cells come together, it can have human intelligence. The dendrites are the part that receives signals from other nerve cells (Input), and the axon is the part that extends very long from the cell body and is the part that transmits signals to other nerve cells (Output). There is a connection called a synapse that connects the axon and dendrites, which transmit signals between nerve cells. Rather than transmitting signals from nerve cells unconditionally, signals are only transmitted when the signal strength exceeds a certain value (threshold). It is done. In other words, not only does each synapse have a different connection strength, but it also determines whether or not to transmit a signal.

Artificial neural network (ANN), a field of artificial intelligence, is a mathematical model modeled by imitating the brain structure (neural network) of biology (usually human). In other words, artificial neural networks are implemented by imitating the information processing and transmission processes of biological nerve cells. Neural networks, which are implemented similarly to the way the human brain solves problems, have excellent parallelism because each nerve cell operates independently. In addition, because information is distributed across many connection lines, even if a problem occurs in a few nerve cells, it does not have a significant impact on the overall system, so it is resistant to a certain level of error and has the ability to learn about a given environment.

Deep neural network can be considered a descendant of artificial neural network, and is the latest version of artificial neural network, surpassing existing limitations and achieving success in areas where many artificial intelligence technologies had failed in the past. Looking at the modeling of an artificial neural network by imitating a biological neural network, in terms of processing units, biological neurons are used as nodes, and in terms of connectivity, synapses are used as weights. It was modeled as shown in Table 1.

biological neural network	artificial neural network
cell body	node
dendrites	input
Axon	output
synapse	weight

Figure 1 is a diagram illustrating the layer structure of an artificial neural network. Just as human biological neurons connect not one but many, but many, meaningful tasks, in the case of an artificial neural network, individual neurons connect with each other through synapses. By connecting, multiple layers are connected to each other, and the connection strength between each layer can be updated with a weight. In this way, it is used in the field of learning and cognition due to its multi-layered structure and connection strength.

Each node is connected by weighted links, and the entire model learns by repeatedly adjusting the weights. Weights are the basic means for long-term memory and express the importance of each node. Simply put, an artificial neural network trains the entire model by initializing these weights and updating and adjusting the weights with the data set to be trained. After training is complete, when new input values come in, an appropriate output value is inferred. The learning principle of artificial neural networks can be viewed as a process in which intelligence is formed from the generalization of experience, and is done in a bottom-up manner. In Figure 1, when there are two or more middle layers (i.e. 5 to 10), the layers are considered to be deeper and are called a deep neural network, and the learning and inference model achieved through such a deep neural network can be referred to as deep learning. there is.

An artificial neural network can perform a certain role even if it has one middle layer (usually referred to as a 'hidden layer') excluding the input and output, but as the complexity of the problem increases, the number of nodes or the number of layers increases. The number must be increased. Among these, it is effective to use a multi-layer structure model by increasing the number of layers, but its scope of use is limited due to the inability to learn efficiently and the large amount of calculations to learn the network.

However, by overcoming the existing limitations as described above, artificial neural networks have been able to achieve deep structures. This makes it possible to build complex and highly expressive models, and groundbreaking results are being announced in various fields such as voice recognition, face recognition, object recognition, and text recognition.

Figure 2 is a diagram showing an example of a deep neural network.

A deep neural network (DNN) is an artificial neural network (ANN) made up of several hidden layers between an input layer and an output layer. A machine learning (Machine Learning) model or algorithm that refers to a Deep Neural Network (DNN) that has one or more hidden layers between the input layer and the output layer. is a set of The connection of a neural network is from the input layer to the hidden layer and from the hidden layer to the output layer.

Deep neural networks, like general artificial neural networks, can model complex non-linear relationships. For example, in a deep neural network structure for an object identification model, each object can be expressed as a hierarchical composition of basic elements of the image. At this time, additional layers can gradually integrate the characteristics of the gathered lower layers. This feature of deep neural networks allows complex data to be modeled with fewer units (nodes) than similarly performed artificial neural networks.

Previous deep neural networks have usually been designed as forward-feeding neural networks, but recent studies have successfully applied deep learning structures to recurrent neural networks (RNNs). For example, there is a case where a deep neural network structure is applied to the field of language modeling. In the case of Convolutional Neural Network (CNN), not only is it well applied in the field of computer vision, but each successful application case is also well documented. More recently, convolutional neural networks have been applied to the field of acoustic modeling for Automatic Speech Recognition (ASR), and are evaluated to be more successful than existing models. Deep neural networks can be trained with the standard error backpropagation algorithm. At this time, the weights can be updated through stochastic gradient descent.

In many fields, deep learning using artificial neural networks is receiving great attention as it significantly surpasses the performance of other existing algorithms. However, the deep learning methods currently mainly used require large power consumption, making them difficult to apply to mobile fields with limited resources. Accordingly, interest in spiking neural networks (SNNs) that can operate at low power is growing. SNN learns synaptic weights using the STDP algorithm, where synaptic weights are adjusted according to the pre- and post-synaptic spike time relationship. Therefore, SNN can learn in various configurations depending on the number of spikes used for learning and the temporal interaction between spikes and the STDP algorithm.

Neuromorphic chips, artificial intelligence computing chips, are attracting attention as the next generation's hottest technology as they can solve the power security problems of existing semiconductor chips and integrate data processing processes. The core of neuromorphic technology is to mimic the human brain and enable memory and computation to proceed simultaneously.

Additionally, by applying the latest insights in neuroscience, the goal is to create a chip that functions more like a human brain than a classical computer. Additionally, the neuromorphic chip models how neurons in the brain communicate and learn using spikes (electrical impulses) and synapses that can be adjusted depending on the situation. These chips are also designed to self-organize and make decisions based on learned patterns and associations.

The goal of implementing neuromorphic chips is to enable them to learn as quickly and efficiently as the human brain, which is far superior to today's most powerful computers. Neuromorphic computing is expected to make development easier and innovatively apply intelligence and automation in the field to a variety of AI edge devices and applications that require continuous learning and adaptation to evolving real-world data in real time.

Neuromorphic Computing

First-generation AI imitated existing logic based on rules and drew reasonable conclusions within a narrowly defined specific problem domain. For example, the first generation of AI was suited to monitoring processes and improving efficiency. The current second generation of AI is closely related to perception and detection, including the use of deep learning networks to analyze video frame content. In the future, next-generation AI will expand into areas that correspond to human perception, such as interpretation and autonomous adaptation. This plays a very important role in overcoming the so-called weak point of AI solutions based on neural network learning and inference, which rely on a deterministic view in situations where understanding of context and common sense is lacking. In order for next-generation AI to automate common human activities, it must be able to solve new situations and concepts.

Computer science research that will enable third-generation AI is underway, with key areas including neuromorphic computing, which involves mimicking the neural structures and behavior of the human brain, and algorithms for handling uncertainty, ambiguity, and contradictions in the natural world. There is a stochastic computing implementation of the method.

The core of neuromorphic computing research

A key challenge in neuromorphic research is to study the ability to learn from unstructured stimuli at a level of energy efficiency comparable to that of the human brain. The computing components of a neuromorphic computing system are logically similar to neurons. Spiking Neural Networks (SNNs) are new models that arrange these elements to mimic the natural neural networks that exist in biological brains.

Each “neuron” in a SNN runs independently of one another, sending pulse signals to other neurons in the network and directly changing the electrical state of those neurons. SNNs, which encode information and timing within the signal itself, simulate natural learning processes by dynamically remapping synapses between artificial neurons in response to stimuli.

Neuromorphic computing or neuromorphic engineering is a field of engineering that attempts to mimic human brain function by creating circuits that mimic the shape of neurons. Circuits and chips created in this way are called neuromorphic circuits and neuromorphic chips. If an artificial neural network is a software-based simulation of the human nervous system, a neuromorphic chip is a hardware-based simulation of nerve cells. In other words, a neuromorphic chip is a computer chip that mimics the structure of a living organism's nervous system (brain).

Because it is a computer chip composed of only the circuits necessary for neural network calculations, it can provide hundreds of times more benefits in terms of power, area, and speed than performing neural network calculations using CPUs and GPUs.

Of course, there are ASIC chips dedicated to abstracted artificial neural network circuits such as DNN or CNN, such as Google's TPU, but these are usually not classified as neuromorphic chips. The implementation method of the TPU is similar to that of a general DSP, and neuromorphic chips that are widely studied usually have individual neurons implemented independently, have higher data locality, and usually have learning algorithms other than backpropagation based on this. One representative example is a spiking neural network that implements Spike-time dependent plasticity (STDP). This method can ultimately have better scalability and higher performance than a typical centrally controlled DSP. Due to difficulties in implementing algorithms and circuits, no significant industrial results have yet been achieved.

Unlike existing computers, the human brain generates very little power even though it processes a lot of data. This is because the structures connecting neurons and synapses are built in parallel. Synapses save energy by connecting and disconnecting when they are working or not. Existing computers consume a lot of electricity in the process of processing data between the CPU and memory, but neuromorphic chips reduce power consumption by imitating the way the brain operates.

Spiking Neural Network (SNN) algorithm model

Figure 3 is a diagram to explain monitoring using the Spiking Neural Network (SNN) algorithm model.

SNN can be called a spiking artificial neural network. The biggest difference from a typical artificial neural network is the presence of a time axis. Rather than simply receiving the values of previous neurons once for each neuron, the value (internal state) of the neuron continuously changes over time. At this time, the concepts created to prevent the network from becoming monotonous and to more closely mimic the actual brain are the threshold and spike. When the internal state value of a neuron exceeds the threshold, a spike is transmitted to connected neurons, and the internal state is initialized. The internal state of a neuron that receives a spike increases or decreases depending on the synapse weight. After receiving this spike, another spike may occur in the next neuron. After determining the time range, the input is given as the “spike frequency of the input neuron” and the output is measured as the “number of spikes appearing in the output neuron.”

For example, the sensors shown in FIG. 3 can detect the occurrence of an earthquake or similar disaster and perform a spiking neural network-based inspection of the condition of the building after the earthquake or disaster occurs. The structural condition of the building is detected by processing the data acquired by the sensor and learning from the spiking neural network through feature point extraction.

Multi-Spiking Neural Network

Here, there is a model called Multi-Spiking Neural Network that mimics the brain more precisely. In contrast, the model introduced earlier is also called a Single-Spiking Neural Network. What is different in this model is that multiple synapses connect the same pair of neurons. Each synapse has a different speed, which results in a different delay when the spike signal is transmitted.

For the PI-MNIST problem, the existing ANN model showed an accuracy of 99.06% (Goodfellow et al., 2013), and the SNN model showed a comparable accuracy of 98.77% (Lee et al., 2016). The N-MNIST problem showed an accuracy of 98.66% (Lee et al., 2016), which is higher than 98.3% in traditional ANN (Neil and Liu, 2016; convolutional neural network (CNN) was used).

The structure of the neural network itself is the same as a typical ANN: it simply consists of an input layer, a hidden layer, and an output layer. It is possible to draw complex structures such as Spiking CNN, but this is out of the discussion. All neurons in each neighboring layer were connected. Now, the unit time (unit of time is "second") is divided into timestamps, and the task of constructing a new state from the state of the neural network one second ago is repeated as many times as the number of timestamps (usually 60). The state of a neural network consists of only one variable, the internal state of the neuron, which will be discussed below. All other arguments are either hyperparameters to be learned or constants.

Each neuron has one variable called an “internal state” (V). This value basically starts with 0 and is a real number below the threshold (Vth). Vth is set to a specific value for each neuron, and is one of the learning targets. However, the input neuron does not have this value and only fires. The utterance period can be determined according to the corresponding input value (mainly using Poisson distribution). Neurons that are not input neurons fire when V hit (threshold voltage), and immediately after firing, V decreases by Vth. Mimicking the refractory period in biology (a period in which neurons that have fired once do not fire again within a certain period of time), neurons that have fired once do not fire again soon.

The V value continuously decreases (in absolute value) unless spikes (firing) are received from all neurons. In the paper, V is decreased in the form of an exponential function, and it decreases by e1*?* times every second. If a spike is transmitted through a synapse from a neuron in the previous layer, V increases by the synapse's weight wij. Whether V increases or decreases is determined depending on the sign of w.

In this way, SNN fires only when the membrane potential of a neuron is higher than the threshold voltage, and transmits information between synapses through fired spikes. It can operate in an event-driven manner, enabling low-power operation compared to other artificial neural networks. Because neurons and synapses are not differentiable in SNN, it is not possible to learn SNN using gradient descent and error backpropagation methods. The most widely known learning method for SNN is STDP (Spiking Timing Dependent Plasticity). STDP is a method of learning synaptic weights through the temporal relationship between pre-synaptic spikes and post-synaptic spikes. Therefore, the number of pre- and post-synaptic spikes considered in STDP learning and the temporal interaction between spikes affect the learning of SNN.

A lot of research is being done on technologies that collect data by measuring biosignals using non-invasive sensors such as wearable devices and detect sleep disorders through real-time monitoring of the collected data using deep neural network models and machine learning. However, due to a memory shortage problem for brain wave data measured and accumulated in real time from a wearable device, the measured data needs to be transmitted to another edge device for monitoring. In addition, because there is overhead that occurs at the data transmission stage, a method for efficiently compressing and transmitting data is needed. Measured biometric data needs to be processed for monitoring on edge devices. However, the most commonly used method for monitoring is monitoring using a deep neural network. Monitoring methods using deep neural networks can accurately detect sleepwalking, but are not suitable for use in edge devices because the high computational load requires a large load to process all incoming data in real time and the power used is very high.

Because edge environments such as wearable devices have limited memory, it is essential to transmit brainwave (EEG) data measured in real time to other devices. Additionally, in order for edge devices to continuously monitor data measured in real time by external devices such as wearable devices, it is important to use low-power-based artificial intelligence models that do not require a high computational amount. Edge devices are devices that generate data and include Internet of Things (IoT) sensors that generate or collect data, video/surveillance cameras, home appliances connected to the Internet, and smart devices such as smartphones.

In the present invention, we propose a compressed sensing-based EEG data compression method for efficient data transmission and a low-power, real-time sleepwalking detection method using a spiking neural network.

FIG. 4 is a block diagram illustrating the function of the edge device 400 according to the present invention.

Referring to FIG. 4, the edge device 400 may include a processor 410, a wireless communication unit 420, a memory 430, and a display unit 440.

The processor 410 performs functions such as inferring results by processing (computing) input data according to the Spiking Neural Networks (SNN) algorithm model according to the present invention. The memory 430 stores various information necessary for neuromorphic computing, such as information necessary for the processor 410 to infer a result and information about the inferred result. The wireless communication unit 420 is equipped to transmit and receive data wirelessly with an external device.

Figures 5 and 6 are exemplary diagrams to explain a process for detecting sleepwalking in the edge device 400 and the external device 500 (hereinafter referred to as a wearable device) according to the present invention.

The wearable device 500 shown in FIG. 5 is a device worn on the user's head and has an EEG (Electroencephalogram) sensor attached to measure or sense EEG signals, that is, brain wave signals. The wearable device 500 compresses the measured or sensed EEG signal by applying a predetermined compressive sensing algorithm model and transmits it to the edge device 400 through a transmitter. The compressed sensing algorithm is one of the sampling methods to convert the measured EEG analog signal into a digital signal. It is a Nyquist-Shannon sampling method that samples the previously mainly used analog signal by placing dots at regular intervals. Unlike this, it is a method of sampling by randomly dotting an analog signal. Because of this randomness, compressed sensing can be expressed with a smaller number of measured values compared to existing sampling methods, resulting in a better compression rate. The sleepwalking detection method proposed in the present invention requires continuous transmission to the edge device 400 because the wearable device 500 continues to measure data while the person sleeps, and the data of the compressed EEG signal is stored in the existing It allows transmission with less overhead than other methods.

As shown in FIG. 5, the wearable device 500 can transmit information about the compressed EEG signal directly to the edge device 400 wirelessly through IoT communication, etc., and as shown in FIG. 6, the wearable device ( 500) shows an example of transmitting information about a compressed EEG signal to the edge device 400 through a network such as a base station. The wearable device 500 vectorizes the measured EEG signal, compresses the signal through matrix operation with a randomly generated matrix, and transmits it.

Referring to Figures 5 and 6, the wireless communication unit 420 of the edge device 400 receives a compressed electroencephalogram (EEG) signal from the wearable device 500. The processor 410 reconstructs the compressed EEG signal into the original EEG signal using a restoration algorithm. After preprocessing the restored EEG signal, the processor 410 analyzes the EEG signal through the SNN model 430, which is learned in advance using the preprocessed signal and embedded in the edge device, to determine whether or not the person is sleepwalking. The processor 410 inputs the restored EEG signal to the input neuron layer (or input layer) of the learned Recurrent SNN (Spiking Neural Network) (algorithm) model 430.

The processor 410 performs spiking encoding in the input neuron layer to generate spatial-temporal spike features. This is because a new time axis was given to prevent information loss while converting continuous multi-channel signal data into discrete spike signal data. Since these spike signal data are also a type of time series data, a Recurrent SNN structure is used to process these time signals. It is desirable to use . The processor 410 applies the learned Recurrent SNN model to the generated spatiotemporal spike characteristics and outputs a result regarding sleepwalking. As described above, in particular, the processor 410 processes the restored EEG signal by applying it to a predetermined learned Recurrent Spiking Neural Network (SNN) (algorithm) model 430.

Figure 7 is a diagram illustrating the Recurrent SNN model structure proposed in the present invention.

Referring to Figure 7, the Recurrent SNN model structure is initialized with random connectivity. In other words, it is not a fully connected method that connects all neurons, and has recurrent characteristics, so it is suitable for cases where continuous EEG data is required by continuously measuring the EEG of a sleeping person, as in the present invention. Additionally, the backpropagation learning algorithm used in DNN is used instead of the unsupervised learning method in existing SNN. The following equation 1 represents the output value from each neuron layer in the existing SNN.

Here, S(t) is the output value from each neuron layer in the existing SNN model.

Since each neuron in the Recurrent SNN model uses a step function with an output of 0 or 1 as shown in Equation 1, differentiation is impossible due to discrete output values, so backpropagation-based learning algorithms cannot be used. To solve this problem, the present invention applies a backpropagation learning algorithm by approximating the Step function in the Recurrent SNN model as a continuous function as shown in Equation 2 below.

Here, S(t) is the output value from each neuron layer in the Recurrent SNN model, and x is the input value from each neuron layer.

The memory 430 stores the results of sleepwalking. The display unit 440 displays the results of sleepwalking under the control of the processor 410 so that the user can check whether the person is sleepwalking.

Figure 8 is a diagram illustrating a recurrent SNN model 430 for sleepwalking detection according to the present invention.

The processor 410 uses the finally restored signal as an example of a recurrent SNN model 430, a learned spiking neural network with three layers, to detect sleepwalking. Due to the nature of deep learning, it requires a lot of data to train the algorithm, so it may be suitable in a wearable device environment where biosignals are continuously measured, but the power consumption used to learn data is quite high for deep learning models. Because it is heavy, it is not suitable for direct use in wearable devices. What the present invention proposes has the advantage of being able to detect sleepwalking through efficient sleep stage classification using only power consumption equivalent to 0.001 times that of a deep learning model.

The processor 410 detects whether sleepwalking is a sleep disorder through inference of the recurrent SNN model 430. In addition, it is very suitable for the edge device (400) environment because it can be monitored in real time and power consumption can be reduced with a small amount of calculation. From this perspective of energy efficiency, the inference method using a spiking neural network can infer whether or not someone is sleepwalking from real-time brain wave data with much lower power consumption than a deep learning model.

If the user is sleepwalking, he or she cannot remember what he or she did while asleep, and this may pose a risk to the user. When looking at the effects from a systemic perspective, when the user is in a sleeping state, the brain wave data measured from the wearable device is shared with an acquaintance or person. By transmitting data to wireless devices owned by users, such as family members, for real-time monitoring, dangerous actions that users engage in unconsciously can be prevented.

Figure 9 is a diagram showing simulation results for power consumption and accuracy between the existing artificial intelligence model (CNN) and the recurrent SNN model used in the present invention.

Referring to Figure 9, as a result of measuring the accuracy of sleepwalking detection/diagnosis through the Recurrent SNN model used in the present invention and the existing deep learning CNN (Convolutional Neural Network) model, the accuracy of the Recurrent SNN model used in the present invention was 79%. The accuracy was high enough to be almost the same as the 81% of the CNN model, and in terms of power, only about 0.0074 of the CNN model was consumed, showing that power efficiency was greatly improved, and it was proven that it can be applied to edge devices.

According to the sleepwalking detection method according to the present invention described above, sleepwalking can be detected in real time with high accuracy and low power.

The embodiments described above are those in which the components and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, it is also possible to configure an embodiment of the present invention by combining some components and/or features. The order of operations described in embodiments of the present invention may be changed. Some features or features of one embodiment may be included in other embodiments or may be replaced with corresponding features or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the patent claims can be combined to form an embodiment or included as a new claim through amendment after filing.

In the present invention, the processor 310 may also be called a controller, microcontroller, microprocessor, microcomputer, etc. Meanwhile, the processor 310 may be implemented by hardware, firmware, software, or a combination thereof. When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable logic devices (PLDs) configured to perform the present invention. , FPGAs (field programmable gate arrays), etc. may be provided in the processor 310.

It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential features of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Edge devices that detect sleepwalking can be used in industries such as the ICT field.

Claims (13)

1. In an edge device that detects sleepwalking,

   A wireless communication unit that receives a compressed EEG (Electroencephalogram) signal from an external device;

   Restore the compressed EEG signal and input the restored EEG signal into the input neuron layer of the learned Recurrent SNN (Spiking Neural Network) model,

   Spiking encoding is performed in the input neuron layer to generate spatiotemporal spike features,

   An edge device comprising a processor that outputs a result of sleepwalking by applying the learned Recurrent SNN model to the generated spatiotemporal spike characteristics.
2. According to clause 1,

   An edge device in which the learned Recurrent SNN model is learned by applying a back propagation learning algorithm based on a predefined approximation function.
3. According to clause 2,

   The approximation function defined in the dictionary can be expressed as the following equation 1,

   [Equation 1]

   

   Here, S(t) is the output value from each neuron layer, and x is the input value from each neuron layer, an edge device.
4. According to clause 1,

   An edge device further comprising a memory that stores a result of whether the person is sleepwalking.
5. According to clause 4,

   An edge device further comprising a display unit displayed so that a user can check the result of sleepwalking under control of the processor.
6. According to clause 1,

   The external device is an edge device that corresponds to a wearable device worn on the user's brain.
7. According to clause 6,

   The wireless communication unit is an edge device that receives the compressed EEG signal directly from the external device or through a network.
8. According to clause 1,

   An edge device wherein the received compressed EEG signal is compressed through a compressed sensing algorithm.
9. In how the edge device determines sleepwalking,

   Receiving a compressed EEG (Electroencephalogram) signal from an external device;

   restoring the compressed EEG signal;

   Inputting the restored EEG signal into the input neuron layer of the learned Recurrent SNN (Spiking Neural Network) model;

   performing spiking encoding in the input neuron layer to generate spatiotemporal spiking features; and

   A method for determining sleepwalking, comprising the step of applying the learned Recurrent SNN model to the generated spatiotemporal spike features and outputting a result on whether or not the person is sleepwalking.
10. According to clause 9,

    A method for determining sleepwalking, further comprising displaying the results on whether sleepwalking occurs on a display unit so that the user can check the results.
11. According to clause 9,

    A method for determining sleepwalking, further comprising learning the learned Recurrent SNN model by applying a back propagation learning algorithm based on a predefined approximation function.
12. According to clause 9,

    The approximation function defined in the dictionary can be expressed as the following equation 1,

    [Equation 1]

    

    Here, S(t) is the output value from each neuron layer and x is the input value from each neuron layer, an edge device.
13. A computer-readable recording medium recording a program for executing the method according to any one of claims 9 to 12 on a computer.

Images