KR102440217B1 - System and device for determining sleep stages based on deep-learning - Google Patents

Abstract

A deep learning-based sleep phase determination apparatus according to an embodiment of the present invention includes: a biosignal detection unit for receiving the biometric data from a sensor system for acquiring the subject's biometric data; a biosignal image generating unit that generates biometric image information obtained by converting the biometric data into a graphic image; a preprocessor for generating biometric preprocessing data by converting the biometric image information into a format corresponding to the input data format of the sleep stage determination model; a deep learning classification unit providing sleep stage determination information obtained by measuring the sleep stage of the examinee by performing deep learning based on the biometric preprocessing data based on the sleep stage determination model; and a result output unit for outputting the sleep stage determination information in a predetermined format. includes

Description

Deep learning-based sleep stage determination device and system {SYSTEM AND DEVICE FOR DETERMINING SLEEP STAGES BASED ON DEEP-LEARNING}

The present invention relates to a sleep stage determination device and system based on deep learning. More particularly, it relates to a deep learning-based sleep stage determination apparatus and system for determining a sleep stage based on a biometric data body of a subject based on a learned deep learning neural network.

Polysomnography is a test to analyze a subject's sleep state based on the subject's biometric data, and comprehensively measures EEG, eye movement, muscle movement, respiration, electrocardiogram, and/or oxygen saturation during sleep. is being used to

Using this polysomnography test, it is possible to estimate the symptoms of sleep disorders including sleep apnea and/or sleep gait for the subject, and an appropriate response therefor can be easily performed.

In general, polysomnography as described above is performed in the classification method of the Association for the Psychological Study of Sleep (APSS) based on the shape of the brain wave that changes according to a person's sleep cycle.

Here, the APSS method refers to biometric data (eg, electroencephalography (EEG), electro-oculogram (EOG), and/or electromyogram (EMG) of the jaw) measured through various sensors). It is a technique in which an expert manually reads whether the corresponding biometric data contains outliers based on subjective judgment.

In other words, the existing polysomnography method has a problem in that its accuracy and reliability are deteriorated because it relies on the subjective judgment of experts based on ambiguous criteria.

In detail, although the criteria for determination in polysomnography may seem clear, the shape and size of bio-signals included in bio-data are different depending on the subject, and due to the atypicality in the bio-signals, there are many errors even among experts. have.

In addition, the identification criteria for the frequency band used when determining the sleep stage of a subject based on polysomnography is ambiguous, so for example, the standard method for dividing sleep stages is a delta wave per cycle of NREM3, which is the third sleep stage. is set at 20% or more and less than 50%, but it is impossible to quantify it accurately by looking at actual data.

Therefore, even for the same biometric data, different judgment results may be shown depending on the expert's personal opinion or skill level, so it is difficult to accurately measure the sleep stage.

In addition, the existing polysomnography method has a problem that the required time is too long as it is manually performed by an expert.

In other words, when performing the existing polysomnography, it takes time for an expert to read the biometric data measured from the subject, so it takes a considerable amount of time to obtain the corresponding test result, and the analysis cost increases accordingly. There is a problem that the inspection is difficult.

KR 10-2004219 B1

The present invention has been devised to solve the problems described above, and a deep learning-based sleep stage that determines a sleep stage based on the subject's biometric data based on a learned deep learning neural network (sleep stage determination model) An object of the present invention is to provide a judgment device and a system therefor.

In detail, the present invention is to provide a deep learning-based sleep stage determination apparatus and system for providing an automated sleep stage determination process based on a learned deep learning neural network.

In addition, the present invention is a deep learning-based sleep stage determination device that implements a sleep stage determination model by performing machine learning based on a deep learning neural network based on a customized training data set optimized for sleep stage determination. and to provide a system therefor.

However, the technical problems to be achieved by the present invention and embodiments of the present invention are not limited to the technical problems as described above, and other technical problems may exist.

A deep learning-based sleep phase determination apparatus according to an embodiment of the present invention includes: a biosignal detection unit for receiving the biometric data from a sensor system for acquiring the subject's biometric data; a biosignal image generating unit that generates biometric image information obtained by converting the biometric data into a graphic image; a preprocessor for generating biometric preprocessing data by converting the biometric image information into a format corresponding to the input data format of the sleep stage determination model; a deep learning classification unit providing sleep stage determination information obtained by measuring the sleep stage of the examinee by performing deep learning based on the biometric preprocessing data based on the sleep stage determination model; and a result output unit for outputting the sleep stage determination information in a predetermined format. includes

In this case, the biometric data is obtained from at least one sensor of an electroencephalography (EEG) sensor, an electro-oculogram (EOG) sensor, an electromyogram (EMG) sensor, a breathing sensor, an electrocardiogram sensor, and an image sensor. It is biosignal measurement data.

In addition, the sleep stage determination model is a deep learning neural network trained to receive the biometric pre-processing data as an input and output the sleep stage determination information based on a predetermined training data set.

In addition, the training data set is implemented based on a set of sleep stage determination information determined by an expert in a pre-performed polysomnography test, and the biometric pre-processing data is filtered out of a plurality of biosignal measurement data. to be removed, matched with the sampling frequency of the plurality of biosignal measurement data, and necessary input data is extracted from the plurality of biosignal measurement data with the same sampling frequency.

In addition, the sleep stage determination information is information in which it is determined which sleep stage the examinee enters among the REM sleep stage, the NREM1 sleep stage, the NREM2 sleep stage, the NREM3 sleep stage, and the NREM4 sleep stage.

In addition, the pre-processing unit filters and removes noise data in the bio-image information, and sets the bio-image information from which the noise data has been removed to a predetermined time interval corresponding to the input data of the sleep stage determination model in epochs units. to generate the biometric pre-processing data.

In addition, the deep learning classification unit, a deep learning neural network comprising at least one of a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN), the training data The sleep stage judgment model is implemented by learning it in three sets.

In addition, the result output unit, based on the sleep stage determination information to generate sleep state information representing the analysis data of the overall sleep of the examinee as a graphic image, the sleep state information, the sleep stage determination information, the sleep and at least one of the biometric preprocessing data and the biometric image information related to the step determination information.

Also, the result output unit outputs the sleep state information based on at least one of a self-display and an external device.

On the other hand, the deep learning-based sleep phase determination system according to an embodiment of the present invention, a sensor system for acquiring the biometric data of the subject; and a sleep stage determination device for providing sleep stage determination information obtained by measuring the sleep stage of the examinee by performing deep learning based on the biometric data; including, wherein the sleep stage determination device generates biometric image information obtained by converting the biometric data into a graphic image, and converts the generated biometric image information into a format corresponding to the input data format of the sleep stage determination model. Generates pre-processing data, provides the sleep stage determination information by performing deep learning based on the biometric pre-processing data based on the sleep stage determination model, and outputs the sleep stage determination information in a predetermined format.

Deep learning-based sleep stage determination apparatus and system according to an embodiment of the present invention determine the sleep stage based on the subject's biometric data based on the learned deep learning neural network (sleep stage determination model), There is an effect that can minimize the time and cost required to determine the sleep stage, and at the same time improve the accuracy and reliability.

In addition, the present invention has the effect of being able to quickly and conveniently determine the sleep stage of the examinee with expert level accuracy by automating the sleep stage determination process of the examinee based on the learned deep learning neural network.

In addition, the deep learning-based sleep stage determination apparatus and system according to an embodiment of the present invention perform machine learning based on a deep learning neural network based on a customized training data set optimized for sleep stage determination Thus, by implementing the sleep stage determination model, there is an effect that the quality of results output from the sleep stage determination model can be further improved.

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood from the description below.

1 is a conceptual diagram of a deep learning-based sleep stage determination system according to an embodiment of the present invention.
2 is an example of a diagram for explaining a sensor system according to an embodiment of the present invention.
3 is a block diagram of an apparatus for determining a sleep stage according to an embodiment of the present invention.
4 is an example of a state showing bio-image information according to an embodiment of the present invention.
5A and 5B are examples of diagrams for explaining sleep stages according to an embodiment of the present invention.
6 is a conceptual diagram for explaining a deep learning neural network learning method according to an embodiment of the present invention.
7 is an example of a configuration diagram of a deep learning neural network according to an embodiment of the present invention.
8 is a flowchart illustrating a method of performing deep learning-based sleep stage determination based on the sleep stage determination apparatus according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and a method for achieving them, will become apparent with reference to the embodiments described below in detail in conjunction with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first, second, etc. are used for the purpose of distinguishing one component from another, not in a limiting sense. Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components will be added. In addition, in the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when described with reference to the drawings, the same or corresponding components are given the same reference numerals, and the overlapping description thereof will be omitted. .

1 is a conceptual diagram of a deep learning-based sleep stage determination system according to an embodiment of the present invention.

Referring to FIG. 1 , a deep learning-based sleep stage determination system (hereinafter, a sleep stage determination system) according to an embodiment of the present invention may include a sensor system SS and a sleep stage determination apparatus 100 .

Here, the sensor system (SS) and the sleep stage determination device 100 according to the embodiment are based on the biometric data of the examinee based on the deep learning neural network (in the embodiment, the sleep stage determination model) learned by interworking. It is possible to provide a deep learning-based sleep stage determination service (hereinafter referred to as sleep stage determination service) for automatically determining the sleep stage of the examinee.

In addition, each component of FIG. 1 may be connected based on a network.

Here, the network refers to a connection structure capable of exchanging information between respective nodes, such as the sensor system (SS) and the sleep stage determination device 100, and an example of such a network is a 3rd Generation Partnership Project (3GPP) network. , LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal) Area Network), Bluetooth (Bluetooth) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, and the like are included, but are not limited thereto.

- Sensor Systems (SS)

In an embodiment of the present invention, the sensor system SS may sense a biosignal of the subject to obtain biometric data for the subject.

Here, the biometric data refers to a person's electroencephalography (EEG), electro-oculogram (EOG), electromyogram (EMG), electrocardiogram (EKG, ECG: Electrocardiogram), respiration, pulse, blood pressure, and heart rate. , past medical history, degree of aging and/or body mass index, etc. may refer to bio-related data, and may be data used as a reference measure for determining a person's health status.

In an embodiment of the present invention, the above biometric data will be described based on biometric data obtained based on the sensor system (SS) in polysomnography, but is not limited thereto.

2 is an example of a diagram for explaining a sensor system SS according to an embodiment of the present invention.

In detail, referring to FIG. 2 , in the embodiment, the sensor system SS is an EEG sensor for measuring EEG, an EEG sensor for measuring eye movement, and an EOG sensor for measuring muscle movement Electromyography (EMG) sensor, respiration sensor and/or respiratory effort sensor to measure respiration, electrocardiogram (EKG, ECG) sensor to measure the potential of the heart in relation to heartbeat, and lower extremity electromyogram (EMG) to measure movement of lower extremity muscles It may include a sensor and/or an image sensor (camera) that captures an image.

Also, in the embodiment, the sensor system SS may provide the sensing data obtained based on the above sensors as biometric data to an external device (in the embodiment, the sleep stage determining device 100 ).

Specifically, in the embodiment, the sensor system SS may provide the biometric data obtained from an electroencephalogram (EEG) sensor, an electro-eye (EOG) sensor, and/or an electromyography (EMG) sensor to the sleep stage determination device 100. In addition, the sleep stage determining apparatus 100 provided with this may perform functional operations such as checking whether a subject is in a sleep state and/or determining a sleep stage based on the biometric data.

In addition, in the embodiment, the sensor system (SS) provides the biometric data obtained from the respiration sensor and/or the respiration effort sensor to the sleep stage determining device 100, and the sleep stage determining device 100 is the breathing state of the examinee. can be confirmed, and through this, it is possible to detect symptoms such as hypopnea, apnea, hypoxia, decreased oxygen saturation and/or arrhythmia during sleep.

In addition, in the embodiment, the sensor system (SS) may provide the biometric data obtained based on the electrocardiogram (EKG, ECG) sensor to the sleep stage determining device 100, and through this, the sleep stage determining device 100 can confirm the state of the heart rhythm during sleep for the subject.

In addition, in the embodiment, the sensor system SS provides the biometric data obtained from the lower extremity electromyogram (EMG) sensor to the sleep stage determination device 100 to determine whether the subject is converted to a non-sleep state due to unintentional movement during sleep. can check whether

In addition, in an embodiment, the sensor system (SS) may provide image data obtained by an image sensor (camera) to the sleep stage determining device 100, and through this, the sleep stage determining device 100 is polysomnia It is possible to check the overall body movement state information along with the biometric data record of the subject who is undergoing the examination.

In addition, in the embodiment, the sensor system (SS), a photoplethysmoGraphy (PPG) sensor for measuring oxygen saturation and heart rate, a thermister and a flow sensor for measuring respiration, and movement of the abdomen and chest At least one biometric data measured based on at least one sensing means of a chest belt measuring chest belt, an abdomen belt, and/or a microphone sensing a voice signal can be further acquired and provided. may be

- Sleep-stage Determination Device (100: Sleep-stage Determination Device)

In an embodiment of the present invention, the sleep stage determination apparatus 100 automatically determines the sleep stage of the examinee based on the subject's biometric data based on the learned deep learning neural network (in the embodiment, the sleep stage determination model) Sleep stage determination service can be provided.

In detail, in the embodiment, the sleep stage determining apparatus 100 may receive biometric data for the subject obtained from the sensor system SS in the course of performing polysomnography from the sensor system SS.

In addition, in the embodiment, the sleep stage determining apparatus 100 may convert the biometric data received from the sensor system SS into a graphic image.

In addition, in the embodiment, the sleep stage determination apparatus 100 may process the information obtained by converting the biometric data into a graphic image into a form suitable for input to a deep learning neural network.

In addition, in the embodiment, the sleep stage determination device 100 inputs the processed data into the learned deep learning neural network (sleep stage determination model) so that the sleep state of the examinee is any sleep stage among predetermined sleep stages. It can be determined whether

In this case, in the embodiment, the sleep stage determination apparatus 100 may perform deep learning neural network learning based on a predetermined training data set in order to implement the sleep stage determination model.

In addition, in the embodiment, the sleep stage determining apparatus 100 may provide the information on the sleep stage of the examinee determined as described above to the outside according to a predetermined format.

3 is a block diagram of an apparatus 100 for determining a sleep stage according to an embodiment of the present invention.

Hereinafter, each component of the sleep stage determining apparatus 100 according to an embodiment of the present invention will be described in detail with reference to FIG. 3 .

Referring to FIG. 3 , in the embodiment of the present invention, the sleep stage determination apparatus 100 includes a biosignal detector 110 , a biosignal image generator 120 , a preprocessor 130 , and a deep learning classification unit 140 . and a result output unit 150 .

<Biosignal detection unit>

In an embodiment, the biosignal detection unit 110 of the sleep stage determining device 100 may receive biometric data for a subject acquired by the sensor system SS from the sensor system SS.

Here, the biometric data according to the embodiment is a living body including an electroencephalogram (EEG), an eye electrocardiogram (EOG), an electromyography (EMG), respiration, an electrocardiogram (EKG, ECG), and/or an electromyogram (EMG) of the lower extremities of the subject. It may mean related data, and may be data used as a reference scale for analyzing the sleep state of the subject.

In an embodiment, the biometric data may be data continuously measured from the sensor system SS for the subject within a preset time range (in the embodiment, the processing time of polysomnography, etc.).

That is, in the embodiment, the biosignal detection unit 110 includes at least one or more of a plurality of sensors including an electroencephalogram (EEG) sensor, an electro-eye (EOG) sensor, and/or an electromyography (EMG) sensor of the sensor system SS. Various types of biometric data may be acquired from the sensor through different channels within a predetermined time.

<Biosignal image generator>

In addition, in an embodiment, the biosignal image generator 120 of the sleep stage determining device 100 may generate biometric image information by graphically imaging each of various types of biometric data acquired by the biosignal detector 110 . .

4 is an example of a state showing bio-image information according to an embodiment of the present invention.

Referring to FIG. 4 , in an embodiment, the biosignal image generator 120 converts biometric data obtained through a predetermined channel from the biosignal detector 110 into a graphic image in the form of a graph to generate biometric image information. can

In detail, in the embodiment, the biosignal image generator 120 may include EEG biometric data obtained from an EEG sensor, EEG biometric data obtained from an EOG sensor, and/or EMG biometric data. Each of the biometric data including electromyography (EMG) biometric data obtained from the (EMG) sensor may be converted into a graphic image in the form of a graph.

And in an embodiment, the biosignal image generator 120 may generate biometric image information based on the biometric data by combining at least one graphic image in the form of a graph for each of the biometric data converted as described above.

<Preprocessor>

In addition, in an embodiment, the pre-processing unit 130 of the sleep stage determining device 100 may process the bio-image information generated by the bio-signal image generating unit 120 into a form suitable for input to the deep learning neural network.

That is, in the embodiment, the preprocessor 130 may convert biometric image information into biometric preprocessing data having a format optimized for deep learning input.

In detail, in an embodiment, the preprocessor 130 may filter and remove unnecessary noise data in the biometric image information, and divide the biometric image information from which the noise data has been removed into units of epochs that are predetermined time intervals. have.

In more detail, in the embodiment, the preprocessor 130 may match the sampling frequency for each sensor corresponding to the biometric data included in the biometric image information.

In an embodiment, the pre-processing unit 130, based on the interpolation process, sampling frequencies of electroencephalogram (EEG) biometric data, electro-ocular electromyography (EOG) biometric data, and/or electromyography (EMG) biometric data having different sampling frequencies from each other can match

In addition, the preprocessor 130 may extract necessary data corresponding to the sleep stage determination model according to the embodiment of the present invention based on the biometric data with the matching sampling frequency.

In this case, in the embodiment, the preprocessor 130 may determine the type of the necessary data based on at least one means of a predetermined criterion and/or a user input for performing sleep phase determination.

For example, the pre-processing unit 130 may separately extract electroencephalogram (EEG) biometric data, eye electroencephalography (EOG) biometric data, and electromyography (EMG) biometric data from among a plurality of biometric data whose sampling frequencies are matched as necessary data. have.

According to an embodiment, the preprocessor 130 may first extract necessary data from the biometric image information, and then match the sampling frequency for each sensor corresponding to the extracted necessary data.

Through this, the preprocessor 130 may filter noise data unnecessary for the sleep stage determination model according to the embodiment.

Also, in the embodiment, the preprocessor 130 may divide the necessary data extracted as above into time intervals (ie, epochs) corresponding to the sleep stage determination model.

In this case, in an embodiment, the preprocessor 130 may determine the epochs based on at least one of a predetermined criterion and/or a user input for determining the sleep stage.

For example, the pre-processing unit 130 may set 30 seconds, which is a time interval generally applied to sleep stage measurement, as epochs, and divide the necessary data into time intervals optimized for the sleep stage determination model. have.

That is, in the embodiment, the pre-processing unit 130 may filter and remove unnecessary noise data in the biometric image information, and convert the biometric image information (necessary data) from which the noise data is removed in epochs at a predetermined time interval. By dividing into , the format of the biometric image information can be converted into a format applicable as an input of a sleep stage determination model (that is, a deep learning neural network that determines a sleep stage), and biometric preprocessing data can be generated based on this. have.

<Deep Learning Classification Unit>

In addition, in the embodiment, the deep learning classification unit 140 of the sleep stage determination device 100 inputs the biometric preprocessing data generated by the preprocessor 130 to the learned deep learning neural network (sleep stage determination model), and the It may be determined whether the sleep state of the examinee corresponds to which sleep stage among predetermined sleep stages.

Here, the sleep stage according to the embodiment may be information obtained by classifying the sleep state of the examinee who is performing the multiple-source examination according to a predetermined criterion.

5A and 5B are examples of diagrams for explaining sleep stages according to an embodiment of the present invention.

In detail, referring to FIGS. 5A and 5B , in the embodiment, the sleep phase can be divided into a REM sleep (Rapid Eye Movement) phase with rapid pupil movement and a NREM (Non Rapid Eye Movement) sleep phase in which rapid eye movement does not occur. have.

In addition, the NREM sleep stage may be further divided into NREM1 sleep stage (stage 1 NREM sleep stage), NREM2 sleep stage (stage 2 NREM sleep stage) and/or NREM3 sleep stage (stage 3 NREM sleep stage).

Specifically, in an embodiment, the NREM1 sleep stage may be a stage between a sleep state and a waking state.

In this NREM1 sleep stage, brain waves (EEG) and muscle activity may start to slow down during sleep, and at this time, the brain waves (EEG) may mainly appear as theta waves (4-7 Hz) of low amplitude.

In addition, in the embodiment, the NREM2 sleep stage is a stage in shallow sleep, and in this stage, a sleep bundle of 12 to 14 Hz and a K-complex may appear.

In addition, in the embodiment, the NREM3 sleep stage is a stage in which the examinee enters deep sleep, and as the brain wave (EEG) slows down, a delta wave (0.5 to 4 Hz) may be detected, in which case the subject's body is less sensitive to external stimuli. can be

Here, the NREM sleep stage may be divided into an NREM3 sleep stage and an NREM4 sleep stage (fourth stage NREM sleep stage) according to the frequency of the delta wave.

At this time, the pulse of the examinee is very slow and changes stably, the eye movement is stopped, and the muscles can be relaxed.

Meanwhile, in the embodiment, in the stage of waking up from sleep, that is, in the stage where NREM sleep ends and enters REM sleep, theta wave of low amplitude appears and rapid eye movement may be detected again.

Returning again, that is, in the embodiment, the deep learning classification unit 140 inputs the biometric pre-processing data generated by the pre-processing unit 130 to the sleep stage determination model so that the sleep state of the examinee is any of the above predetermined sleep stages. It can be determined whether the stage is appropriate.

In addition, in an embodiment, the deep learning classification unit 140 may provide sleep stage determination information for the subject based on the result of the determination.

Here, the sleep stage determination information according to the embodiment may be information in which it is determined which sleep stage the examinee enters among the plurality of sleep stages (eg, REM, NREM1, NREM2 and/or NREM3, etc.).

In detail, in the embodiment, the deep learning classification unit 140 determines the sleep stage of the subject based on the given input data (in the embodiment, biometric preprocessing data) and provides output data (in the embodiment, sleep stage determination information) You can train a deep learning neural network to do this.

6 is a conceptual diagram for explaining a deep learning neural network learning method according to an embodiment of the present invention.

In more detail, referring to FIG. 6 , in the embodiment, the deep learning classification unit 140 performs a polysomnography test previously performed to implement the deep learning neural network (ie, the sleep stage determination model 10) operating as above. It is possible to obtain a predetermined training data set (Training Data Set) based on the result.

In an embodiment, the training data set is based on a set of sleep stage determination result data (in the embodiment, sleep stage determination information) determined by an expert based on the subject's biometric data in the previously performed polysomnography test. can be implemented.

Also, in an embodiment, the deep learning classifier 140 may train a deep learning neural network based on the above training data set.

As such, the deep learning classification unit 140 implements the sleep stage determination model 10 by performing machine learning based on a deep learning neural network based on a customized training data set optimized for sleep stage determination. By doing so, the quality of the result output from the sleep stage determination model 10 can be improved.

7 is an example of a configuration diagram of a deep learning neural network according to an embodiment of the present invention.

Referring to FIG. 7 , in this case, the deep learning neural network according to the embodiment may be a predetermined classification model based on an artificial neural network.

In an embodiment, the deep learning neural network is an algorithm for updating the weight of the neural network using labeled data of an output layer provided in the deep learning neural network. Back propagation It may be a predetermined classification model that performs deep learning based on input data based on an algorithm and/or a weight loss function.

For example, the deep learning neural network is at least one of a predetermined classification model such as a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and/or a Bidirectional Recurrent Deep Neural Network (BRDNN). may be implemented, and according to an embodiment, the deep learning classification unit 140 may utilize various types of artificial neural networks without being limited to the above-described examples.

In addition, in the embodiment, the deep learning classification unit 140 performs deep learning neural network learning based on the training data set, and performs deep learning based on the given input data (eg, biometric preprocessing data) to It is possible to implement the sleep stage determination model 10 that determines the sleep stage and provides it as output data (eg, sleep stage determination information).

That is, in the embodiment, the sleep stage determination model 10 may be a deep learning neural network that takes as an input biometric preprocessing data and outputs sleep stage determination information.

In summary, in the embodiment, the deep learning classification unit 140 can train a deep learning neural network based on the results of the pre-performed polysomnography test, and the learned deep learning neural network (sleep stage determination model 10) ), it is possible to measure and provide a sleep stage based on the subject's biometric data.

As such, in the embodiment, the deep learning classification unit 140 determines the sleep stage based on the subject's biometric data based on the learned deep learning neural network (sleep stage determination model 10), thereby determining the sleep stage of the subject It is possible to improve the accuracy and reliability while minimizing the time and cost required to make a judgment.

In addition, the deep learning classification unit 140, by automating the sleep stage determination process of the examinee based on the sleep stage determination model 10, can quickly and conveniently determine the sleep stage of the examinee with expert level accuracy.

<Result output part>

In addition, in the embodiment, the result output unit 150 of the sleep stage determination device 100 may provide the sleep stage determination information of the examinee obtained based on the deep learning classification unit 140 to the outside according to a predetermined format. have.

In detail, in an embodiment, the result output unit 150 may generate and output sleep state information based on the sleep stage determination information obtained from the deep learning classification unit 140 .

Here, the sleep state information according to the embodiment may be information indicating analysis data of the examinee's overall sleep based on a graphic image (eg, graph and/or text, etc.).

In an embodiment, the sleep state information may include the sleep stage determination information, biometric preprocessing data and/or biometric image information related to the sleep stage determination information.

In more detail, in the embodiment, the result output unit 150 may convert the sleep stage determination information obtained by the deep learning classification unit 140 into a predetermined graphic image format including a graph and/or text.

And the result output unit 150, based on the graphic image converted as described above, may generate sleep state information representing the analysis data of the subject's overall sleep.

In addition, in an embodiment, the result output unit 150 may output the generated sleep state information based on its own display, or transmit and output the generated sleep state information to an external device (eg, a computing device, etc.).

<Other components>

On the other hand, with further reference to FIG. 3 , the apparatus 100 for determining sleep phase in an embodiment of the present invention may further include a communication unit 160 , a database unit 170 , and a processor 180 .

In detail, in the embodiment, the communication unit 160 may transmit/receive various data and/or information for providing a sleep stage determination service.

In an embodiment, the communication unit 160 may include the biosignal detection unit 110 , and may transmit/receive various data related to the sleep stage determination service by communicating with the sensor system SS and/or an external device.

This communication unit 160, the technical standards or communication methods for mobile communication (eg, GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA ( High Speed Uplink Packet Access), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc. can send and receive

In addition, in the embodiment, the database unit 170 , It is possible to store any one or more of various application programs, data, and commands that provide a sleep stage determination service.

In an embodiment, the database unit 170 includes a training data set (Training Data Set), a deep learning model algorithm, a learning algorithm, biometric data, biometric image information, biometric preprocessing data, sleep stage determination information and/or sleep state information, etc. can be stored and managed.

The database unit 170 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, and the like, and web storage that performs the storage function of the database unit 170 on the Internet (internet) ) may be

In addition, in the embodiment the processor 180 , A series of data processing may be performed by controlling the above-mentioned factors to perform the sleep phase determination service.

These processors 180, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers (controllers), It may be implemented using at least one of micro-controllers, microprocessors, and other electrical units for performing functions.

- Operation method of sleep stage determination device

Hereinafter, a method for the processor 180 of the sleep stage determining apparatus 100 according to an embodiment of the present invention to determine the sleep stage of a subject based on deep learning with reference to the accompanying drawings will be described.

8 is a flowchart illustrating a method of performing deep learning-based sleep stage determination based on the sleep stage determination apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 8 , in the embodiment, the processor 180 of the apparatus for determining a sleep stage 100 may obtain biometric data of a subject by interworking with the sensor system SS. (S101)

In detail, in the embodiment, the processor 180 includes a plurality of sensors (eg, an electroencephalogram (EEG) sensor, an electro-eye (EOG) sensor, an electromyography (EMG) sensor, Biometric data for a subject obtained from a respiration sensor, an electrocardiogram (EKG, ECG) sensor, and an image sensor) may be received from the sensor system SS based on the biosignal detector 110 .

Also, in an embodiment, the processor 180 may generate biometric image information based on the biometric data obtained as described above. (S103)

In detail, the processor 180 may generate bio-image information obtained by graphically imaging the bio-data of the subject based on the bio-signal image generating unit 120 .

Specifically, the processor 180 is configured to: EEG biometric data obtained from an EEG sensor of the sensor system SS, EEG biometric data obtained from an EOG sensor, and/or EMG biometric data Each of the biometric data including electromyography (EMG) biometric data obtained from the (EMG) sensor may be converted into a graphic image in the form of a graph.

In an embodiment, the processor 180 may combine at least one graphic image in the form of a graph for each of the biometric data converted as described above to generate biometric image information of the subject based on the biometric data.

Also, in an embodiment, the processor 180 may convert the generated biometric image information into biometric preprocessing data in a deep learning input optimization format. (S105)

In detail, the processor 180 may acquire the biometric preprocessing data by converting the biometric image information into a form suitable for input to the sleep stage determination model 10 based on the preprocessing unit 130 .

First, the processor 180 may filter and remove unnecessary noise data in the biometric image information during the conversion process.

Second, the processor 180 includes a sensor corresponding to each of the biometric data included in the biometric image information (in the embodiment, an EEG sensor, an EOG sensor, an electromyography (EMG) sensor, a respiration sensor, and an electrocardiogram). Sampling frequencies for each (EKG, ECG) sensor and/or image sensor, etc.) may be matched.

Third, in the embodiment, the processor 180 may extract input data necessary for the sleep stage determination model 10 from the biometric data in which each sampling frequency is matched as necessary data.

For example, the processor 180 may convert the electroencephalogram (EEG) biometric data, the eye electroencephalogram (EOG) biometric data, and the electromyography (EMG) biometric data among a plurality of biometric data with matching sampling frequencies, according to an embodiment of the present invention. By extracting as necessary data to be input to the sleep stage determination model 10 , filtered biometric image information may be generated.

Fourth, in the embodiment, the processor 180 may divide the filtered bio-image information into epochs, which is a predetermined time interval for inputting the above-filtered bio-image information to the sleep stage determination model 10 .

In this case, in an embodiment, the processor 180 may determine the epochs based on at least one of a predetermined criterion optimized for determining a sleep stage and/or a user input.

Illustratively, the processor 180 sets, as an epoch, 30 seconds, which is a time interval generally applied when measuring the sleep stage, as an epoch, and sets the filtered biometric image information at a preset time interval of the sleep stage determination model 10 . can be divided into

As described above, in the embodiment, the processor 180 removes unnecessary noise data from the bio-image information and divides the bio-image information from which the noise data has been removed into units of a predetermined time interval (epochs), so that the bio-signal image The biometric preprocessing data obtained by converting the format of the biometric image information generated by the generator 120 into the format of the input format of the sleep stage determination model 10 may be generated.

Also, in an embodiment, the processor 180 may obtain sleep stage determination information by inputting the generated biometric preprocessing data into the sleep stage determination model 10 (deep learning neural network). (S107)

Here, the sleep stage determination information according to the embodiment means that the subject has a plurality of sleep stages (in the embodiment, a REM (Rapid Eye Movement) sleep stage, a Non Rapid Eye Movement (NREM) 1 sleep stage, a NREM2 sleep stage, a NREM3 sleep stage and NREM4 sleep stage, etc.) may be information for determining which sleep stage has entered.

In detail, in the embodiment, the processor 180 determines the sleep stage of the examinee based on the given input data (ie, biometric preprocessing data) based on the deep learning classification unit 140 , and outputs data (ie, sleep phase determination) We can train a deep learning neural network to provide information).

Illustratively, the deep learning neural network is at least one of a predetermined classification model such as a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and/or a Bidirectional Recurrent Deep Neural Network (BRDNN). It may be implemented including the above, but is not limited thereto.

In this case, in an embodiment, the processor 180 may train the deep learning neural network based on a customized training data set optimized for sleep stage determination learning.

In an embodiment, the processor 180 trains a set of sleep phase determination result data (in the embodiment, sleep phase determination information) determined by an expert based on the subject's biometric data in the previously performed polysomnography examination. A deep learning neural network can be trained using a data set.

Thus, in the embodiment, the processor 180 may implement the sleep stage determination model 10 that is learned based on the training data set and outputs sleep stage determination information based on the subject's biometric preprocessing data.

In addition, in the embodiment, the processor 180 inputs the biometric preprocessing data of the examinee to the sleep stage determination model 10 implemented as above, and receives sleep stage determination information for the examinee from the sleep stage determination model 10 can be obtained

The deep learning neural network according to the embodiment may include a first deep learning model and a second deep learning model.

The processor converts the first deep learning model to data from an electroencephalography (EEG) sensor, an electro-oculogram (EOG) sensor, and an electromyogram (EMG) sensor, and the sleep stage determination correct answer label determined by the expert at this time. can be trained with

)을 이용할 수 있다. 제 1-1 모델 손실(

)은 제 1 딥러닝 모델이 출력한 제 1 수면단계 판정 정보(

)와 트레이닝 데이터 셋의 수면단계 판정 정답 레이블(

)을 서로 비교하여 제 1 수면단계 판정 정보가 정답에 가깝게 하도록 하는 손실이다. 제 1-1 모델 손실(

)은 수학식 1과 같다.The first deep learning model loses the 1-1 model during training (

) can be used. 1-1 Model loss (

) is the first sleep stage determination information output by the first deep learning model (

) and the sleep stage determination correct label of the training data set (

) is a loss that makes the first sleep stage determination information close to the correct answer by comparing each other. 1-1 Model loss (

) is the same as in Equation 1.

(Equation 1)

The processor may train the second deep learning model based on an electroencephalography (EEG) sensor value and a sleep stage determination correct answer label determined by an expert.

)을 이용할 수 있다. 이 때, 제 1-1 모델 손실(

)와 비교하여 제 1-1 모델 손실(

)는 제 1 딥러닝 모델의 트레이닝에 사용되는 손실이나 제 1-2 모델 손실(

)은 프로세서의 전체 신경망을 트레이닝할 때 사용되는 기준 손실일 수 있다. And the processor loses the first 1-2 model when training the second deep learning model (

) can be used. At this time, the 1-1 model loss (

) compared to the 1-1 model loss (

) is the loss used for training the first deep learning model or the loss of the first 1-2 model (

) may be the reference loss used when training the entire neural network of the processor.

)은 제2 딥러닝 모델이 출력한 제 2 수면단계 판정 정보(

)와 트레이닝 데이터 셋의 수면단계 판정 정답 레이블(

)을 서로 비교하여 상기 제 2 수면단계 판정정보의 뇌파와 수면단계의 상관관계가 수면단계 판정 정답 레이블의 뇌파와 수면단계의 상관관계에 매칭되도록 하는 손실이다. 제 1-2 모델 손실(

)은 수학식 2와 같다.Section 1-2 model loss (

) is the second sleep stage determination information (

) and the sleep stage determination correct label of the training data set (

) is compared with each other so that the correlation between the brain wave and the sleep stage of the second sleep stage determination information matches the correlation between the brain wave and the sleep stage of the sleep stage determination correct label. Section 1-2 model loss (

) is the same as in Equation 2.

(Equation 2)

)을 이용할 수 있다. 즉, 프로세서는 트레이닝시 GAN 개념을 사용하여 수면판정 정답 레이블과 일치 성능을 향상시킬 수 있다. 제 1-2 모델 손실(

)과 같이 강도 차이 기반 손실의 한계는 REM(Rapid Eye Movement) 수면단계와 NREM(Non Rapid Eye Movement)1 수면단계를 구별하기 어렵다는 것이다. In addition, the processor loses the first hostility loss (

) can be used. That is, the processor can use the GAN concept during training to improve sleep determination correct label and matching performance. Section 1-2 model loss (

), the limitation of intensity-difference-based loss is that it is difficult to differentiate between the rapid eye movement (REM) sleep phase and the non-rapid eye movement (NREM)1 sleep phase.

To solve this, the processor may use the first determining unit. The first discriminating unit may reliably discriminate a rapid eye movement (REM) sleep stage and a non-rapid eye movement (NREM)1 sleep stage by using the eyeball conduction value as an additional parameter. For example, when the first determining unit detects rapid eye movement from the eye conduction value, the second deep learning model may be trained to determine the REM (Rapid Eye Movement) sleep stage regardless of the EEG value.

The deep learning neural network including these two deep learning models has the advantage that simple sleep stage determination is possible by determining the sleep stage through the second deep learning model in a situation where only brain waves can be measured.

In addition, the deep learning neural network trains the first deep learning model including EEG, eyeball conduction, and electromyography values, trains the second deep learning model through the most important EEG to determine the sleep stage, and compares the two models. By learning, it is possible to improve the accuracy of the deep learning model by applying a weight to the EEG values and by learning by comparing the first deep learning model and the second deep learning model in a situation where the amount of data in the training data set is insufficient.

In addition, in an embodiment, the processor 180 may provide the obtained sleep stage determination information to the outside in a predetermined format. (S109)

In detail, the processor 180 may generate sleep state information for the subject based on the sleep stage determination information, based on the result output unit 150 .

Here, the sleep state information according to the embodiment may be information indicating analysis data of the examinee's overall sleep based on a graphic image (eg, graph and/or text, etc.).

In an embodiment, the sleep state information may include the sleep stage determination information, biometric preprocessing data and/or biometric image information related to the sleep stage determination information.

Specifically, in the embodiment, the processor 180 may convert the sleep stage determination information into a predetermined graphic image format including a graph and/or text.

In addition, the processor 180 may generate sleep state information representing analysis data of the examinee's overall sleep based on the converted graphic image.

In addition, in an embodiment, the processor 180 may output the sleep state information generated as described above based on its own display or transmit it to an external device (eg, a computing device, etc.) and provide it.

Above, the deep learning-based sleep stage determination apparatus and system according to an embodiment of the present invention, based on the learned deep learning neural network (sleep stage determination model 10), the sleep stage based on the subject's biometric data By judging, there is an effect of minimizing the time and cost required for determining the sleep stage of the examinee, and at the same time improving the accuracy and reliability.

In addition, the present invention has the effect of being able to quickly and conveniently determine the sleep stage of the examinee with expert level accuracy by automating the sleep stage determination process of the examinee based on the learned deep learning neural network.

In addition, the deep learning-based sleep stage determination apparatus and system according to an embodiment of the present invention perform machine learning based on a deep learning neural network based on a customized training data set optimized for sleep stage determination Thus, by implementing the sleep stage determination model 10 , there is an effect that the quality of the result output from the sleep stage determination model 10 can be further improved.

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and used by those skilled in the art of computer software. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be converted into one or more software modules to perform processing in accordance with the present invention, and vice versa.

The specific implementations described in the present invention are only examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connecting members of the lines between the components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional may be referred to as connections, or circuit connections. In addition, unless there is a specific reference such as “essential” or “importantly”, it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, those skilled in the art or those having ordinary knowledge in the art will appreciate the spirit of the present invention described in the claims to be described later. And it will be understood that various modifications and variations of the present invention can be made without departing from the technical scope. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (10)

a biosignal detection unit for receiving the biometric data from a sensor system for acquiring the subject's biometric data;
a biosignal image generating unit that generates biometric image information obtained by converting the biometric data into a graphic image;
a preprocessor for generating biometric preprocessing data by converting the biometric image information into a format corresponding to the input data format of the sleep stage determination model;
a deep learning classification unit providing sleep stage determination information obtained by measuring the sleep stage of the examinee by performing deep learning based on the biometric preprocessing data based on the sleep stage determination model; and
a result output unit for outputting the sleep stage determination information in a predetermined format; including,
The biometric data is a biosignal obtained from at least one of an electroencephalography (EEG) sensor, an electro-oculogram (EOG) sensor, an electromyogram (EMG) sensor, a breathing sensor, an electrocardiogram sensor, and an image sensor. measurement data,
The sleep stage determination model includes a first deep learning model trained based on biometric data including brain waves, eyeball conduction and electromyography data, a sleep stage determination correct label set in the biometric data, the brain wave data, and the sleep Includes a second deep learning model trained based on the step determination correct label,
The deep learning classification unit, when receiving EEG, EEG and EMG data from the sensor system of the subject, measures the sleep stage of the subject through the first deep learning model, and receives EEG data from the sensor system of the subject Measuring the sleep stage of the subject through the second deep learning model,
The second deep learning model is characterized in that it is comparatively learned with the first deep learning model
Deep learning-based sleep stage determination device. delete The method of claim 1,
The sleep stage determination model is,
A deep learning neural network trained to receive the biometric preprocessing data as an input and output the sleep stage determination information based on a predetermined training data set
Deep learning-based sleep stage determination device. 4. The method of claim 3,
The training data set is
It is implemented based on the set of sleep stage determination information determined by an expert in the previously performed polysomnography test,
The biometric preprocessing data is
Filtering and removing noise data from a plurality of biosignal measurement data,
Match the sampling frequency of the plurality of biosignal measurement data,
Extracting necessary input data from a plurality of biosignal measurement data having the same sampling frequency
Deep learning-based sleep stage determination device. The method of claim 1,
The sleep stage determination information,
It is information that determines which sleep stage the examinee enters among REM sleep stage, NREM1 sleep stage, NREM2 sleep stage, NREM3 sleep stage, and NREM4 sleep stage.
Deep learning-based sleep stage determination device. The method of claim 1,
The preprocessor is
The noise data in the biometric image information is filtered and removed, and the biometric image information from which the noise data is removed is divided into epochs, which are predetermined time intervals corresponding to the input data of the sleep stage determination model, and the biometric preprocessing is performed. generating data
Deep learning-based sleep stage determination device. 4. The method of claim 3,
The deep learning classification unit,
A deep learning neural network including at least one of a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN) is trained with the training data set to determine the sleep stage implementing the model
Deep learning-based sleep stage determination device. The method of claim 1,
The result output unit,
Based on the sleep stage determination information, generating sleep state information representing the analysis data throughout the sleep of the examinee as a graphic image,
The sleep state information,
At least one of the sleep stage determination information, the biometric preprocessing data related to the sleep stage determination information, and the biometric image information
Deep learning-based sleep stage determination device. 9. The method of claim 8,
The result output unit,
outputting the sleep state information based on at least one of a self-display and an external device
Deep learning-based sleep stage determination device. a sensor system for acquiring biometric data of a subject; and
a sleep stage determination device for providing sleep stage determination information obtained by measuring the sleep stage of the examinee by performing deep learning based on the biometric data; including,
The sleep stage determination device,
Generates biometric image information converted from the biometric data into a graphic image, and generates biometric preprocessing data converted from the generated biometric image information into a format corresponding to the input data format of the sleep stage determination model, the sleep stage determination model to provide the sleep stage determination information by performing deep learning based on the biometric preprocessing data based on, and output the sleep stage determination information in a predetermined format,
The biometric data is a biosignal obtained from at least one of an electroencephalography (EEG) sensor, an electro-oculogram (EOG) sensor, an electromyogram (EMG) sensor, a breathing sensor, an electrocardiogram sensor, and an image sensor. measurement data,
The sleep stage determination model includes a first deep learning model trained based on biometric data including brain waves, eyeball conduction and electromyography data, a sleep stage determination correct label set in the biometric data, the brain wave data, and the sleep Includes a second deep learning model trained based on the step determination correct label,
The sleep stage determination device, upon receiving EEG, EEG and EMG data from the sensor system of the subject, measures the sleep stage of the subject through the first deep learning model, and receives EEG data from the sensor system of the subject If the sleep stage of the subject is measured through the second deep learning model,
The second deep learning model is characterized in that it is comparatively learned with the first deep learning model
Deep learning-based sleep stage determination system.

Images