KR102268804B1 - Method and apparatus for providing lighting and music according to sleep conditions using artificial intelligence - Google Patents

Abstract

Disclosed is a method and device for providing lighting and music depending on a sleep state using artificial intelligence (AI). The device for providing the lighting and music depending on the sleep state using the AI comprises: at least one processor; and a memory storing instructions by which the at least one processor performs at least one operation.

Description

A method and apparatus for providing lighting and music according to the sleep state using AI {METHOD AND APPARATUS FOR PROVIDING LIGHTING AND MUSIC ACCORDING TO SLEEP CONDITIONS USING ARTIFICIAL INTELLIGENCE}

The present invention relates to a method and apparatus for detecting a sleep state using AI and providing light and music, and more particularly, detects a user's sleep state by analyzing biometric information and image information using an artificial neural network, A method and apparatus for providing lighting and music corresponding to a sensed sleep state.

Sleep must be provided for one-third or one-quarter of a day in order to recover body fatigue and maintain circadian rhythm. As society is advanced, people's interest in improving the quality of sleep is greatly increasing.

In particular, as various display devices, including smartphones, are widely used, people are exposed to light provided by the display for a long time and often complain of insomnia.

Therefore, there are many people who want to maintain a sleep state more easily and deeply by providing appropriate lighting or providing music during sleep. However, it is difficult to provide the user with lighting or music that is appropriately changed according to the sleeping state, because all of the conventional sleep mood lights and music player devices emit light determined by the user's operation or play music sequentially or randomly. .

On the other hand, artificial intelligence first appeared in 1956, and deep learning-based artificial neural networks were proposed in the 1980s, but the processing power of computing devices did not accommodate deep learning-based artificial neural networks, so it was difficult to spread.

However, recently, as the processing power of computing devices has rapidly improved, the effectiveness and performance of deep learning-based artificial neural networks have been recognized. However, such deep learning-based artificial neural networks show a large performance difference depending on how the data is made in the form of training data and supervised learning.

In addition, in a deep learning-based artificial neural network, a loss function that minimizes the difference between a target vector representing a target result and an output value of the artificial neural network and a loss function that minimizes the difference between the target vector and the output value of the artificial neural network are completely different depending on how the target vector is determined.

Therefore, there is a need for a technology for actively researching such deep learning-based artificial neural networks to provide lighting or music suitable for a user's sleep state.

Republic of Korea Patent Publication No. 10-2019-0100888 (2019.08.29.)

An object of the present invention for solving the above problems is to provide a method and apparatus for providing lighting and music according to a sleep state using AI.

One aspect of the present invention for achieving the above object provides an apparatus for providing lighting and music according to a sleep state using AI (Artificial Intelligence).

An apparatus for providing lighting and music according to a sleep state using artificial intelligence (AI) includes: at least one processor; and a memory for storing instructions for the at least one processor to perform at least one operation.

The at least one operation may include: obtaining biometric information of a user measured using a wearable device, and obtaining image information of the user photographed using a camera; determining a first sleep state by inputting the image information into a convolutional neural network (CNN); determining a second sleep state based on a motion signal included in the biometric information; determining a third sleep state based on a photo-plethysmograph (PPG) signal included in the biometric information; converting the first sleep state, the second sleep state, and the third sleep state into first to third sleep state vectors with reference to a first predefined matching table, respectively; inputting the converted first to third sleep state vectors into a pre-supervised learning recurrent neural network (RNN)-based artificial neural network; obtaining a final sleep state vector indicating the final sleep state of the user as an output of the artificial neural network; and selecting lighting and music corresponding to the final sleep state vector.

The first matching table may indicate preset one-hot vectors to correspond 1:1 with preset sleep states.

The final sleep state vector may be one of the one-hot vectors indicated by the first matching table.

In the selecting of the lighting and music, the level of illumination may be determined based on the component value '1' of the final sleep state vector.

The final sleep state vector may be one of the one-hot vectors indicated by the first matching table.

The step of selecting the lighting and music may include: converting a sound wave signal according to time of each of a plurality of candidate songs to a frequency domain by performing fast Fourier transform (FFT); Filtering and obtaining a sound wave signal corresponding to 20 to 20000 (Hz) corresponding to the audible frequency in the converted frequency domain; dividing the filtered sound wave signal into equal frequency sections corresponding to each of the component values of the final sleep state vector, and integrating the sound wave signal in a frequency domain for each of the divided frequency sections; and determining a frequency section having the largest result value of the integrated sound wave signal, and selecting candidate music in which the component value of the final sleep state vector corresponding to the determined frequency section is '1' as the music to be provided to the user. may include.

The artificial neural network based on the recurrent neural network is composed of input nodes corresponding to components of each of the first to third sleep state vectors, and a first to each component of the first to third sleep state vectors is provided. an input layer that applies a connection strength matrix and transmits it to the hidden layer; a hidden layer that generates an intermediate target vector by using the output value transmitted from the input layer, and transmits the score vector generated by applying a second connection strength matrix to the intermediate target vector to the output layer; and an output layer that determines a probability corresponding to the score vector by applying an activation function to the score vector.

When using the method and apparatus for providing lighting and music according to the sleep state using AI according to the present invention as described above, the first sleep state identified in the image information, the second sleep state identified in the motion signal, and the PPG signal By using the third sleep state identified in , as input vectors to the artificial neural network, an object vector is generated, and the final sleep state is determined by learning the artificial neural network based on the object vector, so the most optimal sleep state is accurately determined. can do.

In addition, since it is possible to determine the primary first sleep state using CNN using image-based input data, there is an advantage in that the processing of inaccurate and ambiguous image information can be standardized and displayed as a sleep state.

1 is a schematic diagram for explaining a method and apparatus for providing lighting and music according to a sleep state using AI according to an embodiment.
FIG. 2 is a schematic diagram schematically illustrating an operation of the service providing apparatus according to FIG. 1 .
3 is a diagram for explaining a process of determining a one-hot vector and illuminance corresponding to sleep states.
4 is a conceptual diagram for explaining a method of selecting music based on a position of a component value 1 of a final sleep state vector according to an embodiment.
FIG. 5 is a block diagram illustrating the structure of the convolutional neural network according to FIG. 2 .
6 is a diagram for explaining a method of determining a second sleep state based on a motion intensity.
7 is a diagram for explaining a method of determining a third sleep state based on a PPG signal.
8 is a block diagram illustrating the structure of an artificial neural network based on a recurrent neural network according to FIG. 2 .
9 is a diagram illustrating a basic hardware configuration of the service providing apparatus according to FIG. 1 .

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram for explaining a method and apparatus for providing lighting and music according to a sleep state using AI according to an embodiment.

Referring to FIG. 1 , an apparatus 100 (which may be abbreviated as a service providing apparatus hereinafter) that provides lighting and music according to a sleep state using AI according to an embodiment of the present invention is worn on a user's body. It can operate in conjunction with the wearable device 200 that measures the user's biometric information.

The service providing apparatus 100 may determine the final sleep state of the user based on the user's image information and biometric information, and provide lighting and music to the user according to the determined final sleep state.

The wearable device 200 is a smart band, a smart watch, smart clothing, and the like, and may include a motion sensor detecting a user's motion and a biosignal sensor detecting a user's heartbeat activity state.

The motion sensor may include, but is not limited to, a geomagnetic sensor, a gyroscope sensor, a gravity sensor, etc. capable of detecting a user's motion.

The biosignal sensor may be a photo-plethysmograph (PPG) sensor for estimating a heartbeat activity state by measuring blood flow in the user's body through optical characteristics, but is not limited thereto.

The wearable device 200 measures biometric information including a motion signal indicating a user's motion detected through a motion sensor and a PPG signal of a user detected through a biosignal sensor, and periodically or In response to the request of the service providing apparatus 100 , the data may be transmitted to the service providing apparatus 100 .

The service providing apparatus 100 may acquire image information including the user's face through an external camera or a camera integrally installed therein. The service providing apparatus 100 may determine the user's first sleep state based on the obtained image information. Specifically, the service providing apparatus 100 inputs a face image corresponding to the user's face to a pre-supervised convolutional neural network 10 , and outputs the first sleep as an output of the convolutional neural network. state can be determined. To this end, the service providing apparatus 100 may supervise the convolutional neural network 10 in advance.

Also, the service providing apparatus 100 may determine the second to third sleep states based on the motion signal and the PPG signal included in the biometric information.

Also, the service providing apparatus 100 may determine the user's final sleep state by analyzing the first to third sleep states using the artificial neural network 20 . That is, the first to third sleep states may be viewed as initial data for determining the user's sleep state, and the final sleep state may be the user's definitively determined sleep state.

As such, the service providing apparatus 100 utilizes, as a primary indicator, the first to third sleep states analyzed through various basic data that may cause unclear or ambiguous results when determining the user's sleep state, and By analyzing the first to third sleep states through the artificial neural network 20, a final sleep state that best represents the user's current sleep state is finally determined. Accordingly, the accuracy of determining the sleep state can be greatly improved.

In addition, when only a motion signal or a PPG signal itself is learned by an artificial neural network, there is a problem in that it is difficult to select the learning data used for supervised learning, and the result of determining the sleep state varies greatly as the learning data changes. Therefore, in the present invention, after determining the second to third sleep states as primary indicators based on the motion signal and the PPG signal, the determined second to third sleep states are combined with the first sleep state to form an artificial neural network (20). ), so the accuracy is much higher and the ease of implementation can be improved.

FIG. 2 is a schematic diagram schematically illustrating an operation of the service providing apparatus according to FIG. 1 .

Referring to FIG. 2 , the service providing apparatus 100 may input a face image corresponding to the user's face into the pre-trained convolutional neural network 10 . In this case, as the face image, a front or side face image of the user may be used.

The service providing apparatus 100 measures the user's brain waves and eye movements and uses the user's sleep state accurately measured as a result value, and receives training data using the user's face image according to the sleep state as an input value in advance, The convolutional neural network 10 may be supervised in advance by using the received training data.

The service providing apparatus 100 may determine the first sleep state indicating the user's sleep state determined based on the user's face image as an output of the convolutional neural network 10 . The service providing apparatus 100 may convert the determined first sleep state into a first sleep state vector Vx.

Meanwhile, the service providing apparatus 100 may determine a second sleep state based on a motion signal included in the biometric information, and convert the determined second sleep state into a second sleep state vector Vx.

Also, the service providing apparatus 100 may determine a third sleep state based on the PPG signal included in the biometric information, and convert the determined third sleep state into a third sleep state vector Vz.

Here, the first to third sleep state vectors Vx, Vy, and Vz may be preset one-hot vectors to correspond 1:1 with preset sleep states. The one-hot vector is a vector in which one component value is '1' and all other component values are '0', and may be used as a vector uniquely representing each of the sleep states. To this end, the service providing apparatus 100 may store in advance a first matching table indicating a one-hot vector corresponding to each of the preset sleep states 1:1. The service providing apparatus 100 determines a one-hot vector corresponding to each of the sleep states by referring to the first matching table, and sets the determined one-hot vector to the first to third sleep states (Vx, Vy, Vz) You can decide on one of these.

On the other hand, the convolutional neural network 10 receives an image as input data, determines a probability indicating which class the received image corresponds to among preset classes, and sets the class having the highest determined probability as an output value can be presented as Here, the preset classes may mean a plurality of preset sleep states.

The service providing apparatus 100 inputs the first to third sleep state vectors Vx, Vy, and Vz corresponding to each of the first to third sleep states to the artificial neural network 20 , and the artificial neural network 20 ), a final sleep state vector Vf corresponding to the final sleep state may be obtained. Here, the final sleep state vector Vf may be converted into the final sleep state with reference to the first matching table. The final sleep state vector Vf may also be a preset one-hot vector to correspond 1:1 with preset sleep states.

Here, the artificial neural network 20 may be a recurrent neural network (RNN) or a Long Shot Term Memory (LSTM) neural network. In a recurrent neural network (RNN), there is a recursive edge between the hidden nodes of the hidden layer. Accordingly, the recurrent neural network (RNN) has an advantage in that it is easy to determine the final sleep state in consideration of changes in the first to third sleep states over time of the user.

In order to learn the artificial neural network 20, first, the service providing apparatus 100 converts the first to third sleep states determined using the user's image information and biometric information into first to third sleep state vectors, respectively. input to the artificial neural network 20 . In this case, the actual sleep state of the user at the time when the first to third sleep states are determined is accurately measured using brain waves or the like. The service providing apparatus 100 converts the measured actual sleep state into a one-hot vector for learning by referring to the first matching table, and converts the converted one-hot vector for learning into a target one-hot in the artificial neural network 20 . used as a vector. In this case, after the first to third sleep states are continuously determined over time, the first to third sleep state vectors may be input to the artificial neural network 20 in the form of a sequence. Accordingly, the artificial neural network 20 receives the input first to third sleep state vectors, and the difference between the output value output through the output layer and the actual sleep state accurately measured using the preceding EEG and the corresponding one-hot vector for learning. supervised learning is minimized.

3 is a diagram for explaining a process of determining a one-hot vector and illuminance corresponding to sleep states.

Specifically, referring to FIG. 3 , the plurality of sleep states include an arousal state divided into n1 stages, a non-REM sleep state divided into n2 stages, and REM divided into n3 stages. may include sleep states. Here, the REM (REM) sleep state corresponds to a sleep in which rapid eye movements are observed among sleep stages, and refers to a sleep state in which the eyes are usually dreaming, and the non-REM sleep state is a sleep stage in which eye movements are not observed. may be, and the arousal state may be a stage other than a sleep state.

REM sleep can be generally confirmed through the appearance of fast and low-voltage beta waves in EEG analysis, but in the present invention, it is possible to determine the REM sleep state as the final sleep state based on motion signals, image information, PPG signals, and the like.

In the present invention, all of n1, n2, and n3 may be natural numbers greater than 1, and may have a size relationship of n2 > n3 > n1. That is, the non-REM sleep state is classified into the most detailed n3 stages, and the arousal state is classified into the smallest n1 stages.

Meanwhile, each of the plurality of sleep states is preset to correspond to a unique one-hot vector. For example, each of the first n1 component values in the one-hot vector may be a component having a '1' when corresponding to the stages of the awakening state. For example, if an arousal state is divided into two stages, and a one-hot vector corresponds to one of the arousal states, the one-hot vector is a vector in which the first component value is '1' and the remaining component values are '0'. Or, it may be a vector in which the first component value is '0' and the remaining component values are '1'.

Also, each of the component values corresponding to the n1+1-th to the n1+n2-th in the one-hot vector may be a component having a '1' when it corresponds to the stages of the non-REM sleep state. In addition, each of the component values corresponding to the n1+n2+1th to the n1+n2+n3th in the one-hot vector may be a component having a '1' when it corresponds to the stages of the REM sleep state.

That is, in the present invention, the one-hot vector corresponding to each of the sleep states is preset to be closer to the awakening state as the component value '1' is positioned earlier in the sequence, and closer to the REM sleep as the '1' is positioned later in the sequence. Therefore, since the one-hot vector itself can represent the depth of the sleep state as it is, when the artificial neural network 20 is trained using this one-hot vector, even the distance relationship between the sleep state steps can be reflected. There are advantages.

As such, each of the n1+n2+n3 sleep states is preset to correspond to a unique one-hot vector, and this correspondence is stored in the service providing server 100 as a first matching table. Accordingly, the first to third sleep states may be converted into one-hot vectors obtainable by referring to the first matching table.

Meanwhile, as described with reference to FIG. 2 , the final sleep state is output from the artificial neural network 20 in the form of a final sleep state vector Vf.

The service providing apparatus 100 may determine the level of illumination based on the position of the component value '1' in the final sleep state vector Vf. For example, the closer the component value '1' is located in front of the final sleep state vector (Vf), the closer to the awakening state, the higher (brighter) the illuminance level is determined, and the component value '1' is the final sleep state vector (Vf) ), it is regarded as closer to the state of REM sleep and the level of illumination is determined to be lower (darker).

4 is a conceptual diagram for explaining a method of selecting music based on a position of a component value 1 of a final sleep state vector according to an embodiment.

In an embodiment, the service providing apparatus 100 may select one of a plurality of pre-stored candidate music based on the position of the component value '1' in the final sleep state vector Vf.

For example, the service providing apparatus 100 converts a sound wave signal according to time of each of a plurality of candidate songs to a frequency domain by performing fast Fourier transform (FFT), and in the converted frequency domain, corresponding to an audible frequency. It can be obtained by filtering the sound wave signal corresponding to 20 to 20000 (Hz).

The service providing apparatus 100 divides the filtered sound wave signal into equal frequency sections (n1 + n2 + n3 sections) corresponding to each of the component values of the final sleep state vector, and for each of the divided sections A sound wave signal can be integrated in the frequency domain. Here, a frequency section having the largest result value of the integrated sound wave signal may be determined.

When the component value of the final sleep state vector Vf corresponding to the determined frequency section is '1', the service providing apparatus 100 may determine the corresponding candidate music as the music to be provided to the user. In this case, the service providing apparatus 100 may preset component values corresponding to each of the frequency sections and store them as a second matching table.

For example, among the frequency sections, the section closest to 20 Hz corresponds to the last component value of the final sleep state vector (Vf), and the section closest to 20 kHz corresponds to the first component value of the final sleep state vector (Vf). can That is, the service providing apparatus 100 may generate the second matching table by matching the first to last component values of the final sleep state vector Vf to each of the frequency sections in the direction in which the frequency decreases.

When configuring the second matching table in this way, as the component value '1' comes after the final sleep state vector Vf, it corresponds to a deeper sleep state, and in this final sleep state vector Vf, the service providing apparatus 100 , the candidate music corresponding to the low frequency in the main band of the frequency is selected as the music to be provided to the user. Accordingly, as the sleep state is deep, music having a lower main band frequency may be provided to the user, and music having a higher main frequency frequency band may be provided to the user as the sleep state is lower or awakened.

FIG. 5 is a block diagram illustrating the structure of the convolutional neural network according to FIG. 2 .

Referring to FIG. 5 , a convolutional neural network (CNN) 10 according to an embodiment of the present invention receives an image (user's face image) of a preset size as an input image, and extracts a feature map A convolutional layer 11 that performs a function, an activation layer 12 that determines whether to activate an output using an activation function on the extracted features, and a pooling layer 13 that performs sampling on the output according to the activation layer 12 ), a fully connected layer 14 that performs classification according to a class, and an output layer 15 that finally outputs an output according to the fully connected layer 14 .

The convolutional layer 11 may be a layer for extracting features of the input data by convolutionally multiplying the input image and the filter. Here, the filter is a function that detects a characteristic part of the input image, and is generally expressed as a matrix. The feature extracted by the convolutional layer 11 may be referred to as a feature map. Also, an interval value for which convolution is performed may be referred to as a stride, and a feature map having a different size may be extracted according to the stride value. In this case, when the size of the filter is smaller than that of the input image, the feature map has a size smaller than that of the existing input image, and a padding process may be additionally performed to prevent the feature from being lost through several steps. In this case, the padding process may be a process of maintaining the same size of the input image and the size of the feature map by adding a preset value (eg, 0) to the outside of the generated feature map.

On the other hand, the convolutional neural network 10 has a lot of computational load because it convolutions a filter to an input image based on image processing. In order to provide lighting or music to a user from such a small mood lamp or speaker, it is necessary to drive the convolutional neural network 10 at the highest speed by minimizing the computational load. To this end, the convolutional layer 11 according to an embodiment of the present invention uses a structure in which a 1×1 convolutional layer and a 3×3 convolutional layer are sequentially and repeatedly connected, thereby minimizing the computational load and performing fast calculation. This may be possible.

The activation layer 12 is a layer that determines whether to activate by changing a feature extracted with a certain value (or matrix) to a non-linear value according to the activation function. The activation functions include a sigmoid function, a ReLU function, and a softmax. (softmax) function or the like may be used. For example, the softmax function may be a function having a characteristic that all input values are normalized to values between 0 and 1 and the sum of output values is always 1.

The pooling layer 130 performs subsampling or pooling on the output of the activation layer 12 to select a feature representing the feature map, and has the largest value for a certain region of the feature map. Max pooling for extracting , average pooling for extracting an average value, etc. may be performed. In this case, the pooling layer is not necessarily performed after the activation function, but may be selectively performed.

Also, here, the convolutional neural network 10 may include a plurality of connection structures of the convolutional layer 11 , the activation layer 12 , and the pooling layer 13 .

6 is a diagram for explaining a method of determining a second sleep state based on a motion intensity.

The service providing server 100 may calculate a movement distance according to time based on the movement signal included in the biometric information, and determine the second sleep state based on the calculated movement distance.

Here, the movement distance (MV) is a value obtained by calculating the distance moved by the user based on the movement signal. When the movement signal includes the user's movement distance in the x, y, and z directions, it can be calculated as in Equation 1 below. have.

Referring to Equation 1, in the present invention, the movement distance MV may mean a distance traveled by a user (or a wearable device worn by the user), and may be in units of m, but may also be in units of mm.

The service providing server 100, the number of pulses exceeding the threshold distance (mth) corresponding to each of the sleep states (pn), the pulse time interval (itv) between the pulses exceeding the threshold distance (mth), and the threshold distance In each of the pulses exceeding (mth), a threshold distance (mth) and a corresponding threshold time interval (w) may be calculated.

Here, the threshold distance mth is individually set for each of the sleep states. For example, as the sleep state moves from the waking state to the REM sleep state, the threshold distance mth may be set smaller.

Specifically, the service providing server 100, based on the interval interval (itv), the number of pulses (pn) exceeding the threshold distance (mth), and the threshold time interval (w), the motion state diagram according to the following equation (2) , and determining whether the calculated motion state diagram is within a threshold range uniquely determined for each of the sleep states may determine the second sleep state.

Here, the motion state diagram may be determined as in Equation 2 below.

Referring to Equation 2, in the motion state diagram, a threshold time interval (w[j] refers to an arbitrary j-th critical time interval) in the sum of the pulse time intervals (itv[i] refers to an arbitrary i-th pulse time interval) ), which is a value obtained by subtracting the sum of the time (min).

7 is a diagram for explaining a method of determining a third sleep state based on a PPG signal.

A photoplethysmography (PPG) signal is a signal used to measure a pulse wave using light, and is called a photoplethysmography signal.

A result of measuring (sampling) the user's PPG signal using the PPG sensor at intervals of every minute to measure the heart rate may be illustrated as shown in FIG. 7 . In this case, the sampled PPG signal may be partially omitted or not measured depending on the performance of the wearable device 200 .

Accordingly, when the heart rate measured from the PPG signal is 0 or less than a preset reference value, the PPG signal may be obtained by performing interpolating in which the heart rate sampled immediately before is replaced. However, since the heart rate according to time represents the heart rate according to a specific time point, it may be somewhat inappropriate to determine the process in which the user enters the sleep state or moves to the awakening state according to time.

In order to more easily determine the sleep state, the service providing apparatus 100 may obtain an average heart rate according to time by averaging heart rates according to each sampled time by a preset time interval. For example, in order to obtain an average heart rate according to time t1, the service providing apparatus 100 calculates an average value of heart rates corresponding to a first time interval immediately preceding (eg, 10 minutes) based on time t1. may be calculated, and the calculated average value may be determined as an average heart rate according to time t1. In the same manner, in order to obtain an average heart rate according to time t2, which is a sampling time immediately after time t1, the service providing apparatus 100 may perform a first time interval immediately before time t2 (for example, 10 minutes) It is possible to calculate an average value of heart rates corresponding to , and determine the calculated average value as an average heart rate according to time t2.

In the sleep state, when entering the sleep state from the waking state, the heart rate decreases, and the REM sleep tends to increase the heart rate rather than the non-REM sleep. Accordingly, the service providing apparatus 100 searches for a time interval in which the average heart rate is equal to or less than the sleep threshold heart rate (HRth1), and in the searched time interval, the change (ie, slope) of the average heart rate is set in advance by a first change amount (first The third sleep state may be determined by finding a time point tx exceeding the time point tx (which may be referred to as a slope) and determining the third sleep state according to a difference value df obtained by subtracting the current time point tc from the time point tx exceeding the time point tx.

For example, when the difference value df is negative, the service providing apparatus 100 may be one of sleep stages corresponding to REM sleep because the current time is after entering REM sleep. Accordingly, in the service providing apparatus 100, the difference value df is negative, and the wake-up threshold heart rate HRth2 corresponds to a value in which the average heart rate at the current time point tc is greater than the sleep threshold heart rate HRth1. In the case of less than or equal to, the greater the absolute value of the difference value df, the deeper the sleep stage among sleep stages of REM sleep may be determined. However, even if the difference value df is negative, if the average heart rate at the current time point tc exceeds the awakening threshold heart rate HRth2, it may be considered that the user has departed from the sleep and entered the awakening state. Accordingly, in this case, the service providing apparatus 100 may determine the user's sleep state as one of the phases of the awakening state so that the greater the absolute value of the difference value df, the closer to the awakening state.

In the same way, when the difference value df is positive, the service providing apparatus 100 may view the current time point as before entering the REM sleep. Accordingly, in this case, as the absolute value of the difference value df decreases, the service providing apparatus 100 may determine the deep sleep phase among sleep phases of non-REM sleep.

Here, as for the relationship between the absolute value of the difference value df and the sleep stages, the sleep stages can be determined in detail by dividing the absolute value equally or unequally according to the number of sleep stages and comparing the absolute values.

8 is a block diagram illustrating the structure of an artificial neural network based on a recurrent neural network according to FIG. 2 .

Referring to FIG. 8 , the artificial neural network 20 may include an input layer 21 , a hidden layer 22 , and an output layer 23 .

The input layer 21 may include the same number (V) of input nodes as the number of components (V) of each of the first to third sleep state vectors Vx, Vy, and Vz. For example, when each of the first to third sleep state vectors Vx, Vy, and Vz has component values corresponding to n1+n2+n3 sleep states, the input layer 21 is n1+n2+ It may consist of n3 input nodes.

The input layer 21 may apply one or more connection strength values corresponding to each of the input nodes to the first sleep state vector Vx and transmit it to the hidden layer 22 .

For example, one or more connection strength values corresponding to each of the input nodes may be expressed as _{a first connection strength matrix (W V×N ) having a size of V×N.} In this case, V may be the same number as the input nodes, and N may be 1/10 or less than V so that the sleep state vectors (Vx, Vy, Vz) composed of V dimensions can be projected into a sufficiently small dimension. can The first connection strength matrix (W _V×N ) may be continuously updated through supervised learning after being set to an arbitrary initial value.

In the same way, the input layer 21 applies one or more connection strength values corresponding to each of the input nodes to the second sleep state vector Vy and transmits them to the hidden layer 22, and the third sleep state vector Vz. One or more connection strength values corresponding to each of the input nodes may be applied to and transmitted to the hidden layer 22 .

In summary, the input layer 21 applies the first intermediate vector (Xa) obtained by the matrix multiplication operation of the received first sleep state vector (Vx) with the first connection strength matrix (W _{V×N ) to the hidden layer 22 .} and transfer the second intermediate vector (Xb) obtained by matrix multiplication operation _{of the first connection strength matrix (W V×N} ) to the second sleep state vector (Vy) to the hidden layer 22, and the third sleep state vector A third intermediate vector (Xc) obtained by performing a matrix multiplication operation on (Vz) of the first connection strength matrix (W _{V×N ) may be transmitted to the hidden layer 22 .}

The hidden layer 22 may be composed of N hidden nodes, and generates an intermediate target vector by synthesizing the first intermediate vector Xa, the second intermediate vector Xb, and the third intermediate vector Xc with each other. can do. For example, the hidden layer 22 may generate an intermediate target vector X′ based on Equation 3 below.

In Equation 3, k1 is a weight constant for the first sleep state determined through image information, k2 is a weight constant for the second sleep state determined through a motion signal, and k1 and k2 are between 0 and 1. It can be an integer.

The hidden layer 22 may transmit a score vector Y′ generated by applying one or more connection strengths corresponding to each of the hidden nodes to the generated intermediate target vector X′ to the output layer 23 . In this case, one or more connection strength values corresponding to each of the hidden nodes may be expressed as _{a second connection strength matrix U N×V having a size of N×V.} That is, the second connection strength matrix (U _N×V ) restores the intermediate target vector (X`) mapped to N dimensions back to V dimensions.

Meanwhile, the initial value of the second connection strength matrix (U _N×V ) may be set to an arbitrary value and then updated as the learning data is continuously supervised.

The output layer 23 may determine the probability p corresponding to the score vector Y′ by applying an activation function to the score vector Y′ received from the hidden layer 22 . The activation function may be a LeRU function or a Softmax function, but is not limited thereto.

For example, the score vector Y′ may represent a particular one-hot vector corresponding to one of the sleep states defined in the first matching table, and thus the probability p corresponding to the score vector Y′. ) represents the probability (p) that the sleep state corresponding to the specific one-hot vector is the final sleep state of the user.

Accordingly, the service providing apparatus 100 may determine the final sleep state corresponding to the score vector Y′ having the highest probability p with reference to the probabilities p output through the output layer 23 .

Meanwhile, in an embodiment, the learning data of the artificial neural network 20 converts the first to third sleep states obtained from the user's image information, motion signal, and PPG signal into one-hot vectors with reference to the first matching table. It may include the learning input data obtained and the learning output data obtained by converting the user's sleep state accurately determined by measuring the user's brain waves into a one-hot vector with reference to the first matching table.

The service providing apparatus 100 compares the score vector Y` obtained by inputting the learning input data to the artificial neural network 20 and the learning output data Y based on a loss function, and the score vector <sub>Supervised learning is performed in such a way that the first connection strength matrix (W V×N</sub> ) and the second connection strength matrix (U <sub>N×V</sub> ) are continuously updated so that the difference between (Y`) and the learning output data (Y) is minimized. can

More specifically, the artificial neural network 10 substitutes the score vector (Y`) and the learning output data (Y) into a predefined loss function, and the first connection strength so that the result value of the loss function is minimized. The matrix (W _V×N ) and the second connection strength matrix (U _N×V ) may be continuously updated.

For example, the loss function may be a cross entropy function. The cross entropy (H(Y, Y')) between the score vector (Y') and the learning output data (Y) may be defined as in Equation 4 below.

In Equation 4, Ym may be the m-th component (m is a natural number greater than or equal to 1 and less than or equal to V) of the learning output data Y, and Y'm may be the m-th component of the score vector Y'.

Meanwhile, in an embodiment of the present invention, a loss function (LF) may be defined as in Equation 5 below to improve accuracy.

Referring to Equation 5, the loss function LF is defined as a function that outputs a larger value among the result values according to the cross entropy H(Y, Y`) and the Minkowski distance. Since the Minkowski distance represents the largest difference between the learning output data (Y) and the score vector (Y`), defining such a loss function (LF) can minimize the large error that occurs when determining the sleep state. have.

Accordingly, it has the advantage of greatly improving the performance of predicting the user's sleep state.

9 is a diagram illustrating a basic hardware configuration of the service providing apparatus according to FIG. 1 .

Referring to FIG. 9 , the service providing apparatus 100 includes at least one processor 110 ; and a memory 120 for storing instructions for the at least one processor to perform at least one operation.

The at least one operation may include: acquiring biometric information of a user measured using a wearable device, and acquiring image information of the user photographed using a camera; determining a first sleep state by inputting the image information into a convolutional neural network (CNN); determining a second sleep state based on a motion signal included in the biometric information; determining a third sleep state based on a photo-plethysmograph (PPG) signal included in the biometric information; converting the first sleep state, the second sleep state, and the third sleep state into first to third sleep state vectors, respectively, with reference to a first predefined matching table; inputting the converted first to third sleep state vectors to a pre-supervised learning recurrent neural network (RNN)-based artificial neural network; obtaining a final sleep state vector indicating the final sleep state of the user as an output of the artificial neural network; and selecting lighting and music corresponding to the final sleep state vector.

In addition, the at least one operation may include at least some of the operations of the service providing apparatus 100 described with reference to FIGS. 1 to 8 .

Here, the at least one processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. can Each of the memory 120 and the storage device 160 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may be one of a read only memory (ROM) and a random access memory (RAM), and the storage device 160 is a flash-memory. , a hard disk drive (HDD), a solid state drive (SSD), or various memory cards (eg, micro SD card).

Also, the service providing apparatus 100 may include a transceiver 130 that performs communication through a wireless network. In addition, the data providing apparatus 100 based on the hierarchical identifier management system may further include an input interface device 140 , an output interface device 150 , a storage device 160 , and the like. Each component included in the service providing apparatus 100 may be connected by a bus 170 to communicate with each other.

For example, the service providing device 100, a communicable desktop computer (desktop computer), a laptop computer (laptop computer), a notebook (notebook), a smart phone (smart phone), a tablet PC (tablet PC), a mobile phone (mobile) phone), smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) ) player, digital audio recorder, digital audio player, digital video recorder, digital video player, PDA (Personal Digital Assistant), and the like.

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Examples of computer-readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or apparatus may be implemented by combining all or part of its configuration or function, or may be implemented separately.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

Claims (5)

A device that provides lighting and music according to the sleep state using AI (Artificial Intelligence),
at least one processor; and
and a memory for storing instructions for the at least one processor to perform at least one operation,
The at least one operation is
acquiring biometric information of a user measured using a wearable device, and acquiring image information of the user photographed using a camera;
determining a first sleep state by inputting the image information into a convolutional neural network (CNN);
determining a second sleep state based on a motion signal included in the biometric information;
determining a third sleep state based on a photo-plethysmograph (PPG) signal included in the biometric information;
converting the first sleep state, the second sleep state, and the third sleep state into first to third sleep state vectors, respectively, with reference to a first predefined matching table;
inputting the converted first to third sleep state vectors to a pre-supervised learning recurrent neural network (RNN)-based artificial neural network;
obtaining a final sleep state vector indicating the final sleep state of the user as an output of the artificial neural network; and
A device for providing lighting and music according to the sleep state using AI, comprising the step of selecting lighting and music corresponding to the final sleep state vector. In claim 1,
The first matching table is
A device that provides lighting and music according to the sleep state using AI, which instructs preset one-hot vectors to correspond 1:1 with preset sleep states. In claim 2,
The final sleep state vector is one of the one-hot vectors indicated by the first matching table,
The step of selecting the lighting and music,
An apparatus for providing lighting and music according to the sleep state using AI, which determines the level of illumination based on the component value '1' of the final sleep state vector. ◈Claim 4 was abandoned when paying the registration fee.◈ In claim 2,
The final sleep state vector is one of the one-hot vectors indicated by the first matching table,
The step of selecting the lighting and music,
frequency transforming (FFT, fast fourier transform) sound wave signals according to time of each of the plurality of candidate songs into a frequency domain;
Filtering and obtaining a sound wave signal corresponding to 20 to 20000 (Hz) corresponding to the audible frequency in the converted frequency domain;
dividing the filtered sound wave signal into equal frequency sections corresponding to each of the component values of the final sleep state vector, and integrating the sound wave signal in a frequency domain for each of the divided frequency sections; and
determining a frequency section having the largest result value of the integrated sound wave signal, and selecting candidate music in which the component value of the final sleep state vector corresponding to the determined frequency section is '1' as the music to be provided to the user; A device that provides lighting and music according to sleep states using AI, including. In claim 1,
The artificial neural network based on the recurrent neural network,
It is composed of input nodes corresponding to components of each of the first to third sleep state vectors, and a first connection strength matrix is applied to each component of the first to third sleep state vectors and transmitted to the hidden layer. the input layer;
a hidden layer that generates an intermediate target vector by using the output value transmitted from the input layer, and transmits the score vector generated by applying a second connection strength matrix to the intermediate target vector to the output layer; and
and an output layer that determines a probability corresponding to the score vector by applying an activation function to the score vector.

Images