CA3056352A1

CA3056352A1 - System and method for detecting and stopping nightmare by sound

Info

Publication number: CA3056352A1
Application number: CA3056352A
Authority: CA
Original assignee: Individual
Current assignee: Individual
Priority date: 2019-09-23
Filing date: 2019-09-23
Publication date: 2021-03-23

Description

TITLE
SYSTEM AND METHOD FOR DETECTING AND STOPPING NIGHTMARE BY SOUND
TECHNICAL FIELD
The invention relates to system and method for monitoring sleep condition. In particular, the invention relates to system and method to detect nightmare occurrence and wake up the sleeper by sound. It also includes method to achieve high recognition rate of nightmare condition.
BACKGROUND
A nightmare is a disturbing dream associated with negative feelings. It is very common for people to have nightmare, especially when they are under stress, or in physically subnormal condition. Nightmare causes bad feeling and distress. Having frequent nightmare could lead to serious illness such as nightmare disorder. Even if having nightmare is not a problem, people still wish to stop their nightmare when it occurs. If there is another person around and aware of the person is having nightmare, the other person can wake up the person in nightmare. But someone who sleeps alone usually needs to struggle for long time before wakes up from nightmare eventually.
A nightmare is often associated with crying or screaming sound and body movement. Such signs can be explored to detect if a sleeper is having a nightmare.
Most of current devices use traditional DSP (digital signal processing) technology as it is relatively simple to implement on a standalone device. In U.S. Patent No.

it describes systems and methods for monitoring EEG signals that can be used in sleep monitoring systems.
Other existing implementations use one or more sound models based on DSP
technology. For example, in US patent No. US20120092171A1 it uses existing sound classification models generated from a hidden Markov model trained or decoded using a Viterbi algorithm which is possible to recognize a snoring sound. In this patent it also describes a mobile device with software application to monitor sleep using environmental sound, which determines a sleep state of the user based on the indicators of sleep activity and generate a report that summarizes the user's sleep states.
Due to limitation of DSP technology, these sound models are for detection of simple sound pattern, such as sound associated with breathing, snoring, or body movement.
They have difficult to detect more complex sound pattern like screaming or bubbling during nightmare.
As such implementation is applied to detect nightmare condition, its accuracy of recognition is very low or liable to trigger false alarm.
In recent years, some devices and software applications can detect signs of nightmare through monitoring heart rate and body movement, such as Apple Watch and Microsoft Band. Most of these devices are wearable with features to monitor sleep condition and bring user out of sleep by slight vibrations if there is excess sign of a nightmare, as gathered from the watch's internal sensors, including a heart rate monitor, gyroscope, and accelerometer.
But none of these devices relies on sound detection alone, and they are costly, inconvenient to use (as they are wearable device and rely on sensor contact to user's body).
Machine learning is a subfield of artificial intelligence (Al). It becomes very popular in recent years to use artificial neural network for image and sound recognition, as it has better recognition rate over traditional digital signal processing (DSP) method.
The present disclosure provides system and method for detecting and stopping nightmare occurrence by taking advantage of artificial neural network for recognition of nightmare sound.
SUMMARY OF THE INVENTION
A system for nightmare alarm is provided in the present disclosure. The system includes a smartphone, a client application software running on smartphone, which connects through internet to cloud server, and a sound recognition software running on cloud server.
A method for detection and recognition of nightmare sound (crying, screaming, or any other kind of sound showing the person is under stress) is provided. The method includes capture of environmental sound, transmission of sound data through internet to a cloud based server for sound processing, and machine learning based software which recognizes if environmental sound shows a situation of nightmare, and upon recognition of nightmare condition, the server software signals back to smartphone application, and the application plays recorded alarm sound to wake up the person.
The application software runs on smartphone during sleep time. The application keeps collecting environmental sound, sending compressed acoustic data over internet to cloud server software. The server software is trained by machine learning technology to recognize the sound pattern of distress, such as screaming, bubbling or crying. Once the server software makes decision that the sound suggests nightmare condition, it sends a signal back to the smartphone application, and the application plays alert sound or vibrates to wake up the user who is having nightmare. This process can be lasting until environmental sound doesn't show nightmare condition (which indicates the user wakes up or nightmare disappears) or forced to stop by the user's intervention.
The core part of this invention is the server software that recognizes nightmare condition. The advantage of this software is high accuracy of detection. The software applies the latest computer technology of artificial neural network, one type of machine learning. The software is trained by sound samples relevant to nightmare. Further, the accuracy of recognition will keep to be improved over time, as the more sleepers use it and give feedback, the more sound data are fed to train the recognition model, hence it acts like a positive feedback loop to improve both accuracy and performance of recognition.
One particularly beneficial use of the method of the present disclosure is cost saving, as smartphone is widely available, the user doesn't need to pay for extra hardware, and what need is to download and install the application software on smartphone.

Another advantage of this system and method is that it is simple and convenient to use, just one touch to activate the function like any other type of mobile application.
In contrast, most of existing sleep monitoring devices are based on detection of one or more other conditions, such as high blood pressure, rapid heart rate, rapid breathing, snoring, etc.
These devices are made wearable as their sensors need body contact, for example, most of them are worn around the wrist.
BRIEF DESCRIPTION OF THE DRAWINGS
The system consists of four parts. Part One, smartphone of any OS (Android, Windows or i0S). Part Two, an application software which runs on smartphone. And part Three, a recognition software runs on cloud server computer. Part four, internet connection between user's smartphone and cloud server.
In the drawings:
FIGURE 1 is an illustration of an example environment using system of the present disclosure;
FIGURE 2 is an illustration of nightmare alarm application with its functional components;
FIGURE 3 is an operational flow chart of an implementation of nightmare alarm application;
FIGURE 4 is an operational flow chart of an implementation of server software for nightmare sound recognition;
FIGURE 5. A diagram shows how the software model (neural network) of sound recognition is trained and built through machine learning and how it is enhanced by user's data;
DETAILED DESCRIPTION
The present disclosure provides a system and method for detecting nightmare occurrence and wake up sleeper out of nightmare. The system includes hardware of existing communication platform like smartphone, internet, and cloud server, as well as client software running on smartphone and server software running on cloud server. The method of detection is recognition of sound made by person who is having nightmare. The method of alert or waking up is playback of recorded sound for specific purpose, such as music, voice, etc. The method of nightmare sound recognition is sound classification using machine learning, the state-of-art computer technology.
FIG. 1 shows the whole system and an example of use case of this invention. A
sleeper 101 turns on software application 103 (aka. smartphone App) for nightmare alert on smartphone 102 before goes to sleep. The application 103 keeps monitoring environmental sound through microphone 104 of smartphone 102. Once detected significant sound (a threshold can be adjusted through user setting of the application), the application starts to send out captured sound stream data through Wi-Fi, or cellular data network, and Internet network 106 to a sound recognition software 108 running on cloud server 107. The sound data can be in one of computer digital audio formats, such as uncompressed format PCM (Pulse-Code Modulation) or compressed format MP3. The sound quality is of high fidelity, which means its sampling rate is equal to or above 44.1 KHz and has a bit depth of 16 bits or more. The sound recognition software 108 is a kind of server-client software which can communicate and provide sound processing service to multiple client applications 103. While receiving incoming sound data, the nightmare recognition software processes the sound data and if it determines the sound suggesting nightmare occurrence, it sends back a signal or command to notify the application 103, and the application 103 plays back a recorded alert sound to wake up the sleeper. By this way, it detects and stops nightmare via a low cost and convenient approach.
FIG. 2 illustrates functional components of the nightmare alarm application 103. The application 103 leverages sound recorder 201, one software component provided by the smartphone operating system (Android OS or Apple i0S), to continuously listen to environment sound via smartphone's microphone 104. A condition detector 202 is to find the start of sound outbreak and start to capture and encode sound into digital audio data, in either uncompressed or compressed format (by sound data compression 203). It ends capture of sound once sound disappears or diluted to silence level. During sound capture period, the server communication 206 transfers the audio data in real time (or very minimum delay in micro-second scale) over communication networks, for example, Wi-Fi, cellular 3G/4G/5G, internet network 106, to destination server software 108. The server communication 206 keeps a continuous bi-directional communication with server software 108. Once the server software 108 recognizes nightmare condition, it sends back a command signal to server communication 206, which in turn, signals to Alert Activator 205 and to trigger Alert Sound Player 204 to play back a pre-recorded alert sound (stored in Alert Sound Storage 207, which can be hardware media storage like smartphone's internal memory device). The application 103 also includes a user API (Application User Interface 208) for user to manage settings of the application, such as sound detection threshold, alert sound selection, etc.
FIG 3 illustrates an operational flow chart of the application 103 to show how it functions bidirectionally to the server end. The application 103 runs in a loop to continuously monitor environmental sound 301, sampling sound at fixed interval, for example, at every one second, and checks the sound amplitude against a trigger threshold 302 (which can be set by the user according to ambient sound condition). If sound level is over the threshold 302, the application103 starts to capture environmental sound, encode acoustic data in digital audio format and send data stream in real time to server software 108. The data are usually compressed (encoded) to save data amount and network bandwidth. Meanwhile, the application 103 runs in another loop to wait and receive command 304 from server software 108. Once it received a command 305 indicating nightmare condition detected, it plays alert sound 306 to wake up sleeper till it receives further command to stop (or it can be configured to play for preset duration of time).
In general, the application 103 is a client software which just performs light tasks such as data collection and sound playback in the front end. In contrast, the heavy task part in this implementation is nightmare sound recognition, which needs high performance and heavy computation load of neural network inference, so it is allocated to software running on server machine.

FIG. 4 illustrates an operational flow chart of an implementation of server software for sound recognition. The communication interface 401 maintains the signal link over internet network 106 with client application 103, which includes receiving service request 402 for sound recognition and sending command 408 of playing alert sound to client application 103. When there is sound recognition request coming in, it starts to receive acoustic stream data 403 and feed data into recognition system model 406. The model 406 uses artificial neural network, the state-of-art computer technology widely applied nowadays to recognize image, speech and sound. If a nightmare sound made by sleeper is recognized 407, such as crying, screaming, or any other kind of sounds showing the person is under stress, it sends the command "play alert sound" to client application 103. If the sound is not recognized as nightmare sound, there is no command to send out and the system model 406 just keeps processing incoming acoustic data till the end of data stream. Meanwhile, the received acoustic data 403 is stored in the acoustic data storage 404, which can form a sound data library and can be used to train the system model or artificial neural network 406. The artificial neural network needs to be trained by large amount of data in order to improve the recognition accuracy.
FIG 5 is a diagram showing how the software model of sound recognition 406 is trained and built through machine learning and how it is enhanced by user's data. The software model 406 is an artificial neural network for specific sound recognition, namely, nightmare sound made by sleeper. This is the core part of the present disclosure, as the recognition rate and accuracy directly affect the user's experience with this application. Either false alarm or missed detection does not meet user's expectation. In the diagram, 501, 502, 503, and 504 shows the build process of the recognition model 406. The training and build process is a typical machine learning process for sound/speech recognition. First, a sound data library or called labelled data set 501 of nightmare sound made by sleeper are processed by extracting sound feature and results in spectrogram 502. Then selects the architecture 503 of artificial neural network to be CNN (convolutional neural network) or RNN (recurrent neural network).
Finally, train the neural network 504 using labelled data set 501, using a process called supervised machine learning.
One feature of this invention is to leverage real-world environmental sound 506 and user's feedback 507 from users of this application to fine tune parameters of neural network by evaluating accuracy of prediction. The user's environmental sound 506 captured by this application could be sound in nightmare case or ambient sound in non-nightmare case. After each use, a user can send a feedback of correct or incorrect detection through client application 108 to server recognition software. By comparing the prediction of recognition model 505 and user's feedback 504, if the prediction is correct, it means the captured sound is of nightmare sound, and this piece of sound is added into data set of nightmare sound 501, which can be used for re-training the recognition model. Or if the prediction is wrong, it means the sound is not of nightmare but recognition model makes incorrect prediction. Such piece of sound can be used to tune parameters of the recognition model to reduce probability of prediction error. It forms a positive feedback loop to keep training and tuning the recognition model with all users' real experience. In this way, the recognition rate of the model will be enhanced along with time.
Data set is crucial to machine learning. The iterative aspect of machine learning is the key because model can independently adapt as it is exposed to new data. There are three types of data set used in this invention - initial nightmare sound used to generate the model, user's environment sound to tune the model, and specific ambient sound available online to exclude nightmare condition.
A method is also proposed in this invention to further enhance prediction accuracy by leveraging ambient sound recognition. The goal is to achieve higher probability of recognition.
The method is to combine prediction by the model of nightmare sound and another model of non-nightmare sound (ambient sound such as air conditioner, siren, etc.) For example, when an environmental sound is recognized as nightmare sound and not ambient sound, the probability of nightmare occurrence is higher than prediction probability by the single model of nightmare sound recognition. There are algorithms to be considered to combine two model's probabilities to enhance the final prediction rate with evaluation of the two models.
But the multiple software models and specific algorithm to combine their prediction results is not necessary requirement to the present disclosure.
AN IMPLEMENTATION EXAMPLE OF THE INVENTION
As an example, the client software in this invention can be implemented as an application installed on either Android or iOS smartphone. The recognition software can be installed on a server computer provided by any cloud service platform such as Amazon AWS web services.
And the recognition software can be trained and built using Google TensorFlow, a free and open-source software development platform for machine learning applications.
One use case is like this: a user downloads and installs the client application on smartphone and activates the application before goes to sleep. Since then, the application keeps monitoring environmental sound and communicating with the server recognition software. An ambient sound would not trigger an alert sound. Only when the user has nightmare and makes sound recognized as nightmare sound, the smartphone plays alert sound trying to wake up the user out of nightmare condition. If waken up by alert sound, the user can stop the sound play by a touch on smartphone. The user intervene is not necessary, as the application is "smart"
enough to stop playing after it detects that no more nightmare sound persists, which could be the case that the sleeper doesn't fully wake up but nightmare condition is gone. The application can be set up to auto activate and deactivate at programmed time.
The server software can provide recognition service to multiple users for all time. What a user needs are the client application and an internet connection.