GB2588036A

GB2588036A - Sound monitoring system and method

Info

Publication number: GB2588036A
Application number: GB2019008.8A
Authority: GB
Inventors: Moorhead Paul
Original assignee: Kraydel Ltd
Current assignee: Kraydel Ltd
Priority date: 2017-06-28
Filing date: 2017-06-28
Publication date: 2021-04-14
Anticipated expiration: 2037-06-28
Also published as: GB2588036B; GB202019008D0

Abstract

A method of monitoring sounds in an environment in which sound recognition is performed on audio signals produced by microphones to produce detected sound data representing recognised sound events. If the detected sound events indicate that there is abnormal activity in the monitored environment, an alert message may be sent to, for example, a carer of a patient. Determinating if detected sound data is indicative of abnormal activity involves identifying an absence of an expected sound event. Audio signal processing technology is used to recognise, classify, and identify patterns in household noises such that the occurrence of noises consistent with illness, or increases/changes in such noises consistent with deterioration in an existing condition, may be identified and made known to the carer. The method may also include recognizing human speech and /or utterances.

Description

Sound Monitoring System and Method

Field of the Invention

The present invention relates to sound monitoring systems. The invention relates parflcularly to systems for monitoring activity in domestic environments.

Background to the Invention

In the western world, approximately 50% of people over the age of 75 live alone (2 million people in the UK in 2016) and they are vulnerable to injury through falls, life-threatening infections such as 'flu which much more greatly impact the elderly, and many suffer from emphysema. COPD and other diseases of the lungs and cardio-vascular system.

Advances in low-cost computing and sensor technology, along with developments in machine-learning are making it possible to monitor the home environment of vulnerable people at low cost. Speech recognition has advanced greatly in the last few years giving rise to devices such as Google Home and Amazon Echo, and services such as Shazam.

It would be desirable to provide a system for monitoring the well-being of vulnerable people in a domestic environment.

Summary of the Invention

A flrst aspect of the invenflon provides a method of monitoring sounds in an environment using at least one microphone, the method comprising: detecting at least one sound using said at least one microphone; producing a corresponding audio signal from the or each microphone; performing sound recognition on the corresponding audio signal from the or each microphone to produce detected sound data representing one or more recognised sound events; determining if said detected sound data is indicative of abnormal activity in said environment; and performing at least one action in response to determining that there is abnormal activity.

A second aspect of the invention provides a sound monitoring device for monitoring sounds in an environment. the device comprising: at least one microphone, audio signal processing means configured to perform sound recognition on audio signals from the or each microphone to produce detected sound data representing one or more recognised 40 sound events, and to determine if said detected sound data is indicative of abnormal activity in said environment; and means for performing at least one action in response to determining that there is abnormal activity.

A third aspect of the invention provides a sound monitoring system comprising: at least one sound monitoring device comprising at least one microphone for detecting sounds in said environment; means for performing sound recognition on audio signals produced by the or each microphone to produce detected sound data representing one or more recognised sound events; means for determining if said detected sound data is indicative of abnormal activity in said 10 environment; and means for performing at least one action in response to determining that there is abnormal activity.

Determining if said detected sound data is indicative of abnormal activity in said environment may 15 comprise comparing said detected sound data against reference sound data.

Preferred embodiments further include creating said reference sound data by: detecting sounds in said environment using said at least one microphone; producing corresponding audio signals from the or each microphone; and performing sound recognition on the corresponding audio signals to produce said reference sound data representing one or more recognised sound events.

Said reference sound data may created during a training period Preferred embodiments include revising said reference sound data outside of said training period using said detected sound data.

Preferred embodiments include determining that said detected sound data is indicative of abnormal activity in said environment by identifying one or more designated deviation between said detected 30 sound data and said reference sound data.

Determining that said detected sound data is indicative of abnormal activity in said environment may comprise identifying from said detected sound data a change in frequency of occurrence of one or more sound events.

Determining that said detected sound data is indicative of abnormal activity in said environment may comprise identifying from said detected sound data an absence of one or more sound event.

Determining that said detected sound data is indicative of abnormal activity in said environment may 40 comprise identifying from said detected sound data, a change in duration of one or more sound event.

Determining that said detected sound data is indicative of abnormal activity in said environment may comprise identifying from said detected sound data a change in time of occurrence of one or more sound event.

Determining that said detected sound data is indicative of abnormal activity in said environment may comprise identifying from said detected sound data one or more instances of a sound of a type that is designated as being abnormal.

Determining that said detected sound data is indicative of abnormal activity in said environment may comprise identifying from said detected sound data sounds of a combination of types that is designated as being abnormal.

Performing at least one action in response to determining that there is abnormal activity in said environment may comprise sending a message to one or more electronic message-receiving device.

Performing sound recognition may comprise determining if said audio signals correspond to any one or more of said recognisable sound events.

Performing sound recognition may comprise comparing one or more audio samples from said audio signals to audio sample data representing each recognisable sound event.

Said recognisable sound events comprise human utterances including speech and non-speech utterances (e.g. coughs, sneezes, nose blowing. yells, laughter) and/or non-human sounds, e.g. of 25 types that commonly occur in a domestic environment.

Preferred systems include at least one message-receiving device in communication with said at least one sound monitoring device across a telecommunications network, wherein said at least one action comprises sending a message to at least one of said at least one message-receiving device in 30 response to determining that there is abnormal activity.

In preferred embodiments. audio signal processing technology is used to recognise, classify. and identify patterns in household noises such that the occurrence of noises consistent with illness, or increases/changes in such noises consistent with deterioration in an existing condition, may be 35 identified and made known to a care-giver.

Further advantageous aspects of the invention will be apparent to those ordinarily skilled in the art upon review of the following specific embodiment and with reference to the accompanying drawings.

Brief Description of the Drawings

A embodiment of the invention is now described by way of example and with reference to the accompanying drawings in which Figure 1 is a schematic diagram of a room in which a sound monitoring device embodying one aspect of the invention is installed; Figure 2 is a block diagram of the sound monitoring device of Figure 1; Figure 3 is a schematic diagram of a sound monitoring system embodying another aspect of the invention; and Figure 4 is a flow diagram illustrating a preferred operation of the device of Figure 1

Detailed Description of the Drawings

Referring now to the drawings there is shown a sound monitoring device 10 embodying one aspect of the invention. The device 10 is shown installed in a room 12. In the illustrated example the room 20 12 is a typical living room but this is not limiting to the invention.

In preferred embodiments, the sound monitoring device 10 is a device for monitoring activity in its environment using detected sounds. Typically, the environment in which the device 10 is located in use is a domestic environment, e.g. in a room of a person's home, although this need not necessary be the case. The device 10 is part of a sound monitoring system 15 embodying another aspect of the invention, the system 15 including at least one of the devices 10 in communication with a central monitoring system 14 and/or one or more carer devices 16 across a telecommunications network 17, which may for example comprise a computer network (e.g. the internet) and or a telephone network. The central monitoring device 14 Of present) may comprise one or more computers and may provide server functionality to the monitoring devices 10 and/or the carer devices 16. The carer devices 16 may comprise any electronic device capable of receiving messages from the devices 10 and/or the central monitoring device 14, for example a computer or computing device, or a telephone, especially a smartphone. Where there are multiple devices 10, a respective one may be located in a respective room of the same building, and/or in other buildings as required.

The sound monitoring device 10 is an electronic apparatus that detects sounds in its environment (the room 12 in the illustrated example) and supports sound recognition of the detected sounds. This may be achieved using any conventional sound recognition techniques. As part of the sound recognition, the device 10 preferably configured to perform classification of detected sounds.

The device 10 is configured to monitor the environment and, using the detected and recognised sounds, determine whether or not there is any abnormal activity in the environment. If any abnormal activity is detected, then the device 10 may take one or more actions. The or each action may depend on the type of abnormal activity that is detected. Typically, the actions include notifying one or more carers of the abnormal activity. This may involve sending a message to one or more of the carer devices 16. Such messages may be sent directly to the carer device(s) 16 by the monitoring device 10 across the network 17, or may be sent by the central monitoring system 14 to the carer device(s) 16 in response to notification by the monitoring device 10 of abnormal activity. The message may comprise an email, SMS message. MMS message, pager notification, phone call 10 and/or any other messaging means supported by the relevant components of the system 15.

The device 10 may be configured to take one or more action in response to sounds that it recognises as being of a known type as determined by the classification process. Other actions that may be taken as well as or instead of sending a message to a carer include, but are not limited to: issuing an alarm (e.g. an audio and/or visual alarm generated by a suitable audio and/or visual output device with which the device 10 may be provided); operating some other equipment to which the device 10 is connected; and/or reporting the detected sound(s) and/or an analysis thereof to the central monitoring device 14.

The device 10 typically includes a controller 11 for controlling the overall operation of the device 10. The controller 11 may comprise any suitably configured or programmed processor(s), for example a microprocessor, microcontroller or multi-core processor. Typically the controller 11 causes the device 10 to take whichever action(s) are required in response to detection of recognised sounds and the analysis of detected sounds. Typically the device 10 is implemented using a multi-core processor running a plurality of processes, one of which may be designated as the controller and the others performing the other tasks described herein as required. Each process may be performed in software, hardware or a combination of software as is convenient. One or more hardware digital signal processors may be provided to perform one or more of the processes as is convenient and as applicable.

Figure 2 is a block diagram of a typical embodiment of the sound monitoring device 10. The device 10 comprises at least one microphone 18. Typical embodiments include two or more (four or more is preferred) microphones 18 to facilitate determining the location of sound sources and for isolating sounds of interest from background noise. Preferably one or more sensitive microphone, for example a MEMS microphone is provided.

The device 10 comprises an audio signal processor 20 for receiving and processing audio signals produced by the microphones 18 in response to detecting sounds in the room 12 or other environment. The audio signal processor 20 may take any convenient conventional form, being implemented in hardware, software or a combination of hardware and software. Accordingly, the audio signal processor 20 may be implemented by one or more suitably configured ASIC. FPGA or other integrated circuit, and/or a computing device with suitably programmed microprocessor(s). In preferred embodiments the audio signal processor 20 is configured to perform sound recognition (which typically includes sound classification) on the microphone output signals. After sound recognition, the device 10 may analyse the detected recognised sounds to determine whether or not there is any abnormal activity in the environment. The analysis may involve applying a set of rules to the detected recognised sounds and to this end the device 10 may include a rules engine 24. The analysis, including implementation of the rules engine 24, may be performed by the audio signal processor 20, or by any other suitable component of the sound monitoring device 10 as is convenient. The audio signal processor 20 includes components or other means for performing the relevant audio signal processing and/or analysis functions, as indicated in the example of Figure 2 as 22 and 24. Sound recognition may be performed using any convention sound recognition techniques. For example, sound recognition may be performed by comparing the audio signals captured by the microphones 18 against machine-learned audio templates for each sound to be recognised. For example the system may employ machine-learning training based on a large database of audio-samples, combined, optionally, with specific training on the household environment.

In alternative embodiments, some or all of the audio signal processing and/or the analysis of the detected sounds may be performed by other component(s) of the system 15; for example by the central monitoring device 14. For example the sound recognition 22 and/or the rules engine 24 may be implemented by the central monitoring device 14 (instead of or as well as being performed by the device 10). This allows the analysis of detected sounds to be performed not only in relation to sounds detected by a single device 10, but to sounds detected by multiple devices 10, for example multiple devices 10 located in different rooms of the same building.

The analysis of detected recognised sounds is intended to determine if any abnormal activity is occurring in the environment being monitored (which may be a single room 12 monitored by one or more device 10, or multiple rooms each monitored by one or more device 10) in respect of which one or more actions needs to be taken by the relevant device 10 or system 15.

In preferred embodiments the system 15 and/or the device 10 as applicable may be configured to identify abnormal activity by any one or more of the following: 1. Detection of one or more instances of a sound of a type that is designated as being 35 abnormal (for example a scream or a gun shot). where one or more types may be designated as abnormal; 2. Detection of sounds of a combination of types that is designated as being abnormal (for example a cough and a sneeze), where one or more combination of types may be designated as abnormal; 3. Detection of one or more sound events that deviates from sound reference data; 4. Detection of changes in one or more sounds, or in one or more patterns of one or more sounds (for example a cough becoming louder and/or more frequent, or occurrence of a sound at an unusual time).

Any of the foregoing may take into account not only detected sounds but also detected silences and/or the absence of one or more expected sounds.

Any of the foregoing may take into account sounds of types that emanate from a person (e.g. speech, shouts, coughs, sneezes or snores) or from the environment (e.g. a door bell. telephone 10 ring. running tap and so on). or a combination of both.

Any of the foregoing may be performed in respect of a specified time period which may be long or short depending on whether it is desired to monitor sudden or progressive activities.

The sound reference data may be obtained by any convenient means. For example the or each device 10 may monitor sounds in its environment during a training period and generate sound reference data from the sounds detected during that period. The sound reference data may be periodically or continually updated by the system 15 or device 10 as applicable depending on the sounds that it detects during use (for example using any conventional machine learning techniques).

The sound reference data represents a profile of sound events that is deemed to be normal for the relevant environment. The reference sound data may identify any one or more of: types of sounds; combinations of types of sounds; patterns of one or more sounds; time of occurrence of one or more sounds; frequency of occurrence of one or more sounds; volume of sounds; absence(s) of one or more sounds; silence(s), and/or any other sound characteristics that may be used to define a normal sound environment. The reference sound data may be defined over a specified period of time, for example a day, week or month.

During use, the or each device 10 detects sound data from the environment. The detected sound data therefore represents one or more recognised sound events detected by the device 10. The detected sound data may identify any one or more of: types of detected sounds; combinations of types of detected sounds; patterns of one or more detected sounds; time of occurrence of one or more detected sounds; frequency of occurrence of one or more detected sounds; volume of detected sounds; absence(s) of one or more sounds; silence(s), and/or any other sound characteristics that may be used to define the detected sound events. The detected sound data may be defined over a specified period of time, for example a day, week or month. The detected sound data is analysed, by the rules engine 24 in the present example, to determine if any one or more of the recognised sound events indicates that there is abnormal activity in the environment. This may involve comparing the detected sound data against the reference sound data. Any deviations between the detected sound data and the reference sound data may be indicative of abnormal activity in the environment.

Typically any such deviations are analysed (conveniently by the rules engine 24) to determine if they are indicative of abnormal activity of a type that warrants further action by the system 15, e.g. notification of a carer or generation of an alarm. Such deviations may be referred to as designated deviations and may include for example a change in frequency of one or more particular sound events (e.g. increased coughing or reduced conversation), the absence of one or more sound event that was previously present, a change in duration of one or more particular sound event (e.g. increased periods of silence), a change in time of occurrence of one or more particular sound events (e.g. door slams at unusual times).

Figure 4 shows a typical operation of the device 10 or system 15. At 301 the or each device 10 captures sound samples from the audio signals produced by the microphones 18. The sound samples are subjected to sound recognition to identify the type of sound(s) detected (302). Recognised sounds are sent to the rules engine 24 which applies a set of rules to the detected recognised sounds to determine if one or more abnormal activity is detected (303). If abnormal activity is detected, then appropriate action is taken, typically involving notification of a carer (304).

In typical embodiments, household sounds that may be detected and analysed by the device 10/system 15 may include any one or more of the following (without limitation): telephone; doorbell; door slams; footsteps; boiling kettle; coughs; sneezes; wheezing; conversation; laughter; crying; cries of pain or for help; breaking glass, or other household items; TV or radio; running water (e.g. indicative of a shower or bath).

During a training period (e.g. of 1 to 2 weeks) the device 10/system 15 can learn the frequency, and time of occurrence of such sounds to create or adapt the reference data for the environment. Using the reference data as a baseline, which may be allowed to evolve over time, the system/device can then use rules to identify both sudden and progressive changes in the environment. particularly those that are deemed to be abnormal and that warrant further action being taken. Examples of abnormal activities may include any one or more of the following (without limitation): o Sudden Changes: * Silence * Cries of pain or for help * Door slams at unusual hours * Footsteps at unusual hours * Crying * Sneezing * Coughing * Vomiting Breaking of glass, or smashing of other household items o Progressive Changes * Increased coughing * Increased wheezing * Reduction in conversation * Decrease in doorbell and/or telephone rings indicating growing social isolation * Lengthening periods of silence during normal waking hours potentially indicating depression, dementia or other illness * Decrease in frequency of incidence of running water indicating reduced self care indicating depression or other illness.

Advantageously knowledge of time of day, and weather conditions (which may readily be obtained from web APIs for example) may be used to eliminate false alarms, e.g. prolonged silence in the middle of a sunny day may mean that the monitored individual has just gone out, whereas prolonged silence on a cold, wet, afternoon is more concerning.

Advantageously aggregation of machine-learning across multiple devices 10 improves the accuracy of recognition of sounds.

Advantageously aggregation of learning across localities can give evidence of increasing levels of coughing/sneezing/vomiting consistent with possible 'flu epidemics.

Over time it is possible to assign a "health score" based on a large corpus of data from multiple 20 monitored residences and trigger an appropriate response from care givers using this score (or changes to it).

The system 15/device10 may be trained to recognise a noise specific to its environment, for example to enable a learning mode, then trigger the door bell (or other sound) one or more times. and then 25 label that sound appropriately within the system (as a door bell in this case).

The invention is not limited to the embodiment(s) described herein but can be amended or modified without departing from the scope of the present invention.

Claims

CLAIMS: 1. A method of monitoring sounds in an environment using at least one microphone, the method comprising: detecting at least one sound using said at least one microphone; producing a corresponding audio signal from the or each microphone performing sound recognition on the corresponding audio signal from the or each microphone to produce detected sound data representing one or more recognised sound events; determining if said detected sound data is indicative of abnormal activity in said environment; 10 and performing at least one action in response to determining that there is abnormal activity, wherein determining that said detected sound data is indicative of abnormal activity in said environment involves identifying from said detected sound data an absence of one or more sound event.
2. The method of claim 1 wherein determining if said detected sound data is indicative of abnormal activity in said environment comprises comparing said detected sound data against reference sound data.
3. The method of claim 2 further including creating said reference sound data by: detecting sounds in said environment using said at least one microphone; producing corresponding audio signals from the or each microphone; and performing sound recognition on the corresponding audio signals to produce said reference sound data representing one or more recognised sound events.
4. The method of claim 3, wherein said reference sound data is created during a training period.
5. The method of claim 3 or 4, including revising said reference sound data outside of said training period using said detected sound data.
6. The method of any one of claims 2 to 5 including determining that said detected sound data is indicative of abnormal activity in said environment by identifying one or more designated deviation between said detected sound data and said reference sound data.
7. The method of any preceding claim including determining that said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data a change in frequency of occurrence of one or more sound events.
8. The method of claim 2, wherein comparing said detected sound data against reference sound 40 data comprises comparing said detected sound data against reference sound data representing a profile of sound events that is deemed to be normal for the environment, and identifying one or more deviations between said detected sound data and said reference sound data.
9. The method of any preceding claim including determining that said detected sound data is 5 indicative of abnormal activity in said environment by identifying from said detected sound data, a change in duration of one or more sound event.
10. The method of any preceding claim including determining that said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data a 10 change in time of occurrence of one or more sound event.
11. The method of any preceding claim including determining that said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data one or more instances of a sound of a type that is designated as being abnormal.
12. The method of any preceding claim including determining that said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data sounds of a combination of types that is designated as being abnormal.
13. The method of any preceding claim wherein performing at least one action in response to determining that there is abnormal activity in said environment comprises sending a message to one or more electronic message-receiving device.
14. The method of any preceding claim wherein performing sound recognition comprises 25 determining if said audio signals correspond to any one or more of said recognisable sound events.
15. The method of any preceding claim wherein performing sound recognition comprises comparing one or more audio samples from said audio signals to audio sample data representing each recognisable sound event.
16. The method of any preceding claim wherein said recognisable sound events comprise human utterances, wherein said human utterances may comprise any one or more of speech, a cough, a sneeze, nose blowing, a yell, laughter, crying, vomiting or wheezing.
17. A sound monitoring device for monitoring sounds in an environment, the device comprising: at least one microphone, audio signal processing means configured to perform sound recognition on audio signals from the or each microphone to produce detected sound data representing one or more recognised sound events, and to determine if said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data an absence of one or more sound event; and means for performing at least one action in response to determining that there is abnormal activity.
18. A sound monitoring system comprising: at least one sound monitoring device comprising at least one microphone for detecting sounds in said environment; means for performing sound recognition on audio signals produced by the or each microphone to produce detected sound data representing one or more recognised sound events; means for determining if said detected sound data is indicative of abnormal activity in said environment by identifying from said detected sound data an absence of one or more sound event; and means for performing at least one action in response to determining that there is abnormal activity.
19. The system of claim 18, further including at least one message-receiving device in communication with said at least one sound monitoring device across a telecommunications network, wherein said at least one action comprises sending a message to at least one of said at least one message-receiving device in response to determining that there is abnormal activity.