EP4006856A1

EP4006856A1 - Computer-implemented method and system for the triggering of an alarm in an emergency communication system

Info

Publication number: EP4006856A1
Application number: EP20210722.3A
Authority: EP
Inventors: Foteini ANDRIOPOULOU; Vasileios Giatilis
Original assignee: Unify Patente GmbH and Co KG
Current assignee: Unify Patente GmbH and Co KG
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2022-06-01

Abstract

The present invention relates to a computer-implemented method of triggering an alarm in an emergency communication system, wherein the emergency system comprises a pre-recording system (1) that is connected to at least one PSAP (8) and a plurality of peripheral devcies (6) that respectively are equipped with a microphone via an emergency communication network, the method comprising the steps of receiving, at the pre-recording system (1), audio data input from at least one peripheral device (6), the audio data comprising speech and/or sound, determining, if the speech and/or sound exceeds a noise level threshold for a predetermined period of time, and if the noice level threshold is exceeded for a predetermined period of time, identifying a possible emergency incident; recording the audio data comprising the speech and/or sound related to the possible emergency incident, and initiating an emergency call to at least one PSAP (8). Further, the present invention relates to an emergency communication system comprising a pre-recording system (1) adapted for carrying out the computer-implemented method of triggering an alarm.

Description

The present invention relates to a computer-implemented method of triggering an alarm in an emergency communication system and to a corresponding emergency communication system.
Nowadays, an increase of violence is observed resulting amongst others from current pandemic situation caused by the world-wide spreading of the SARS-CoV-2 virus. In particular, the risk for domestic violence and violence against women has increased. Namely, the statistical analysis of the World Health Organization (WHO) has shown that violence against women remains a major threat to global public health during emergencies, including epidemics or the above mentioned pandemic. However, these violent events may be overlooked, excused, or denied either because the victims are scared or because they experience difficulties or barriers in accessing the emergency services and Public Safety Answering Points (PSAPs) for reporting emergencies.
Although nowadays, there are lots of devices, peripherals, and network connected devices, Internet of Things (IoT) devices such as microphones, cameras, etc, that are able to monitor and gather contextual ambient information, these devices still are not able to automatically trigger the emergency systems and PSAPs in order to report the emergency incident.
Also, according to prior art, existing recording systems can be initiated at the point of time when an emergency call is established. However, they do not contain any kind of information for any prior actions. Even if the personnel of the emergency system could have access to the IoT devices in order to request such prior recordings, in an emergency event, it would take too much time to analyze the data and identify the context and the circumstances under which a violent event has occurred.
Therefore, the present invention is based on the object to solve the above described problem and, in particular, to provide a computer-implemented method of triggering an alarm in an emergency communication system and a corresponding emergency communication system for overcoming the above mentioned problem. In particular, the present invention is based on the object to provide a method and a corresponding system for efficiently detecting and handling a possible emergency incident.
This object is solved according to the present invention by a method of triggering an alarm having the features according to claim 1, and a corresponding emergency communication system having the features according to claim 14. Preferred embodiments of the invention are specified in the respective dependent claims.
Thus, according to the present invention, a computer-implemented method of triggering an alarm in an emergency communication system is provided, wherein the emergency system comprises a pre-recording system that is connected to at least one PSAP and a plurality of peripheral devcies that respectively are equipped with a microphone via an emergency communication network, the method comprising the steps of

receiving, at a pre-recording system, audio data input from at least one peripheral device, the audio data comprising speech and/or sound,
determining, if the speech and/or sound exceeds a noise level threshold for a predetermined period of time, and
if the noice level threshold is exceeded for a predetermined period of time, identifying a possible emergency incident;
recording the audio data comprising the speech and/or sound related to the possible emergency incident, and
initiating an emergency call to at least one PSAP.

Enabling peripheral devices, as IoT devices, to generate automatic emergency calls to PSAPs and transmit the previously collected and recorded audio data data marking and tagging the time slots in which a scene of possible violence has occurred, enables immediate emergency response, thus eliminating the time to analyze and identify the circumstances. As a consequence, the present invention provides a significant tool for call takers at PSAPs to handle pre-recorded audio data and evaluate the severity of an emergency incident in an accurate, fast, and thus efficient manner.
Computer-implemented method according to claim 1, wherein the method further comprises a step of tagging the recorded audio data with respect to time of occurrence.
Computer-implemented method according to claim 1 or claim 2, wherein the method further comprises a step of classifying speech and/or sound according to a predetermined classification algorithm.
Computer-implemented method according to claim 3, wherein classes for the classification step comprise human screams, yelling, shouting, crying, asking for help, combat sounds, and/or gunshots.
Computer-implemented method according to any one of the preceding steps, wherein the method further comprises a step of evaluating the severity of the possible emergency incident on the basis of the noise level and the duration.
Computer-implemented method according to any one of the preceding claims, wherein the method further comprises a step of determining the location of the possible emergency incident based on GPS data received from the respective peripheral device that has transmitted the audio data.
Computer-implemented method according to any one of the preceding claims, wherein the peripheral devices comprise smart devices, in particular, smart phones, tablets, desktop phones, cameras, microphones, and/or sensors.
Computer-implemented method according to any one of claims 6 or 7, wherein the emergency call comprises information of the location of the possible emergency incident and the nature of the emergency incident.
Computer-implemented method according to any one of the preceding claims, wherein the emergency call comprises the audio data that has been recorded, tagged, and classified.
Computer-implemented method according to any one of claims 6 to 9, wherein the method further comprises a step of setting an area having a predetermined radius around the location of the possible emergency incident, and sending the location data and the predetermined radius to a social media server for identifying mobile devices used by its subscribers that are currently located within the predetermined radius around the location of the possible emergency incident, and for activating a push notification service to the subscribers for questioning the subscribers about the possible emergency incident.
Computer-implemented method according to claim 10, wherein the questions are yes-or-no questions.
Computer-implemented method according to claim 11, wherein the subscribers are provided with different questions.
Computer-implemented method according to any one of claims 10 to 12, wherein the location of the subscribers is determined based on GPS data.
Emergency communication system comprising a pre-recording system adapted for carrying out the computer-implemented method according to any one of the preceding claims.
The present invention provides a method and a system that enable network connected devices, referred to as peripheral devices comprising amongst others smart devices and individual sensor devices, to monitor and gather ambient contextual information. The system comprises speech and sound recognition modules that gather, process and evaluate audio segments from the peripheral devices in order to detect events that may deal with important emergency incidents such as human screams, yelling, crying, asking for help in combination with physical assault accusation or battering as well as gunshots. People in these situations are at a high risk of being seriously injured or killed. When the inventive system and method identify a deviation from normal beyond a particular level value (threshold) of sound(s) measured in decibel (dB), for a predefined time frame, then it starts the recording to a local or remote server. The recording is enhanced with tags related to the time slot that the potential emergency incidents occurred and also, the recordings are labelled with the classification of the algorithm used for the recognition such as scream, gunshot, broken glass or fights. Moreover, the deviation of the normal sound levels measured in dB are identified and marked with different colours or patterns depending on the intensity of the dB level and on how long it lasted. Based on the severity of the sound and the duration, an emergency call to a PSAP is triggered automatically. After the establishment of the emergency call, the call taker at the PSAP may have access and retrieve the pre-recorded media stream with marked slots and labels based on the time and the event that initiated the call. The call taker may navigate to these slots or just inspect the slot marked with colours or patterns that correspond to more serious events and evaluate the severity of the incident in an accurate and efficient manner eliminating the time response to a possible emergency incident.
The invention and embodiments thereof will be described below in further detail in connection with the drawing.

Fig. 1: shows a schematic overview of a system architecture for carrying out the method of triggering an alarm in an emergency communication system according to an embodiment of the invention;
Fig. 2: is a flow chart illustrating the steps of the method of triggering an alarm in an emergency communication system according to an embodiment of the invention;
Fig. 3: shows a classification scheme for different types of sounds;
Fig. 4: shows an labelling scheme for audio recordings;
Fig. 5: shows a map indicating a range around a location at which an emergency has occurred;
Fig. 6: shows a chart illustrating the clustering of artificial responses from neighbors;
Fig. 7: schematically illustrates a messging flow in case of an emergency incident; and
Fig. 8: schematically illustrates a messaging flow when neighbors within a predetermined radius of the emergency incident provide feedback.

Fig. 1 shows a schematic overview of a system architecture system architecture for carrying out the method of triggering an alarm in an emergency communication system according to an embodiment of the invention. The illustrated system comprises a pre-recording system 1 with speech and sound recognition modules that gather, process and evaluate audio segments from peripheral devices in order to detect events that can deal with important emergency incidents such as human screams, yelling, crying, asking for help in correlation with physical assault accusation or battering as well as gunshots. In particular, the pre-recording system 1 comprises an audio acquisition module 2, a noise filter 3, a sound pattern recognition unit 4, and a speech pattern recognition unit 5. The pre-recording system 1 receives input from sensors and/or smart devices 6, which may be smartphones, cameras, desktop phones, or other individual microphones included in devices, like televisions, radios or the like. These sensors and/or smart devices 6 are able to record sound within their vicinity and to transmit it to the pre-recording system 1 which analyzes the audio data received, and in case it is verified that the audio data may indicate an abnormal event that may be an emergency event, it starts recording the audio data to a remote server (not shown). The identification of an abonormal event may be reflected by identifying a deviation from normal beyond a particular level value (threshold) of sound(s) measured in decibel (dB), for example, for a predefined time frame.
The thus recorded audio segment is tagged and labeled, i.e., it is provided with a tag that indicates the time slot of the potential emergency incident and labelled with the classification of the algorithm used for the recognition, such as scream, gunshot, broken glass or fights; then it is stored. Moreover, the deviation of the normal sound levels measured in dB is identified and marked with different colours depending on the intensity of the dB level and on how long it lasted. Based on the severity of the sound and the duration, an emergency call may be triggered automatically by calling, e.g., 911 to reach a PSAP. It is noted that the pre-recording system 1 may be included in the sensors and/or smart devices 6 or it may be a separate unit that receives the audio data, for example, via any kind of suitable communication network, from the sensors and/or smart devices 6.
After the establishment of the emergency call, the call taker at the PSAP 8 may have access and retrieve the pre-recording media stream with marked slots and labels based on the time and the event that initiated the call. The call taker may navigate to these slots or just inspect the slot marked with colours that correspond to more serious events and evaluate the severity of the incident in an accurate and efficient manner eliminating the time response to the emergency incident.
Fig. 2 is a flow chart illustrating the steps of the method of triggering an alarm in an emergency communication system according to an embodiment of the invention. As already explained with respect to Fig. 1, the method starts at step S1 with audio acquisition while receiving audio data input from sensors and/or smart devices. In step S2, the audio data is filtered, and subsequently, in step S3, an analysis of the audio data is carried out. In step S4, a sound pattern recognition is carried out, followed by a sound event detection in step S5. At the same time, in step S4', a speech pattern recognition is carried out, followed by a speech event detection in step S5'. In step S6 it is verified, wether an abnormal event is detected in either one of steps S5 or S5'. If negative, then triggering an alarm is rejected in step S12. If positive, recording is started in step S7 that is stored in step S8. In step S9, audio frames are labelled with respect to duration and classification, as outlined with respect to Fig. 1. If the duriation is determined to be abnormal, in step S10, it is assumed that an emergency incident has occurred, and an emergency system is triggered in step S11. In the illustrated scenario, an emergency call is initiated and forwarded to a PSAP 8 at which an agent or call taker will take care of the emergency incident.
In real life, the above described procedure may be implemented, for example, for the cases described below. For example, as to domestic violence, a home of a couple may be equipped with IoT devices sound and speech detectors. The husband of the couple may have a severe drinking problem. Suddenly, with no reason, he may pick up some objects and throw them at his wife. The smart devices and sound detectors detect the wife's screaming, shouting for help. Moreover, the sound of objects dropping is identifying. The pre-recording system 1 described with respect to Fig. 1 gathers the sounds and speech, and processes and identifies these events. The recording starts, the time slots with the suspicious events are marked and a label is tagged with the classification of shouting help, screaming and fighting. The suspicious event is identified as an emergency event. Thus, an automatic trigger is activated and the PSAP 8 is notified automatically. The recording is set to the call taker with marked time slots and labels related to the sequences that initiated the emergency call. The call taker may quickly navigate to the recorded data and evaluate the severity of the emergency event more accurately while decreasing the response time.
Another case may be conceived, in which a young girl having a mental handicap is living in a State government funded group home, and although she is classified as having "high support needs", she receives only two hours of support each day. For the remaining 22 hours, she is left unsupervised and unsupported. During the unsupported time, another girl from the residential facility attacks her. The attacked girl fights, screams, and cries. Sounds and speech are received by the peripherical devices located at the residential facility. The pre-recording system processes and identifies these events starting the recording. The time slots with identified screams, cries are marked and labelled with the respective classification event. Based on the deviation and the variance of the sound and speech thresholds, dB values, the severity of the event is identified as emergency or non-emergency. In emergency events, an automated emergency call is performed at the PSAP side with the information regarding the location, the nature of the event and the recorded audio data. The call taker may quickly navigate to the recorded data and evaluate the severity of the emergency event, inform the security of the residential facility and the ambulance to respond immediately to the event.
Fig. 3 shows a classification scheme for different types of sounds. As illustrated, the speech and sound recognition modules gather, process and evaluate the audio segments received from peripheral devices in order to detect events that can deal with important emergency incidents such as human screams, yelling, crying, asking for help in correlation with physical assault accusation or battering as well as gunshots.
Fig. 4 shows an labelling scheme for audio recordings. Namely, the sound levels measured in dB are identified and marked with different colours (indicated here by different shading and patterns) depending on the intensity of the dB level and on how long it lasted. Based on the severity of the sound and the duration, the PSAP may be triggered automatically.
Fig. 5 shows a map indicating a range around a location at which an emergency has occurred. This figure will be used for explaining how social media may be exploited for assisting an agent at the PSAP in processing the emergency incident. Involving social media will reduce the processing time during an emergency call. As mentioned with respect to Fig. 1, there may be a variety of physical reactions upon any kind of assault or attack like crying, shouting, etc.. All these instances are recorded and categorized according to the aforementioned method. A proliferation of methods known from prior art may be used in terms of detecting different types of screaming and grouping them accordingly. Usually, upon an assault, there exist versatile types of reactions and this makes it difficult for the call taker at the PSAP to identify what is really going on during the incident. In order to evaluate the latter, she/he would have to check the entire audio stream which will take a lot of time.
Considering former times, whenever domestic violence happened, there were complaints from neighbors reporting the incident. However, many times, the reporting or such incidents to the relevant authorities by neigbours or other eye witnesses mostly was done too late and thus, not in time for taking appropriate measures to prevent further escalation or the like.
Nowaday, social media and more particularly chat messages offer an interface with subjective anonymity where people may express their feelings or opinions more freely. Therefore, if a neighbor who witnesses an emergency incident of any kind would use a text message for reporting the incident, this would make things a lot easier and faster. Although text messaging provides an ostensible method to increase the probability that a neighbor would proceed in reporting the incident, smart phone push notification services bring two more factors that would increase the chances even more:

(1) Offering a notification to the chat user of a possible assault. For example, a push notification asking to confirm the assault.
(2) The security that there will be no follow up calls or questions about the incident. For example, a question that asks specifically and unambiguously about an assault with no specific questions about the place of the attack, names etc.

In the Fig. 5, a circle is illustrated that represents a predetermined radius around an emergency incident. Within the circle, there is a plurality of smart phones 7 that lie wihin the vicinity of the emergency incident, the location of which is indicated by reference numeral 10. The identification whether a phone in fact is located within this range may be given by a social media application that is aware of the respective GPS location. Push notifications based on classified events may thus be triggered by the PSAP to Social Media servers and is forwarded by the latter only to the phones within the predetermined radius.
Notification messages may have two forms:

(1) A plain question when there is only one type of reaction (shouting, screaming, etc.), or
(2) A specific questions to multiple users based on the outcome of the reaction detection.

As to the first form (1), the PSAP may receive all answers and may decide automatically if the assault can be confirmed and may provide an update to the call taker indicating the severity of the incident. If no answer is received, the case may be dropped or can be processed according to a normal priority.
According to another embodiment, another approach may be taken according to which not all users in the vicinity of the emergency event receive the same message. Since there are multiple reaction types reported, a different set of questions may be pushed to the neighbors' end devices. For example:

Do you hear screaming in your neighborhood?
Is someone fighting with another?
Do you hear someone shouting for help?

The answer to the above listed questions may be yes or no, without a follow up as the call taker will have all the necessary details about the nature of the assault. Upon responding a 'thank you' message may be forwarded offering the opportunity to call the emergency service for providing further information.
Fig. 6 shows a chart illustrating the clustering of artificial responses from neighbors. In the example shown here, the shouting for help was observed by the majority of neigbors in the vicinity of the emergency incident, followed by screaming. Only few responses revealed crying and sounds of fighting.
Fig. 7 schematically illustrates a messging flow in case of an emergency incident. An emergency service or PSAP 8 receives an emergency message, indicated by A. The PSAP 8 now processes the emergency incident as described above and sets a radius of x m measured from the location of the emergency incident, indicated by reference numeral 10. The PSAP 8 triggers a message to a social media server, wherein a push notification service 9 of a social media server 11 is informed about the location of the emergency incident and the radius. As the push notification service 9 knows the GPS location of its subscribers, here, mobile devices 7, it thus sends messages as outlined with respect to Fig. 5 only to those mobile devices 7 that are specified by the GPS location and radius set by the emergency service 8. The content of the message may be a single question to all recipients or multiple diverse questions. A time window is set by the emergency service 8 and it signifies the period that the emergency service anticipates that the incident will last. For instance, if a single beating is heard then the time window would be significantly less compared to an assault that contains multiple occurrences of the same or diverse reaction types. The time window may be also a moving window that updates its end time.
Fig. 8 schematically illustrates a messaging flow when neighbors within a predetermined radius of the emergency incident provide feedback. Here, the social media server 11 sends the information back to the emergency service during the above mentioned time window. The radius value is also sent by the PSAP 8 since it is the authority that is able to predict the severity of the incident. The PSAP 8 may also decide to enlarge the radius if the product sound level is high and can be heard even from distant neighbors (for example, in case a shooting is heard).

Reference numerals

1: pre-recording system
2: audio acquisition module
3: noise filter
4: sound pattern recognition
5: speech pattern recognition
6: sensor and/or smart devices
7: smart phones
8: PSAP
9: Push notification service
10: Location of emergency incident
11: Social media server

Claims

Computer-implemented method of triggering an alarm in an emergency communication system, wherein the emergency system comprises a pre-recording system (1) that is connected to at least one PSAP (8) and a plurality of peripheral devcies (6) that respectively are equipped with a microphone via an emergency communication network, the method comprising the steps of
- receiving, at the pre-recording system (1), audio data input from at least one peripheral device (6), the audio data comprising speech and/or sound,

- determining, if the speech and/or sound exceeds a noise level threshold for a predetermined period of time, and

- if the noice level threshold is exceeded for a predetermined period of time, identifying a possible emergency incident;

- recording the audio data comprising the speech and/or sound related to the possible emergency incident, and

- initiating an emergency call to at least one PSAP (8).
Computer-implemented method according to claim 1, wherein the method further comprises a step of tagging the recorded audio data with respect to time of occurrence.
Computer-implemented method according to claim 1 or claim 2, wherein the method further comprises a step of classifying speech and/or sound according to a predetermined classification algorithm.
Computer-implemented method according to claim 3, wherein classes for the classification step comprise human screams, yelling, shouting, crying, asking for help, combat sounds, and/or gunshots.
Computer-implemented method according to any one of the preceding steps, wherein the method further comprises a step of evaluating the severity of the possible emergency incident on the basis of the noise level and the duration.
Computer-implemented method according to any one of the preceding claims, wherein the method further comprises a step of determining the location of the possible emergency incident based on GPS data received from the respective peripheral device (6) that has transmitted the audio data.
Computer-implemented method according to any one of the preceding claims, wherein the peripheral devices (6) comprise smart devices, in particular, smart phones, tablets, desktop phones, cameras, microphones, and/or sensors.
Computer-implemented method according to any one of claims 6 or 7, wherein the emergency call comprises information of the location of the possible emergency incident and the nature of the emergency incident.
Computer-implemented method according to any one of the preceding claims, wherein the emergency call comprises the audio data that has been recorded, tagged, and classified.
Computer-implemented method according to any one of claims 6 to 9, wherein the method further comprises a step of setting an area having a predetermined radius around the location (10) of the possible emergency incident, and sending the location data and the predetermined radius to a social media server for identifying mobile devices (7) used by its subscribers that are currently located within the predetermined radius around the location (10) of the possible emergency incident, and for activating a push notification service to the subscribers for questioning the subscribers about the possible emergency incident.
Computer-implemented method according to claim 10, wherein the questions are yes-or-no questions.
Computer-implemented method according to claim 11, wherein the subscribers are provided with different questions.
Computer-implemented method according to any one of claims 10 to 12, wherein the location of the subscribers is determined based on GPS data.
Emergency communication system comprising a pre-recording system (1) adapted for carrying out the computer-implemented method according to any one of the preceding claims.