CA2879204A1 - Emergency detection and response system and method - Google Patents

Abstract

Disclosed herein are automated emergency detection and response systems and methods, in accordance with different embodiments of the invention. In some embodiments, respective data sets generated in respect of distinctly located sensors are compared, and an appropriate response protocol is selected based on this comparison and as a function of at least one of the data sets.

Description

EMERGENCY DETECTION AND RESPONSE SYSTEM AND METHOD
REFERENCE TO COPENDING APPLICATIONS
100011 The application is a Continuation-in-Part application of U.S.
Patent application no. 12/471,213, filed May 22, 2009, which claims benefit under 35 U.S.C.
119 of U.S. Provisional application no. 61/071,939, filed May 27, 2008, the entire subject matter of which applications is incorporated herein by reference in its entirety.
FIELD OF THE DISCLOSURE
100021 The present disclosure relates to emergency detection systems, an in particular, to automated emergency detection and response systems and methods.
BACKGROUND
10003) Falls in the home are the most common cause of injury among older adults or those with disabilities, and are a significant challenge to older adults maintaining their mobility. To address this issue, several personal emergency response (PERS) products have been developed that allow an older adult to call for help using a wearable panic button. However, these devices often are ineffective because users forget to wear the button, user training is required, and the system cannot be operated if the person is severely injured or left unconscious. These limitations are further exacerbated if the older adult has a cognitive impairment, such as Alzheimer's disease.
f00041 Therefore, there remains a need for an emergency detection and response system and method that overcomes at least some of the drawbacks of known techniques, or at least, provides a useful alternative.
[0005] This background information is provided to reveal information believed by the applicant to be of possible relevance to the invention. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the invention.

SUMMARY
[0006] An object of the invention is to provide an emergency detection and response system and method that overcomes at least some of the drawbacks of known techniques, .
or at least, provides a useful alternative thereto.
100071 In accordance with one embodiment of the invention, there is provided a system for detecting and responding to an emergency event occurring in or near a predetermined area, the system comprising: two or more detection modules each adapted for operative coupling to one or more sensors distinctly disposed in or near the area to detect the event, and each configured to generate a respective data set representative of the detected event; and a controller operatively coupled to said detection modules and configured to compare each said respective data set and implement a response protocol as a result thereof as a function of at least one said respective data set.
[00081 In accordance with another embodiment of the invention, there is provided a method for responding to an emergency event occurring in or near a predetermined area, is the system comprising: monitoring the area with two or more sensors distinctly disposed in or near the area; generating respective data sets representative of activity within the area via said two or more sensors; comparing said respective data sets;
automatically detecting the event from at least one of said respective data sets; selecting a response protocol as a function of said comparison and at least one of said respective data sets; and implementing said selected response protocol.
10009] In accordance with one embodiment, there is provided oone or more computer-readable media having statements and instructions embodied therein to implement, upon execution by one or more processors, the above method.
[0010] In an exemplary embodiment, there is provided a system for detecting and responding to emergency events occurring in a predetermined local area, comprising a plurality of local emergency detection and response units positioned in the local area.
Each local emergency detection and response unit includes one or more local sensing agents and a local detection manager. The local detection manager includes a local processor and each local sensing agent is operable to detect a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor. The detection manager is operable to receive the data and to assign a value to the emergency factor according to the data. A local interface agent is provided for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor. A central location controller unit is also provided and includes a central processor and a central interface agent in communication with the local interface agent on the data path to receive the event status messages therefrom. Each local emergency detection and response unit includes a local emergency event response agent for responding to at least one person in or near the local area. The central location controller unit and/or the local emergency detection and response unit are operable for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification, and which includes dispatching a message to the person and recording a response therefrom The central processor includes a ranking agent for ranking status signals being received from more than one local emergency detection and response unit in the same local area.
[00111 In some exemplary embodiments, the ranking agent is operable to rank each of the emergency detection and response units according to one or more ranking criteria.
The central processor is also operable to select one of the emergency detection and response units according to the ranking as an active local emergency detection and response unit to initiate the emergency event response protocol.
100121 In some exemplary embodiments, the central processor is operable to receive status signals over regular, or periodic time intervals and to change the selection of the active emergency detection and response unit according to changes in the status signals.
[0013] In some exemplary embodiments, the at least one emergency factor is selected from a plurality of video variables and/or thresholds, a plurality of audio variables and/or thresholds, and a plurality of environmental variables and/or thresholds.

[0014] In some exemplary embodiments, the environmental variables and/or thresholds, include smoke concentration, carbon monoxide concentration, oxygen concentration, and/or environmental pollutant concentration.
[0015] In some exemplary embodiments, the ranking agent is operable to access and compare a plurality of emergency factors received from the plurality of reporting local emergency detection and response units.
[0016] In exemplary some embodiments, the emergency factor includes a video image, the variable including size, shape, and motion of object being tracked.
[0017] In some exemplary embodiments, the emergency factor includes an audio signal, the variable including amplitude and type of the audio signal.
100181 In some exemplary embodiments, the ranking agent assigns, as active, the active local emergency detection and response unit possessing the highest ranking for a given emergency factor.
[0019] In some exemplary embodiments, the local processor or the central processor includes a voice processing agent for issuing voice signals to the person or receiving and interpreting voice signals from the person.
[0020] In some exemplary embodiments, one of the sensing functions includes a fixed or steerable digital camera.
[0021] In some exemplary embodiments, one of the sensing functions includes a microphone.
[0022] In some exemplary embodiments, one of the sensing functions includes an environmental detector.
[0023] In another exemplary embodiment, there is provided a system for detecting and responding to emergency events occurring in a predetermined local area.
The system comprises a plurality of local emergency detection and response units positioned in the local area. Each local emergency detection and response unit includes one or more local sensing agents and a local detection manager. The local detection manager includes a local processor. Each local sensing agent is operable to detect emergency events by a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor. The detection manager is operable to receive the data and to assign a value to the emergency factor according to the data. A
local interface agent is provided for issuing, on a data path, one or more event status messages including the assigned value for the emergency factor. A central location controller unit is also provided which includes a central processor and a central interface agent in communication with the local interface agent on the data path to receive the event status to messages therefrom. Each local emergency detection and response unit includes a local emergency event response agent for responding to at least one person in or near the predetermined area. The central location controller unit and/or the local emergency detection and response unit are operable for classifying the assigned value of the emergency function to form an assigned value classification and for initiating the local emergency event response agent to implement a response protocol according to the assigned value classification.
100241 In some exemplary embodiments, the central processor is operable to receive regular or periodic messages from each emergency detection and response unit after detecting activity in the local area. The central processor may then record the activity to track a subject between multiple emergency detection and response units as the subject moves through the local area.
100251 In another exemplary embodiment, there is provided a system for detecting and responding to emergency events occurring in a predetermined local area.
The system comprises a plurality of local emergency detection and response units to be positioned in the local area. Each local emergency detection and response unit includes one or more local sensing agents and a local detection manager. The local detection manager includes a local processor. Each local sensing agent is operable to detect a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor. The detection manager is operable to receive the data and to assign a value to the emergency factor according to the data. A local interface agent is also provided for issuing, on a data path, one or more event status signals includes the assigned value for the emergency factor. Each local emergency detection and response unit includes a local emergency event response agent for responding to at least one person in or near the local area, the local emergency detection and response unit being operable for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification. The response protocol includes dispatching a message to the person and recording a response therefrom. One or more of the local detection and response units are responsive to data received on the data path for ranking status signals received from more than one local emergency detection and response unit in the same local area. One or more of the local emergency detection and response units are operable to select one of the emergency detection and response units according to the ranking as an active local emergency detection and response unit to initiate the emergency event response protocol.
[0026] In some exemplary embodiments, the local emergency detection and response units are each operable for exchanging the assigned values from one another to form an assigned value group.
[0027] In some exemplary embodiments, one or more of the local emergency detection and response units are operable to select an active emergency detection and response unit according to a ranking of individual values in the value group.
[0028] In some exemplary embodiments, the central location controller unit and/or the local emergency detection and response unit is operable for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification.
[0029] In still another exemplary embodiment, there is provided a method for detecting and responding to emergency events occurring in a predetermined local area.
The method comprises providing a plurality of local emergency detection and response units positioned in the local area, providing each local emergency detection and response unit with one or more local sensing agents and a local detection manager with a local processor, arranging the local sensing agents to detect a change in a given emergency factor in the local area and to convey first data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the first data, providing a local interface agent for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor, providing a central location controller unit including a central processor and a central interface agent in communication with the local interface agent on the data path to receive the event status messages therefrom, providing in each local emergency detection and response unit a local emergency event response agent for responding to at least one person in or near the predetermined area, arranging the central location controller unit and/or the local emergency detection and response unit for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification, providing the central processor with a ranking agent for ranking status signals being received from more than one local emergency detection and response unit in the same predetermined local area.
[0030] In still another exemplary embodiment, there is provided a method for detecting and responding to emergency events occurring in a predetermined local area.
The method comprises providing a plurality of local emergency detection and response sites in the local area, providing each local emergency detection and response site with one or more local sensing agents, sensing at each of the sites a change in a given emergency factor in the local area, providing a detection manager to receive from each site a first data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the first data, dispatching one or more event status signals including the assigned value for the emergency factor, providing a central location controller to receive the event status messages therefrom, arranging the emergency detection and response site and/or the central location controller to classify the assigned value of the emergency function to form an assigned value classification, arranging the central location controller to rank status signals received from more than one local emergency detection and response site in the same predetermined local area, according to one or more ranking criteria and to select one of the emergency detection and response sites according to the ranking as an active local emergency detection and response site to initiate an emergency event response protocol.
NOM In yet another exemplary embodiment, there is provided a method for detecting and responding to emergency events occurring in a predetermined local area.
The method comprises: providing a local emergency detection and response unit in the local area, providing the local emergency detection and response unit with one or more local sensing agents and a local detection manager with a local processor, configuring the local sensing agents to detect an emergency event by a change in a corresponding emergency factor in the local area, and to generate data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the first data, providing a local interface agent for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor, providing a local emergency event response agent for responding to at least one person in or near the local area, and configuring the local interface agent to receive an initiation signal for initiating the local emergency event response agent to implement a response protocol according to an assigned value.
[00321 In yet another exemplary embodiment, there is provided a method for detecting and responding to emergency events occurring in a predetermined focal area.
The method comprises: providing a plurality of local emergency detection and response sites in the local area; providing each local emergency detection and response site with one or more local sensing agents; configuring the sensing agents to sense a change in a predetermined emergency factor in the local area; providing a detection manager at each site to receive, from each sensing local sensing agent, data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the data; dispatching, over a data path, one or more event status messages including the assigned value for the emergency factor;
providing a central location controller to receive, in a first operative phase, the event status messages over the data path; and configuring the central location controller, in a second operative phase, to: classify the assigned value of the emergency factor to form an assigned value classification; rank the status messages according to one or more ranking criteria, select one of the emergency detection and response sites according to the ranking as an active local emergency detection and response site to initiate an emergency event response protocol; and instruct the selected active local emergency detection and response site to initiate the emergency event response protocol.
[0033] Some exemplary embodiments further comprise configuring the central location controller to: operate in the first operative phase receive the event status messages; operate in the second operative phase concurrently with the first operative phase to: select a new active local emergency detection and response site according to changes in the assigned values of the emergency factors; and instruct the new active local emergency detection and response site to initiate the emergency event response protocol.
[0034] Other aims, objects, advantages and features of the invention will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE FIGURES
[0035] Several embodiments of the present disclosure will be provided, by way of examples only, with reference to the appended drawings, wherein:
[0036] Figure 1 is a schematic diagram of an emergency detection and response system, in accordance with one embodiment of the invention;
[0037] Figure 2 is a schematic diagram of the system of Figure 1, further comprising respective response modules for each detection module, in accordance with one embodiment of the invention;
[0038] Figures 3A and 3B are operational perspective views of an emergency detection and response system, in accordance with an exemplary embodiment of the invention;

[0039] Figures 4 and 5 are schematic views of portions of the system of Figure 3;
[0040] Figure 6 is a flow diagram showing an operative mode of the system of Figure 3A, in accordance with an exemplary embodiment of the invention;
[0041] Figures 7A and 7B are raw and processed video images, respectively, captured by an emergency detection and response system in tracking a user in a predetermined area and in identifying regular activity, in accordance with one embodiment of the invention; and [0042] Figures 8A and 8B are raw and processed video images, respectively, captured by the emergency detection and response system of Figures 7A and 7B, identifying a possible emergency event, in accordance with one embodiment of the invention.
DETAILED DESCRIPTION
100431 It should be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having"
and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless limited otherwise, the terms "connected," "coupled," and "mounted," and variations thereof herein are used broadly and encompass direct and indirect connections, couplings, and mountings. In addition, the terms "connected" and "coupled" and variations thereof are not restricted to physical or mechanical or electrical connections or couplings. Furthermore, and as described in subsequent paragraphs, the specific mechanical or electrical configurations illustrated in the drawings are intended to exemplify embodiments of the disclosure. However, other alternative mechanical or electrical configurations are possible which are considered to be within the teachings of the instant disclosure. Furthermore, unless otherwise indicated, the term "or" is to be considered inclusive.

100441 Described below are various embodiments of an emergency detection and response system and method, wherein multiple sensors distributed throughout one or more predetermined areas can be jointly used in detecting an adverse situation in one of these areas and responding accordingly. For example, in one embodiment, the various sensors may be used to generate respective data sets to be processed in identifying an emergency event. For example, each data set may be processed locally and/or centrally to identify an emergency event, referred to hereinbelow in various examples as a local emergency factor, or again be processed in defining a globally assessed emergency event or factor. Upon identification of an emergency event from this data, the system can be to configured to implement an appropriate response protocol. For example, in some embodiments, an appropriate response may be established based on a single data set upon this data set identifying an emergency event of concern, whereas in other embodiments, two or more data sets, for example as generated from distinctly located sensors, may be used to improve emergency identification and response.
100451 In one such embodiment, respective data sets may be compared and ranked, whereby a highest ranking data set (or emergency factor) may be used to further identify the emergency event and/or implement an appropriate response protocol. For example, where audio signals are captured by two or more audio sensors distinctly located in or near a predetermined area where an emergency event may have taken place, the audio signals captured by each of these sensors may be ranked based on one or more ranking criteria (e.g. signal strength, background noise, proximity as defined by complimentary sensory means, speech recognition reliability, etc.), and a response protocol implemented as a function of this highest ranking signal. In some embodiments, this protocol may involve the dispatch of an emergency communication to an external party based, at least in part, on a content of the highest ranking audio signal (e.g. identified speech element/spoken word(s), high intensity sound (yell, break-in, etc.)). In some embodiments, this protocol may rather or also involve prompting a user in or near the predetermined area via a prompting device associated with a highest ranking sensor (i.e.
the sensor having produced the highest ranking data set), whereby a user response captured by the highest ranking sensor and/or other sensors/input devices associated therewith (e.g. commonly located) may be used to further ascertain the situation in implementing the response protocol.
[00461 In other such embodiments, respective data sets may rather or also be compared and merged based on overlap identified therebetween, thereby allowing to produce a more accurate rendition of the emergency event taking place, and thus leading to the rendition of a more accurate or informative emergency factor. For example, where imaging signals are captured by two or more distinctly disposed imaging sensors (e.g.
video, infrared (1R), multispectral imaging, heat sensor, range sensor, microphone array, etc.), data sets representative thereof may be compared to identify overlap therebetween (e.g. via computer vision techniques, 2D/3D mapping, etc.) to enhance emergency identification and thus improve response thereto. For instance, a merged data set may provide for a more accurate depiction or imaging of the event, for example, where different imaging sensors may be used to piece together a more complete image of a user in distress located in partial view of these sensors. Alternatively, or in addition thereto, merged data sets may be used to improve locational techniques for accurately identifying A location of such a user, for example, using various 2D/3D mapping techniques.
100471 As will be appreciated by the skilled artisan, both data set ranking and data set merging may be implemented in concert to increase system accuracy, or may be implemented independently depending on the application at hand, or again exclusively depending on the intended application for the system.
[00481 Referring now to Figure 1, and in accordance with one embodiment of the invention, an emergency detection and response system, generally referred to using the numeral 100 and in accordance with one embodiment of the invention, will now be described. The system 100 generally comprises two or more detection modules 102A, 102B each adapted for operative coupling to one or more sensors, as in sensors 104A and 104B, respectively, which are distinctly disposed in or near a predetermined area 106 to detect emergency events occurring in or near this area (e.g. at location A).
The sensors 104A, 104B, in concert with detection modules 102A, 102B, are generally configured to generate respective data sets representative of a detected event.

[0049] It will be appreciated that while distinct detection modules are depicted in this example to schematically represent the generation of distinct data sets representative of distinct data perspectives, viewpoints and/or origins as prescribed by the respectively located sensors from which they originate, various data/signal processing architectures and platforms may be considered herein without departing from the general scope and nature of the present disclosure. For example, distinctly located sensors may themselves be configured to generate respective data sets via integral detection modules, or again be configured to communicate raw or pre-processed data to a common detection module for further processing, albeit retaining some original signature in identifying the sensor from which such data originated. These and other such permutations will be readily apparent to the person of ordinary skill in the art, and is thus considered to fall within the scope of the present disclosure.
[0050] The system 100 further comprises one or more controllers 108 operatively coupled to each detection module 102A, 102B and configured to compare each of the respective data sets (e.g. via ranking and/or merging module 109) and implement a response protocol as a result of this comparison and as a function of at least one of these respective data sets. For example, in one embodiment, the controller 108 may be configured to process locally derived emergency factors provided by the respective data sets of each detection module to select and implement an appropriate response protocol.
Alternatively, the controller may derive each local emergency factor from raw and/or processed data in making this comparison. In yet another embodiment, the controller may rather derive a global emergency factor from distinctly provided data sets, for example where such data sets are compared for ranking and/or merging purposes. As will described in greater detail below, various techniques may be employed to achieve intended results, such as machine learning techniques, artificial intelligence and/or computer vision techniques implemented on the local and/or global data set(s).
[0051] In one embodiment, the controller 108 may be configured to dispatch an emergency message or warning to an external user, such as a friend, family member, neighbour, medial practitioner, security unit, external emergency response unit and the like, via one or more wired and/or wireless communication networks, this functionality commonly depicted in this illustrative embodiment by antenna 110. Examples of such communication links may include, but are not limited to, a landline phone link, a cellular phone or data link, a residential Wi-Fl network in concert with an associated communication platform (e.g. home computer network for dispatching an e-mail, text message or the like), and the like, as will be readily appreciated by the skilled artisan.
Alternatively, or in addition thereto, a given response protocol may include the dispatch of related sensory data, be it raw, pre-processed and/or processed data, for example in qualifying the detected emergency. Dispatched data may include, but is not limited to, audio data, for example as recorded in identifying the emergency event and/or as a response to an emergency initiated prompt (discussed below), video/imaging data depicting the user's emergency/condition, positional data to identify a location of the user within the area or amongst plural areas commonly monitored by the system, environmental data, etc.
[00521 Again, as will be appreciated by the skilled artisan, while a singular controller 108 is depicted in this example to interface with respective detection modules 102A, 102B, alternative processing architectures may also be contemplated without departing form the general scope of the present disclosure. For example, the computational and/or communicative functionalities of the controller 108 as described in the context of this example, may be singularly implemented within a same global control module, wherefrom emergency detection and response protocols may be commonly initiated in respect of a given area, but also in respect of plural areas commonly monitored by an expanded embodiment of system 100. In other embodiments, these functionalities may rather be implemented locally and/or distributively between control modules respective to each detection module, that is, wherein a network of emergency detection units may be configured to cooperate in processing respective data sets and implementing a common response protocol. In such examples, each detection unit may comprise a respective sensor(s), processor, data storage device and network communication interface to implement various computational and communicative functionalities of the system 100 locally. Accordingly, while the below examples will describe various modular architectures for the acquisition, processing and manipulation of sensory data, and response protocols implemented as a function thereof, it will be appreciated that such modularity is only provided for the purpose of clarifying various functions and features of the herein described embodiments of the invention, and should thus not be taken as limiting to the scope of the present= disclosure. For example, the controller may be described as comprising a merging module and/or a ranking module, as introduced above and further described below, for the purpose of managing multiple data sets.
Clearly, such processing modules may be implemented independently within the context of a common controller or control module, or again implemented distributively within the context of two or more central and/or distributed controllers. Furthermore, the modularity of the processing techniques contemplated herein may be more or less defined in different embodiments, whereby data may be processed in parallel, in sequence and/or cooperatively depending on the intended outcome of the system's implementation.
100531 Referring now to Figure 2, and in accordance with another embodiment of the invention, a similar system 200, will now be described. In this embodiment, the system 200 again generally comprises two or more detection modules 202A, 202B each adapted for operative coupling to one or more sensors, as in sensors 204A and 204B, respectively, which are distinctly disposed in or near a predetermined area 206 to detect emergency events occurring in or near this area (e.g. in respect of user B). The sensors 204A, 204B, in concert with detection modules 202A, 202B, are generally configured to generate respective data sets representative of a detected event.
[00541 The system 200 further comprises one or more controllers 208 operatively coupled to each detection module 202A, 202B and configured to compare each of the respective data sets (e.g. via ranking and/or merging module 209) and implement a response protocol as a result of this comparison and as a function of at least one of these respective data sets.
100551 In this embodiment, a respective response module 212A, 212B is associated with each of the detection modules 202A, 202B and adapted for operative coupling to a respective prompting device 214A, 214B and input device, which in this embodiment, is commonly operated via sensors 204A, 2048. Accordingly, upon the controller 208 processing each data set, a response protocol may be selected and implemented in which the user B is first prompted via an appropriate response module and prompting device to provide a user input in return, which user input can then be relayed to the controller (or local response module) to refine, adjust and/or terminate the response protocol. For example, in one embodiment, a highest ranking data set may be used to identify a most reliable sensory location (e.g. from which a user response may have the greatest likelihood of being understood or captured by the system 200). Based on this ranking, a response module and prompting device associated with this location (e.g.
associated with a highest ranking detection module/data set) may be operated by the controller to proceed with the next step of the response protocol. In the examples described below, a prompt is provided via a selected prompting device, and a user input is recorded in response thereto. In the event where the sensors 204A, 204B include an audio sensor (e.g.
microphone, microphone array), this audio sensor may double as input device for recording the user's response.
100561 It will be appreciated that various device combinations may be used to achieve this result without departing from the general scope and nature of the present disclosure.
For example, a combined detection and response unit encompassing the functionality of both detection and response modules may be considered, as can a self-contained processing unit further encompassing at least some of the processing functionalities of the controller, to name a few. Furthermore, and as depicted by prompting device 216, an additional or alternative prompting device, such as a visual prompting device for the hearing impaired (e.g. communal or dedicated screen) may be used by respective response modules or shared between them in prompting a user response.
100571 In one embodiment, the controller 208 may be configured to dispatch an emergency message or warning to an external user (e.g. via antenna 210), based on the processed sensory data and/or user responses.
[0058] Referring now to Figures 3 to 5, and in accordance with an exemplary embodiment of the invention, an emergency detection and response system 10 will now be described. The system 10 is generally provided for detecting and responding to emergency events occurring in a predetermined local area 12. The system includes a plurality of local emergency detection and response units (or hereinafter referred to as EDR units) 14 positioned in the local area 12.
100591 Each EDR
unit 14 includes one or more local sensing agents or sensors 16 and a local detection manager 18 (i.e. detection module). The local detection manager includes a local processor 20 with a control module 22 which communicates with the local sensing agents 16, including a local video sensing agent 24, a local audio sensing agent 26 and, in this example, an environmental sensing agent 28. Each local sensing agent is operable to detect a change in a given emergency factor in the local area and to report to the control module accordingly. Each local sensing agent conveys data representative of the change in the emergency factor to the control module 22.
[00601 In this case, the local video sensing agent 24 includes a video camera 30 monitored by a video processing module 32. As will be described, the video processing module 32 is thus operable to detect a change in a given emergency factor in the local area 12, in this case, for example, by detecting the presence of a person, subject, user or other object in the local area 12. In this case, the person may include, among others, a patient in a healthcare facility, or a resident of a residential or other facility, with or without disabilities, or others who may be prone to falling or to disorientation or suffer from other condition worthy of monitoring and response, in the manner to be described.
Also in this case, the audio sensing agent 26 includes a microphone 34 monitored by a speech dialog module 36. As will be described, the speech dialog module 36 is thus operable to detect a change in a given emergency factor in the local area 12, in this case, for example, by detecting a verbal message from a person in the local area, which may be in response to automated questions being asked by the EDR units 14 to determine the severity of the emergency, as will be described. In both cases, the video processing module 32 and the speech dialog module 36 convey data to the control module 22 for further processing. In this case, the control module 22, the video processing module 32, and the speech dialog module 36 may be applications running on the local processor 20 or be one or more distinct processors, such as by way of video cards and the like.

[0061] The control module 22 is, in turn, operable to assign a value to the emergency factor according to the data received from one or more of the video processing module, the speech dialog module and the environmental monitor 28, as the case may be.
[0062] A local interface agent 38 is provided for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor.
[0063] A central location controller unit 40 is provided in the form of a central server, with one or more central processors 42 communicating with a central interface agent 44 to receive event status messages therefrom on a communication channel shown at 45. In this case, the central and local interface agents may include wired or wireless network cards, REID tags, Bluetooth, or other forms of transmitter receivers as the case may be.
10064j The system 10 is operable in a communication network which, in this example, is computer implemented and may be provided in a number of forms, by way of one or more software programs configured to run on one or more general purpose computers, such as a personal computer, or on a single custom built computer, such as a programmable logic controller (PLC) which is dedicated to the function of the system alone. A system controlling such a communication network may, alternatively, be executed on a more substantial computer mainframe. The general purpose computer may work within a network involving several general purpose computers, for example those sold under the trade names APPLE or IBM, or clones thereof, which are programmed with operating systems known by the trade names WINDOWS, LINUX or other equivalents of these. The system may involve pre-programmed software using a number of possible languages or a custom designed version of a programming software.
The computer network may be include a wired local area network, or a wide area network such as the Internet, or a combination of the two, with or without added security, authentication protocols, or under "peer-to-peer" or "client-server" or other networking architectures. The network may also be a wireless network or a combination of wired and wireless networks. The wireless network may operate under frequencies such as those referred to as 'radio frequency' or "RF" using protocols such as the 802.11, TCP/IP, BLUE TOOTH and the like, or other wireless, satellite or cell packet protocols.

While the assembly 10 collects location data from the EDR units 14, each EDR
alone or the central server 40 may have the ability to determine its location within the local area by use of other locating methods, such as by the use of network addresses, GPS
positions or the like.
[0065] Each local EDR unit 14 further includes a local emergency event response agent or module 46 for responding to at least one person in or near the predetermined local area (e.g. via a dedicated or shared prompting device and input device).
In this case, the emergency event response agent is provided by the speech dialog module 36 and a loudspeaker 48.
[0066] In this case, each local EDR unit 14 includes a housing 14a containing the local processor 20, the control module 22, the video camera 30, the video processing module 32, the speech dialog module 36, the microphone 34 and the loudspeaker 48.
However, other variations may see one or more of these components being located outside the housing 14a, as desired. If desired, the surface of the housing 14a may be paintable, allowing for custom colouring of the housing according to the decor of a monitored location. The housing may also be provided in varying shapes and styles creating different options for the product's appearance, as desired. To guard against the possibility of a power outage, each EDR unit may be provided with a backup battery.
[0067] As will be described below, the central location controller unit 40 and/or the EDR units 14 are operable for classifying the assigned value of the emergency function to form an assigned value classification and for initiating the local emergency event response agent 46 to implement a response protocol according to the assigned value classification.
[0068] The central processor 42 includes a ranking agent 50 for ranking status signals being received from more than one local EDR unit 14 in the same predetermined local area 12.
100691 The ranking agent 50 is operable to rank each of the EDR units 14 according to one or more ranking criteria. The central processor is thus operable to select one of the EDR units 14 according to the ranking as an "active" EDR unit to initiate the emergency event response protocol. Alternatively, one or more of the EDR units themselves may be configured to selective an "active" EDR unit. In this case, the data paths 14b may be configured to form a local emergency and detection and response network, in which the local emergency detection and response units are each operable for exchanging the assigned values from one another to form an assigned value group. In this case, one or more of the local emergency detection and response units being operable to select an active emergency detection and response unit according to a ranking of individual values in the value group.
[0070) In this case, at least one emergency factor may include a plurality of video variables and/or thresholds, a plurality of audio variables and/or thresholds, and a plurality of environmental variables and/or thresholds. The environmental variables and/or thresholds may, in this case, include temperature, atmospheric pressure, humidity, smoke concentration, carbon monoxide concentration, oxygen concentration, and/or environmental pollutant concentration.
[0071] The ranking agent 50 may access and compare a plurality of emergency factors received from the plurality of reporting local emergency detection and response units 14. The emergency factor may include, in this case, a video image, the variable including size, shape, and motion of object being tracked. Alternatively, the emergency factor may include an audio signal, the variable including amplitude and type of the audio signal.
[0072] Thus, Figures 3 to 5 shows a general schematic of the various parts of a single unit, while Figure 6 is a flow diagram outlining an exemplary decision-making process performed by the central server, to communicate with multiple EDR units simultaneously.
[0073] The system 10 is configured so that the EDR units 14 may be located throughout a person's living space. The central server 40 makes overall decisions about which EDR unit 14 is actively monitoring or communicating with the human user at a given point in time. In the event of an emergency, the central server 40 may also facilitate communications with the outside world (e.g. contact a neighbour, relative or 911), by way of an external interface unit 52, for example.
[0074] Each EDR unit 14 may thus include one or several hardware components which may be installed in a common housing, such as one or more cameras (e.g.
webcam or `steerable' camera, infrared or multispectral imaging device, heat sensor, range sensor, etc.) one or more small loudspeakers, a single, multiple or small array of microphones, a computer processor, or an environmental monitor, such as a smoke and/or carbon monoxide detector.
[0075] Each EDR unit 14 may be portable or mobile, such as on a movable robot, or it may be stationary and installed in an appropriate location in a user's house or long-term care facility, such as on the ceiling of the living room or bedroom. In one example, the EDR unit, or a component thereof, may be mounted on the user. This might include a blood pressure or heart rate monitor, or the like.
[0076] The EDR unit 14 may use the camera(s) or microphone(s) to monitor the living environment of a human subject in real-time. The camera may be fixed within the housing, with a static field of view, or `steerable' allowing it to follow the movement of a given subject. The local processor 20 in this case performs real-time analysis of the video and audio inputs to determine if an emergency event, such as a fall, has occurred.
In the event of an emergency, the EDR unit 14 can communicate with the subject via the microphone 34 and loudspeaker 48 and initiate a dialog using speech recognition software. Communicating with the subject in this way, allows the system to determine the level of assistance required. If external assistance is required (e.g.
from relatives, neighbours or emergency services), the local processor 20 can relay this information to the central server 40 located at a convenient location in the house or other facility.
Information between the local processor 20 and the central server 40 can occur via either a standard wired or wireless (Wi-Fi) communication network. The server may send information about an emergency event to the outside world via a variety of possible communications methods (e.g. landline or cell phone network, text messaging, email), via the external interface 52.

[0077] It should be noted that, in one example, the system 10 may ensure that the privacy of the subject is maintained at all times by configuring the local processor 20 to relay only computer vision and speech recognition results, as well as information about a possible emergency event, to the central server, but not any of the original video or audio information. In another example, original video or audio information may be relayed to the central server for further processing, as well as other or alternative types of data such as blob information, feature vectors, etc.
[0078] In this example, the environmental sensing agent 28 or Environmental Monitor may include sub-components such as a smoke detector and a carbon monoxide detector. Thus, in the event of a smoke, fire or carbon monoxide emergency the Environmental Monitor may also relay this information to the appropriate emergency services via the central server 40.
[0079] The video processing module 32 takes real-time video input from the video camera 30, and performs computer vision algorithms to determine if an emergency has occurred. The employed computer vision algorithms may include object extraction and tracking techniques such as adaptive background subtraction, color analysis, image gradient estimation, and connected component analysis. These techniques allow the system to isolate a human subject from various static and dynamic unwanted features of the video scene, including the static background, dynamic cast shadows and varying light conditions. As such, characteristics of the subject's movement, posture and behaviour may be monitored in real-time to determine if an emergency (e.g. a fall) has occurred.
The video processing module 32 may relay information of this emergency event to the control module 22. For instance, Figures 7A and 7B illustrate tracking results for a subject walking, with an original "webcam" image and the extracted silhouette of the subject (light blue, or white as shown in the black and white version presented herein) and their shadow (dark blue or grey in the black and white version presented herein). As the subject is not in need of assistance, the tracking box on the shows "green" (in this case in chain dotted lines). When a fall is detected, as shown in Figures 8A
and 8B, the tracking box may then change state, such as to the colour "red" (as shown by the solid lines) and the silhouette is elongated with very little shadow. In the event of an emergency, the control module 22 may instruct the speech dialog module 36 to initiate a conversation with the subject using speech recognition software, such as a small vocabulary, speaker-independent automatic speech recognition (ASR) software.
However, other configurations of ASR software, such as speaker-dependent or adaptable ASR, may also be used. This ASR software may be specially trained on population-specific and/or situation-specific voice data (e.g. older adult voices, voices under distress, atypical voice patterns caused by affliction or emergency events). The system may also learn its users' specific voice patterns in an offline or on-line manner during the lifetime of its usage to maximize speech recognition results. Other audio processing techniques, such as real-time background noise suppression, and the recognition of environmental sounds (e.g. falling objects, slam sounds, etc.) may also be employed to ensure robust system performance.
100801 The speech dialog module 36 may communicate directly with the subject by outputting speech prompts via the loudspeaker(s) 48 and listening to audio input via the microphone(s) 34, to determine the level of assistance that the subject may require. The outcome of this speech dialog can be sent to the control module 22 and, if further assistance is required, the control module 22 can relay this to the central server 40 via the communications network 45. Alternatively, or in combination therewith, visual prompts may be used to prompt the hearing impaired (e.g. via images, text and/or speech-to-text displays in or near the area), as can one or more physical input interfaces (e.g. a specialized keypad or touchscreen) be used for those unable to respond orally to reply to such prompts. In some embodiments, speech synthesis technology (e.g. text-to-speech), may be used in addressing occupants in rooms. For example, the voice pattern of the speech synthesis system may be customized or trained with the voice patterns of a particular person, e.g. a familiar and trusted voice, for instance to allow people with afflictions such as Alzheimer's or dementia to more easily co-operate with it.
[00811 An alternative example implementation of the system 10 may allow for emergency situations to be detected by either the video processing module 32 or the speech dialog module 36 simultaneously. In this configuration, the microphone 34 may be on at all times, allowing the speech dialog module 36 to listen for key emergency words or audio events (e.g. a cry for "help!", a loud crash), or again to detect distressed and/or atypical speech. This implementation may be particularly useful if the video processing module 32 is unable to detect a given emergency situation (e.g. if the subject is outside the field of view of the camera(s), or during low light conditions such as nighttime).
[0082] As an optional feature, the EDR unit 14 may also include a motion sensor 54, such as an infrared sensor. The video camera 30 may also be equipped for "night vision".
This may add additional functionality to the system, such as the ability for a given unit to automatically turn on when significant motion is detected (e.g. when a person enters a room), or for more robust vision tracking in low light conditions (e.g. at nighttime). This functionality may allow the system to also operate in an "away" mode, thereby to detect in-home disturbances or intrusions when the person is not home. Therefore, an additional application for the system may be to act as a home security system or to augment existing home security systems. A light bulb may also be fitted in each EDR unit, so as to be activated by a light switch, for example on a neighbouring wall. If desired, the light on the EDR unit may operate in the same fashion as a conventional ceiling-mounted light fixture, enabling a user to replace or augment existing ceiling mounted light fixtures with functions of the device 10.
[0083] The central server 40 is able to handle simultaneous communications with one or multiple EDR units 14, allowing for multiple EDR units 14 to be installed in different rooms of a house, assisted living facility or long-term care facility.
Therefore, at a given point in time, the central server may analyze the information simultaneously received from multiple EDR units 14 and determine which EDR unit is currently "active"
(i.e., which camera currently has the subject of interest within its field of view).
This may be accomplished by comparing the audio and/or computer vision tracking and/or audio processing results from each local processor 20. For example, the EDR unit currently tagged as "active" may be the one currently tracking the object with the largest size or with significant movement. This methodology allows for the central server 40 to track a subject between the cameras of multiple EDR units installed in the same room, or throughout various rooms of a house, ensuring that an emergency event is detected robustly.
[0084] Similarly, where more than one EDR is operable to detect a user within or near a same predetermined area, the system may be configured to select as "active" the EDR for which respective data generated thereby achieves a highest ranking, e.g. a highest reliability measure. Also, respective data sets processed from multiple EDRs can be compared to identify overlap therebetween, whereby multiple data sets may be merged based on this identified overlap to achieve greater monitoring, emergency detection and/or response. Accordingly, where more than one EDR can actively track the user, the system may be configured to actively process data from each active EDR to generate a merged data set. Merged data sets may, for example, provide greater locational data with respect to the user (e.g. 2D/3D mapping) as well as greater depiction of the user's status, particularly where only partial images can be rendered by each EDR, for example.
[0085] Each EDR may also be adapted for automatic global localization (e.g. real-world coordinates such as longitude/latitude or address), for example via GPS
and/or network geolocation, with a possible manual override. Such global localization may prove useful in initiating emergency responses and dispatching an exact location of the emergency event to external responders.
[0086] Different embodiments may further include one or more of the following optional features. For example, a given embodiment may include an event interface, whereby a central location controller unit may include an externally addressable interface allowing authorized third party clients to receive event data, such as emergency factors and other information gathered by the local processors, and act on it. For example, the event interface may allow authorized users to send event information to external notification methods, publish emergency events, import contact data and data storage, interface with web-services and social networks, GPS/locational information for emergency response and positional information for tracking the position of people in their environment (e.g. used in applications such as extracting, analyzing, or using patterns of living). In addition, such data may be correlated with other sensors in the environment such as pressure sensors, motion sensors, "smart-home" sensors, etc., for instance for improving the accuracy of emergency detection and 3rd party applications.
[0087] In some implementations, smart-home devices may also be used in the context of the herein described system to ensure user safety. For example, ensuring lights are on in the rooms people are present in or about to enter to prevent falls, or ensuring that the stove is turned off when a person leaves it unattended, e.g. when user is detected leaving the kitchen, may contribute to improved user safety. The central controller may also implement the functionality of a central "smart-home" controller, allowing user interfaces to smart home devices, for example.
[0088] In some implementations, the central location controller unit and/or the local units may be interfaced by users and/or administrators over a local or remotely accessible interface, such as a database-backed or configuration-file based webpage configuration.
This interface may allow the users to set preferences on event notification, account settings for 3rd party interfaces, etc. In addition, an administrator configuration may include advanced fine-tuning options for the emergency factor along with debugging information.
[0089] In some implementations, online learning may also be implemented to adjust parameters of the system in response to learned user preferences, such as a user's preference for emergency response criteria (e.g. detection sensitivity, timeout on user prompts for automatic emergency calls, etc.). For example some users may have slower responses to voice prompts than others, and the system may learn to wait longer for a response. Some users may be more or less independent than others, allowing the system to learn user's tolerance preferences for help/emergency response in order to preserve dignity and a sense of independence, for example..
[0090] Figure 6 is a flow diagram outlining this decision-making process performed by the central server 40, which may communicate with multiple EDR units 14 simultaneously. At step 301, the central server 40 receives emergency monitoring results from the EDR units 14, as shown in Figure 3A, with the three EDR units 14 on the left side of the space 12 reporting with the message on data paths 14b. This connection 45 may be a wireless network connection (Wi-Fi) or some other type of communications connection. The emergency monitoring information received by the central server 40 may include complete or partial results from the analysis of video input and/or audio input from each EDR unit 14. For example, video analysis results may include information such as: the present or absence of an object of interest (e.g.
human user) in the camera's field of view, size and movement of the object, etc. Similarly, audio analysis results may include: the presence or absence of a significant audio event of interest (e.g. speech, loud crash noise), the detected loudness of the audio event, etc.
Other important video/audio monitoring results may also be received at this step depending on the nature of the video/audio analysis performed by the EDR
units.
100911 At step 302, the monitoring results received from all EDR units 14 are compared, and the central server 40 decides which EDR unit 14 is currently in a fully "active" state. In order to do this, the central server 40, via ranking agent 50, ranks all the received emergency monitoring results according to a set of ranking criteria (e.g. digital imaging, video and/or audio analysis criteria and thresholds). These analysis criteria and thresholds may be fixed or dynamic depending on a given video or audio scenario. Such ranking criteria may include video analysis metrics such as the current size of the object being tracked, if present, by each video camera 30, and audio analysis metrics such as the current amplitude of audio or speech, if present, captured by each microphone 34.
For a given point in time, the EDR unit 14 with the highest rank may then be chosen to be the currently "active" EDR unit 14. Using these ranking criteria may ensure that the chosen active EDR unit will be the unit that is currently best suited for monitoring the human user and detecting and responding to possible emergency events. In this case, the system will continue to receive EDR data on a regular basis as the subject progresses through a particular monitoring time period, either in a single monitored location or in a plurality of monitored location. This may involve several ceiling mounted EDR
units mounted throughout the location(s) and one central controller that supervises and coordinates the monitoring operations of the EDR units.
[0092] In the interests of minimizing power consumption and extending the life of the EDR units, the central controller may be configured to employ predictive modeling to determine a predicted location of an occupant. In this case, the central controller may activate (power-on) tracking units that are located in areas that are proximal to and currently containing the occupant(s). Areas that are not in use by the occupant(s) may be kept in a standby (power-save) mode. Predictive algorithms may be employed to determine the areas the occupant is most likely to move to, activating tracking units that are located in the areas along the occupant(s)' likely path of travel. During periods of inactivity, the central controller may be used to place all units on standby except those monitoring areas of critical interest. Examples of this include monitoring only a bedroom location, while the occupant is asleep or only the entrance area if the occupant has left to his/her home.
[0093] Once the active EDR unit has been chosen, the central server 40 may notify all EDR units 14 of this decision, and/or notify only the currently active unit to continue with the full detection of and response to an emergency event, the latter case shown by path 14c in Figure 3B. This decision-making framework by the server may prevent multiple EDR units from detecting the same emergency event and confusing the user by all starting a dialog with the user simultaneously. Thus, the EDR units are operable to sense data either continuously, regularly or periodically, thus dispatching signals to the central controller continuously, regularly or periodically and then being responsive to the central controller to be selected for implementing dialog with a subject.
Therefore, the communication with the subject occurs following (and not before) selection of an active EDR, according to prevailing emergency factor data. That being said, the active status of a selected EDR at time T = T1 may not be effective at time T = T2, if the emergency factor data indicates a change, as may occur if the subject were to slip and fall in a bedroom (at T = Ti) and then stand up and stumble into a neighbouring hallway.
In this case, a successive train of EDR's may be activated and then deactivated, in succession, as the subject moves.
[0094] The active EDR unit 14 is then operable to monitor the user and detect if an emergency occurs (e.g. a fall). If such an event does occur, this information may be relayed back to the server, which may in turn instruct the active EDR unit 14 to initiate a dialog with the subject using speech recognition software to determine what level of assistance may be required (steps 304 and 307).
100951 If the active EDR unit determines that the user does require assistance, this information may be relayed to the central server 40 (step 308). The central server 40 may then initiate the appropriate communication about the detected emergency situation to the outside world (step 309). Depending on the type of assistance required the system could notify a neighbour, relative, medical staff (e.g. family doctor), emergency services (e.g.
911), or other appropriate personnel, via the external interface unit 52.
Furthermore, this communication may occur via a variety of possible communications protocols, such as landline phone, cell phone, email, text message (SMS), or some other communication system.
100961 As an optional configuration, at step 303, information about the current monitored behaviour of the user may be used to update a learned model of the user's daily "normal" behaviour and routines. This model may contain information about time of day and duration spent in certain locations of a house, etc. In this example, the EDR
unit or central server may employ artificial intelligence algorithms (e.g.
Markov Decision Process methods) to enable the system to gradually learn the expected daily behaviours of a given subject. This may allow the overall system to detect emergency situations more robustly by further characterizing "normal" versus "abnormal" behaviour for a given subject, which is done at steps 305 and 306. For example, if the subject is spending significantly more time in bed and less time in the living room than he/she normally does on a given day, this may be an indication of declining health (e.g. physical health issue such as flu, or a mental health issue such as depression). Since such a situation may not indicate an immediate emergency, the central server 40 may still tell the active EDR unit 14 to initiate a dialog with the user (step 307) that is adaptable and appropriate for the particular situation to determine if any assistance is required. For example, the system could ask the user if everything is ok because it appears they have been in the bedroom all afternoon. The active EDR unit 14 could perform these tasks by employing adaptable speech dialog technologies such as Text-To-Speech (ITS) software in conjunction with automatic speech recognition (ASR) software.

100971 As a further optional configuration, in the case where multiple EDR units 14 are used, some or all of the EDR units may have a scaled-down local processor or may even have no processor at all. In this case, such EDR units may not perform full video processing or speech/audio processing, but may send video and audio data over the communications network 45 in real-time to either be processed by central server 40 or by a nearby EDR unit that does contain a fully functional local processor 20.
This type of network configuration may be advantageous in some scenarios due to the centralization of video and audio processing for the system 10, which may make the system more affordable by reducing hardware and software costs of the EDR units.
Furthermore, EDR
units without a fully functional local processor 20 may be constructed using a smaller housing 14a, which may make them easier to install throughout a home as well as more aesthetically pleasing to the user.
[0098] In yet another optional configuration, there may be instances when the occupant will be located on or across the edge of the field of view of an EDR
unit. When possible and necessary, images from overlapping EDR units' field of views may be delivered to the central computer and be stitched together to provide complete image of the subject, for later analysis. In cases such as this, the control module 22 may not assign a value to the emergency factor according to the data received from either the video processing module or the speech dialog module, or both, as the case may be, but rather transmit the raw video data image or images to the central controller, which may then assemble an image or images from the control unit of a neighbouring EDR unit in order to assemble a full image of the combined field of view, so that the emergency factor may then be valued based on this merged data set.
[0099] For exemplary purposes, the device 10 may be operated as follows.
In the event of an adverse event, such as a fall, the units that are closest to the occupant may "view" and "listen" for input (e.g., commands and responses) from the occupant. The input received by each EDR unit (hereinafter referred to below as a ceiling mounted unit (CMU)) may then assigned a confidence factor (CF), which may indicate what the likelihood is that the input from the occupant perceived by the system matches an actual verbal input from the occupant. The CF may be influenced by aspects such as (but not limited to) CMU proximity to the occupant, ambient noise levels, and word recognition error rate (WRER, which is estimated by the speech recognition software), and may be calculated using a function similar to the one described in Equation I:
CF = 1/[131(proximity) + 132(ambient noise level) + 133(1-WRER)] (1) where 131, 132, and 133 are constants determined through a combination of within-lab experimentation and calibration once the system is installed in an environment. Post-installation calibration may be necessary, as the layout of the environment may influence the acoustics, to determine each CMUs' "hearing" range.
[00100] The CMU that rates the occupant's response with the highest CF may then dictate action by the central control unit. This unit may not necessarily be the CMU
closest to the occupant (as demonstrated in the scenarios below).
[00101] A situation may occur where an occupant has an adverse event in a visual monitoring area covered by two or more CMUs. As the images from the units may be stitched together, an entire image of the occupant may be available to the system.
[00102] The system may also be designed so that the occupant may activate the system using a keyword (e.g., "Heir), to enable the occupant to procure assistance regardless of whether s/he is visible by the system. In this instance, the system may determine the occupant's most likely responses using verbal input only. The unit that has the highest CF
may be the one relaying responses, although since the proximity will be unknown, Equation 1 may be altered to reflect this (such as a '0' value for proximity).
[00103] Reference will now be made to the following non-limiting example, in which some of the above-proposed approaches to signal classification are applied to the classification of physiological signals, in accordance with exemplary embodiments of the invention.
EXAMPLE
1001041 The following scenarios are presented for illustrative purposes only.
Consider that a subject experiences an adverse event near a source of sound interference, such as a television, as depicted at 13 in Figure 3A. For the sake of simplicity, assume the distance to CMU #1 is 3 meters (d1=3) and to CMU #2 is 8 meters (d2=8) and (notionally for the purposes of this example), that proximity has the least impact on a CMU's ability to "hear" verbal input (131=0.2), ambient noise has the most impact (132=0.6), and WRC is somewhere in between (133=0.3). Assume that the constants 132 and 133 are the same for both units. Note that as distance is in meters, ambient noise is an exponential measure (dB), and the WRER is expressed as a percentage, the constants 131, 132, and 133 for a particular application may be established through empirical testing. The actual values for 131, 132, and 133 will depend on the specific configuration of the device 10, the physical to characteristics of the location being monitored, as well as the general health of the subject. For instance, for an application in which a subject is only able to whisper, the constant 132 may be set to a higher value, when compared with the same location with a subject who is able to speak at a normal speaking volume.
SCENARIO 1: TELEVISION OFF
100105] If the television is off when the same adverse event occurs, there may be no dominant competing noise to interfere with communication. Assuming the room is quiet (25dB) and the WRER is 5%, or 0.05, the CF for the two CMUs may be calculated as:
CFI = 140.2(3) + 0.6(25) + 0.3(1-0.05)] 0.060 CF2 = 1/[0.2(8) + 0.6(25) + 0.3(1-0.05)] 0.057 1001061 As the CF for CMU #1 is higher than the one for CMU #2 (i.e., CFI >
CF2), the input received by CMU #1 will be considered to be more reliable and therefore used by the central control unit to make decisions.
SCENARIO 2: TELEVISION ON
1001071 Assume that the television is on at a volume of 70dB when the adverse event occurs, creating a source of interference. As CMU #2 is further from the occupant than CMU #1, it is also further from the competing noise source and thus has less competing ambient noise, for example, 50dB and 65dB respectively. Moreover, while the television and the occupant are approximately equidistant from CMU #1, the occupant is closer to CMU #2 than the television is. The CF for the two CMUs might be:
CFI = 1/[0.2(3) + 0.6(65) + 0.3(1-0.05)1 0.025 CF2 1/[0.2(8) + 0.6(50) + 0.3(1-0.05)] 0.031 100108] In this case, the CF for CMU #2 is higher than the CF for CMU #1 (i.e., CF1 < CF2). Thus the verbal input received by CMU #2 may then be used by the central control unit to determine the appropriate action(s) to take.
1001091 Thus, an exemplary embodiment operates through one or more CMU's and a central control unit, where the CMU's are mounted to the ceiling, resulting in improved monitoring coverage while ensuring that the EDR units are unobtrusive and discrete.
Each CMU includes a vision sensor, in this case as a non-recording video camera. The CMU also includes a microphone, one or more speakers, a processor, and a smoke alarm.
Multiple CMR's may thus be used and networked together to ensure that designated areas of a monitored facility, such as a subject's home, may be monitored. One central controller may thus be installed in each home and be responsible for coordinating the CMU's and relaying communications to the outside world.
[00110] In one example, the system 10 may use computer vision techniques to track a subject as the subject moves about the subject's home and detects if there is an acute emergency event, such as a fall or a fire. In this case, the vision sensor does not record, transmit, or store collected images. The system does not provide continuous data about the subject's status and activities to an external source, thus preserving the subject's privacy. When an acute event is detected, one or more CMU's sensing the event dispatch event data to the central controller which then selects either the closest CMU, or another CMU with improved sensing results for the event, to be an active CMU to employ speech recognition algorithms to have a dialogue with the subject in distress to determine if and what type of assistance is required. In this case, 911 and the monitoring provider's live switchboard may be made available as needed, as well as one or more respondents that were predefined by the subject (e.g. a neighbour, a family member, and/or a friend). By verbally answering a short series of simple "yes"/"no" questions, or other simple verbal phrases, the subject may select which respondent s/he would like to contact.
[00111] To further improve the subject's safety, if the system 10 does not hear a response from the user during an acute event, the system may connect to a live operator at the provider's switchboard to assess the situation and provide appropriate assistance.
Through the selection of the type of aid s/he would like to receive, the subject may thus remain fully in control of the situation and his/her health, promoting feelings the dignity, autonomy, and independence without compromising safety.
[00112] Thus, the system 10 provides, in one example, an automated emergency detection and response system that uses computer vision to detect an emergency event, such as a fall, and speech recognition and artificial intelligence to then determine the level of assistance that is required by an individual. These devices may be networked together with a central server 40 to provide more comprehensive monitoring throughout an entire living environment, without the necessity of having an older adult wear an alert device or call button.
[00113] While the present disclosure describes various exemplary embodiments, the disclosure is not so limited. To the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims (51)

CLAIMS:

1. A system for detecting and responding to an emergency event occurring in or near a predetermined area, the system comprising:
two or more detection modules each adapted for operative coupling to one or more sensors distinctly disposed in or near the area to detect the event, and each configured to generate a respective data set representative of the detected event; and a controller operatively coupled to said detection modules and configured to compare each said respective data set and implement a response protocol as a result thereof as a function of at least one said respective data set.

2. The system of claim 1, said controller comprising a merging module configured to merge each said data set based on an overlap automatically identified therebetween to generate a merged data set, and configured to implement at least part of the response protocol as a function of said merged data set.

3. The system of claim 2, wherein each said data set comprises imaging data captured via respective imaging sensors, said merging module configured to merge said imaging data into a combined image thereby improving overall imaging data representative of the event.

4. The system of claim 3, wherein each said data set comprises complimentary partial imaging data of a user in or near the area.

5. The system of claim 3 or claim 4, said controller further configured to process said overall imaging data in automatically identifying a position of a user in or near the area.

6. The system of any one of claims 3 to 5, said controller further configured to process said overall imaging data in automatically identifying a location of a user in or near the area.

7. The system of any one of claims 1 to 6, said controller comprising a ranking module configured to rank each said data set, and configured to implement at least part of the response protocol as a function of a highest ranking data set.

8. The system of claim 7, wherein said ranking module is configured to automatically perform a reliability measure of each said data set, and rank each said data set at least in part as a function of said reliability measure.

9. The system of any one of claims 1 to 8, further comprising a response module adapted for operative coupling to a prompting device and an input device, said controller further configured to prompt a user in or near the area and record a user response therefrom via said response module, and implement at least part of the response protocol as a function of said user response.

10. The system of claim 7 or claim 8, each of said detection modules having a respective response module associated therewith, each said response module having a prompting device and an input device operatively associated therewith, said controller further configured to set one said response module as active as a function of said highest ranking data set, prompt a user in or near the area and record a user response therefrom via said active response module, and implement at least part of the response protocol as a function of said user response.

11. The system of claim 9 or claim 10, wherein said controller is further configured to operate one of said one or more sensors as said input device.

12. The system of any one of claims 9 to 11, wherein said prompting device comprises a speaker and wherein said input device comprises a microphone or microphone array.

13. The system of anyone one of claims 9 to 12, wherein said prompting device comprises a visual display.

14. The system of any one of claims 7 to 13, wherein said ranking is at least in part calculated as a function of a respective distance of a user from each said one or more sensors.

15. The system of any one of claims 7 to 14, wherein at least one of said one or more sensors comprises an audio sensor for generating an audio signal representative of sounds in or near the area, the system further comprising a speech recognition module for automatically processing said audio signal in detecting one or more preset speech elements representative of the event, wherein said ranking is at least in part calculated as a function of a speech recognition reliability measure output from said speech recognition module.

16. The system of claim 15, wherein said ranking is at least in part calculated as a function of background noise captured by said audio sensor.

17. The system of any one of claims 1 to 16, the system further comprising each said one or more sensors.

18. The system of any one of claims 1 to 17, said one or more sensors comprising at least one of a video sensor, a range sensor, a multi-spectral sensor, a heat sensors, a microphone, a microphone array, a pressure sensor, a motion sensor and an infrared sensor.

19. The system of any one of claims 1 to 18, each said respective data set representative of at least one emergency factor selected from at least one of a plurality of video variables and/or thresholds, a plurality of audio variables and/or thresholds, and a plurality of environmental variables and/or thresholds.

20. The system of any one of claims 1 to 19, said controller comprising a central controller.

21. The system of any one of claims 1 to 20, further comprising an external communication interface for communicating the event to an external respondent, said communication interface comprising at least one of a landline voice interface, a cellular voice interface, a cellular data interface, an e-mail interface and a text messaging interface.

22. The system of claim 10, each of said detection modules and said response module respectively associated therewith forming a combined emergency detection and response unit disposed in or near the area and comprising said one or more sensors, said input device, said output device, a processor for generating said respective data set and a communication interface for communicating same to said controller and for receiving instructions therefrom to implement said response protocol.

23. The system of any one of claims 1 to 21, each of said detection modules and said controller being centrally operated to interface remotely with each said one or more sensors.

24. The system of any one of claims 1 to 21, each of said detection modules and said controller being centrally operated to interface remotely with each said one or more sensors.

25. The system of any one of claims 9 to 13, each of said detection modules, response modules and said controller being centrally operated to interface remotely with each said one or more sensors, input device and prompting device.

26. The system of claim 1, said controller operatively configured to compare each said respective data set to select one of said detection modules as active, and implement a response protocol as a function of said respective data set of said active detection module.

27. A method for responding to an emergency event occurring in or near a predetermined area, the system comprising:
monitoring the area with two or more sensors distinctly disposed in or near the area;
generating respective data sets representative of activity within the area via said two or more sensors;
comparing said respective data sets;
automatically detecting the event from at least one of said respective data sets;
selecting a response protocol as a function of said comparison and at least one of said respective data sets; and implementing said selected response protocol.

28. The method of claim 27, said comparing step comprising automatically identifying overlap between said respective data sets and merging said data sets as a function of said overlap; said selecting step comprising selecting a response protocol as a function of merged data set.

29. The method of claim 28, said respective data sets comprising imaging data, said merging step comprising merging said imaging data into a combined image.

30. The method of claim 29, wherein said merging step is automatically implemented via a computer vision technique.

31. The method of claim 29, wherein said merging step comprises automatically aggregating partial user imaging data.

32. The method of any one of claims 29 to 31, further comprising automatically identifying a position of a user in or near the area from said combined image.

33. The method of any one of claims 29 to 32, further comprising automatically identifying a location of a user in or near the area from said combined image.

34. The method of any one of claims 27 to 33, said comparing step comprising ranking each of said respective data sets, said selecting step comprising selecting a response protocol as a function of a highest ranking data set.

35. The method of claim 34, said ranking step comprising automatically measuring a reliability of each of said data sets, and ranking each of said data sets at least in part as a function of said reliability measure.

36. The method of any one of claims 27 to 35, further comprising the steps of prompting a user in or near the area; and recording a user response therefrom;
said selecting step comprising selecting said response protocol at least in part as a function of said user response.

37. The method of any one of claims 27 to 35, said implementing step comprising the steps of prompting a user in or near the area; and recording a user response therefrom;
said response protocol being implemented at least in part as a function of said user response.

38. The method of claim 34 or claim 35, further comprising the steps of:
selecting an input device associated with said highest ranking data set;
prompting a user in or near the area; and recording a user response via said selected input device;
said response protocol being selected and/or implemented at least in part as a function of said user response.

39. The method of claim 38, wherein said selected input device is selected from said two or more sensors.

40. The method of any one of claims 37 to 39, wherein said prompting step comprises visually prompting the user.

41. The method of any one of claims 34, 35 and 38 to 40, said ranking step comprising automatically calculating a respective distance of a user from said two or more sensors, and ranking said respective data sets as a function of said respective distance.

42. The method of any one of claims 34, 35 and 38 to 41, wherein said two or more sensors comprise audio sensors for generating an audio signal representative of sounds in or near the area, said generating step comprising processing said audio signal via an automated speech recognition process, said detecting step comprising detecting one or more preset speech elements from said processed audio signal representative of the event, said ranking step comprising calculating a speech recognition reliability measure for each of said respective data sets, and ranking same at least in part as a function of said speech recognition reliability measure.

43. The method of claim 42, wherein said ranking step comprises ranking said respective data sets at least in part as a function of background noise respectively captured by said audio sensors.

44. The method of any one of claims 27 to 43, further comprising the step of dispatching an emergency message to an external respondent.

45. The method of claim 26, further comprising selecting one of said sensors as active based on results of said comparing step, said selecting step comprising selecting a response protocol as a function of an active sensor data set.

46. One or more computer-readable media having statements and instructions embodied therein to implement, upon execution by one or more processors, the method of any one of claims 27 to 45.

47. A system for detecting and responding to emergency events occurring in a predetermined local area, comprising a plurality of local emergency detection and response units to be positioned in the local area, each local emergency detection and response unit including one or more local sensing agents and a local detection manager, the local detection manager including a local processor, each local sensing agent being operable to detect a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor, the detection manager being operable to receive the data and to assign a value to the emergency factor according to the data, and a local interface agent for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor, a central location controller unit including a central processor and a central interface agent in communication with the local interface agent on the data path to receive the event status messages therefrom, each local emergency detection and response unit including a local emergency event response agent for responding to at least one person in or near the local area, the central location controller unit and/or the local emergency detection and response unit being operable for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification, the response protocol including dispatching a message to the person and recording a response therefrom, the central processor including a ranking agent for ranking status signals being received from more than one local emergency detection and response unit in the same local area.

48. A system for detecting and responding to emergency events occurring in a predetermined local area, comprising a plurality of local emergency detection and response units positioned in the local area, each local emergency detection and response unit including one or more local sensing agents and a local detection manager, the local detection manager including a local processor, each local sensing agent being operable to detect emergency events by a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor, the detection manager being operable to receive the data and to assign a value to the emergency factor according to the data, and a local interface agent for issuing, on data path, one or more event status messages including the assigned value for the emergency factor, a central location controller unit including a central processor and a central interface agent in communication with the local interface agent on the data path to receive the event status messages therefrom, each local emergency detection and response unit including a local emergency event response agent for responding to at least one person in or near the predetermined area, the central location controller unit and/or the local emergency detection and response unit being operable for classifying the assigned value of the emergency function to form an assigned value classification and for initiating the local emergency event response agent to implement a response protocol according to the assigned value classification.

49. A system for detecting and responding to emergency events occurring in a predetermined local area, comprising a plurality of local emergency detection and response units to be positioned in the local area, each local emergency detection and response unit including one or more local sensing agents and a local detection manager, the local detection manager including a local processor, each local sensing agent being operable to detect a change in a given emergency factor in the local area and to convey data representative of the change in the emergency factor, the detection manager being operable to receive the data and to assign a value to the emergency factor according to the data, and a local interface agent for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor, each local emergency detection and response unit including a local emergency event response agent for responding to at least one person in or near the local area, the local emergency detection and response unit being operable for classifying the assigned value of the emergency function to form an assigned value classification and initiating the local emergency event response agent to implement a response protocol according to the assigned value classification, the response protocol including dispatching a message to the person and recording a response therefrom, one or more of the local detection and response units responsive to data received on the data path for ranking status signals received from more than one local emergency detection and response unit in the same local area, one or more of the local emergency detection and response units being operable to select one of the emergency detection and response units according to the ranking as an active local emergency detection and response unit to initiate the emergency event response protocol.

50. A method for detecting and responding to emergency events occurring in a predetermined local area, comprising:
providing a local emergency detection and response unit in the local area, providing the local emergency detection and response unit with one or more local sensing agents and a local detection manager with a local processor, configuring the local sensing agents to detect an emergency event by a change in a corresponding emergency factor in the local area, and to generate data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the first data, providing a local interface agent for issuing, on a data path, one or more event status signals including the assigned value for the emergency factor, providing a local emergency event response agent for responding to at least one person in or near the local area, and configuring the local interface agent to receive an initiation signal for initiating the local emergency event response agent to implement a response protocol according to an assigned value.

51. A method for detecting and responding to emergency events occurring in a predetermined local area, comprising:
providing a plurality of local emergency detection and response sites in the local area;
providing each local emergency detection and response site with one or more local sensing agents;
configuring the sensing agents to sense a change in a predetermined emergency factor in the local area;
providing a detection manager at each site to receive, from each sensing local sensing agent, data representative of the change in the emergency factor, arranging the detection manager to assign a value to the emergency factor according to the data;
dispatching, over a data path, one or more event status messages including the assigned value for the emergency factor;
providing a central location controller to receive, in a first operative phase, the event status messages over the data path; and configuring the central location controller, in a second operative phase, to:
classify the assigned value of the emergency factor to form an assigned value classification;
rank the status messages according to one or more ranking criteria;
select one of the emergency detection and response sites according to the ranking as an active local emergency detection and response site to initiate an emergency event response protocol; and instruct the selected active local emergency detection and response site to initiate the emergency event response protocol.