WO2017001848A1

WO2017001848A1 - A sensing system for detecting the behavior of a user

Info

Publication number: WO2017001848A1
Application number: PCT/GB2016/051952
Authority: WO
Inventors: Robert David MERRIFIELD; Benny Ping Lai Lo; Guang-Zhong Yang
Original assignee: Sensixia Limited
Priority date: 2015-06-30
Filing date: 2016-06-29
Publication date: 2017-01-05
Also published as: GB201511483D0

Abstract

A system for detecting a behavior of a user, the system comprising at least one wearable sensing device and a mobile device wherein the at least one sensing device is arranged to communicate with the mobile device.

Description

A SENSING SYSTEM FOR DETECTING THE BEHAVIOR OF A

USER

The present disclosure relates to a system for detecting challenging behaviors, enabling a customizable learning experience, and providing context-aware, intelligent daily assistance for people with learning disabilities.

In addition to general health problems, people with learning disabilities are known to have higher incidents of dementia, respiratory diseases, gastrointestinal cancer, ADHD/hyperkinesis and conduct disorders, epilepsy, physical and sensory impairments, dysphagia, poor oral health, and tend to be prone to injuries, accidents and falls. Often due to the lack of expressive skills, people with learning disabilities are more likely to have undiagnosed long-term conditions and which leads to high risk of premature death. With the aim of improving the care of people with learning disabilities, a new wearable sensing system is provided which comprises a new miniaturized wearable sensor, for example that may be worn either as a wrist worn or ear worn sensor, and a seamlessly integrated mobile app that adapts to individual care needs.

Aspects of the present disclosure are set forth in the accompanying independent claims. Optional features of embodiments are set out in the dependent claims. Optional features of embodiments of the disclosure are also set out in the following clauses:

Clause 1. A system, which comprises of one or more wearable sensing devices and a mobile device/computer, for providing a personalized learning experience, and/or assisting/monitoring the daily activities of a user with learning disabilities. For example, the user may be a child. Clause 2. A system, according to clause 1, wherein the wearable sensing devices can be worn on and attached to any part of the body and consist of a combination of different sensors, which includes motion sensors (e.g. accelerometers, gyroscopes, etc.), physiological sensors (e.g. heart rate, galvanic skin response, etc.), position sensors (e.g. GPS, altitude), and/or ambient sensors (e.g. temperature, humidity). The devices have wireless connectivity, memory storage (for storing the sensing data), and computing capabilities for detecting the relevant movement, behaviors, and activities. The sensing information will be transmitted and displayed via the mobile device/computer.

Clause 3. A system, according to clause 1 or 2, wherein the wearable devices can capture the normal and challenging behaviors of people with learning disabilities, and the behaviors profiles of the user can be derived to identify precursors of challenging behaviors. Such information will be provided to the teachers/carers/parents/therapists, and which enable early intervention.

Clause 4. A system, according to any of clauses 1-3, wherein the wearable sensor can capture the gait and activity patterns of the user, based on the acceleration patterns. From which the information will be used in the software application to encourage active and healthy lifestyle.

Clause 5. A system, according to any preceding clause, wherein the software application on the smartphone/computer allows health professionals, teachers, carers, and parents to create their own health education/monitoring Apps.

Clause 6. A system, according to any preceding clause, the software uses the built-in and attached camera of the computer/ smartphone to capture a drawing or photo. A visual timetable, a matching game, a sorting task or an information arrangement game can be created by dragging and dropping the visual objects in the captured image. Clause 7. A system, according to any preceding clause, wherein the software is able to analyse the sensor data to determine how closely a visual timetable is followed (e.g. based on the sensor data/activities detected in Clause 2 and 3). Clause 8. A system, according to any preceding clause, can interface with other location sensing devices, such as GPS and iBeacon (Bluetooth low energy (BLE) beacons), to identify the location of the user.

Clause 9. A system, according to clause 8, can use the location information to capture challenging behaviors, such as wondering around the corridor at night.

Clause 10. A system, according to Clause 8 or 9, can use the location information to create interactive games for help the people with learning difficulties (LD) to learn and carry out daily activities. For example, ordinary objects, such as fridge can be tagged with iBeacon and the interaction with the fridge can be used as a point in the game.

Specific embodiments are described below by way of example only and with reference to the accompanying drawings in which:

Figure 1 is a schematic diagram of the system;

Figure 2 illustrates the wearable sensor device design;

Figure 3 shows how the user can draw visual assets for the app using pen and paper;

Figure 4 illustrates how a user can use the camera of the smartphone/mobile device to take a photo of the drawing of the visual assets;

Figure 5 depicts how a user can add logic to the app by cutting out (by drawing a rectangle around the visual asset) and drag-and-drop the captured visual assets (i.e. the photo of the drawing) to form a puzzle game;

Figure 6 demonstrates how a user can test and use the app, where the cut out visual assets can be dragged-and-dropped to void spaces, similar to a puzzle game;

Figure 7 illustrate some example health Apps created. System Design

Devices for people with learning disabilities need to be robust, lightweight, easy to use, and resilient against potential misuse. The new sensing device 2 is designed based on Bluetooth Low Energy (BLE) and integrated with inertial, physiological, ambient, and position sensors. With BLE, the new device can seamlessly be integrated with mobile devices. Figure 1 depicts the design of the system.

As shown in Figure 1, the miniaturised sensing platform can be packaged as a wrist-worn or ear-worn sensor 2 for the user (e.g. a person with a learning disability. For example, the person may be a child.). The sensor will be integrated with a smartphone/mobile device 3 via the wireless (BLE) link 6. Apart from acting as a data aggregator, the smartphone/mobile device 3 will also work as a gateway to forward the information gathered to the cloud server 4 for behavioural profiling and long term trend analysis. The carer/parent/teacher/therapist, on the other hand, can use the associated mobile app 5 to monitor the progress of the user and introduce events/instructions/games to teach and help the user to learn to carry out daily tasks. In addition, by using the BLE link 6, the wearable device will be able to integrated with ambient sensors 1 and enable indoor localisation, as shown in Figure 1.

Wearable Sensing Device

Figure 2 illustrates the main components of the wearable device 2.

As shown in Figure 2, the device 2 mainly consists of the following:

• An ultra-low power microcontroller 7

• Wireless (BLE) transceiver 8

• A rechargeable battery 9

• A battery charging circuit 10

o A battery charging circuit 10 will be designed in the sensor node to protect the battery 9 and enable safe battery charging. User can charge up the battery 9 by plugging it onto a conventional 5V charger.

• LEDs /vibration motor 11

o To provide feedback to the user, the device 2 is embedded with a few bright LEDs and/or a vibration motor.

• Touch sensor control 12

o A touch sensor will be incorporated in the sensor node as the control interface; as such, the user can control the device 2 by simply touching it.

• Sensors and Interface 13

o To detect challenging behaviors, different sensors, such as inertial sensors and temperature sensors, may be required, and various interfaces are incorporated in the device 2 to facilitate sensor integration.

o Different sensors, such as motion sensors (inertial sensors), physiological sensors (heart rate), ambient (temperature), positional (GPS) sensors, etc., can be integrated into the device. By default, each device 2 consists of an inertial measurement unit, a humidity sensor, and a skin temperature sensor.

• On-node memory 14

o A low power flash memory is integrated into the device 2 for data storage.

Customisable Health /Education Application

A software application (e.g. a mobile App) is designed and created to allow health professionals, teachers, carers, and parents to create their own health education/monitoring Apps. The programme operates with simple drag-and- drop actions. Within a 3-20 minute timeframe, a custom app can be created to help a person with a particular health need. The following steps describe the process of creating the custom app:

Step (1) - Draw visual assets for the app using pen and paper (Figure 3) The software application allows the user to create their own healthcare app using pen and paper. Apps can take the form of a visual timetable (as used widely in special needs schools), a matching game, a sorting task or an information arrangement game. Example apps are provided on the home screen to give the user an understanding of the types of apps that can be created. In the case of a tooth-brushing app, the user can draw a puzzle that teaches the key aspects of dental hygiene.

Step (2) - Take a photo of the assets using the in-software viewfinder (Figure 4)

The software features a virtual viewfinder with which the user can align the paper-based design and take a photo to import the assets into the software. Assets are imported at HD resolution to take advantage of the high resolution screens on modern tablet computers.

Step (3) - Add logic to the app to define user interaction (Figure 5)

Step (4) - Test or use the app (Figure 6) The user adds logic to their application by delineating rectangular tiles on the visual assets and repositioning them to new locations. The software stores the original and final location of each tile enabling the software to determine whether it has been moved to the correct location.

Tapping on the play button allows the user to test their app. The app enters a full-screen mode to reduce visual distraction. The user must rearrange the tiles to their correct positions. For example, in the hand-washing app, the user must put the steps involved with hand-washing in the correct order. Placing tiles in the incorrect order/position causes the tile position to be reset.

Step (5) - Connect the app to wearable and ambient sensors The wearable sensor devices can be connected to each tile of the app. This can be used for creating interactive visual timetables of health- related activities. For example, the tooth-brushing sensor event can be attached to tooth-brushing visual asset in a morning routine app. Step (6) - Provide monitoring of health activities

The software is able to analyse the sensor data to determine how closely a visual timetable is followed. In the case of the tooth-brushing app it is possible to determine the duration of brushing and how suitable a brushing action was used. Example Health Apps

In addition to allowing users to create their custom applications, a library of example health-related apps has been created in the system (Figure 7). These include a tooth-brushing app to teach the theory of dental hygiene and to test the person's performance in practice, a hand-washing app to teach the person the order in which different aspects of hand-washing should be performed, an obesity awareness app that allows the person to identify healthy and unhealthy foods, a morning routine app that encourages the person to wash hands before eating, to eat a healthy breakfast and to brush teeth after eating and an exercise app that tracks the level of activity of the person. For example, the person may be a child.

Locational aware interactive learning

With the BLE transceiver 8 and interface with the mobile device/smartphone/computer 3, the wearable device 2 can detect the iBeacon 1 signals (i.e. other wireless transceiver in proximity) and GPS signals to identify the location of the user. With the location information, interactive learning tools and games can be developed using the software application. For instance, from the location information, the sensor can detect if the user has been to the bathroom in the morning to brush his/her teeth, or if the person is wondering around the corridor at night. Such information can be fed to the carers/teachers/parents/therapists for them to design targeted training games/exercises app using the software application (mobile app). Furthermore, the location information can also be used to help the person navigate. For instance, the wearable device 2 will show a message or vibrate if the person should get off the bus in the next bus stop, when he/she is travelling to school. With the integration with iBeacon 1 and other ambient sensors 1, the system will enable the development of interactive games for the people with learning disabilities to learn or remind them to carry out daily activities. In other words, the living environment can be converted into a gaming environment. Every object (such as washing machines, etc.,) can be tagged, and the interaction with the objects can be captured and the interaction can be considered as a scoring point in a game.

It will be understood that the above description is of specific embodiments by way of example only and that many modifications and alterations will be within the skilled person's reach and are intended to be covered by the scope of the appendent claims.

Claims

CLAIMS:

1. A system for detecting a behavior of a user, the system comprising at least one wearable sensing device and a mobile device wherein the at least one sensing device is arranged to communicate with the mobile device.

2. A system according to claim 1, wherein the at least one wearable sensing device can be worn on and/ or attached to any part of the user's body.

3. A system according to either of claim 1 or 2, wherein the at least one sensing device comprises at least one of: a motion sensor for example an accelerometer and/or gyroscope, a physiological sensor for example a heart rate and/or galvanic skin response sensor, a position sensor for example a GPS and/or an altitude sensor, and/or an ambient sensor for example a temperature and/or a humidity sensor.

4. A system according to any preceding claim, where the at least one sensing device and the mobile device communicate via a wireless communication means.

5. A system according to any preceding claim, wherein the system further comprises at least one location sensing device arranged to communicate with the mobile device wherein the at least one location sensing device is arranged to determine the location of the user.

6. A system according to claim 5, wherein the at least one location sensing device is a GPS and/or Bluetooth beacon.

7. A system according to any preceding claim, wherein the mobile device comprises a processor for determining a behavior of a user based on information received from the at least one sensing device and/or the at least one or a location sensing device.

8. A system according to any preceding claim, wherein the mobile device is arranged to communicate with an external computing device.

9. A system according to claim 8, wherein the external computing device is arranged to determine a behavior of a user.

10. A system according to claim 7 or 9, wherein a behavior is detected by comparing information with known sensing patterns indicative of the behavior, wherein the information is received from the at least one sensing device and/or the at least one or a location sensing device.

11. A system according to any preceding claim, wherein the mobile device comprises memory storage for storing the information is received from the at least one sensing device and/or the at least one or a location sensing device.

12. A system according to any preceding claim, wherein the mobile device comprises a display unit for displaying information relating to the information received from the at least one sensing device and/or the at least one or a location sensing device and/or the or a behavior detected.

13. A system according to any preceding claim, wherein information received from the at least one sensing device and/or the at least one or a location sensing device may be used to identify precursors of a given behavior.

14. A system according to any preceding claim, where the at least one

wearable sensor comprises a gait sensor arranged to capture the gait of the user.

15. A system according to claim 14, wherein the gait sensor comprises an accelerometer.

16. A system according to any preceding claim, wherein the mobile device comprises functionality to enable a user to create a health education and/or monitoring application which may be run on the mobile device.

17. A system according to claims 16, wherein the application may comprise a visual timetable, a game, a sorting task and/or an information arrangement game.

18. A system according to claim 16 or 17, wherein the mobile device comprises a camera and wherein information obtained from the camera is used as an input to create the application.

19. A system according to any of claims 16-18, wherein information obtained from the at least one sensing device and/or the at least one or a location sensing device is used as an input to the application.

20. A system according to claim 19, wherein the application is arranged to display information on the mobile device dependent on the information received from the at least one sensing device and/or the at least one or a location sensing device.

21. A computer program product comprising coded instructions which, when run on a processor, implement a system according to any preceding claim.

22. A tangible computer readable medium comprising a computer program product according to claim 21.