WO2015116126A1

WO2015116126A1 - Notifying users of mobile devices

Info

Publication number: WO2015116126A1
Application number: PCT/US2014/013991
Authority: WO
Inventors: Akihiko Ikeda
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2014-01-31
Filing date: 2014-01-31
Publication date: 2015-08-06

Abstract

The present disclosure describes a system for notifying a user of a mobile device. The system comprises a distance sensor to measure a distance from the system to an object and an orientation sensor associated with the distance sensor to measure an orientation of the distance sensor and to measure a direction of movement of the system. The system further comprises a processor to determine based on the distance, based on the orientation and based on the direction of movement that the system is about to cross a threshold distance from the object. The system also comprises a notification system to generate a notification based on the determination of the processor.

Description

NOTIFYING USERS OF MOBILE DEVICES

[01] Many users of mobile devices, such as phones, e- readers and tablet computers operate their devices while walking. They read e-books, browse the internet, read their emails and write SMS messages.

Brief Description of the Drawings

[02] By way of non-limiting examples, systems, mobile communication devices and methods for notifying users according to the present disclosure will be described with reference to the following drawings in which

[03] Figure 1 is a block diagram of a system, for

notifying a user according to an example implementation,

[04] Figure 2 illustrates a method for notifying a user- according to an example implementation.

[051 Figure 3 is a plan view of an example scenario.

[06] Figure 4 is a three-dimensional depth map according to an example implementation.

[07] Figure 5 is a cutaway plan view of a mobile

communication device according to an example

implementation .

Detailed Description

[08] The present disclosure describes a system for notifying a user of a mobile device. The system comprises a distance sensor which measures a distance from the system to an object. A direction sensor is associated with the distance sensor and measures an orientation of the distance sensor. The direction sensor also measures a direction of movement of the system. The system, further comprises a processor. The processor determines based on the distance, based on the orientation and based on the direction of movement that the system is about to cross a threshold distance from the object. The system also comprises a notification system. The notification system, generates a notification based, on. the determination of the processor .

[09] Users of mobile communication devices operate their devices while they are unav/are of the dangers around them. As a result, some of these users fall down stairs, walk in front of cars or bump into other people

[10] Since the system measures the orientation of the distance sensor, such as towards the ground, in front or towards the sky, the processor can compensate for

different orientations and the determination that the system is about to cross a threshold distance from the object is more accurate than without considering the orientation .

[11] For example, initially the orientation of the system is such that the distance sensor is facing in front of the user and there are no obstacles within 5 meters. Then the user changes the orientation of the system such. that, the distance sensor faces to the ground. As a result, the system may detect the ground, itself as an object because the distance to the ground is within 1 meter. However, since the system measures the orientation of the distance sensor, the ground can be disregarded as a relevant obj ect . [12] Users like to change the orientation of their devices often v/hile operating them and therefore,

restraining the orientation of the device would reduce the enjoyment, for the user. However, the disclosed device measures the orientation of the distance sensor and can therefore disregard the ground as a relevant object. As a result, the user of the mobile communication device can change the orientation often while the device can still warn the user effectively and. prevent the user from causing the above mentioned accidents.

[13] Figure 1 illustrates a computer system 100 for notifying a user of a mobile device. The computer system 100 comprises a processor 102 connected to a program memory 104, a data memory 106, a communication port 108 and a user port 110. The program memory 104 is a non- transitory computer readable medium., such as a hard drive, a solid state disk, flash, read only memory (ROM) , or compact disk (CD) ROM. Software, that is, an executable program stored on program memory 104 causes the processor 102 to execute the program,

[14] Data memory 106 may comprise volatile or nonvolatile memory, such as a hard drive, a solid state disk, CD-ROM, random access memory (RAM) , dynamic random access memory (DRAM) or flash ROM. In order to execute machine readable instructions being stored on program memory 104, processor 102 may load program code into volatile memory, such as RAM, and. execute the program code from there.

[15] The user port 110 is connected to a. display 112, which displays a user interface 114 to a user 116. User 116 interacts with the user interface 114 while walking. For example, user 116 browses the Internet using user- interface 114. Processor 102 sends and receives data from a web-server over communication port. 108 using a third generation (3G) cellular communication network, Long Terra Evolution (LTE) , Wifi or other network protocols. In another exaraple, user 116 uses mobi1e communi cation device 100 as a telephone, such that the transmitted data is voice data.

[16] Processor 102 is further connected to a distance sensor 120 and a direction sensor 122. Distance sensor 120 senses a distance 124 of an object 126 from the device 100.

[17] The object 126 may be anything- that the user could perceive or see if the user was not distracted by using the mobile device and includes a step, an obstacle on the road, a moving object, such .as a car, a hole in the ground, a light pole or any other object.

[18] In one example, the distance sensor 120 is an optical sensor and comprises a video graphics array (VGA) camera with three channels for red, green and blue (RGB) , respectively, with a resolution of 640x480 pixels and a frame rate of 30 frames per second. The distance sensor 120 may further comprise a depth sensor having an infrared projector and a monochrome complementary metal-oxide semiconductor (CMOS) infrared sensor. Firmware integrated into the distance sensor 120 may fuse the data from the camera and the infrared sensor to identify objects. The distance sensor 120 may be interfaced using a proprietary or open source software development kit (SDK) , which allows direct computation of a depth map. The SDK also provides an application programming interface (API) to the firmware of the distance sensor 120.

[19] The direction sensor 122 may be an inertial sensor, such as accelerometer or gyroscope, that is integrated together with processor 102 into a mobile phone, such as a smart phone, a tablet computer or other mobile device.

The direction sensor 122 may also comprise an inertial component to measure the orientation of the distance sensor 120 and a Global Positioning System (GPS) component, to measure the direction and speed of movement of the device 100. The direction sensor 122 may also comprise a compass to determine the orientation of the distance sensor 120 with reference to the earth magnetic field.

[20] In one example, an operating system is installed, on. program memory 104 and executed by processor 102,

Processor 102 accesses a SensorEvent.valu.es [0] attribute of the API to receive the sensor data for a particular sensor .

[21] The direction sensor 122 is associated, with the distance sensor 120 such that, movement of the direction sensor 122 also causes movement of the distance sensor 120. For example, the distance sensor 120 may be

externally attached to the device 100 by inserting a plug into the appropriate socket of device 100 and the distance sensor 120 is directly attached to the plug without the use of a cable. As a result, the distance sensor 120 is rigidly fastened to device 100. The direction sensor 122 is integrated into the device 100 and therefore, device 100 and direction sensor 120 move together, which allows direction sensor 122 to sense the orientation of the distance sensor 120,

[22] In one example, the processor 102 receives and processes the sensing data, in real time, such as every 100 ms . The processor 102 determines that the device 100 is about to cross a threshold distance from the object every time sensing data is received from the sensors and completes this calculation before the sensors send, the next sensing data update. [23] Figure 2 illustrates a method 200 as performed by- processor 102 for notifying a user using a mobile device, such as device 100. The method commences by the processor 102 receiving 202 sensing d.ata. In one example, processor 102 receives the sensing data directly from the sensors 120 and 122. In another example, the sensors 120 and 122 write the sensing data to a buffer data structure on data memory 106 and the processor 102 requests the sensor data from data memory 106.

[24] Processor 102 may also temporarily store the sensing data by defining an onSensorChanged event of the API and pre-process the sensing data, such as by noise filtering or normalising, and store the pre-processed sensing data on data store 106, such as RAM or on an internal processor- register. After that the processor 102 receives the sensing data from the data store 106.

[25] The sensor data is indicative of a distance of an object 126 from the device 100, an orientation in which the distance is sensed, that is, the orientation of the distance sensor 120 and. a direction of movement of the device 100. In one example, the sensing data explicitly includes these values, such as a distance of 10 m.

[26] In a different example, the sensing data comprises raw data, such as the reflection time of a laser range finder. The processor 102 may compute the explicit distance value based on the raw data or may simply use the reflection time as a distance value. It is to be

understood that, throughout this specification 'distance' , 'direction' and 'orientation' is not limited to the explicit distance value, direction value or orientation value but may also be an implicit value from which the distance, direction or orientation can be derived as in the example above of the raw range finder reflection data. [27] Figure 3 is a plan view of an example scenario 300 where the user 116 with device 100 is walking towards first object 126, second object. 302 and third object. 304 while the user is operating the device 100.

[28] Optical sensor 120 senses a distance 124 of the first object 126 from the device 100, which is also the radius of a semi-circle 305 centred at the location of device 100. Inertial sensor 122 senses an orientation 306 of the distance sensor 120 in which the distance 124 is sensed., In this example, the orientation of the distance sensor 120 defines a sensing sector limited by two radii 308 and 310.

[29] Figure 3 shows that the orientation 306 of the distance sensor 120 may be different, to the direction from the device 100 to the object 126. Orientation 306 of the distance sensor 120 is to be understood as the direction the sensor is measuring or facing- or as the orientation of the sensor 120 in which the sensor 120 senses the distance 124 to the object 126. In one example, the distance sensor 120 has an angular offset from, the inertial.

sensors, such as 90 degrees. As a result, processor 102 measures the orientation of the distance sensor by retrieving the raw readout from the inertial sensors and applying the constant angular offset.

[30] Inertial sensors further sense a direction of movement. 312 of the device 100. The example of Figure 3 shows that user 116 is walking towards first, object. 126 and if the communication device 100 does not notify the user 116, the user 116 will cross a threshold distance from the object 126. As a result, the user 116 is at risk of encountering the object 126, which may result in inj ury . [31] In one example, the system 100 is configured such that the threshold distance is 1 m. For example, coming closer than 1 m to another person may be undesirable since the personal space of that person is entered.

[32] In one example, distance sensor 120 is a three- dimensional depth sensor and generates a three-dimensional depth map -

[33] Figure 4 is a three-dimensional depth map 400 comprising, first, second and third objects 126, 302 and 304, respectively. The first two dimensions are

represented by the horizontal and the vertical axis while the third dimension is the distance of the measured objects represented by different colours, generally shown as different shading in Figure 4. First object 126 is at about the same distance away from device 100 as third object 304 and therefore, first object 126 and third object 304 are depicted with equal shading. In contrast, second object 302 is further away, which is represented by different shading.

[34] Depth map 400 is received by processor 102 from, distance sensor 120 and stored on a data store, such as RAM, hard disk drive or solid state disk. As mentioned before, in one example, processor 102 executes function calls of the API to retrieve the depth map 400. Depth map 400 also shows the orientation 306 of the distance sensor 120 and the direction of movement 312 of the device 100, which are not stored within the depth map but separately i n this examp1e .

[35] In this example, depth map 400 is a projection of the sensing sector onto a rectangular image plane, such that radii 308 and 310 form the sides of the image plane, the semi-circle 305 forms the top edge and user location 116 forms a bottom edge 402, As a result, the bottom edge 402 can be labelled with a scale that indicates the angle of the orientation 306 of the distance sensor 120. For example, depth map 400 shows that the direction of movement 312 of the system 100 is -10" from the

orientation 306 of the distance sensor 120.

[36] First radius 308 of depth map 400 is annotated with a further scale that indicates the vertical angle of the distance sensor 120. The natural horizon is located at 0°, while the feet of the user would be located at -90°. In this example, the orientation 306 of the distance sensor 120 also includes the vertical angle of the distance sensor 120 measured by direction sensor 122.

Based on the vertical angle processor 102 can transform the received depth map.

[37] For example, processor 102 can determine a depth gradient that would be expected if the measured object is the ground. Processor 102 can then exclude pixels of the depth map that have depth values according to this gradient. Further, processor 102 can vertically translate the received depth map such that it is located at the correct location with respect, to the vertical and.

horizontal angles shown in Figure 4.

[38] In one example, the length of the arc of semi-circle 305 between radii 308 and 310 is 1 m. It is further noted that an object may also be defined by the distance being greater than the area surrounding the object, for example, when approaching stairs or a hole.

[39] Referring- back to Figure 2, the processor 102 then determines 204 that the system 100 is about to cross a threshold distance from the object 126 based on the sensing data indicative of the distance 124, the orientation306, and the direction of movement 312. In this example, processor 102 operates on the depth map 400 and a vector representing the direction of movement. 312. If the vector representing the direction of movement 312 intersects with the object 126, the system is about to cross a threshold distance from the object 126.

Therefore, processor 102 determines whether there is an intersection and if there is, processor notifies 206 the user that the system 100 is about to cross a threshold distance from the object. 126.

[40] In one example, a library of functions implemented as executable program code is installed on program memory 104, v/hich provides functions for image segmentation.

Executing these functions, processor 102 performs

segmentation of depth map 400 and edge detection to obtain the shape of object 126. For example, Processor 102 executes a function named intersectLinePolygon. to

determine whether direction of movement 312 intersects with the shape of object 126.

[41] Notifying the user may be performed by various different means, such as by activation of a vibration of the device 100, by an audio system, such as a. beeping sound. The beeping sound may become louder or higher pitched as the device 100 approaches the object 126.

Another option is to use the display 112 as an optical notification system and display a warning message to the user 116, which can be particularly useful if the user is reading emails, writing SMS messages or browsing the Internet, while walking. The warning message may be a popup message similar to a message that alerts the user 116 of running low on battery.

[42] For clarification, Figures 3 and 4 further comprise second object 302 and. third object 304. Second object 302 is further away than first object 126 and processor therefore does not. identify object 302 as relevant. Third object 304 is in a different direction to the direction of movement of the system 100 and therefore, third object 304 is also not relevant.

[43] In one example, the inertial sensor 122 further- senses the speed of the system 100, The vector

representing the direction of movement 312 may be scaled by the speed of movement, such that the vector 312 intersects the object 126 in cases where the speed is sufficiently high. If the speed is low, the user 116 can be notified later, that is, when the system 100 is closer to the object, without significantly increasing the risk for the user 116. Speed may be measured by detecting the steps of the user 116 and deriving a speed measure from the rate of steps assuming a fixed length of each step of 1 m, for example .

[44] In examples where the notification is found to be reliable for walking speeds but not reliable for higher speeds, such as when driving a car, the notification may^¬ be limited and provided in cases where the speed of movement, of the system 100 is less than a predetermined threshold, such as 30 km/h, which also includes riding a bicycle at a moderate speed.

[45] As mentioned before, processor 102 may repeat the processing of sensing data in real. time. When multiple samples of sensing data are received within a certain time frame, the processor 102 can calculate the difference between the distance to the object 126 in the first sample and the distance to the object 126 in the second sample. This difference divided by the time between the two samples yields the rate of change in the distance 124. In one example, processor 102 stores 50 samples of sensing data on data store 106 and determines the rate of change based on these 50 samples, such as by computing an average rate of change. The number of samples is adjustable and another possible value is 10 samples.

[46] The threshold distance may be based on the rate of change of the distance 124, For example, if the rate of change is faster than 2 m/s, the threshold distance is 5 m. If the rate of change is less than 1 m/^'s, the threshold distance is 2 ra and if the rate of change is between 2 m/s and 1 m/s, the threshold distance is 3 m.

[47] Subtracting the speed of the movement of the system 100 from the rate of change the processor 102 may

determine the speed of the object itself. In some situations, faster objects are more dangerous and

therefore, processor 102 employs a greater distance threshold for objects with a determined speed that is above a predetermined speed threshold. In one example, the speed threshold is 5 km/h to distinguish betv/een other walking persons and faster objects, such as cars.

[48] In one example, processor 102 also determines a type of the object 126 based on the distance information and the threshold distance is based on the determined type of the object 126. For example, the distance sensor 120 and API provide the feature of detecting humans using a skeleton model. Processor 102 then queries a database for a distance associated -with that, particular object type, such, as 'person' , 'non-person object' or 'car' , As a. result, a low distance threshold can be applied for less dangerous encounters, such as encountering other people, while a high distance threshold can be applied for encountering more dangerous situations, such as falling off a railway platform.. [49] Figure 5 illustrates a mobile communication device 500, such as a tablet computer, for notifying a user that, the mobile communication device 500 is about, to cross a threshold distance from the object 126. The communication device 500 comprises a processing- module 502, such as a mobile communication signal processor, connected to program memory 504 and data memory 506. Processing module 502 is further connected to communications port 508 to send and. receive data and voice signals over a 3G, LTE, Wifi or other physical network. Processing module 502 is also connected via. user port 510 with touch sensitive display 512. A user (not shown) operates display 512 to create or access information stored on the device or retrieved via data port 508.

[50] Communication device 500 further comprises a distance sensing module 520 to sense a. distance 124 of an object from the mobile communication device 500 and a. direction sensing module 522 associated with the distance sensing module 520 to sense an orientation 306 of the distance sensor 120 in which the distance 124 is sensed, and to sense a direction of movement, of the mobile communication device 500. The direction sensing module 522 may be a nricroelectromechanical system (MEMS) .

[511 The processing module 502 determines that the mobile communication device 500 is about to cross a threshold distance from the object 126 based on the distance, the orientation and the direction of movement as explained above with reference to Figures 3 and 4.

[52] A notification system 524 notifies the user in response to determining that the mobile communication device 500 is .about to cross a threshold distance from the object 126. The notification system. 524 may be a buzzer, loudspeaker, vibrating element, or integrated, into the display 512, such that a notification message 530 is shown to the user 116 on display 512.

[53] Throughout this specification the word, "have", or variations such as "has" or "having", will be understood to have the same meaning as the word "comprise" and to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[54] It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not. restrictive.

Claims

CLAIMS :

1. A system for notifying a user of a mobile device, the system comprising:

a first sensor to measure a. distance from the system, to an object;

a second sensor associated with the first sensor to measure an orientation of the first sensor and to measure a direction of movement of the system;

a processor to determine based on the distance, based on the orientation and based on the direction of movement that the system is about to cross a threshold distance from the object; and

a notification system to generate a notification based on the determination of the processor.

2. The system of claim. 1 wherein the first sensor is a 3-dimensional depth sensor,

3. The system of claim 1, wherein the second sensor is an inertial sensor.

4. The system of claim 1, wherein the second sensor is an inertial sensor of the mobile device and. the processor i s the processor o f the mobile device .

5. A method for notifying a user using- a mobile device, the method comprising:

receiving sensing data indicative of

a distance of an object from the mobile device, an orientation in which the distance is sensed, and

a direction of movement of the mobile device; determining that the mobile device is about to cross a threshold distance from the object based on the sensing data indicative of the distance, the orientation, and the direction of movement; and notifying the user in response to determining that the mobile device is about to cross a threshold distance from the object.

6. The method of claim 5, further comprising sensing speed of movement of the mobile device, wherein

determining that the mobile device is about to cross a threshold distance from the object is based on the speed of movement of the mobile device.

7. The method, of claim 6, wherein notifying the user comprises selectively notifying the user where the speed of the mobile device is less than 30 kilometres per hour.

8. The method of claim 5, further comprising determining a rate of change in the distance based on the distance, wherein determining that the mobile device is about to cross a threshold distance from the object is based on the rate of change of the distance.

9. The method of claim 8, further comprising determining speed of the object based on the rate of change and. based on the speed of movement of the mobile device,

wherein determining that the mobile dev.ice is about to cross a threshold distance from the object is based on the speed of the object.

10. The method of claim 5, further comprising determining a type of the object, wherein the threshold distance is based on the type of the object.

11. The method of claim 10, further comprising

determining the threshold distance based on the speed of the object.

12. A mobile communication device for notifying a user of the mobile communication device, the mobile communication device comprising :

a first sensing module to sense a distance of an object from the mobile communication device;

a second sensing module associated with the distance sensing module to sense an orientation of the distance sensing module ;

a third sensing module to sense a direction of movement of the mobile communication device;

a processing module to determine that the mobile communication device is about to cross a threshold distance from the object based on the distance, the orientation and the direction of movement; and

a notification system to notify the user in response to determining that the mobile communication device is about to cross a threshold, distance from the object.

13. The mobile communication device of claim 12, wherein the notification system is a display to display to the user a notification message.

14. The mobile communication device of claim. 12, wherein the notification system is a vibration system to cause vibration of the mobile communication device.

15. The mobile communication device of claim 12, wherein the notification system is an audio system to generate an audio signal.