WO2011108914A2

WO2011108914A2 - A visual object tracking and searching system and a method thereof

Info

Publication number: WO2011108914A2
Application number: PCT/MY2010/000301
Authority: WO
Inventors: Norshuhada Samudin; Tomas Maul
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2010-03-01
Filing date: 2010-11-25
Publication date: 2011-09-09
Anticipated expiration: 2012-09-01
Also published as: WO2011108914A3; MY174438A

Abstract

A visual object tracking and searching system of a location is provided, the system includes at least one auditory acquisition means, at least one visual acquisition means, at least one textual input means, wherein the at least one auditory acquisition means, the at least one visual acquisition means and the at least one textual input means are connectable to at least one processing means, wherein the at least one processing means is further connectable to at least one display means such that the at least one processing means receives auditory, visual and textual input from the at least one auditory acquisition means, the at least one visual acquisition means, the at least one textual input means to produce an output in the at least one display means.

Description

A VISUAL OBJECT TRACKING AND SEARCHING SYSTEM AND A METHOD

THEREOF

FIELD OF INVENTION

The present invention relates to a visual object tracking and searching system of a location and a method thereof.

BACKGROUND OF INVENTION

Computer vision solutions are used for keeping track of and searching for any class of object in an indoor environment. The known computer vision solutions are restricted to academia that tends to be not too concerned with lost objects, but rather with object tracking.

Most known solutions use RFID tags to track objects which require the user to continuously increase total number of tags based on number of objects that are being tracked. Further, RFID tags run a risk of becoming faulty over time. This can prove to be a rather costly exercise.

U.S. 1999/6127928 describes a method and an apparatus for locating and tracking documents such as office files in an office environment using RFID technology. However, this method is only limited for the tracking of documents and not for any other objects that may be lost as well.

In a known computer vision based application, a camera is fixed on a pair of glasses. A user is required to wear the glasses during process of registration of items and the searching process. However, this method is limited to a single individual and depends on the user to store the glasses safely in order to use them to track objects. Further, it is rather cumbersome to use the glasses at all times and this causes the method to be more prone to human error.

Another problem in vision based solutions for lost objects pertain to objects that are not visible because of being concealed by body or contained in an enclosure. The objects are not locatable because of these reasons. A solution is needed to overcome concealed objects that need to be tracked.

Object disambiguation is another problem faced in vision based solutions as there may be several objects that are similar to requested object. A solution is needed to be able to narrow down the search to the requested object. U.S. 1999/6184789 describes a method and apparatus for visually determining object locations using an object associated with the object locater. A camera is used to only get the display of the object location for the user after the object has been detected.

There is therefore a need for a practical solution for locating general lost objects within an environment while considering object disambiguation and objects that are concealed away from plain sight.

SUMMARY OF INVENTION

Accordingly there is provided a visual object tracking and searching system of a location, the system includes at least one auditory acquisition means, at least one visual acquisition means, at least one textual input means, wherein the at least one auditory acquisition means, the at least one visual acquisition means and the at least one textual input means are connectable to at least one processing means, wherein the at least one processing means is further connectable to at least one display means such that the at least one processing means receives auditory, visual and textual input from the at least one auditory acquisition means, the at least one visual acquisition means, the at least one textual input means to produce an output in the at least one display means.

There is also provided a method of tracking and searching for objects visually in a location, the method includes the steps of registering data of objects of interest (01) from a location in a database in reference to a plurality of frames acquired from an at least one visual acquisition means, categorizing said data of Ol based on predetermined attributes, searching the database for an object of interest (Ol) using categorized data in database, searching for the Ol at a specified previous location as updated in the database, displaying at least one location of the Ol to a user and updating the database with current location of Ol.

The present invention consists of several novel features and a combination of parts hereinafter fully described and illustrated in the accompanying description and drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be fully understood from the detailed description given herein below and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, wherein:

Figure 1 depicts a block diagram of a visual object tracking and searching system of a location in a preferred embodiment of the present invention;

Figure 2 is a high level flow chart of the method involving detecting of translocation behaviors and updating of object locations in a preferred embodiment of the present invention;

Figure 3 is a flowchart showing a method of input request for an object of Interest (Ol) in the preferred embodiment of the present invention;

Figure 4 is a flowchart showing a method of input request for an 01 using a microphone as means of input in the preferred embodiment of the present invention; and Figure 5 is a flowchart showing a method of input request for an Ol using a keyboard as means of input in the preferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention relates to a visual object tracking and searching system of a location and a method thereof. A detailed description of a preferred embodiment of the invention is disclosed herein. It should be understood, however, that the disclosed preferred embodiment are merely exemplary of the invention, which may be embodied in various forms. Therefore, the details disclosed herein are not to be interpreted as limiting, but merely as the basis for the claims and for teaching one skilled in the art of the invention.

The following detailed description of the preferred embodiment will now be described in accordance with the attached drawings, either individually or in combination.

While the following description may describe example embodiment of the present invention in relation to locating lost objects, the invention is not limited to and can be applicable to other types of object searching where similar advantages may be obtained.

Figure 1 depicts a block diagram of a visual object tracking and searching system of a location. The system includes at least one auditory acquisition means, at least one visual acquisition means and at least one textual input means which are connectable to at least one processing means. The at least one processing means is further connectable to at least one display means such that the at least one processing means receives auditory, visual and textual input from the at least one auditory acquisition means, the at least one visual acquisition means, the at least one textual input means to produce an output in the at least one display means. An example of the at least one auditory acquisition means is a microphone, used for command interpretation. At least one visual acquisition means such as high resolution cameras are used to acquire visual input for tracking objects. An example of the at least one textual input means is a keyboard. At least one processing means used in this embodiment is a typical personal computer with peripherals. The at least one display means used in this embodiment is selected from Liquid Crystal Displays (LCDs) and motorized lasers.

The typical computer further includes a plurality of modules such as speech recognition, command interpretation, object registration, feature extraction, pattern matching, object tracker, object searcher, posture extractor, translocation detector, spatial reasoning, location uncertainty reasoning, query interpreter and disambiguator. However, it is to be understood that the plurality of modules described herein may further include other modules that are seen fit according to application.

Output of the system further includes a sound synthesizer for producing synthetic speech or sound for interaction with a user. A log manager is further used to manage a database of locations, translocations, characteristics of objects and previous queries. A camera controller may also be used to switch on or off the cameras or zoom as seen fit. The system is used to search for lost objects such as pens, sets of keys, remote controls and other similar objects.

A method of tracking and searching for objects visually in a location is described. Figure 2 shows a high level flow chart of the method involving detection of translocation behaviors and updating of object locations. T e method includes the steps of registering data of objects of interest (01) from a location in a database in reference to a plurality of frames acquired from an at least one visual acquisition means, categorizing said data of Ol based on predetermined attributes, searching the database for an object of interest (Ol) using categorized data in database, searching for the Ol at a specified previous location as updated in the database, displaying at least one location of the Ol to a user and updating the database with current location of Ol.

Registration of data of Ol is done by collecting data of all objects and locations involved are stored in a database. The registration is conducted by means of speech recognition through microphone, by means of text input through keyboard and visual feature extraction through cameras. The collected location data is then categorized upon capturing a plurality of images from a respective camera. Each segment of a location assigned to a camera is identified upon coordination by view found by the camera. Categorization is conducted based on attributes such as color, shape and semantic type. Examples of semantic types are as follows, but not restricted to, electronic devices, kitchen appliances, personal belongings and hazardous household items. However, it is to be appreciated that these items and categories are to be defined by users based on applications that the invention is used for.

As seen in Figure 2, for each new frame captured from a camera, behavior of Ol is computed by detecting any translocation of the Ol from frame to frame. If no translocation is found, each subsequent frame is continuously checked for each Ol. However, if translocation is detected for a specific Ol, the database is searched for the Ol if it is registered. If the Ol is registered in the database, location of the Ol is tracked and stored in the database. This aspect of the invention is detection of translocation behaviors and updating of object locations. Figure 3 depicts a flowchart of input request for an 01. Once the system receives the request, the database is referred for information updates, such as locations and categories of said 01. From the database, a previously specified location of the 01 is found. The location is then traced and searched to locate the Ol. Upon detecting the 01, size of the Ol is considered. If the size of the Ol on display is smaller than a predetermined limit, the camera assigned to the location is instructed to zoom in to the 01. In this example, the predetermined limit of the size of view on display is set at 25 x 25 pixels. Further, the system identifies if there are more than one of the Ol that is detected. If there is more than one Ol, an output indicating the same is displayed via a display means or a loudspeaker to users. A user is then able to manually select the Ol as required. Upon selecting the Ol, the database is then updated with a latest location of the 01. The searching step is then considered as successful.

Figure 4 shows a flowchart of the method as described above using a microphone as means of input. A search keyword associated with an 01 is pronounced to trigger the search in the system. If a related Ol is found, object name is cross checked in the database for registration. If the Ol is not found, characteristics of the Ol are pronounced carefully to detect the Ol accurately. Name of the Ol is informed to user by auditory means such as a loudspeaker. Inaccurate pronunciations may require user to pronounce the 01 name and characteristics repetitively until the correct 01 is detected. Further steps are then carried out as per the method as described in Figure 3.

Figure 5 shows a flowchart of the method as described above using a keyboard as means of input. A search keyword associated with an Ol is input into the system and the search is initiated. A list of attributes associated with the Ol, such as list name, classification, shape and color of 01 is displayed. Required attributes are then selected from the list. A resulting 01 is then displayed. If the resulting 01 is correct, further steps are then carried out as per the method described in Figure 3. If the resulting 01 is incorrect, the list of attributes is displayed again for the user to select most accurate attributes associated with the 01. This step is repeated until a correct Ol is displayed.

The system and method as described above solves a problem of object disambiguation and a problem of tracking unseen objects. Object disambiguation is addressed by allowing the user to be able to manually select the required Ol from a list of possible 01 displayed as seen in Figure 3. Tracking of unseen objects is solved by identifying locations previously inhabited by the 01 and assuming that behavioral characteristics of the Ol is predictable based on the previously inhabited locations as seen in Figure 3. It is also assumed that one of the reasons why an object is thought to be lost is because it is occluded from user's vision. With the assistance of the invention, the user is able to detect hidden Ols.

It can be seen that the described embodiment of the invention is able to detect object translocation behaviors and requests the user to reveal the object that is being relocated by representing and reasoning about uncertainties of object location. This is an improvement over prior art in this field as this combination of features are not found in other computer vision based solutions.

It is to be understood that the embodiments of the invention described are exchangeable for other variations of the same in order to be used in various applications. The present embodiment of the invention is intended for, but not restricted to, locating lost objects in a specific location.

Claims

1. A visual object tracking and searching system of a location, the system includes: at least one auditory acquisition means, at least one visual acquisition means, at least one textual input means,

wherein the at least one auditory acquisition means, the at least one visual acquisition means and the at least one textual input means are connectable to at least one processing means,

wherein the at least one processing means is further connectable to at least one display means such that the at least one processing means receives auditory, visual and textual input from the at least one auditory acquisition means, the at least one visual acquisition means, the at least one textual input means to produce an output in the at least one display means.

2. The system as claimed in claim 1 , wherein the system further produces auditory output from at least one loudspeaker connectable to the at least one processing means.

3. The system as claimed in claim 1, wherein the at least one auditory acquisition means is a microphone.

4. The system as claimed in claim 1 , wherein the system produces visual output in Liquid Crystal Display (LCD).

5. The system as claimed in claim 1 , wherein the system produces visual output in motorized lasers.

The system as claimed in claim 1 , wherein the at least one textual input means is a keyboard.

The system as claimed in claim 1 , wherein the at least one processing means is a personal computer.

The system as claimed in claim 1 , wherein the at least one visual acquisition means is a camera.

A method of tracking and searching for objects visually in a location, the method includes the steps of:

i. registering data of objects of interest (Ol) from a location in a database in reference to a plurality of frames acquired from an at least one visual acquisition means;

ii. categorizing said data of Ol based on predetermined attributes;

iii. searching the database for an object of interest (Ol) using categorized data in database;

iv. searching for the Ol at a specified previous location as updated in the database;

v. displaying at least one location of the Ol to a user; and

vi. updating the database with current location of Ol.

10. The method as claimed in claim 9, wherein searching the database for the Ol further includes the steps of determining translocation of Ol within the plurality of frames and updating the translocation in the database.

11. The method as claimed in claim 9, wherein registration of data is conducted by means of speech recognition, text input and visual feature extraction.

12. The method as claimed in claim 9, wherein searching for the Ol at a specified previous location is used to compute behavior of Ol.

13. The method as claimed in claim 10, wherein if no translocation of Ol is found, subsequent plurality of frames are continuously checked for each Ol.

14. The method as claimed in claim 10, wherein if the Ol is registered in database, location of the Ol is tracked and stored in the database.

15. The method as claimed in claim 10, wherein if a list of Ol is found, a user selects the Ol manually.

16. The method as claimed in claim 15, wherein data of the manually selected Ol is updated in the database.