WO2012093365A1

WO2012093365A1 - System for fusing geographic and locally acquired data for providing real world interoperability

Info

Publication number: WO2012093365A1
Application number: PCT/IB2012/050046
Authority: WO
Inventors: Jason A HADDAD; Michael Y UOHARA
Original assignee: Seenow Ltd; GROSSMAN, Ruth, Leslie
Priority date: 2011-01-04
Filing date: 2012-01-04
Publication date: 2012-07-12
Also published as: WO2012093365A4

Abstract

A system for providing to a user orientation and facilitates interaction with the real world in three hierarchical levels. The a user operates a mobile network accessing unit, having a connection to a GIS database. Associated with the user are also one to several real world spatial data acquisition units such as a camera and a laser range finder. An input device for obtaining real world data on a micro scale is also associated with the mobile network accessing unit. The data collected by the real world data acquiring facilities is registrable to the GIS database.

Description

SYSTEM FOR FUSING GEOGRAPHIC AND LOCALLY ACQUIRED DATA FOR PROVIDING REAL WORLD INTEROPERABILITY

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

[001 ] This application claims priority from US provisional application serial number 61/429,614 filed 4 January 201 1 .

TECHNICAL FIELD OF THE INVENTION

[002] The present invention relates to the acquisition of visual and other spatial data and fusing that visual data with data derived from geographic information systems and web maps.

BACKGROUND ART

[003] Most personal navigation systems that are freely available today use the GPS technology as a geographic reference. In some built up locations and heavily wood covered localities such a system may not work or produce poor results. Navigation systems based on the TDMA (time division multiple access) standard based cellular network system can provide a rather accurate navigation and geo-location system that may back up or replace the satellite system in certain situations, notably inside buildings.

[004] Field measurements made for the purpose of quantifying areas and distances resort to a digital method in order to insert newly acquired data into existing geographic databases referred to in the profession as GIS (geographic information systems). In order to incorporate a measurement into an existing geographic database, it is required to register (i.e. match coordinates) the newly acquired spatial data to the existing GIS. This can be done by using GPS or enhanced GPS (such as differential GPS for more accurate results) measurements made in the field to exactly overlay the new data with an existing database. Digital photogrammetry is used to fuse acquired images with existing geographic data. A typical GIS contains layers of information, for example topography layer, roads layer, houses layer, electric lines layer, all registered to the geographic grid used by the particular GIS system.

BRIEF DESCRIPTION OF THE DRAWINGS

[005] The present invention will be understood and appreciated more fully from the following detailed description taken in conjunction with the appended drawings in which:

[006] Fig. 1 is a schematic representation of the major modules in a framework in which the present invention is implemented;

[007] Fig. 2 is a schematic sequence of events taking place starting from a user expressing a query verbally, in one embodiment of the invention;

[008] Fig. 3 is a schematic representation of a personal component of a geographic data base registered to a large GIS system.

[009] The following detailed description of embodiments of the invention refers to the accompanying drawings referred to above. Dimensions of components and features shown in the figures are chosen for convenience or clarity of presentation and are not necessarily shown to scale. Wherever possible, the same reference numbers will be used throughout the drawings and the following description to refer to the same and like parts.

DISCLOSURE OF THE INVENTION [010] Illustrative embodiments of the invention are described below. In the interest of clarity, not all features/components of an actual implementation are necessarily described. The aim of the present invention to combine large geographic information with local information to provide a technological framework for providing user orientation in and interaction with, the real world.

[01 1 ] In accordance with the present invention, spatial geographic data is collected from various sources and combined in a meaningful fashion to create an accessible spatial hierarchical data base. The meaning of the term "access" in the context of the present invention relates to a mobile user borne network accessing unit, typically such that can probe a spatial database residing in a network or in the unit itself. Typically such a unit is an enhanced mobile telephone having Internet or other network accessing capabilities.

[012] Fig. 1 shows a generalized overview of the technological framework in which the invention is implemented. Input devices 30 and network available sources 32 report to a data processing device 34, which is made accessible to a user handling a specific information and guidance conveying device 36. Although described in the singular, all sources and devices referred to above may be embodied in the plural.

[013] Registration to the geographic grid

[014] Registration to the geographic grid is a fundamental requirement whenever any reference is to be made to the contents of a GIS or other maps, any of the layers of the GIS or all of the layers. Practically, once a user is registered to the geographic grid, any available spatial data that the user obtains must be registered to the geographic grid, in order that meaningful calculations and assertions can be made based on the data stored in the GIS. Registration to the world grid can be made by using the GPS satellite navigation system, in which user operated satellite signal receiving device accepts timing data from several satellites and the device calculates in situ X, Y and Z coordinates. More satellite systems like these are planned by large organisations and their use is not expected to be significantly different that the GPS system presently in use. There are less well known, more localized navigation systems which may be used (e.g. LORAN) but are generally not as abundant or easily accessible as the satellite system are. Another wide spread system is the location system based on the cellular networks. A navigation satellite receiver, known also as GPS receiver, is now available in many cellular phone personal terminals which makes the registration to the world grid easy and cheap though not always accurate. Often a GIS system will need to apply a transition calculation in order to accommodate world coordinates to the GIS local coordinate system.

[015] A less formal coordination with a geographical grid is available by interrogation of local networks other than cellular networks, such as WiFi IEEE 802.1 1 standard, and Wimax IEEE 802.16 standard. A "hot spot" is a common name in which public access to a network, (typically WiFi) is provided freely or for payment. In transportation terminals, coffee houses and malls such a service may connectable by a user to derive information. A more simple but efficient approach for a registration with the GIS grid can be achieved by looking for several items in the GIS which can be distinctly spotted in the real world, thus providing for discrete registration to the GIS.

[016] Network available geographic data sources

[017] The field of geographic information systems (GISs) is a well developed and useful geographic discipline in which spatial data organized in thematic layers relating to topography, public networks and infrastructure are interlinked, based on a geographic grid. Such databases are managed and updated by public authorities or private enterprises to include as much spatial information as can be sponsored. GISs, used and maintained by municipal, private and state-wide, nation-wide enterprises, include vast sources of information regarding surface coverage, underground infrastructure (cables, sewage, water) and the like. The term GIS is usually used by the professionally oriented people, and not so much in the public domain. Nevertheless, throughout the present disclosure the term shall relate also to computerized maps in general. [018] Devices for acquiring spatial information of the real world

[019] The number and nature of devices that can potentially acquire with the required real time spatial data are numerous. Examples include regular electronic cameras that are able to gather, visible, infra red, still or streaming real world information upon automatic pre-setting, or upon activation by the user. Likewise, thermal infrared cameras for night vision, laser range finders or proximity sensors in general for determining range, sonar devices for gathering information about the ambient , and radar devices can potentially supply spatial information, either in the form of image as array of pixels or in a point measurement.

[020] Devices for conveying information and or alerts to a user

[021 ] In accordance with the present invention, a user who is incapable of browsing a network browser, or otherwise engaged in user graphic interface manipulation, is able to receive information from a personal receiving device through the acceptance of information or alerts via senses other than sight. For example, a bracelet worn by the user (see US patent application 2004/0192423 to Nevermann) is capable of providing an electroshock to the user in order to silently alert the wearer regarding a certain event. Other more common means of conveying information are speakers, typically for that matter accepting synthetic voice produced by a voice synthesizer attached or constituting a part of the user's personal receiver. For users who are blind or sight impaired and who are acquainted with the Braille alphabet, information can be conveyed through a Braille synthesizer. In the other direction, conveying commands or queries to the system is made in accordance with the present invention by means other than graphic screen requiring visual attention to operate. Talking into a microphone such that the commands are processed and machine coded to perform a task such as "look for a road crossing in the closest road heading south" is a preferred option for loading data and or commands to a data base interface. Interpretation of such a query may be achieved by a knowledge base that interprets the meaning and provides answers. Reference is made now to Fig. 2 to summarize the flow of information beginning in a spoken command or query. In step 42 words, typically an expression is uttered by the user. Then in step 44 the microphone receives the word/s the microphone connected to the user's network access device (e.g. smartphone) supplies the electric equivalent of speech to the supporting circuits in the access device which converts it to computer code in step 46. Then the piece of computer code is communicated to a knowledge base, typically a web service, where it is analysed and resulting answer is synthesized in step 50. Then, in step 52 the answer is communicated to the user.

[022] Hierarchical spatial analysis

[023] To associate oneself with a specific location, the user may be required to access his/her GIS service and as a result of presenting a query, the system performs a hierarchical spatial analysis. Typically the GIS is accessed via the user borne network accessing device. For example, if a specific railway station is sought, the system accesses the GIS of the railway company, one or more. Presenting the query, the system calculates and instructs the user of the possible ways to arrive to the desired destination. While progressing towards and when the user eventually arrives at the desired destination, the personal data collection device (e.g. camera) acquires images of the surroundings and an automated program (including for example an OCR - optical character recognition) scans the images looking for a sign that would indicate that the user arrived at the desired location (e.g. railway station). The implementation of the hierarchical approach aims to provide more information to the user than can be expected from using just one level at a time. For example, the nation - wide GIS is not expected supply real time data. The hierarchical structure of the data in accordance with the present invention and the analysis thereof will be dealt with in more detail below.

[024] Data fusion and spatial data base production

[025] Hierarchical spatial analysis - examples [026] Country level GISs can provide, for example, topography, road layout, and railway station locations. Municipal GISs on the other hand is typically more oriented to local governments including for example topics such as sewage lines, gardens, and information regarding all municipal infrastructure and transportation. Thus, a user arriving at a city for the first time, can register to all layers of a GIS once the user receives the coordinates from a GPS directly to the user's own mobile phone. Using one of the navigation applications, the user can receive instructions of the route to a certain address such as an address of a certain shop in another part of the city. The above described information level will be referred to hereinafter as the large scale level.

[027] Getting down to the next lower level in the hierarchy of the spatial information, or smaller scale, in other words, the user can be directed to, for example, an underground train station via a database of the train company. Then, a calculation of the shortest route to a train station can be made using the municipal GIS information or a GPS system. In another example, the user can be directed to a road crossing and while crossing the road, his/her own camera may be used to assess sources of danger such as traffic, by implementing a motion detection algorithm. For crossing the road, a video application can again be implemented for deciphering street crossing signals that guide pedestrians to cross the road. It is again mentioned that finding a location may also be accomplished using a video processing algorithm. Such interactions will be referred to hereinafter as intermediate level spatial interactions, typically measuring meters or tens of meters between the user and targeted objects. Referring now to a still lower hierarchical level of the spatial information and associated interactions, hereinafter the micro level, a user is guided to objects in the framework of the range of centimeters - a few meters. Going into a shop, for example, the user may be alerted by an RFID reader that detects a RFID tag attached to his/her mobile telephone. In another option as disclosed in WO 201 1/014292 an ultrasonic emitter emits coded ultrasonic radiation to notify consumers within a shop of offers. Such communications facilities enable real time data to be transmitted to the user in accordance with the present invention. An alternative navigation system that can replace the GPS technology altogether in the context of the present invention, or use as a complement to it, is inertial navigation. The implementation of inertial navigation for all three levels of the spatial information related to in the present disclosure is possible, however it is specifically useful in the micro level orientation. Inertial navigation can provide accuracies that the GPS system cannot provide. To that end, accelerometers and gyroscopes can be carried by a user preferably attached to his/her personal phone, and with the proper electronic support, the distance and direction that the user has accomplished since a specified time up to a time. As an example of the exploitation of inertial navigation, a person who digs drainage trenches in the field and is required to apply caution to avoid digging into an underground communications cable, logs on to a computerized cadatsral map service, which in fact a sort of a GIS service, and to a GIS database of the communications company, can be potentially made aware of the distance to the cable in a certain field. However, in order to calculate his own distance from the cable the operator of the trencher at each moment, the operator activates an inertial navigation system at a certain location, the coordinates of which are completely known in the GIS, and as the trencher moves, the inertial navigation system measures distance and issues an alert when the trencher gets too close to the cable.

[028] Non-hierarchical spatial analysis

[029] To explain non-hierarchal spatial analysis, an example is given to an instance in which a meeting is scheduled for a user with a specific person. In such a case the user may apply facial recognition software, once a person pertaining to be a specific individual has been encountered. In such a case, there is no hierarchical analysis because the specific person to be met is a unique individual notwithstanding the location of the meeting.

[030] Location based services (LBS), is technological trend developing in the last few years, typically provided by cellular communications systems that fairly accurately follow a user and may provide certain services including 91 1 service (emergency services) but also commercial applications relating to commerce, and services, shopping and the like. In such a case, the cellular communications network keeps a contact with the user of the network at the level of accuracy and resolution it can provide. However, the user, in accordance with the instant invention, is able to choose a smaller scale search and real world interaction using further technological facilities not afforded by the LBS.

[031 ] The user, who is to benefit from the implementation of the invention ?

[032] The invention is targeted at such people who are busy exploiting their sensory resources in a way that prevents or limits their paying attention to user interfaces. In addition, individuals who spend time in poorly lit areas or are visually impaired and are unable to conveniently attend to a graphic user interface. Other examples of those who may benefit from the present invention Include, for example, a farmer tilling the soil is careful about the results of his/her work and cannot be attentive to a browser screen to help interact with it regarding for example the whereabouts of other manned machines in the farm. Another example is a foreman of a team or several teams working in outdoor construction jobs, where his/her attention is completely taken over by current activities. Police work is a further example of an activity that may require full attention of men and women and supervising and thus could benefit from the instant invention.

[033] Learning and knowledge base augmentation

[034] As the user acquires more experience and information, provisions are made to augment the accessible GISs and knowledge base so as to include the personalized data. It would seem obvious that authorities and private organizations would not give a permit for strangers to manipulate their GIS. Therefore, in order to improve the performance of the instant system, user data that has been acquired and is not available to the public is added to a localized portion of the data base. To explain this aspect of the invention, reference is made to Fig. 3. Publically available GIS data base 122 is registered to a geographic grid (shown schematically in the figure). A user who uses GIS data base 122 is located at a specific geographical region covered by rectangular portion 124 of the data base. The user may choose to add features to his/her personal copy 126 of the data base which is registered to that data base.

[035] More generally, the user may update and create a version of the spatial data base in all three hierarchical level discussed above. At the large scale level, the availability of public GIS is common. And the issue of keeping an updated version of a part or a plurality of parts of the GIS was discussed in the foregoing augmentation section. In the intermediate hierarchy level, which may be for example an electronic map of a mall, user spatial database augmentation is also applicable.

[036] Knowledge bases as known in the art are different from GIS in several respects. In the context of the present invention, it can relate to description of objects distributed geographically. For example qualitative and quantitative descriptions of transportation facilities, government institution etc. A knowledge base may be augmented perpetually by collecting data from various sources and can therefore benefit from sharing among users, unless limited for cfident6iality.

[037] Working in the micro scale, registration to the higher hierarchical layers allows to store micro level data in relation to a GIS level database. For example, a user having, in addition to the network accessing device also an inertial navigation system, can augment his/her personal spatial data by adding trek routes. Accelerometers and gyroscopes stashed for example in the shoes can report the navigational properties of the walking user to the system to be later on retrieved and instruct the walker when he/she should follow the appropriate direction in order to re do the stored track.

[038] Location of the modules of the system

[039] A user of the system is expected to carry with a receiving unit for gaining access to the Internet or other connections for use as data acquisition unit from the GIS or additional resources. The computer memory units harboring the data and or program tools (such a various software applications, scripts, data bases) are partially or wholly resident in the access tool (i.e. smartphone), or more realistically are situated partially in the access unit (operation system, input/output drivers) while GIS and other shared services are typically resident in servers over the network or in the what is known as cloud. The individually obtained and inferred data base portions (as discussed above in the learning and data base augmentation section), may reside on the personal access device or in the network, or both.

[040] It should be understood that the above description is merely exemplary and that there are various embodiments of the present invention that may be devised, mutatis mutandis, and that the features described in the above- described embodiments, and those not described herein, may be used separately or in any suitable combination; and the invention can be devised in accordance with embodiments not necessarily described above.

Claims

A system for providing to a user spatial orientation in and facilitates interaction with the real world in three hierarchical levels, said system comprising:

• a user operated mobile network accessing unit;

• a connection to at least one GIS, said connection accomplished via said accessing unit;

• at least one real world spatial data acquisition unit associated with said user;

• a connection between said spatial data acquisition unit and said GIS data accessing unit;

• an input device for receiving instructions and or data from said user;

• at least one input device for obtaining real world data on a micro scale, and wherein said GIS is augmentable.

The system according to claim 1 , wherein said mobile network accessing unit and said GIS data acquisition unit are packaged together.

The system according to claim 1 , wherein a GPS receiver is used for registering to a GIS database, and wherein inertial navigation system is used to navigate in the micro scale level of the real world.

4. The system according to claim 1 , wherein said real world spatial data acquisition unit acquires real time data.

5. The system according to claim 4, wherein said real world spatial data acquisition unit is a camera.