DE202018100752U1 - Bayesian methodology for geospatial object / feature recognition - Google Patents

Abstract

A computer-readable storage storing instructions executable by one or more processors to perform a method of determining a location of an object of interest, the method comprising: identifying a set of many images from an image database relating to a region of interest wherein each of the plurality of images is associated with image capture information including at least one image capture site; applying an image recognition tool to each image in the set of multiple images; determining, based on the application of the image recognition tool, which of the images is the object of Include interest and which of the images do not include the object of interest; for each of a plurality of candidate sites in the region of interest, calculate a probability value of the object of interest present at the candidate site using the image capture information for images in the set of many images that are determined to include the object of interest, and using the image capture information for images in the set of many images that are determined not to include the object of interest; anddetermining the location of the object using the probability values for the plurality of candidate sites.

Description

In accordance with the provisions of the Utility Model Law, only devices as defined in the appended claims are protected and covered by utility model, but not methods. Insofar as reference is made in the description below to methods, these instructions serve only to exemplify the device (s) protected by the attached claims.

BACKGROUND

Over time, ever larger amounts of geodata have become available. Performing semantic searches through such images has become more demanding as volume increases. The semantic search is the search for a specific object or environmental attribute, such. A lamppost or parkland, by name and not by exact pixel pattern. Conventional methods of semantic search require manual checking, and new methods require elaborately trained and matched, dedicated image recognition models. For example, manual triangulation is labor-intensive, slow and costly, while automatic triangulation requires sophisticated and cumbersome localization-specific image recognition models or very high-resolution images, yet can provide an inaccurate location. Although general image classification models are available for storing photos or searching the Internet, these non-specialized models lack the precision to properly locate objects.

SHORT DESCRIPTION OF THE REVELATION

One aspect of the disclosure provides a computer-readable storage that stores instructions that are executable by one or more processors to perform a method of determining a location of an object of interest. This method includes identifying a set of many images from an image database relating to a region of interest, wherein each of the plurality of images is associated with image capture information that includes at least one image capture site. The method further includes applying an image recognition tool to each image in the set of many images; and determining, based on the application of the image recognition tool, which of the images include the object of interest and which of the images do not include the object of interest. For each of a plurality of candidate sites in the region of interest, a probability value of the object existing at the candidate site is calculated using the image capture information for images in the set of many images determined to include the object of interest, and under Use the image capture information for images in the set of many images that are determined not to contain the object of interest. The location of the object is determined using the probability values for the multiple candidate sites.

According to some examples, the image capture information for at least some of the images includes a capture location and an image capture orientation, while calculating the probability score of the object present at a particular location includes using the capture orientation.

In one of the foregoing embodiments, determining which of the images include an object of interest and which of the images do not include the object of interest may include determining a confidence factor that the images include or do not include the object of interest while calculating the probability value of Using the specific confidence factor may be present in the location of the object of interest.

In one of the foregoing embodiments, computing the probability value of the object of interest at the candidate site may include applying a factor having a first character to the images in the set of many images determined to include the object of interest and applying a factor having a second character for the images in the set of many images that are determined not to include the object of interest, the second character being opposite to the first character.

In one of the previous embodiments, the plurality of candidate sites in the region of interest may include a grid of multiple sites.

In one of the preceding embodiments, the plurality of candidate sites may include locations included in a field of view of each image in the set of many images.

In one of the previous embodiments, the image recognition tool may be configured to recognize discrete objects.

In one of the preceding embodiments, the image recognition tool may be configured to recognize objects having specific characteristics.

In any of the foregoing embodiments, the image recognition tool may be selected from a library of image recognition tools.

Another aspect of the disclosure provides a system that provides a memory that stores a plurality of images associated with location information and one or more processors that communicate with the memory. The one or more processors are programmed to identify, from among the plurality of images, a set of images relating to a region of interest, using image recognition to determine which of the images include the object of interest, and using of image recognition determine which of the images do not contain the object of interest. For each of a plurality of candidate sites in the region of interest, a probability value of the object existing at the candidate site is calculated using the location information for images in the set of many images determined to include the object of interest, and using the Location information for images in the set of images that are determined not to include the object of interest. The one or more processors are further programmed to determine the location of the object using the probability values for the plurality of candidate sites. In some examples, the system may further include an image recognition tool that can identify objects or attributes in the images. Further, the one or more processors may also be configured to provide the determined location information for display on a display.

Another aspect of the disclosure provides a computer-readable medium storing instructions executable by a processor to perform a method of determining the location of an object of interest. Those instructions provide to identify from an image database a set of many images relating to a region of interest, each of the images being associated with image capture information comprising at least one image capture site, an image recognition tool on each image in the sentence apply images based on applying the image recognition tool to determine which of the images include the object of interest and which of the images do not include the object of interest. For each of multiple candidate sites in the region of interest, the instructions further provide a probability value of the object present at the candidate site using the image capture information for images in the set of many images determined to include the object of interest, and using the image capture information for images in the set of many images that are determined not to include the object of interest, and determining the location of the object using the probability values for the multiple candidate sites.

list of figures

• 1 FIG. 12 is a block diagram of an exemplary method in accordance with aspects of the disclosure. FIG.
• 2 FIG. 12 shows a top view illustration of exemplary observations and non-observations in accordance with aspects of the disclosure. FIG.
• 3 illustrates street views in conjunction with the top view illustration of observations and non-observations of 2 ,
• 4 FIG. 11 shows a top view illustration of observations and non-observations relative to a plurality of cells in accordance with aspects of the disclosure. FIG.
• 5 FIG. 12 is a plan view illustration of another example of observations and non-observations in accordance with aspects of the disclosure. FIG.
• 6 FIG. 12 is a plan view illustration of another example of observations and non-observations in accordance with aspects of the disclosure. FIG.
• 7 FIG. 10 is a plan view illustration of an exemplary focused observation using a bounding box in accordance with aspects of the disclosure. FIG.
• 8th FIG. 10 is a plan view illustration of exemplary obstacles in accordance with aspects of the disclosure. FIG.
• 9 FIG. 10 is a plan view illustration of an exemplary likelihood of obstacles affecting observations in accordance with aspects of the disclosure. FIG.
• 10 FIG. 12 illustrates an exemplary outputting of semantic searches using observations and non-observations in accordance with aspects of the disclosure. FIG.
• 11 FIG. 12 is a flowchart illustrating an exemplary method of determining object locations using observations and non-observations, in accordance with aspects of the disclosure. FIG.

DETAILED DESCRIPTION

Overview

The technology generally relates to image recognition and mapping. In particular, information from observations and non-observations is used to locate a particular object or feature. For example, to locate all fire hydrants within a geographic target region, image recognition may be performed on images of the region, with images including hydrants and images containing no hydrants used to more accurately calculate the locations of each hydrant. The non-observations near an observation significantly limit the range of locations where an object could be located. By using an image recognition tool and calculating multi-site probabilities, the location of the object can be determined relatively easily and with relatively high security, even if the images are not high-resolution and / or associated with low resolution or unreliable image capture information. This may enable object location identification also using non-professional content, e.g. B. from smartphones to execute. By using information for images that are not intended to capture the object, object location identification can be improved. In addition, this can be achieved without, or only with relatively low calculation costs.

An initialized value is determined for a set of candidate sites for the particular object or feature. For example, the initialized value for each candidate site may be set to zero, or, in a more sophisticated version, set to an earlier probability value prior to image recognition analysis that represents a likelihood that the object or feature exists at that site. For example, for each site prior to application of an image recognition tool, the probability that the site contains a fire hydrant may be x%, based on a number of fire hydrants in the target area and a size of the target area. The candidate locations can, for. A raster or grid division of the target region into cells, a rasterized map of the target region (eg, each pixel of the raster is a cell), or a continuous vector image. and gradient definition of regions.

For each image in a plurality of images, an image recognition tool is applied to the image to obtain a score or confidence rating that the respective object or feature is visible in the image. This score or confidence score may be converted to a normalized value indicating an amount that the likelihood that the object is in a region represented in the image has increased after the image recognition. For example, if the previous probability (before image recognition analysis) that the image contains a fire hydrant is x%, the probability of image recognition may increase from x% to (x + n)% or from (x%) to (xm)% decrease depending on whether or not a hydrant has been detected in the image. And after processing successive images, the previous probability may increase or decrease sequentially (in addition or otherwise) to a later probability and / or a normalized probability score. In some examples, a Log-Bayesian factor may be used. This may provide a particularly computationally efficient way of using both object observations and non-observations of the object to identify the location of the object.

Candidate locations contained in a field of view of each image are e.g. B. identified using camera features and pose. In some examples, the camera geometry and some standard horizon distances may define a sector that is compared to the initialized set of candidate sites to determine which candidate sites are included in the sector. In other examples, a falloff function may be used that represents a likelihood that an object will be visible in the image, if it actually exists, e.g. Considering a possible occlusion of the object in the image. The falloff function may not be radial and / or location dependent, e.g. Taking into account other factors such as local population density or vegetation density that would increase the risk of concealment. The use of information in conjunction with the orientation of an image capture device may allow for improved object location recognition. However, object location detection may be performed even if some or all of the images do not contain image capture alignment information.

A probability value can be calculated for each candidate location that is visible in the image based on the normalized value and the initialized value for each of these location candidates. For example, the normalized value may be added to or multiplied by the initialized value if a falloff function is used. In some examples, the likelihood value may be compared to a threshold at which the object may be determined to be at the candidate location corresponding to that likelihood score. In other examples, the probability score may be converted to a probability that the object or attribute exists at that candidate location. This may allow information for object location identification to be used, even if there is no absolute reliance on object identification by the image recognition tool.

Exemplary systems

1 illustrates an exemplary system for the semantic search of images. This should not be taken as limiting the scope of the disclosure or the usefulness of the features described herein. In this example, system 100 computer equipment 110 involve, over network 150 in communication with one or more client devices 160 . 170 as well as storage system 140 stand. Every computer device 110 can be one or more processors 120 , Random access memory 130 and other components typically found in general purpose computer devices. random access memory 130 from each one of computing device 110 can store information on the one or more processors 120 can be accessed, including instructions 134 that of the one or more processors 120 can be executed.

random access memory 130 can also data 132 which can be retrieved, edited or stored by the processor. The random access memory may be any non-volatile type that is capable of storing information that the one or more processors may access, such as, for example: As a hard disk, a memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory.

The instructions 134 can be any set of statements that directly, such as. As machine code, or indirectly, such as. For example, scripts to be executed by the one or more processors. In this regard, the terms "instructions,""application,""steps," and "programs" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by a processor or in any other computer device language that includes scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Functions, procedures and routines of the instructions are explained in more detail below.

dates 132 can be from the one or more processors 220 according to the instructions 134 be retrieved, saved or edited. For example, although the subject matter described herein is not limited by any particular data structure, the data may be stored in computer registers as a table having many different fields and records, or XML documents. The data may also be formatted in any computer-readable format such as, but not limited to, binary, ASCII, or Unicode. In addition, the data may include any information sufficient to identify the pertinent information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, such as memory. Other network locations, or include information used by a function to calculate the relevant data.

The one or more processors 120 can any conventional processor, such as. A commercially available CPU. Alternatively, the processor may be dedicated components such as An application-specific integrated circuit ("ASIC") or other hardware-based processor. Although not necessary, one or more of the computer devices 110 special hardware components to specific calculation methods, such. As image recognition, object recognition, image coding, tagging, etc. to execute.

Even though 1 the processor, memory and other elements of the computing device 110 As illustrated within the same block, the processor, computer, computing device, or memory may actually include multiple processors, computers, computing devices, or memory that may or may not be stored within the same physical enclosure. For example, the working memory may be a hard disk or other storage medium that is in a different housing than that of the computing devices 110 located. Accordingly, references to a processor, computer, computing device, or memory are understood to include references to a collection of processors, computers, computing devices, or memory that may or may not operate in parallel. For example, the computer equipment 110 Includes server computing devices that operate as a load-balanced server farm, distributed system, and so on. Furthermore, while some of the functions described below may be presented as occurring in a single computing device having a single processor, various aspects of the subject matter described herein may also be provided by a variety of computing devices, e.g. In the "cloud". Likewise, storage components at different locations may share different parts of the instructions 134 store and together form a medium for storing the instructions. Various operations described herein that are performed by a computing device may be performed by a virtual machine. For example, the statement 134 may be specific to a first type of server, but the corresponding operations may be performed by a second type of server running a hypervisor that emulates the first type of server. The operations can also be performed by a container, e.g. A computer environment that does not rely on an operating system that is bound to specific types of hardware.

Each of the computer devices 110 . 160 . 170 can connect to different nodes of a network 150 located and either directly or indirectly with other nodes of network 150 communicate. Although only a few computer devices in 1 It should be understood that a typical system may include a large number of connected computing devices, each different computing device at a different node of the network 150 located. The network 150 and intervening nodes described herein may be networked using various protocols and systems, such that the network may be part of the Internet, Word Wide Web, specific intranets, long distance networks, or local area networks. The network may utilize standard communication protocols such as Ethernet, WLAN and HTTP, protocols that are proprietary to one or more companies, and various combinations of the foregoing. Although certain advantages are achieved when transmitting or receiving information as described above, other aspects of the subject matter described herein are not limited to any particular type of information transfer.

For example, any of the computing devices 110 Web servers that are capable of containing the storage system 140 and computer devices 160 . 170 over the network 150 to communicate. For example, one or more server computing devices 110 network 150 use to transfer information and present it to a user on a display such as B. Display 165 from computer device 160 to represent. In this regard, computing devices 160 . 170 be considered as client computer devices and perform all or some of the functions described herein.

Each of the client computer devices 160 . 170 can, as described above, similar to the server computer devices 110 , be configured with one or more processors, memory and instructions. Any client computing device 160 . 170 may be a personal computing device that is intended for use by a user and that may include all of the components typically associated with a personal computing device, such as a personal computing device. A central processing unit (CPU), a random access memory (e.g., RAM and internal hard disks) that stores data and instructions, a display such as a display. B. Display 165 (eg a monitor with a screen, a touch screen, a projector, a TV or other device that can be used to display information), and an input device 166 (such as a mouse, keyboard, touch screen, or microphone). The client computing device may also have a camera 167 for recording video streams and / or capturing images, speakers, a network interface device, and any components used to interconnect these elements. The client computing device 160 can also be a location system, such. B. a GPS 168 , include. Other examples of location systems may determine the location based on the strength of the wireless access signal, images of geographic objects such as landmarks, semantic indicators such as light or noise levels, and so on.

Although the client computer devices 160 . 170 optionally, each comprise a full size personal computing device, they may alternatively include mobile computing devices that are capable of wirelessly exchanging data with a server over a network, e.g. As the Internet, are capable. For example, client computer device 160 a mobile phone or a device such as a wireless PDA, a tablet PC, a netbook, a smartwatch, a head mounted computer system, or any other device capable of receiving information over the Internet. For example, the user may enter information via a small keyboard, keypad, microphone, visual signals with a camera, or a touchscreen.

In other examples, one or more of the client devices may 160 . 170 primarily for input to the server 110 or memory 140 be used. For example, the client device 170 an image capture device that captures geographic images. For example, that can Image capture device 170 a still camera or video camera mounted on a vehicle to collect road images. As another example, the image capture device 170 Use sonar, LIDAR, radar, laser or other imaging techniques. In other examples, the device may 160 an audio recorder such. A microphone or other device for obtaining audio. In other examples, the image capture device may be a radio frequency transceiver or an electromagnetic field detector. In addition, the image capture device 170 be portable, or on a device such. As an unmanned aerial vehicle, are attached. Accordingly, the image capture device 170 Capture street images, aerial photographs or images at any angle.

As with memory 130 can the storage system 140 of any type of computerized storage that is capable of storing information to which the server computing devices 110 can access, such. Hard disk, memory card, ROM, RAM, DVD, CD-ROM, recordable and read-only memory. In addition, the storage system 140 include a distributed storage system, wherein data such. For example, data may be stored on a variety of different storage devices that may physically reside at the same or different geographic locations. storage system 140 can over the network 150 , as in 1 represented, connected to the computer devices and / or directly to one of the computer devices 110 be connected.

storage system 140 can data such. B. geographic images store. The geographic images can be used in conjunction with other data, such as As location data, are stored. For example, each geographic image may include metadata that identifies a location where the image was captured, camera angle, time, date, environmental conditions, and so forth. As another example, the geographic images, e.g. For example, by region, date or other information, categorized or grouped.

2 shows a plan view illustration of exemplary observations and non-observations of a particular object 205 , In this example captures an image capture device that is located along a road 210 moves pictures at each of the multiple detection locations 221 - 225 along the road 210 , Each image has an associated field of view 23 - 235 on. Based on a position and an angle of the image acquisition device at the detection location 221 For example, it captures images of everything in the field of view 231 located. Likewise, the image capture device would be at the subsequent capture location 223 Images of objects in field of view 233 to capture. In the search for the respective object 205 the fields of view apply 233 and 234 that the object 205 include, as observations of this object. fields of view 231 . 232 and 235 that the object 205 do not include, are considered non-observations.

Both the observations and the non-observations are used to locate the object 205 to be determined exactly. For example, the consideration helps where the object is 205 is not located to limit the possibilities where the object is located 205 located. As further explained below, determining the location of the object 205 using observations and non-observations, initialize a candidate set of recognition sites, such as By setting it to zero or setting it to a probability that the candidate location will be determined, with the probability prior to checking images. This determination may be based on available information, such as. Zoning, population density, non-visual signals, etc. Then each image is checked to get a rating as to whether the object is visible in the image. Using a normalization function, the confidence score is converted to a value approximately equal to the amount by which the quotas for reviewing the object that is in the field of view of the image exceed the previous quotas, depending on the observation of that image. On the basis of the camera properties and the pose, it is determined which candidate locations in this image are contained in the field of view. The candidate locations can, for. B. sectors, which are defined by the camera geometry and a given horizon distance. The normalized value is then added or multiplied to the initialized value for each of the candidate locations that are determined to be visible in the image. As a result, a probability score is created for each candidate site. The probability score may be compared to a threshold at which it is determined that the object is at the candidate location.

In some examples, greater precision may be achieved by using a falloff function such as A radial falloff function representing the likelihood that the object will be visible in each image when it is actually there. For example, the object in the image may be obscured by other objects such as trees, cars, fences, and so forth. Even greater accuracy can be achieved by using a non-radial or location-dependent falloff function. This feature may include other factors such as the local Account for population density or vegetation density that would increase the risk of In these examples, the result is not only a simple Boolean Yes / No for inclusion, but also a discount factor that reduces the likelihood of a false negative for reasons other than non-existence, such as the presence of a false negative. As occlusion reflects.

In addition to recognizing objects, these techniques can also be used to determine the exact location of attributes. So z. B. are searched for sunny geographical areas, z. B. for the real estate search. Accordingly, images representing a natural light level above a threshold may be identified as observations, and images representing a natural light level below the threshold may be identified as non-observations. Relevant timestamp information may also be considered, e.g. For example, by restricting the images you are looking for to images captured at a particular time of day (eg, during the day). The image acquisition information associated with the non-observations can be used to precisely locate the sunny areas.

3 Figure 11 illustrates an example of road images corresponding to the images taken by the detection sites 221 - 225 were recorded. The pictures contain a large number of single pictures 341 - 345 that overlap, resulting in an extended or panoramic image. A field of view of every picture 341 - 345 corresponds to the field of view 231 - 235 from image capture sites 221 - 225 , In this example, the object corresponds 205 the fire hydrant 305 , However, any other type of object or attribute may be searched, such as: B. Toilets 362 , Playground equipment 364 , Weather conditions 366 or a variety of other objects or attributes that are not displayed.

In some examples, regions of interest may be defined by a grid having a plurality of cells. For example 4 a top view of observations and non-observations with respect to a variety of cells 455 in a grid 450 each cell 455 can correspond to a subdivision of a geographic region, where each cell is generally the same in size and shape. In one example, the cells may be defined by latitude and longitude points.

The grid 450 can be used to find candidate sites for the object 205 define. So z. For example, prior to viewing an image, a probability for each cell containing the object 205 contains, be determined. Is this the object 205 z. For example, around a fire hydrant, a number of fire hydrants in the geographic region may be known by the number of cells 455 can be shared. For every picture that is along the street 210 is then determined whether the image points in a direction that includes a cell. For example, each image may be stored in conjunction with the image capture site and also a direction, e.g. In a direction in which the image capture device was directed. The direction can be determined by conventional coordinates, such. B. North, West, etc., in relation to other objects, such. Between First Street and Second Street, or any other mechanism. Locations of the cells 355 may be known so that a comparison can be made to determine if the image includes cells. Each cell can be treated as a discrete data point, positive or negative. In the example of 4 can any of the cells 455 as candidate location for the object 205 to be viewed as. An initial probability of each cell, including the object 205 , before viewing the images for each of the cells 455 be about the same size.

The pictures 341 - 345 can then z. B. using an image recognition tool to detect objects in the images. Based on the review, a score or trust score may be calculated for each candidate site, where the score or trust score indicates a likelihood that the object is within the candidate site. For each image, a normalization function can be used to convert that confidence score to a value. In some examples, normalization may match the previously calculated probabilities for each cell 455 include. For example, for each observation, the cells that are in the fields of view 233 . 234 These observations are included, with the likelihood that the object will be increased 205 located in this cell. Cells in fields of view 231 . 232 . 235 of non-observations can be proven with a reduced probability.

Some cells in a certain field of view can, for. Based on the position of the cell compared to a general position of the object 205 in the reviewed image, with a higher score or a higher confidence rating than others. For example, the hydrant 305 as in a left part of the picture 343 positioned shown. Accordingly, cells may be in a left part of the field of view 233 attributed a higher confidence than cells in a right part of the field of view 233 , In other examples, other camera properties and Pose information is used to calculate candidate site trust scores. For example, depth of field, focus or other information can be used.

In some examples, the confidence score may be increased or further refined when examining successive images. For example, if the picture 343 that the detection site 223 matches, first checked, every cell in field of view 233 an increased probability can be attributed to the object 205 take. If picture 344 according to the detection location 224 Subsequently, each cell can be field of view 234 an increased probability that they are the object 205 includes. Because that's the object 205 containing cells in both fields of view 233 . 234 These cells may result in a higher probability than others.

The normalized value indicating an amount indicating that the likelihood that the object is in an area displayed in the image has increased after the image recognition is then added or multiplied by the initialized value for each candidate location. The result for each candidate site can be compared to other results for other candidate sites. If the resulting value, e.g. B. by a predetermined numerical factor, for a first candidate location is relatively higher than the result for other locations, the first candidate location can be determined as the location of the object. In other examples, the result is compared to a threshold. In this context, a candidate site whose result value is above the threshold may be determined as a location for the object.

While in the above examples each image is stored with a location and a direction, in some examples objects can be located without the use of directional information. 5 shows a plan view illustration of such an example. Pictures in different locations 521 - 525 ma have corresponding fields of vision 531 - 535 on. In this example, one can assume that every field of view 531 - 535 a radius around its respective location 521 - 525 having. The candidate locations within such radial fields of view may cover the entire radius, or a portion thereof, such as e.g. As cells or other subdivisions include. The initialization and calculation of confidence values based on the image analysis may be performed as previously described.

6 represents another example in which directional information is not used for the images although images are acquired from different angles. In this example, a first road intersects 610 with a second street 615 , Pictures are taken from locations 621 . 622 on the second street 615 and of locations 623 - 625 captured on the second street. While the fields of view 631 - 635 are radial, accordingly, the positions of the detection sites 621 - 625 along different axes the accuracy of the localization of object 605 continue to increase. The object 605 is located z. B. as shown in the fields of view 632 and 633 but not in the fields of view 631 . 634 . 635 , The confidence score for candidate sites in the two fields of view 632 . 633 should be greater than the one calculated for other locations. Accordingly, the normalized value for such candidate sites should be within both fields of view 632 . 633 , the non-observations from adjacent fields of view 631 . 634 . 635 takes into account a high probability of the presence of the object.

Although in many of the above examples images are described as being taken from locations along a road, it will be understood that the images used to locate objects or attributes may be captured by any number of non-road locations. For example, images of users may include uploaded photos taken in parks, within buildings, or elsewhere.

7 illustrates another example in which a bounding box is used to reference the candidate sites 705 narrow. In this example, when reviewing image 743 that the field of view 733 corresponds to a bounding box 760 around the object of interest 705 drawn. In this example, the object is 705 a fire hydrant. The bounding box 760 corresponds to a narrower slice 762 of the field of view 733 , In this regard, candidate sites for which confidence score and normalized values are calculated may point to locations within the narrower slice 762 be limited. Accordingly, locations outside the disc 762 but still within the field of view 733 , regarded as non-observations and for more precise localization of the object 705 be used. In addition, the calculation can be accelerated since a reduced number of candidate sites can be analyzed.

As previously mentioned, the calculation of the normalized confidence score may be affected by occlusion, e.g. For example, if other objects are the view of an object of interest to impede a picture. 8th represents a top view illustration of the exemplary obstacles 871 - 873 ready to cause at least partial obscuration of the object 805 to lead. The obstacles 871 - 873 In this example, trees may be, though occluded by weather conditions such as fog or other objects such as humans, monuments, buildings, animals, etc. When calculating the confidence score for an image, the field of view 833 corresponds, such a cover can be taken into account. For example, a radial falloff function representing the probability that the object may be used 805 visible in this picture, if it actually exists. As another example, a non-radial or location-dependent falloff function may include factors such as: For example, consider other factors such as local population density or vegetation density that would increase the risk of concealment. Accordingly, the confidence score includes a discount factor that reduces the likelihood of false negatives for reasons other than non-existence, such as the absence of a false negative. As occlusion reflects.

According to some examples, the occlusion may occur as a result of the performance of an image recognition tool. For example, while there are no objects between the camera and the object, the image recognition tool may not recognize the object of interest, e.g. B. if the object is far away, small and / or shown only by a few pixels. Accordingly, the importance of observation or non-observation can be neglected when an object is far away, small, represented by a few pixels, and so on.

The falloff feature may be responsible for various types of occlusion or effects. So z. For example, when using the falloff function, different types of occlusion may be added or multiplied. Furthermore, the falloff function can be applied to observations other than non-observations. Non-observations can z. B. for masking, but not for observations are discounted. The falloff function can also be trained, eg. When a location of a particular object is known. If the locations of each fire hydrant in a particular city are known and images are available to the city, various calculations can be made with different falloff functions, and the function that produces a result closest to the known locations can be selected ,

The likelihood of occlusion may increase with increasing distance between an image capture site and an object of interest. 9 shows an example of the probability of obstacles affecting observations. object 905 is in a field of view 933 of an image by location 923 was recorded. The field of vision 933 is based on the probability of concealment in different regions 982 . 984 . 986 divided. For example, for all objects within the first region 982 that the detection site 923 closest, the probability of occlusion is lowest. For objects within the second region 984 , such as B. object 905 , the occlusion probability is higher than for the first region 982 , For the region 986 farthest from the image capture site 923 removed, the risk of concealment is greatest. In some examples, the probabilities may be adjusted using iterative training based on observations.

The computed location of objects of interest resulting from a semantic search can be used in any number of applications. For example, the locations may be used to populate a map to identify certain features, such as a map. B. locations of skate parks, dog parks, fire hydrants, street lamps, traffic signs, etc. represent. Locations can also be used by end users looking for specific destinations or trying to explore the landscape of a specific neighborhood. For example, a user who wants to buy a home may search for power plants or power lines near a potential home, and determine the exact location of such home-related objects. The potential homeowner may wish to conduct a more general search for areas that may be, for example: For example, they may appear as "rural", in which case localized objects or attributes such as trees or "green" may be output. A marketer may want to determine where certain industries are located to advertise these industries. If necessary, the marketer also wants to know which of his competitors advertise in a particular area. An employee of a utility company can search and locate all street lights. These are just a few of many possible application examples.

10 illustrates an exemplary output of semantic search using observations and non-observations. This example becomes a map 1000 updated to specify locations of specific objects. The object of interest in the semantic search can be z. B. be a fire hydrant. Accordingly, the markers 1012 . 1014 . 1016 . 1018 and 1020 that highlight the locations of the fire hydrants in the displayed region. Although the markers in 10 be shown, it is understood that Other indications, such. As symbols, texts, etc., can be used.

Exemplary procedure

Illustrated in addition to the exemplary systems described above 11 an exemplary procedure 1100 to accurately locate objects based on semantic image search. Those methods may be practiced using the systems described above, their modifications, or a variety of systems having different configurations. It should be understood that the operations contained in the following methods need not be performed in the precise order described. Rather, various operations may be processed in a different order or concurrently, and operations may be added or omitted.

In block 1110 Geographical images are identified that relate to a region of interest. For example, various images may be stored in association with information indicating a capture location for the images. If a user wants to search for objects in a particular neighborhood, city, country, or other region, images with detection location information within that region can be identified.

In block 1120 it is determined which of the identified images contain the object of interest. For example, each of the identified images, e.g. By applying an image recognition tool.

In block 1130 it is determined which of the identified images do not contain the object of interest. For example, when applying the image recognition tool in block 1120 Images in which the object of interest is not recognized, separately identified and / or marked.

Each one in the blocks 1120 and 1130 Identified images can include a variety of candidate locations. In block 1140 For each of the plurality of candidate sites, though not necessarily all, a probability of the object of interest present at the candidate site is calculated. The probability may be in block by using image acquisition location information for the images 1120 and the pictures in block 1130 be calculated. In some examples, the likelihood may also be obtained using directional or pose information and other information, such as directional information. As optical properties of the camera or factors that relate to possible occlusion calculated.

In block 1150 a location of the object of interest is determined using the probability value. For example, the probability value can be normalized, added to an initial value and compared to a threshold value. If the resulting value for a particular candidate site is above the threshold, the location of the object of interest may be determined to be the same as that of the candidate site.

By using observations and non-observations to locate a particular object or feature, the location of the object can be determined relatively easily and with relatively high security, even if the images are not high resolution and / or associated with low resolution or unreliable image acquisition information. This may enable object location identification also using non-professional content, e.g. B. from smartphones to execute. By using information for images that are not intended to capture the object, object location identification can be improved. In addition, this can be achieved without, or only with relatively low calculation costs.

Although the foregoing examples are described with respect to images, it will be understood that those images are conventional images, such as images. A collection of pixels, or frames from a video, video observations (e.g., a portion of a video), LIDAR imaging, radar, imaging, sonar imaging, or even "listening" observations such as audio or radio frequency reception , can act. Likewise, the image recognition performed in these examples may be video detection, LIDAR detection, audio detection, etc. Accordingly, observations and non-observations using these or other types of imaging may be used to precisely locate an object or attribute.

Unless otherwise stated, most of the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve characteristic advantages. While these and other variations and combinations of the features described above may be utilized without departing from the subject matter as defined by the claims, the foregoing description of the embodiments should be taken as illustrative rather than as an opportunity to address the subject matter defined by the claims To limit the invention. Moreover, the provision of the examples described herein, as well as terms formulated with terms such as "for example," "including," and the like, should not be interpreted as limiting the scope of the claims to these specific examples, as the examples serve this purpose to illustrate just one of the many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

Claims (20)

A computer-readable storage storing instructions executable by one or more processors to perform a method of determining a location of an object of interest, the method comprising: Identifying a set of many images from an image database relating to a region of interest, each of the plurality of images associated with image capture information including at least one image capture site; Applying an image recognition tool to each image in the set of many images; Determining, based on the application of the image recognition tool, which of the images include the object of interest and which of the images do not include the object of interest; for each of a plurality of candidate sites in the region of interest, calculating a probability value of the object of interest present at the candidate site using the image capture information for images in the set of many images determined to include the object of interest, and using the image capture information for images in the set of many images that are determined not to include the object of interest; and Determining the location of the object using the probability values for the plurality of candidate sites. Computer readable memory after Claim 1 wherein the image capture information for at least some of the images includes an image capture location and an image capture orientation, and wherein calculating the probability value of the object of interest present at a particular location comprises using the image capture orientation. Computer readable memory after Claim 1 wherein determining which of the images include an object of interest and which of the images do not include the object of interest, determining a confidence factor includes whether or not the images include the object of interest, and calculating the probability value of the one present at the location Object of interest using the specific confidence factor. Computer readable memory after Claim 1 wherein calculating the probability value of the object of interest at the candidate site comprises applying a factor having a first character to the images in the set of many images determined to include the object of interest and applying a factor a second character for the images in the set of many images that are determined not to include the object of interest, the second character being opposite the first character. Computer readable memory after Claim 1 wherein the plurality of candidate sites in the region of interest comprise a grid of multiple sites. Computer readable memory after Claim 1 wherein the plurality of candidate sites include locations included in a field of view of each image in the set of many images. Computer readable memory after Claim 1 where the image recognition tool is configured to recognize discrete objects. Computer readable memory after Claim 1 wherein the image recognition tool is configured to recognize objects having specific properties. Computer readable memory after Claim 1 where the image recognition tool is selected from a library of image recognition tools. A system comprising: a work memory storing a plurality of images in association with location information; and one or more processors communicating with the memory programmed to: identify a set of images from the plurality of images relating to a region of interest; Determining, using image recognition, which of the images include the object of interest; Determining, using image recognition, which of the images does not include the object of interest; for each of a plurality of candidate sites in the region of interest, calculate a probability value of the object present at the candidate site using the location information for images in the set of many images determined to include the object of interest, and using the location information for images in the set of images that are determined not to contain the object of interest; and determining the location of the object using the probability values for the plurality of candidate sites. System after Claim 10 wherein the location information includes image capture information. System after Claim 11 wherein the image capture information for at least some of the images includes an image capture location and an image capture orientation, and wherein calculating the probability value of the object of interest present at a particular location comprises using the image capture orientation. System after Claim 10 wherein determining which of the images include an object of interest and which of the images do not include the object of interest, determining a confidence factor includes whether or not the images include the object of interest, and calculating the probability value of the one present at the location Object of interest using the specific confidence factor. System after Claim 10 wherein calculating the probability value of the object of interest at the candidate site comprises applying a factor having a first character to the images in the set of many images determined to include the object of interest and applying a factor a second character for the images in the set of many images that are determined not to include the object of interest, the second character being opposite the first character. System after Claim 10 wherein the plurality of candidate sites in the region of interest comprise a grid of multiple sites. System after Claim 10 wherein the plurality of candidate sites include locations included in a field of view of each image in the set of images. System after Claim 10 wherein an image recognition tool is configured to recognize discrete objects. System after Claim 10 wherein the image recognition tool is configured to recognize objects having specific properties. System after Claim 10 wherein the one or more processors are further configured to provide the determined location information for output on a display. A computer-readable medium storing instructions executable by a processor to perform a method of determining the location of an object of interest, the method comprising: Identifying a set of many images from an image database relating to a region of interest, each of the plurality of images associated with image capture information including at least one image capture site; Applying an image recognition tool to each image in the set of images; Determining, based on the application of the image recognition tool, which of the images include the object of interest and which of the images do not include the object of interest; for each of a plurality of candidate sites in the region of interest, calculating a probability value of the object of interest present at the candidate site using the image capture information for images in the set of many images determined to include the object of interest, and using the image capture information for images in the set of many images that are determined not to include the object of interest; and Determining the location of the object using the probability values for the plurality of candidate sites.

Images