JP6397144B2 - Business discovery from images - Google Patents

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation of US patent application Ser. No. 14 / 821,128 filed Aug. 7, 2015, the disclosure of which is incorporated herein by reference.

  Numerous street-level photos based on the geographic location available today on the Internet provide a unique opportunity to detect and monitor man-made structures that help create accurate maps. Examples of such structures may include local businesses such as restaurants, clothing stores, gas stations, pharmacies, coin laundry. There is a high consumer interest in searching for such businesses through local related queries of popular search engines. Accurately identifying whether such a local business exists globally is not an easy task.

  Aspects of the present disclosure provide a method. The method comprises using one or more computing devices to train a deep neural network using a set of training images and data identifying one or more business store locations within the training images. A deep neural network outputting a first plurality of bounding boxes on each training image; and receiving the first image using one or more computing devices; Evaluating the first image using one or more computing devices and deep neural networks, and using one or more computing devices and deep neural networks in the first image Identify business locations And generating a second plurality of bounding boxes that.

  In one example, the method also includes detecting business information at each of the identified business store locations using one or more computing devices and a deep neural network, and the one or more computing devices. To update the business information database by adding information from each bounding box in the second plurality of bounding boxes to the business information detected at the business store location identified by the bounding box. Steps. In this example, the method also includes receiving a request from a user for business information using one or more computing devices, and updating using one or more computing devices. Retrieving the requested business information from the requested database.

  In another example, the second plurality of bounding boxes includes two bounding boxes arranged side by side in a first image that identifies two distinct business store locations. In one example, training a deep neural network includes applying a coarse sliding window to a portion of a given training image and one or more bounding boxes based on the location of the portion of the given training image. And a step of removing. In another example, generating the second plurality of bounding boxes also includes applying a coarse sliding window to a portion of the first image and 1 based on the location of the portion of the given training image. Removing one or more bounding boxes.

  In yet another example, training the deep neural network also includes determining a confidence score for each bounding box that represents the likelihood that the bounding box contains business store images, and less than a set threshold. Removing a bounding box corresponding to the bounding box having a confidence score. In a further example, generating the second plurality of bounding boxes also includes determining a confidence score for each bounding box that represents the likelihood that the bounding box includes an image of the business store, and a set threshold value. Removing a bounding box position corresponding to a bounding box having a confidence score of less than. In another example, training the deep neural network also includes using post-classification, and generating the second plurality of bounding boxes further includes using post-classification. Prepare.

  In a further example, generating the second plurality of bounding boxes also includes calculating a probability that the given bounding box includes a business store, and based on the calculated probability, the second plurality of bounding boxes. And removing one or more bounding boxes based on the ranking. In yet another example, generating the second plurality of bounding boxes also includes removing objects in the second plurality of bounding boxes that interfere with the prospect of identified business store locations. In another example, the training image and the first image are panoramas.

  Another aspect of the present disclosure provides a system. The system includes a deep neural network and one or more computing devices. One or more computing devices are training a deep neural network using a set of training images and data identifying one or more business store locations within the training images, where the deep neural network Output a first plurality of bounding boxes on each training image, train, receive the first image in the deep neural network, and evaluate the first image using the deep neural network And generating a second plurality of bounding boxes that identify the business store location in the first image using the deep neural network.

  In one example, one or more computing devices can also train a deep neural network by applying a coarse sliding window to a portion of a given training image and the location of a portion of the given training image. And removing one or more bounding boxes based on. In another example, the one or more computing devices also apply one or more coarse sliding windows to a portion of the first image and one or more based on the location of the portion of the given training image. Is configured to generate a second plurality of bounding boxes.

  In yet another example, one or more computing devices may also determine a confidence score for each bounding box that represents the likelihood that the bounding box will contain business store images, and less than a set threshold. The deep neural network is configured to be trained by removing bounding boxes corresponding to bounding boxes having a confidence score of. In a further example, the one or more computing devices may also determine a confidence score for each bounding box that represents the likelihood that the bounding box includes an image of the business store, and a confidence that is below a set threshold. The second plurality of bounding boxes are configured to be generated by removing bounding box positions corresponding to the bounding box having the score. In another example, one or more computing devices can also train a deep neural network by using posterior classification and generate a second plurality of bounding boxes by using posterior classification. It is comprised so that it may perform.

  In a further example, one or more computing devices may also calculate a probability that a given bounding box includes a business store and rank a second plurality of bounding boxes based on the calculated probability. And removing the one or more bounding boxes based on the ranking to generate a second plurality of bounding boxes. In yet another example, the one or more computing devices may also remove the second plurality of bounding boxes by removing objects in the second plurality of bounding boxes that impede visibility of the identified business store location. Is configured to generate

  A further aspect of the present disclosure provides a non-transitory tangible computer readable storage medium having stored thereon computer readable instructions of a program. The instructions, when executed by one or more computing devices, cause the one or more computing devices to perform the method. The method comprises training a deep neural network using a set of training images and data identifying one or more business store locations within the training image, wherein the deep neural network is on each training image. Outputting a first plurality of bounding boxes; receiving a first image in the deep neural network; evaluating the first image using the deep neural network; and deep neural network Using to generate a second plurality of bounding boxes that identify business store locations in the first image.

FIG. 3 is a functional diagram of an example system in accordance with aspects of the present disclosure. FIG. 2 is a pictorial diagram of the exemplary system of FIG. FIG. 4 is an exemplary diagram according to aspects of the disclosure. FIG. 4 is another exemplary diagram according to an aspect of the present disclosure. FIG. 6 illustrates exemplary inputs and outputs in accordance with aspects of the present disclosure. 4 is an exemplary flow diagram in accordance with aspects of the present disclosure.

Overview The present technology relates to automatically generating a bounding box that identifies different business stores in an image. In other words, a single convolution network evaluation can be used to predict multiple bounding boxes directly along with their confidence scores. By using convolutional neural networks and deep learning in post-classification, stores in a panoramic image can be identified with higher accuracy and speed than other methods. Accurate business segment detection and segmentation provides an opportunity to extract information about a specific business during post-processing. For example, text and images may be extracted to provide information about the identified business, and in some cases it may be used to more accurately determine the location of the business.

  Extracting arbitrary business stores from street-level photos is a difficult problem. Complexity is a high degree of intra-class variability in store appearance across business categories and regions, intrinsic ambiguity of store physical scale, businesses adjacent to each other in urban areas, and stores appearing around the world It comes from the scale itself. These factors make this an ambiguous task for human annotators. Image acquisition factors such as noise, subject blur, occlusion, illumination changes, specular reflection, perspective, and geolocation errors further contribute to the complexity of this problem. There are probably hundreds of millions of companies around the world and billions of street-level images. Given the size of this problem and the rate at which companies change, manual annotation is an outrageous task and not a sustainable solution. Automated techniques are very demanding on run-time efficiency to detect companies around the world in a reasonable time frame.

  Detecting business stores is the first and most important step in the multi-step process for extracting available business listings from images. Accurate store detection allows further downstream processing such as store geolocation, text OCR, business name and other attribute extraction, and categorization.

  A convolutional neural network may be used to detect business stores in the image. A convolutional network is a neural network that includes a set of nodes with coupled parameters. Combining increased size of available training data and availability of computational power with algorithmic advances such as piecewise linear units and dropout training results in significant improvements in many computer vision tasks. For large data sets that are available today for many tasks, overfitting is not a problem, and increasing the size of the network improves test accuracy. Optimal use of computing resources becomes a limiting factor. For this purpose, a distributed and scalable implementation of deep neural networks may be used.

  Conventionally, object detection is performed by exhaustively searching for objects of interest in the image. Such an approach generates a probability map corresponding to the presence of the object at that location. A post-processing of this probability map by either non-maximum suppression or an average shift based approach is then generated to produce discrete detection results. To counter the computational complexity of exhaustive search, selective search that uses image segmentation techniques to generate several suggestions may significantly reduce the number of parameters to search.

  The technique disclosed herein uses a deep neural network to assign a final detection score and employs a fully learned approach from pixels to discrete bounding boxes. The end-to-end learned approach has the advantage of integrating proposal generation and post-processing using a single network to simultaneously predict multiple proposals and trust. Relying solely on the confidence output of this approach may produce high quality results, but it may be further improved by implementing a special dedicated post-classifier network for the most reliable proposals. Good. Even with this additional post-classification step, this technique can be orders of magnitude faster than the previous generation.

  Data that identifies a set of training images and one or more business store locations may be used to train a deep neural network. By using the training image, the deep neural network may output the first plurality of bounding boxes along with their respective confidence scores. The confidence score for each bounding box may represent the likelihood that the bounding box includes an image of the business store. Each bounding box may be matched with a business store location. In training, training images may be evaluated using a coarse sliding window, also called multi-crop evaluation. In order to further train the deep neural network, post-classification may be applied to improve the results of the multi-crop evaluation. Subsequent classification may include calculating the probability that a given bounding box includes a business store.

  A trained deep neural network may receive an image to be evaluated. Image features may be identified and evaluated using multi-crop evaluation and post-classification. Based on the evaluation, the deep neural network may generate a second plurality of bounding boxes that identify possible business store locations, and each bounding box may include an image of only one business store.

Exemplary System FIGS. 1 and 2 include an exemplary system 100 in which the features described above can be implemented. This should not be considered as limiting the scope of the present disclosure or the usefulness of the features described herein. In this example, system 100 can include one or more computing devices 110, 120, 130, and 140 and storage system 150. Each of the computing devices 110 may include one or more processors 112, memory 114, and other components that typically reside in general purpose computing devices. The memory 114 of the computing device 110 may store information accessible by the one or more processors 112, including instructions 116 executable by the one or more processors 112.

  The memory 114 can also include data 118 that can be retrieved, manipulated, or stored by the processor. The data may include an image. The image may be a panoramic image or an image having a field of view larger than 180 degrees, for example, a maximum of 360 degrees. Further, the panoramic image may be spherical or nearly spherical. The image may represent various business stores associated with information regarding the location of each business store in each image. This information can identify a range of pixels that represent a single business store, for example, some images may include a bounding box located on an image where each business store is located. Some of these images may be identified as training images. Other images that are not associated with information about the location of the business store may also be stored in the memory. The data may also include the geolocation of each business store. Memory can be any non-transitory type that can store information accessible by the processor, such as hard drives, memory cards, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory. Can be a thing.

  The instructions 116 may be any set of instructions that are executed directly, such as machine code, or indirectly, such as a script, by one or more processors. In this regard, the terms “instructions”, “applications”, “steps”, and “programs” may be used interchangeably herein. The instructions are in object code form for direct processing by the processor, or any other computing device that includes a set of scripts or independent source code modules that are interpreted on demand or pre-compiled It may be stored in a language. The instructions may include instructions that cause one or more computing devices, such as computing device 110, to operate as a deep neural network. Instruction functions, methods, and routines are described in more detail below.

  Data 118 may be retrieved, stored, or modified by one or more processors 112 in accordance with instructions 116. For example, the subject matter described herein is not limited by any particular data structure, but data can be stored in a relational database as a table with many different fields and records, or XML documents, in computer registers. . The data may also be formatted in any computing device readable format such as, but not limited to, binary values, ASCII, or Unicode. In addition, data can be related, such as numbers, descriptive text, dedicated codes, pointers, references to data stored in other memories such as network locations, or information used by functions to calculate related data Any information sufficient to identify the information can be provided.

  The one or more processors 112 may be any conventional processor such as a commercially available CPU. Alternatively, the processor may be a dedicated component such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, one or more of the computing devices 110 may make certain computing processes faster, such as video decoding, video frame and image matching, video transformation, and modified video encoding. Or it may include specialized hardware components for more efficient execution.

  Although FIG. 1 functionally illustrates the processor, memory, and other elements of the computing device 110 as being in the same block, the processor, computer, computing device, or memory may actually be the same physical A plurality of processors, computers, computing devices, or memory may or may not be housed in a typical enclosure. For example, the memory may be a hard drive arranged in a case different from the case of the computing device 110, or may be another storage medium. Thus, a reference to a processor, computer, computing device or memory is understood to include a reference to a collection of processors, computers, computing devices or memory that may or may not operate in parallel. The For example, the computing device 110 may include a server computing device that operates as a load balancing server farm. Moreover, although some functions described below are illustrated as being performed on a single computing device having a single processor, various aspects of the subject matter described herein are not limited to May be implemented by multiple computing devices that communicate information over a network 160, for example.

  Each of the computing devices 110 may be at different nodes of the network 160 and can communicate directly and indirectly with other nodes of the network 160. Although only a few computing devices are shown in FIGS. 1-2, a typical system can include a number of connected computing devices, each of the different computing devices being different on the network 160. It should be understood that it is in the node. The network 160 may be a deep neural network that uses multiple layers of the model, and the lower layer outputs are used to build higher level layer outputs. The network 160 and intervening nodes described herein interact with each other using various protocols and systems so that the network can be part of the Internet, the World Wide Web, a specific intranet, a wide area network, or a local network. Can be connected. The network can utilize standard communication protocols such as Ethernet, WiFi, and HTTP, protocols that are proprietary to one or more companies, and various combinations of the foregoing. Although certain advantages are obtained when information is transmitted or received as described above, other aspects of the subject matter described herein are not limited to any particular method of transmitting information.

  As an example, each of the computing devices 110 may include a web server that can communicate with the storage system 150 and the computing devices 120, 130, and 140 via a network. For example, one or more of the server computing devices 110 send information to a user such as a user 220, 230, or 240 on a display such as the display 122, 132, or 142 of the computing device 120, 130, or 140 And network 160 may be used to present and present. In this regard, computing devices 120, 130, and 140 may be considered client computing devices and may perform all or part of the functionality described herein.

  Each of the client computing devices may be configured similarly to the server computing device 110 that includes one or more processors, memory, and instructions as described above. Each client computing device 120, 130, or 140 may be a personal computing device intended for use by users 220, 230, 240, such as a central processing unit (CPU), memory that stores data and instructions (e.g., RAM and internal hard drive), displays such as displays 122, 132, or 142 (e.g., screens, touch screens, projectors, televisions, or monitors with other devices operable to display information), and users It has all the components normally used in connection with a personal computing device such as an input device 124 (eg, a mouse, keyboard, touch screen, or microphone). The client computing device also includes a camera 126 for capturing still images or recording video streams, speakers, network interface devices, and all components used to connect these elements to each other. But you can.

  Client computing devices 120, 130, and 140 may each comprise a full-size personal computing device, but they can instead exchange data wirelessly with a server over a network such as the Internet. A device may be provided. By way of example only, the client computing device 120 may be a mobile phone or a device such as a wireless enabled PDA, tablet PC, or netbook that can obtain information over the Internet. In another example, client computing device 130 may be a head mounted computing system. As an example, a user may enter information using a small keyboard, keypad, microphone, using visual signals from a camera, or using a touch screen.

  Similar to memory 114, storage system 150 stores information accessible by server computing device 110, such as hard drives, memory cards, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory. It can be any type of computerized storage that can. Further, the storage system 150 may include a distributed storage system in which data is stored on a plurality of different storage devices that may be physically located at the same or different geographic locations. Storage system 150 may be connected to computing devices via network 160 as shown in FIG. 1 and / or directly connected to the memory of any of computing devices 110-140 (not shown). Or may be incorporated into it.

  The storage system 150 may also store images. These images may include various types of images, such as panoramic images that represent one or more business stores, or images that have a field of view greater than 180 degrees, for example, up to 360 degrees. In some examples, a given image may be associated with store information that identifies the location of each business store within the given image. For example, store information for a given image may be stored in a given image corresponding to one or more stores and / or image coordinates corresponding to the shape of one or more business stores in the given image. It may include a range of pixels. As an example, the store information may be represented by a bounding box corresponding to each business store position in the image. As described below, at least some of the images may be identified as training images. The storage system 150 may also include geolocation information for some business stores, or information about the geographic location.

Exemplary Method As shown in FIG. 3, the deep neural network 310 may be trained by using a set of training images 320. These training images 320 may include images of the storage system 150 associated with store information 330 that identifies one or more business store locations within the set of training images. As described above, the store information may be a range of pixels of an image that represents one or more business stores located within the associated image.

  Deep neural network 310 may be used to evaluate the training image using business store location data for the location of the business store in the training image. A coarse sliding window, also called multi-crop evaluation, may be applied to an image using a deep neural network. The position of each window may be considered the “crop” of the image. Compared to the high-density sliding window method, the coarse sliding window method reduces the number of sliding windows by several orders of magnitude. For example, a coarse sliding window may evaluate 100 windows for a full 360 degree panoramic image instead of 300,000 windows that are likely to be used by a high density sliding window. Although a single crop evaluation may work, high resolution panoramic images cannot reliably detect smaller stores from a low resolution version of a single panoramic image. Therefore, the detection quality of the store can be actually improved by using a rough sliding window.

  During training of the deep neural network 310, a first plurality of bounding boxes 340 superimposed on the image may be identified. The bounding box may be a rectangle on the image that identifies a portion of the image in the image. The bounding box may also be any other polygon or shape. The shape and size of each bounding box may depend on the shape of each business store location.

  Each bounding box 340 may be matched with a business store location based on data associated with each image. Only one business store location may be surrounded by a single bounding box so that business store locations directly adjacent to each other in the image are defined by separate bounding boxes. Matching may include maximum weight matching where the edge weight between the business store location and a given bounding box is related to the amount of box overlap. For example, the edge weight may be a Jacquard similarity coefficient defined as the size of an intersection divided by the size of the union of a given bounding box and business store location.

  For the set of training images evaluated by the deep neural network, the deep neural network may be used to determine the coordinates of each bounding box of the first plurality of bounding boxes. The coordinates may be image coordinates such as image coordinates corresponding to the business store position. The image coordinates may define the position of the bounding box using a coordinate system relative to the image itself, or may be latitude / longitude coordinates or any other geolocation coordinates.

  A confidence score 350 may be calculated for each bounding box 340. The confidence score 350 for each bounding box of the first plurality of bounding boxes may represent the likelihood that the bounding box includes an image of the business store.

  When evaluating training images, the bounding box may be removed under certain circumstances. For example, bounding boxes that have a confidence score below a set threshold may be removed. Further, a bounding box adjacent to one of the crop edges of the image may be removed as long as the crop edges are not image edges. In this way, the bounding box that does not completely contain the object can be removed, and the business store can be detected more accurately. Furthermore, any bounding box that is larger than a given distance from any given crop may be removed as well.

  Deep neural network 310 may also be trained by post classification. An affine transformation may be applied to the receptive field of the deep neural network when preparing for post classification. In the post-classification, the first plurality of bounding boxes identified from the plurality of crop evaluations are further classified in order to narrow down the results. In other words, another classifier is applied to the result to increase the confidence that each bounding box includes the business store location. For example, a second confidence score may be calculated for each bounding box. The probability that a given bounding box includes a business store may be calculated based on the calculated confidence score. This probability may be calculated by summing the product of the confidence score of each bounding box in the deep neural network and the confidence score of each bounding box in the post classification. Alternatively, the probability may be calculated by multiplying the confidence score in the deep neural network by the posterior classification for a given bounding box.

  The probability may be used to filter the first plurality of bounding boxes by removing bounding boxes associated with a probability that is lower than a set threshold. This probability may also be used to rank the bounding box. The first plurality of bounding boxes may be filtered based on the ranking. For example, bounding boxes ranked below a set number may be removed.

  While being trained, the deep neural network may output a first plurality of bounding boxes 340 on the training image having a respective confidence score 350. Coordinates determined for each bounding box and probabilities calculated by posterior classification may be included in the output of the network. This information may be stored in the storage system 150 for later use.

  After being trained, the deep neural network 310 may evaluate one or more images 420 as shown in FIG. Similar to training image 320, image 420 may also be stored in storage system 150. However, unlike the training image, the image 420 may not be associated with data identifying the business store location in the image. The image 420 may be a panoramic image or an image having a field of view larger than 180 degrees, for example, a maximum of 360 degrees. Further, the panoramic image may be spherical or nearly spherical. Detection in the panorama avoids the loss of recall that can result in images with a smaller field of view. By using the deep neural network 310, the image 420 may be evaluated to identify image features. As described above with respect to training, evaluation may include multi-crop evaluation and post classification.

  Based on the evaluation, the deep neural network 310 may generate a second plurality of bounding boxes 440 that identify possible business store locations in the image, as shown in FIG. Each bounding box 440 may include an image of only one business store. Accordingly, the rows of adjacent business stores may be segmented by a plurality of bounding boxes surrounding each business store in the row. Further, each bounding box 440 may be associated with a confidence score 450 representing the likelihood that each bounding box includes an image of the business store.

  As shown in FIG. 5, the image 510 may be evaluated by the deep neural network 310. As a result, the deep neural network 310 may identify a plurality of bounding boxes including 522, 524, 526, and 528, each including an image of an individual business store. Images 524, 526, and 528 identify adjacent business stores as individual stores.

  In some examples, the second plurality of bounding boxes 440 may be filtered by removing bounding boxes that have a confidence score below a set threshold. Additionally or alternatively, bounding boxes in multiple images associated with the same or similar geographic location (or in other words, the same business store) may be merged. Merging these bounding boxes may include removing objects that are false positives. An example of a false positive is a vehicle that is temporarily stopped in front of a business store. The filtered bounding box may then be associated with the evaluated image and stored in the storage system 150 for later use.

  Business information in each bounding box 440 may be detected. Known methods of information extraction such as optical character recognition may be used. The detected business information may include names, words, logos, merchandise, or other items that are visible within a particular bounding box. The business information may then be added to the business information database. The database may be stored in the storage system 150 for later use.

  Users 220, 230, 240 may request business information using computing devices 120, 130, 140. In response to the user request, business information may be retrieved by the computing device 110 from a database in the storage system 150 and sent to the computing devices 120, 130, 140.

  FIG. 6 is an exemplary flowchart 600 according to some of the above-described aspects that may be performed in the deep neural network 310. However, the described features may be implemented by any of a variety of systems having different configurations. Further, the actions involved in the method need not be performed in the exact order described. Rather, the various operations may be processed in different orders or simultaneously, and operations may be added or omitted.

  At block 610, the deep neural network may be trained using a set of training images and data identifying one or more business store locations within the training images. At block 620, a first image may be received at the deep neural network. The first image may be evaluated by the deep neural network at block 630. At block 640, a set of two or more bounding boxes that identify business store locations in the first image may be generated.

  Using the functions described above, stores in a large database of images can be identified with speed and accuracy that cannot be achieved using other methods. Specifically, this feature allows the bounding box to be a direct output of the analysis without using intermediate outputs such as heat maps and probability maps that require further analysis and / or processing. In addition, this feature allows adjacent business stores to be appropriately segmented instead of being identified as one store. Compared to using selective search to generate bounding boxes around store images, the computational cost for using the above method is much lower and the speed is much faster. Using the method described above requires extensive post-processing to convert the heat map into a meaningful bounding box, rather than using a trained heat map approach that is sensitive to label noise. It may be efficient. The described method of training and using deep convolutional neural networks automates what would require significant effort to produce while producing accurate results. After the bounding box is generated, the image may be used to automatically extract available business listings, more accurate store geolocation, and more information available in the image.

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, many modifications may be made to an exemplary embodiment, and other configurations may be devised without departing from the spirit and scope of the invention as defined by the appended claims. Please understand that it is good.

100 system
110 Computing devices, server computing devices
112 processor
114 memory
116 instructions
118 data
120 computing devices, client computing devices
122 display
124 User input devices
126 camera
130 Computing devices, client computing devices
132 displays
140 Computing devices
142 display
150 storage systems
160 network
220 users
230 users
240 users
310 deep neural network
320 training images
330 Business location, data
340 bounding box
350 confidence score
420 images
440 bounding box
450 confidence score
510 images
522 Bounding box
524 bounding box
526 bounding box
528 bounding box
600 Flow chart

Claims (20)

Training a deep neural network using a set of training images and data identifying one or more business store locations within the training images using one or more computing devices, comprising: The deep neural network outputs a first plurality of bounding boxes on each training image; and
Receiving a first image using the one or more computing devices;
Evaluating the first image using the one or more computing devices and the deep neural network;
Generating a second plurality of bounding boxes identifying two or more business store locations in the first image using the one or more computing devices and the deep neural network. ,Method. Detecting business information at each of the identified business store locations using the one or more computing devices and the deep neural network;
Using the one or more computing devices, the information from each bounding box in the second plurality of bounding boxes is converted into business information detected at the business store location identified by the bounding box. Updating the business information database by adding;
Receiving a request from a user for business information using the one or more computing devices;
2. The method of claim 1, further comprising: using the one or more computing devices to retrieve the requested business information from the updated database.   The method of claim 1, wherein the second plurality of bounding boxes includes two bounding boxes arranged side by side in the first image that identifies two distinct business store locations. Training the deep neural network comprises
Applying a coarse sliding window to a portion of a given training image;
The method of claim 1, further comprising: removing one or more bounding boxes based on the position of the portion of the given training image. Generating the second plurality of bounding boxes;
Applying a coarse sliding window to a portion of the first image;
The method of claim 1, further comprising: removing one or more bounding boxes based on the position of the portion of the first image. Training the deep neural network comprises
Determining a confidence score for each bounding box that represents the likelihood that the bounding box includes an image of a business store;
The method of claim 1, further comprising: removing a bounding box corresponding to a bounding box having a confidence score that is less than a set threshold. Generating the second plurality of bounding boxes;
Determining a confidence score for each bounding box that represents the likelihood that the bounding box includes an image of a business store;
The method of claim 1, further comprising: removing a bounding box position corresponding to a bounding box having a confidence score less than a set threshold. Training the deep neural network further comprises using post-classification;
Generating the second plurality of bounding boxes further comprises using posterior classification;
The method of claim 1. Generating the second plurality of bounding boxes;
Calculating the probability that a given bounding box includes a business store;
Ranking the second plurality of bounding boxes based on the calculated probabilities;
The method of claim 1, further comprising: removing one or more bounding boxes based on the ranking.   The method of claim 1, wherein generating the second plurality of bounding boxes further comprises removing objects in the second plurality of bounding boxes that impede visibility of the identified business store location. Method.   The method of claim 1, wherein the training image and the first image are panoramic. Deep neural network,
One or more computing devices,
Training the deep neural network using a set of training images and data identifying one or more business store locations within the training images, wherein the deep neural network includes a first on each training image; Output multiple bounding boxes in one, training,
Receiving a first image in the deep neural network;
Using the deep neural network to evaluate the first image;
One or more computing devices configured to use the deep neural network to generate a second plurality of bounding boxes that identify business store locations in the first image; A system comprising: The one or more computing devices are:
Applying a coarse sliding window to a portion of a given training image;
The system of claim 12, further configured to train the deep neural network by removing one or more bounding boxes based on the position of the portion of the given training image. . The one or more computing devices are:
Applying a coarse sliding window to a portion of the first image;
13. The apparatus of claim 12, further configured to generate the second plurality of bounding boxes by removing one or more bounding boxes based on the position of the portion of the first image. The system described. The one or more computing devices are:
Determining a confidence score for each bounding box representing the likelihood that the bounding box includes an image of a business store;
The system of claim 12, further configured to train the deep neural network by removing bounding boxes corresponding to bounding boxes having confidence scores below a set threshold. The one or more computing devices are:
Determining a confidence score for each bounding box representing the likelihood that the bounding box includes an image of a business store;
13. The apparatus of claim 12, further configured to generate the second plurality of bounding boxes by removing bounding box positions corresponding to bounding boxes having a confidence score less than a set threshold. System. The one or more computing devices are:
Training the deep neural network by using post-classification;
13. The system of claim 12, further configured to generate the second plurality of bounding boxes by using post classification. The one or more computing devices are:
Calculating the probability that a given bounding box contains a business store;
Ranking the second plurality of bounding boxes based on the calculated probabilities;
13. The system of claim 12, further configured to generate the second plurality of bounding boxes by removing one or more bounding boxes based on the ranking.   The one or more computing devices generate the second plurality of bounding boxes by removing objects in the second plurality of bounding boxes that impede visibility of the identified business store location. The system of claim 12, further configured as follows. A non-transitory tangible computer readable storage medium having stored thereon computer readable instructions of a program, wherein the instructions are executed by one or more computing devices when the instructions are executed by the one or more computing devices. And the method comprises
Training a deep neural network using a set of training images and data identifying one or more business store locations within the training image, wherein the deep neural network includes a first on each training image. Outputting a plurality of bounding boxes
Receiving a first image in the deep neural network;
Using the deep neural network to evaluate the first image;
Using the deep neural network to generate a second plurality of bounding boxes that identify business store locations in the first image.

Images