JP2016500885A - System and method for merging multiple maps for computer vision based tracking - Google Patents

Abstract

A method, apparatus, and computer program product for merging multiple maps for computer vision based tracking are disclosed. In one embodiment, a method for merging a plurality of maps for computer vision-based tracking includes receiving a plurality of maps of a scene at a location from at least one mobile device; and a plurality of maps of the scenes Identifying a key frame, and merging the plurality of key frames to generate a global map of the scene.

Description

This application claims the benefit of US Application No. 13 / 830,664, filed March 14, 2013, entitled `` Systems and Methods of Merging Multiple Maps for Computer Vision Based Mobile Device Tracking ''. This application claims the benefit of US Provisional Application No. 61 / 720,804, filed Oct. 31, 2012, entitled “Systems and Methods of Merging Multiple Maps for Computer Vision Based Mobile Device Tracking”. The aforementioned US application is incorporated herein by reference in its entirety.

  The present disclosure relates to the fields of wireless communications, computer vision, and augmented reality. In particular, this disclosure relates to a system and method for merging multiple maps for computer vision-based tracking.

  A number of conventional methods are available for generating an entire map from multiple submaps. These conventional methods store redundant submaps in memory to support merging of multiple submaps. Such redundant submaps may have a low contribution to the overall map quality. However, these redundant submaps can consume considerable storage. In addition, valuable computing resources and time may be wasted processing such redundant submaps.

  The present disclosure relates to systems and methods for merging multiple maps for computer vision based tracking. In one embodiment, a method for merging a plurality of maps for computer vision-based tracking includes receiving a plurality of maps of a scene at a location from at least one mobile device; and a plurality of maps of the scenes Identifying a key frame, and merging the plurality of key frames to generate a global map of the scene. The method further comprises tracking the location of at least one mobile device at the location.

  A method for receiving a plurality of maps of a scene at a location comprises receiving a plurality of maps of the scene from a plurality of mobile devices. The method further comprises evaluating redundancy of a plurality of key frames from a plurality of maps of a scene from a plurality of mobile devices, and deleting the redundant key frames. The method includes generating a self-localization and simultaneous map construction (SLAM) map using multiple maps of a scene from multiple mobile devices and sharing the SLAM map between the multiple mobile devices. And further comprising.

  A method of identifying a plurality of key frames of a plurality of maps of a scene includes: identifying a plurality of key frames representing a plurality of viewpoints of the scene from a plurality of angles; At least one of identifying a key frame, the plurality of key frames including common features, position coordinates, and appearance of the plurality of maps. The plurality of scales comprises at least one of a plurality of distances between the at least one mobile device and the scene and a plurality of aspect ratios of the scene from the at least one mobile device.

  The method of merging multiple keyframes to generate a global map of a scene determines a contribution score based at least in part on keyframe angular diversity and keyframe scale diversity for each keyframe of the multiple keyframes And deleting a key frame in response to a contribution score less than a predetermined threshold. A method for determining a contribution score based at least in part on keyframe angle diversity and scale diversity calculates an angle diversity scoring criterion by applying a first weighting factor to the keyframe angle diversity; Calculating a scale diversity scoring reference value by applying a second weighting factor to the scale diversity of the keyframe, and calculating an angle diversity scoring reference value and a scale diversity scoring reference value to generate a contribution score. Combining steps.

  A method of merging a plurality of key frames to generate a global map of a scene includes determining a contribution score based at least in part on the key frame angular diversity for each key frame of the plurality of key frames; Deleting a key frame in response to a contribution score less than the threshold. A method of merging a plurality of key frames to generate a global map of a scene includes determining a contribution score based at least in part on the key frame scale diversity for each key frame of the plurality of key frames; Deleting a key frame in response to a contribution score less than the threshold.

  The method of merging multiple keyframes to generate a global map of a scene further comprises counting the multiple keyframes in an angle histogram, the angle histogram representing different viewing angles of the scene. It includes a plurality of bins, each bin of the plurality of bins including a plurality of sections that represent different viewing scales of the scene. The method includes determining, for each keyframe of the plurality of keyframes in the angle histogram, a contribution score based at least in part on the keyframe angle diversity and the keyframe scale diversity, and a contribution less than a predetermined threshold And deleting a key frame according to the score.

  A method for determining a contribution score based at least in part on the angular diversity and scale diversity of a key frame includes a first weight factor based on at least a part of the number of key frames in the bin containing the key frame. Calculating an angle diversity scoring criterion by applying to angle diversity, and a second weighting factor based on the number of keyframes in the partition containing the keyframe to the keyframe scale diversity Calculating a scale diversity scoring reference value by applying and combining the angle diversity scoring reference value and the scale diversity scoring reference value to generate a contribution score.

  In another embodiment, an apparatus is configured to identify logic configured to receive a plurality of maps of a scene at a location from at least one mobile device and a plurality of key frames of the maps of the scene. And logic configured to merge a plurality of key frames to generate a global map of the scene.

  In yet another embodiment, a computer program product comprises a non-transitory medium that stores instructions executed by one or more computer systems. These instructions include instructions for receiving multiple maps of a scene at a location from at least one mobile device, instructions for identifying multiple keyframes of the multiple maps of the scene, and a global map of the scene Instructions for merging a plurality of keyframes for generation.

  In yet another embodiment, the system includes means for receiving a plurality of maps of a scene at a location from at least one mobile device; means for identifying a plurality of key frames of the maps of the scene; Means for merging a plurality of key frames to generate a global map of the scene.

  The foregoing features and advantages of the present disclosure, as well as additional features and advantages thereof, will be more clearly understood when the detailed description of the embodiments of the present disclosure is read in conjunction with the following drawings.

FIG. 3 illustrates an example location in accordance with certain aspects of the present disclosure. FIG. 6 illustrates a method for recording multiple viewpoints of a scene from multiple angles according to some aspects of the present disclosure. FIG. 6 illustrates a method for recording multiple viewpoints of a scene from multiple distances according to some aspects of the present disclosure. FIG. 6 illustrates a method for recording multiple viewpoints of a scene with multiple aspect ratios according to some aspects of the present disclosure. FIG. 6 illustrates an example apparatus for performing a method for merging multiple maps for computer vision-based tracking in accordance with certain aspects of the present disclosure. 6 is an exemplary flowchart for implementing a method of merging multiple maps for computer vision based tracking according to some aspects of the present disclosure. FIG. 6 is an exemplary block diagram of a mobile device in accordance with certain aspects of the present disclosure. FIG. 6 illustrates an example method for determining key frame angular diversity and scale diversity in accordance with certain aspects of the present disclosure. FIG. 4 illustrates another exemplary method for determining key frame angular diversity and scale diversity in accordance with certain aspects of the present disclosure.

  Similar numbers are used throughout the drawings.

  Embodiments for merging multiple maps for computer vision based tracking are disclosed. The following description is presented to enable any person skilled in the art to make and use the present disclosure. Descriptions of specific embodiments and applications are provided as examples only. Various modifications and combinations of the examples described herein will be readily apparent to those skilled in the art, and the generic principles defined herein may be used in other examples and applications without departing from the spirit and scope of the disclosure. Applicable to. Accordingly, this disclosure is not intended to be limited to the examples described and shown, but is intended to be accorded the maximum scope consistent with the principles and features disclosed herein.

  According to some aspects of the present disclosure, a self-localization and simultaneous map construction (SLAM) based application is observed by at least one mobile device to track the location of at least one mobile device in an environment or location Build and maintain information about the environment or location This information includes, but is not limited to, feature structure (3D position) and appearance, which may also be referred to as an environment map. Techniques for map construction can be beneficial to support different use cases in different environments or locations. For example, building a global map of a large location using multiple mobile devices can be useful for tracking the location of multiple mobile devices in the environment. It should be noted that using multiple mobile devices may be preferable for building a global map as opposed to building a global map using a single mobile device.

  In some implementations, these global maps may be shared by multiple mobile devices in the environment when performing self-location estimation. If there are multiple maps of the scene from different mobile devices, one approach is to merge these maps to obtain a global map. According to aspects of the present disclosure, the global map can be generated in real time or can be generated offline at the server. Another approach is to eliminate redundancy while simultaneously merging multiple maps can result in smaller global map sizes and lower tracking times using global maps. Methods for generating merge maps that represent information about the environment or location in an efficient manner are described in the following sections.

  According to some aspects of the present disclosure, a map of a scene may be represented as key frames that are images recorded from various viewpoints. Each keyframe may include a set of images in the scene and observations of features of 3D features (points, lines, etc.) (location coordinates and appearance of points / lines). In some implementations, past observations and key frames are stored and used to refine the geometry with new incoming observations or constraints. One approach is to store feature observations in key frames. Keyframes in the map can record observations of scene features at various viewpoints. In addition, key frames can provide observations of various parts of the scene that have minimal redundancy in the stored information. Feature tracking using appearances has observations of features from multiple viewpoints, as described in connection with FIG. 2, and multiple features as described in connection with FIGS. 3 and 4. It is useful to have observations of features on a scale. Such a feature tracking method is useful for improving tracking quality and robustness.

  Note that keeping all keyframes when merging information between maps may increase the joint map size if there is redundant information. In addition, it is desirable that reducing key frames (abandonment) may not affect tracking quality because key frames are important for tracking performance. One approach may preserve keyframe field features at various angles, but may abandon keyframe field features at various scales. However, this approach does not retain information useful for continuously tracking features at various distances. This approach retains only the information necessary to perform triangulation or analyze the structure. In another approach, key frames that observe features closer may be preferred. In yet another approach, key frames that observe features at various distances may be retained, but key frames that observe features at various angles may be discarded. A key frame is evaluated for its contribution in generating the entire map or image.

  According to aspects of the present disclosure, keyframes that observe features at various scales can be used to record changes in appearance with changes in scale, as described in connection with FIGS. Can be done. In addition, keyframes can be used to predict the appearance of the appropriate scale of the various scales so that the predicted appearance has minimal workpiece or minimal appearance distortion.

  In one exemplary implementation, a contribution score associated with a key frame may be determined based at least in part on the angular diversity and scale diversity of the key frame. For example, a portion of the contribution score may be derived from the keyframe contribution to the angular diversity of features observed by the keyframe. Another part of the contribution score may be derived from the key frame contribution to the scale diversity of the features observed by the key frame. The key frame can then be deleted if the contribution score is below a predetermined threshold. According to aspects of the present disclosure, the predetermined threshold for the contribution score may be programmable depending on the application. In some other implementations, the contribution score is calculated for a keyframe that already exists in a bin configured to count the occurrence of the number of keyframes from a constant viewpoint angle and from a constant viewpoint scale or distance. May represent a number. For example, in some applications, the predetermined threshold for the number of keyframes in a bin can be set to 2, and in some other applications, a predetermined threshold for the number of keyframes in a bin. The value can be set to 5.

  In other words, the keyframe scoring method may use a first scoring reference value to measure angular diversity and a second scoring reference value to measure keyframe scale diversity. it can. The two reference values can then be combined with various weight factors and summed. The weighting factor can be used to more emphasize certain scoring reference values relative to other scoring reference values. According to aspects of the present disclosure, the angular diversity and scale diversity weighting factors are programmable, and their ranges can vary from 0 (0%) to 1 (100%). For example, in some implementations, the angle diversity weight may be 0, the scale diversity weight may be 1, and vice versa. In some other implementations, the angular diversity weight may be 5 and the scale diversity weight may be 5. In still some other implementations, the angular diversity weight may be 6, the scale diversity weight may be 4, and so on. The joint reference value can then be used to evaluate the keyframe contribution for mapping and tracking. As described above, a key frame can be deleted (or reduced) if its contribution score is below a predetermined threshold. For example, if there were nine other key frames that looked at the same object in the scene from very similar distances and angles, these nine key frames would have a low angle diversity score and a scale diversity score, but that The reason is that removing any one of those key frames is not important because the other key frames have the same information content.

  In another exemplary implementation, the information content of each observation of the scene may be calculated when determining the angle diversity reference value for the scene. The normalized score can then be integrated to form a key frame contribution score. Note that this approach encourages angular diversity but prefers keyframes representing scenes with corresponding distances closer to the mobile device.

  In determining the scale diversity reference value for the scene, the scale diversity scale is used. The method can be configured to use keyframes at distance x to synthesize views from a distance of 0.5x to 2x, as an example, and the method can be downsampled by one pyramid level / Supports upsampling. In this approach, for all key frames within this bracket, the method assigns a contribution score that may be a score divided by the number (N) of key frames in that bracket. The method then sums the contribution scores of the observed map points. In accordance with aspects of the present disclosure, various approaches can be used to assign scores based on scale diversity, for example, one approach is −0.25 * N + 1.5 if N ≦ 5, N> 5 In the case of 1 / N can be used.

  FIG. 1 illustrates an exemplary location according to some aspects of the present disclosure. As shown in FIG. 1, the exemplary location 102 may be a complex football stadium. In such a complex location, a typical mobile device may not be able to record an image that represents the entire stadium. It is desirable to merge multiple images recorded by one or more mobile devices to build a global image that can be used in a tracking self-location estimation application.

  FIG. 2 illustrates a method for recording multiple viewpoints of a scene from multiple angles according to some aspects of the present disclosure. In the example shown in FIG. 2, the scene 202 may be recorded by one or more mobile devices (201a-201f) at various viewpoint angles. For example, mobile device 201a has a viewing angle A1, mobile device 201b has a viewing angle A2, and so on, mobile device 201f has a viewing angle A6. In some implementations, multiple viewpoints (A1-A6) may be recorded by one mobile device. In some other implementations, multiple viewpoints (A1-A6) may be recorded by multiple mobile devices.

  FIG. 3 illustrates a method for recording multiple viewpoints of a scene from multiple distances according to some aspects of the present disclosure. In the example shown in FIG. 3, the scene 302 may be recorded by one or more mobile devices (301a-301c) from various distances. For example, mobile device 301a records scene 302 at distance D1, mobile device 301b records scene 302 at distance D2, and mobile device 301c records scene 302 at distance D3. In some implementations, multiple viewpoints from various distances (D1-D3) may be recorded by one mobile device. In some other implementations, multiple viewpoints from various distances (D1-D3) may be recorded by multiple mobile devices.

  FIG. 4 illustrates a method for recording multiple viewpoints of a scene with multiple aspect ratios, according to some aspects of the present disclosure. In the example shown in FIG. 4, multiple viewpoints of the scene 402 may be recorded by one or more mobile devices with various aspect ratio settings of one or more mobile devices (401a-401c). For example, mobile device 401a records scene 402 at an aspect ratio AR1 at distance D, mobile device 401b records scene 402 at an aspect ratio AR2 at distance D, and mobile device 401c records scene 402 at an aspect ratio AR3 at distance D. Record. In some implementations, multiple viewpoints of the scene 402 with various aspect ratios (AR1-AR3) may be recorded by one mobile device. In some other implementations, multiple viewpoints of the scene 402 with various aspect ratios (AR1-AR3) may be recorded by multiple mobile devices.

  FIG. 5 illustrates an example apparatus for performing a method of merging multiple maps for computer vision-based tracking according to some aspects of the present disclosure. In the example shown in FIG. 5, apparatus 500 includes one or more processors 502, a network interface 504, a map database 506, a key frame processing module 508, and a memory 510. One or more processors 502 may be configured to control the operation of apparatus 500. Network interface 504 can be configured to communicate with a network (not shown), and the network can be configured to communicate with servers, computers, and mobile devices on the network. The map database 506 may be configured to store images of various locations, landmarks, maps, and other user defined information. The key frame processing module 538 may be configured to implement a method for merging multiple maps for computer vision based tracking. For example, the key frame processing module 508 may be configured to perform the methods described above and the methods described below in connection with FIGS. 6, 8, and 9 while operating in conjunction with the processor 502. Memory 510 may be configured to store program codes and data for device 500. In other embodiments, the method described below in connection with FIGS. 6, 8, and 9 is performed by one of the mobile devices shown in FIGS. 2, 3, and 4, and the server Can also be implemented.

  FIG. 6 illustrates an exemplary flowchart for implementing a method of merging multiple maps for computer vision-based tracking according to some aspects of the present disclosure. In the exemplary implementation shown in FIG. 6, at block 602, the method receives a plurality of maps of a scene at a location from at least one mobile device. At block 604, the method identifies multiple key frames of multiple maps of the scene. At block 606, the method merges multiple key frames to generate a global map of the scene. According to aspects of the present disclosure, global maps may include, but are not limited to, a three-dimensional model of a place, a self-localization and simultaneous map construction (SLAM) map, and a two-dimensional model. In addition, the global map can be updated dynamically or can be updated offline at a remote server. In some situations, none of the multiple maps may overlap each other, and in such situations, none of the corresponding keyframes in the multiple maps may be removed because of these reasons This is because a key frame can be considered to have high angular diversity and / or scale diversity relative to its respective one or more adjacent key frames. In some other situations, some of the maps may overlap each other, and in such situations, some of the corresponding keyframes in the maps may be removed. This is because some of the redundant key frames may be considered to have low angle diversity and / or scale diversity relative to their respective one or more adjacent key frames. is there. In some special situations, two or more maps may substantially overlap each other, in which case most of the corresponding keyframes from at least one of the maps are removed In some cases, the reason is that most of the redundant key frames can be considered to have low angle diversity and / or scale diversity with respect to their respective one or more adjacent key frames. .

  According to some aspects of the present disclosure, the method performed at block 602 can further include the method performed at block 608, and the method performed at block 608 is performed at blocks 609 and 610. The method may further be included. At block 608, the method receives a plurality of maps of the scene from a plurality of mobile devices. At block 609, the method evaluates the redundancy of multiple key frames from multiple maps of the scene from multiple mobile devices and removes redundant key frames. At block 610, the method generates a global map using multiple maps of the scene from multiple mobile devices and shares the global map between the multiple mobile devices.

  According to some aspects of the present disclosure, the method performed at block 604 can further include the method performed at block 612, and the method performed at block 612 further includes the elements of block 614. be able to. At block 612, the method identifies a plurality of keyframes that represent a plurality of viewpoints of the scene from a plurality of angles, and / or identifies a plurality of keyframes that represent the plurality of viewpoints of the scene at a plurality of scales. These multiple keyframes include multiple map features, 3D coordinates, and appearance. At block 614, the plurality of scales comprises at least one of a plurality of distances between the at least one mobile device and the scene and a plurality of aspect ratios of the scene from the at least one mobile device.

  According to some aspects of the present disclosure, the method performed at block 606 can further include the method performed at block 616, and the method performed at block 616 is the method performed at block 618. Can further be included. At block 616, the method determines a contribution score for each keyframe of the plurality of keyframes based at least in part on the keyframe angular diversity and / or keyframe scale diversity and less than a predetermined threshold value. Delete keyframes according to contribution score. At block 618, the method calculates an angle diversity scoring criterion by applying a first weighting factor to keyframe angular diversity and applying a second weighting factor to keyframe scale diversity. A scale diversity scoring reference value is calculated and the angle diversity scoring reference value and the scale diversity scoring reference value are combined to generate a contribution score. At block 620, the method may optionally track the location of at least one mobile device at the location.

  FIG. 7 illustrates an example block diagram of a mobile device according to some aspects of the present disclosure. Mobile device 201 (which may be configured to implement 201a-201f) includes, but is not limited to, transceiver 106 for communicating with other computing devices, including but not limited to servers and other mobile devices, and individual photos or videos. And a camera 108 for recording an image of the environment, which can be any of the frames. The mobile device 201 can also include a sensor 116 that can be used to provide data with which the mobile device 201 can determine its position and orientation, i.e., pose. Examples of sensors that can be used with mobile device 201 include accelerometers used as linear accelerometers, crystal sensors, gyros, micro-electromechanical system (MEMS) sensors, and magnetometers.

  The mobile device 201 may also include a user interface 110 that includes a display 112 that can display images. The user interface 110 may also include a keypad 114 or other input device that allows a user to enter information into the mobile device 201. If desired, the keypad 114 can be removed by incorporating a virtual keypad into the display 112 with touch sensor. The user interface 110 may also include a microphone 117 and one or more speakers 118, for example when the mobile platform is a cellular phone. Of course, the mobile device 201 can include other components not relevant to the present disclosure.

  Mobile device 201, along with any other desired features, further includes a control unit 120 connected to and in communication with camera 108 and sensor 116 and user interface 110. The control unit 120 may be provided by one or more processors 122 and associated memory / storage devices 124. The control unit 120 can also include software 126 and hardware 128 and firmware 130. The control unit 120 includes a key frame processing module 132 that is configured to process key frames based on observations recorded by the mobile device 201. The control unit 120 further includes a map generation module 134 configured to generate a map based on the plurality of submaps. For clarity, the keyframe processing module 132 and map generation module 134 are illustrated separately from the processor 122 and / or hardware 128, but based on instructions in the software 126 and firmware 130, and in the processor 122 and It can be combined and / or implemented in hardware 128. Note that the control unit 120 may be configured to implement keyframe processing and map generation methods. For example, the control unit 120 may be configured to perform the functions of the mobile device 201 described in FIGS. 2, 3, and 4.

  FIG. 8 illustrates an exemplary method for determining angle diversity and scale diversity of a key frame, according to some aspects of the present disclosure. As shown in FIG. 8, an object 802, shown as a black square, can be observed from different angles, from different distances, and at different scales. In one approach, possible directions may be discretized into sphere 800 bins, such as 804 and 806 shown in FIG. According to aspects of the present disclosure, a bin may represent a sector of a sphere 800 (shown approximately as a pyramid). A bin can also be viewed as an angular histogram that records keyframes that contain observations, where the observations are represented by one or more stars in the bin. For illustrative purposes, the bin 804 is shown separately on the right side of the sphere 800. In this example, bin 804 may include observations 810a, 810b, 810c, 810d, 810e, and 810f. In some implementations, the bin may be further partitioned into multiple sections, such as Section 1 and Section 2 shown in FIG. Each observation in the bin can be evaluated against other observations in the sector or can be evaluated against other observations from other bins.

  In some implementations, if the number of observations (map points) in a bin is less than a predetermined number, e.g., 3, the observations may be considered as having high angular importance, Those observations can be retained. On the other hand, if the number of observations is greater than the predetermined number, the additional observations may be considered as having a low angular importance and the additional observations may be discarded.

  According to an aspect of the present disclosure, assuming an observed value of a map point from a certain viewpoint, its appearance from a different viewpoint and a similar viewpoint angle are 1 / f times to f times the distance from the camera to the map point. Can be expected to be generated accurately within. In some implementations, f may be equal to 2, and in other implementations, the value of f may be lower depending on how much scale range is desired. The logarithm of the distance from the camera to the map point for each observation can be assigned to a bin representing the angle histogram of the observation. The distance can be recorded on a uniform scale or a logarithmic scale.

  In some implementations, if the number of observations in the bin is less than a predetermined number, e.g., 2, the observation may be considered to have a high scale importance, and such an observation Can be retained. On the other hand, if the number of observations in the bin is greater than the predetermined number, the additional observations in the bin may be considered to have a low scale importance and the additional observations are discarded. obtain.

  According to aspects of the present disclosure, if an observation has a high degree of angular importance or scale importance, the observation can be flagged as important. If an observation has a low angular importance and a low scale importance, the observation may be flagged as insignificant and the observation may be removable. If a keyframe is greater than a predetermined ratio of removable observations (e.g., 0.8) and less than a predetermined ratio of important observations (e.g., 0.1), the keyframe is flagged for removal. Can be stood.

  FIG. 9 illustrates another exemplary method for determining the angular diversity and scale diversity of a key frame, according to some aspects of the present disclosure. In the example shown in FIG. 9, the object 902, shown as a black square, can be observed from different angles, from different distances, and at different scales. In one approach, the possible directions may be discretized into sphere 900 bins, such as 904 and 906 shown in FIG. Similar to the example shown in FIG. 8, a bin may represent a sector of a sphere 900 (shown as a cone). A bin can also be viewed as an angular histogram that records keyframes that contain observations, where the observations are represented by one or more stars in the bin. For illustration, the bin 904 is shown separately on the right side of the sphere 900. In this example, bin 904 may include observations 910a, 910b, 910c, 910d, 910e, 910f, and 910g. In some implementations, the bin may be further partitioned into multiple sections, such as Section 1 and Section 2, as shown in FIG. Each observation in the bin can be evaluated against other observations in the sector or can be evaluated against other observations from other bins.

  According to aspects of the present disclosure, assuming that the first observation is already in the first bin, having the second observation in the second bin is the second in the first bin. Will contribute to higher angular diversity (ie, importance) than having For example, if observation 910a already exists in bin 904, having observation 912a in bin 906 will contribute to higher angular diversity than having observation 910b in bin 904. In some implementations, the contribution of angular diversity due to additional observations to a bin may be inversely proportional to the number of observations in that bin.

  According to aspects of this disclosure, assuming that an observation already exists in the first partition of the bin, having an additional observation in the second partition of the bin adds to the first partition. Will contribute to higher scale diversity (ie, importance) than having For example, if observation 910a already exists in bin 904 segment 1, having an additional observation 910f in bin 904 segment 2 is better than having observation 910b in bin 904 segment 1. It will contribute to high scale diversity. In some implementations, the contribution of scale diversity due to additional observations in a bin segment may be inversely proportional to the number of observations in that bin segment.

  According to aspects of this disclosure, several techniques may be used to identify and track features in an image frame. In one approach, the method of identifying features may be performed by examining the minimum eigenvalue of each 2 × 2 gradient matrix. The features are then tracked using a Newton-Raphson method that minimizes the difference between the two windows. Multi-resolution tracking methods allow for relatively large displacements between images. Note that errors can accumulate while tracking features from one frame to the next. In order to detect potentially bad features, the mobile device determines whether the image signal in the window surrounding the feature in the current frame is still similar to the image signal in the window surrounding the feature in the previous frame. May be configured to monitor. The image content may be deformed because features can be tracked through many frames. To address this challenge, consistency checks can be performed using similarity or affine mapping.

  According to aspects of the present disclosure, to identify an object in an image, a point on the object is extracted to provide a feature descriptor (also referred to as a keypoint, feature point, or feature for short) of that object be able to. This descriptor extracted from the training image can then be used to identify that object when attempting to locate an object in a test image that contains many other objects. In order to perform reliable recognition, features extracted from training images may be detectable even under changes in image scale, noise, and illumination. Such points are usually present in high contrast areas of the image, such as object edges.

  Another characteristic of these features is that the relative position between these features in the original scene may not change from image to image. For example, if only the four corners of the door are used as features, these four corners can function regardless of the position of the door, but if a point in the frame is used, the door is opened or closed. Recognition may fail. Similarly, features located in connected or flexible objects may generally not work if any changes in their internal geometry occur between two images in the set being processed. There is. In some implementations, SIFT detects and uses multiple features from the image, which may reduce the error contribution caused by local changes in the average error of all feature matching errors. Thus, the disclosed method can identify objects even in noise and even under partial occlusion, but this is because SIFT feature descriptors are invariant to constant scaling and direction. This is possible because it can be partially invariant to affine distortion and illumination changes.

  For example, object keypoints can first be extracted from a set of reference images and stored in a database. Objects in the new image are recognized by comparing each feature from the new image to this database and finding candidate matching features based on the Euclidean distance of those feature vectors. From the complete set of matches, the object in the new image and the subset of keypoints that matched with respect to its position, scale, and orientation can be identified to filter out good matches. The determination of a consistent cluster can be performed by using a generated Hough transform hash table implementation. Each cluster of three or more features that are matched with respect to the object and its pose can then undergo further detailed model verification, after which outliers can be discarded. Based on the accuracy of the fit and the number of estimated mismatches, it is then possible to calculate the probability that a particular set of features indicates the presence of the object. Object matches that pass the test can be identified as accurate and reliable.

  In accordance with aspects of the present disclosure, image feature generation can be invariant to illumination changes, as well as local to each image transformation, scaling, and rotation. Convert to a large collection of feature vectors that can be robust against geometric distortions. These features have properties similar to those of neurons in the inferior temporal cortex used for object recognition in primate vision. The key position may be defined as the maximum and minimum results of a Gaussian function difference applied in scale space to a series of smoothed and resampled images. Low contrast candidate points and edge reaction points along the edge can be discarded. Assign the dominant orientations to local keypoints. This approach ensures that the key points are more stable with respect to matching and recognition. A SIFT descriptor that is robust against local affine distortion can then be obtained by blurring and resampling the local image orientation plane, taking into account the pixels surrounding the radius of the key position.

  Feature matching and indexing may include storing SIFT keys and identifying matching keys from new images. In one approach, a modification of the k-d tree algorithm, also called a best-bin-first search method, that can be used to identify nearest neighbors with high probability using a limited amount of computation. The best bin first algorithm uses a modified search order with respect to the k-d tree algorithm so that bins in the feature space can be searched in order of closest distance from the query location. This search order requires the use of a heap-based priority queue to efficiently determine the search order. By identifying its nearest neighbor in the keypoint database from the training image, the best candidate match for each keypoint can be found. The nearest neighbor can be defined as the keypoint that has the minimum Euclidean distance from a given descriptor vector. The probability that the matching is accurate can be determined by taking the ratio of the distance from the nearest neighbor to the second nearest neighbor.

  In one exemplary implementation, matching with a distance ratio exceeding 0.8 may be rejected, which eliminates 90% of false matches while discarding less than 5% of exact matches. To further improve the efficiency of the best bin first algorithm, the search can be stopped after examining a predetermined number (eg, 100) of nearest neighbor candidates. For a database of 100,000 keypoints, this can achieve approximately two orders of magnitude acceleration for accurate nearest neighbor search, further resulting in less than 5% loss in the number of exact matches.

  For this exemplary implementation, the Hough transform can be used to cluster reliable model hypotheses and search for matching keys for a particular model pose. A Hough transform can be used to identify a cluster of features using a consistent interpretation by using each feature to determine an object pose that can match that feature. When a cluster of features is found to determine the same pose of an object, the probability of interpretation being accurate may be higher than for any single feature. Entries can be created in the hash table to predict the position, orientation, and scale of the model from the matching hypothesis. A hash table can be searched to identify clusters of at least three entries in the bin, and bins can be sorted in descending size order.

  According to aspects of the present disclosure, each of the SIFT keypoints can specify a 2D position, a scale, and a direction. In addition, each matching keypoint in the database may have a record of its parameters for the training image where the matching keypoint is found. The similarity transformation implied by these four parameters can be an approximation of a 6-DOF pose space for a 3D object and does not correspond to any non-rigid deformation. Thus, an exemplary implementation can use a wide bin size of 30 degrees with respect to direction, a scale of 2 times, and 0.25 times the maximum expected training image size (using the predicted scale) with respect to position. . SIFT key samples generated at a larger scale can be given twice as much weight as SIFT key samples at a smaller scale. With this approach, a larger scale can effectively filter the neighbors that are most likely to test on a smaller scale. This approach also improves recognition performance by giving greater weight to the lowest noise scale. According to aspects of the present disclosure, to avoid the bin assignment boundary effect problem, each keypoint matching determines two adjacent bins in each dimension, making a total of 16 entries for each hypothesis. Given this, the range of poses can be further expanded.

  In accordance with aspects of the present disclosure, outliers can be removed by examining the match between each image feature and model for a given parameter solution. For example, given a linear least squares solution, each match may need to match within half the error used for the parameters in the Hough transform bin. Once the outliers are discarded, the remaining points can be used to solve the linear least squares solution and this process can be repeated. In some implementations, matching may be rejected if fewer than a predetermined number of points (eg, 3 points) remain after discarding outliers. In addition, the top-down matching stage can be used to add any further matching that matches the predicted model location that may be missing from the Hough transform bin due to similarity transformation approximations or other errors. .

  The decision to accept or reject the model hypothesis can be based on a detailed probabilistic model. This method first calculates the expected number of false matches for a model pose, taking into account the predicted size of the model, the number of features in the region, and the accuracy of matching. Bayesian probability analysis can then give a probability that the object may exist based on the actual number of matching features found. If the final probability for correct interpretation exceeds a predetermined percentage (eg, 95%), the model can be accepted.

  According to an aspect of the present disclosure, in one approach, a rotation invariant feature transformation (RIFT) is used as a generalization of SIFT rotation invariants in order to deal with clutter or partial occlusion situations. invariant feature transform) method can be used. It is possible to construct RIFT descriptors using circular normalized patches divided into concentric rings of equal width, and to compute gradient direction histograms within each ring . In order to maintain rotation invariance, the direction at each point can be measured relative to the direction from the center outward.

  In another approach, a generalized robust invariant feature (G-RIF) method can be used. G-RIF encodes edge direction, edge density, and hue information in a unified format that combines perceptual information with spatial coding. Object recognition schemes use proximity context-based determination to estimate the object model.

  Yet another approach uses a scale and rotation-invariant interest detector / descriptor that may outperform previously proposed methods for repeatability, uniqueness, and robustness. A speeded up robust feature (SURF) method can be used. SURF relies on integral images for image convolution to reduce computation time, and superior existing detectors and descriptions (using fast Hessian-based measurements on detectors and distribution-based descriptors) Based on the strength of the child. The SURF method describes the distribution of Haar wavelet responses within the neighborhood of interest points. Integral images can be used for speed, and 64 dimensions can be used to reduce time for feature calculation and matching. The indexing step can be based on Laplacian signals that enhance descriptor matching speed and robustness.

  In yet another technique, a principal component analysis SIFT (PCA-SIFT) method can be used. In some implementations, the PCA-SIFT descriptor is a vector of image gradients in the x and y directions calculated within the support region. The gradient region can be sampled at 39 × 39 locations. Thus, the vector can be of dimension 3042. Using PCA, the dimension can be reduced to 36. Yet another approach may use a Gradient location-orientation histogram (GLOH) method, which is an extension of the SIFT descriptor designed to enhance its robustness and uniqueness. In some implementations, the SIFT descriptor has 3 bins in the radial direction (with radii set to 6, 11, and 15) and 8 bins in the angular direction, resulting in 17 It can be calculated for a log-polar position grid with a number of position bins. The central bin need not be divided in the angular direction. The gradient direction can be quantized within 16 bins, resulting in a 272 bin histogram. PCA can be used to reduce the size of this descriptor. For image patches collected from various images, a covariance matrix for PCA can be estimated. The 128 largest eigenvectors can then be used for the description.

  Yet another approach can be used to use a two-object recognition algorithm within current mobile device limits. In contrast to typical SIFT techniques, a feature from accelerated segment test (FAST) corner detector can be used for feature detection. This approach distinguishes between an offline preparation stage where features can be created at different scale levels and an online stage where features can be created at the current fixed scale level of the camera image of the mobile device. In one exemplary implementation, features can be created from a predetermined fixed patch size (eg, 15 × 15 pixels), forming a SIFT descriptor having 36 dimensions. This approach can be further extended by integrating a scalable vocabulary tree within the recognition pipeline. This allows efficient recognition of a large number of objects on the mobile device.

  According to aspects of the present disclosure, detection and description of local image features may be useful for object recognition. SIFT features can be local, can be based on the appearance of the object at a particular point of interest, and can be invariant to the scale and rotation of the image. SIFT features can also be robust to illuminance, noise changes, and small changes in the field of view. In addition to these properties, these features can be very characteristic, can be relatively easy to extract and can allow accurate object identification with low mismatch probability. These features can be matched relatively easily against a (large) database of local features, and generally use probabilistic algorithms such as K-dimensional (kd) trees with best bin first search. be able to. Object descriptions with a set of SIFT features can be robust against partial occlusion. For example, as few as three SIFT features from an object may be sufficient to calculate their position and pose. In some implementations, recognition may be performed in pseudo real time on modern computer hardware with respect to a small database.

  In accordance with aspects of the present disclosure, random sample consensus (RANSAC) techniques can be used to remove outliers caused by moving objects within the camera's field of view. Note that RANSAC uses an iterative method to estimate mathematical model parameters from a set of observed data, including outliers. Because this method produces reasonable results with associated probabilities, this method can be non-deterministic and can increase the probability as more iterations are performed.

In one exemplary implementation, a set of observed data values, a parameterized model that can be adapted to observations with corresponding reliability parameters. In this exemplary implementation, the method iteratively selects a random subset of the original data. These data can be hypothetical normal values, which can then be tested as follows.
1. The model can fit hypothetical normal values, ie all free parameters of the model are reconstructed from normal values.
2. All other data can then be tested against the fitted model, and if the point fits well into the estimated model, the point can be considered hypothetically normal.
3. If a sufficient number of points are classified as hypothetical normal values, the estimation model can be considered acceptable.
4. Since the model is estimated only from the initial set of hypothetical normal values, the model can be estimated again from all hypothetical normal values.
5. Finally, the model can be evaluated by estimating the normal value error for the model.

  The above procedure can be repeated a predetermined number of times to produce either a model that can be rejected because too few points are classified as normal values, or a refined model with corresponding error measurements. In the latter case, the refined model can be maintained if the error is lower than the previously stored model.

  In another exemplary implementation, a model-based motion tracking method may be used to actively identify and remove moving objects in the camera's field of view. In one approach, the tracking object can be treated as a model recognition problem. The binary representation of the target can be tracked, and the Hausdorff distance-based search can be used to search the area of the image for the object. In the case of a binary representation of the target (model), the output from a standard canny edge detector of a Gaussian smoothed image can be extended using the concept of model history. At each frame, a Hausdorff search can be performed for each target using the current image and the canny edge from the current model. In addition, affine estimation can be performed to approximate the net background motion. From these two search results, this information can be used to gather information about the target to approximate the target's motion, as well as to separate the background from motion within the target's region. The past movement of the target to allow it to deal with harmful / abnormal conditions (e.g. objects are blocked and become shadows, objects leave the frame, or camera image distortion results in poor image quality) And keep historical data about the target, such as resizing, target characteristic field of view (snapshot over time, providing an accurate representation of the various ways the target has been tracked), and past matching quality Is possible.

  The history of tracking the target may be more useful than simply assisting in harmful / abnormal conditions, and some solid motion tracking methods may require historical data as well as frame-by-frame motion comparison methods. is there. How this history state determines what should be considered part of the target (for example, something that moves close to an object moving at the same speed should be incorporated into the object) With information about motion and size, this method can predictively where a lost object has gone or where it can be reproduced ( This is useful in leaving the frame and recovering the target to be reproduced at a later time).

  The essential challenge of motion tracking methods is that a camera may have arbitrary motion (as opposed to a static camera), which develops a tracking system that can cope with unpredictable changes in camera motion. May be caused by making it difficult. A computationally efficient affine background estimation scheme can be used to provide information about camera and scene motion.

  According to aspects of the present disclosure, an affine transformation on an image can be performed on an image at time t + dt at time t, which allows the motion in the two images to be correlated. . This background information allows this method to synthesize an image at time t + dt from an image at time t and an affine transformation that can be an approximation of the net scene motion. Since the image feature can be removed from the space surrounding the target using the difference between the actual image at t + dt and the generated image at t + dt, this composite image generates new model information, It can be useful in removing background clutter from model space.

  In addition to using the affine transformation as a tool to clean up the search space, it is also possible to use this composite image to normalize the target coordinate movement and how the background can move By using a vector to track the target and a vector to track how the target can move, the difference between the two vectors is used to generate a vector that describes the movement of the target relative to the background can do. This vector allows the method to predictively match where the target should be and anticipate a fault condition (e.g., anticipating the direction of motion provides clues about the fault that is approaching. Similarly, in the case of a fault condition, we can track where an object can exist). When an object enters a fault state, this method can still estimate the background motion and use that estimate along with knowledge of the previous motion of the model, where the model may be reproduced, Alternatively, it can be estimated whether the frame may be entered again.

  Background estimation can be a major factor in long-term tracking of objects. It is possible to perform short-term tracking without background estimation, but after a certain period of time, it can be difficult to effectively deal with object distortion and danger without good estimation of the background Please note that.

  According to aspects of the present disclosure, one of the advantages of using the Hausdorff distance as a matching operator is that the Hausdorff distance can be extremely resistant to shape changes during matching. Although using the Hausdorff distance as a matching operator may require more precise definition of the object being tracked.

  In one approach, a straight dilation based method that captures a new model from the time t + 1 image can be used. In some situations, where there may be non-object features that are close to the object (which is very likely to occur), the dilation method can slowly incorporate the entire scene into the model, so this method is effective. Note that this may not be the case. Therefore, the model is updated every frame, which can be resistant to changes in model shape, but is not relaxed, so that the incorporation of non-model pixels into the model can be employed. One exemplary implementation adds background removal and the previous model to the current model matching window, what appears to be a stable pixel, as well as the model over time since it may not be stable Use a combination with the use of new pixels surrounding those pixels that may either be excluded from or incorporated into the model. This approach can be effective in keeping the model relatively clean from clutter in the image. For example, using this approach, roads close to the truck are no longer drawn into the model pixel by pixel. The models may appear to be inflated, but this may be the result of a historical effect of how those models are built, but this method can be Note that it may also have features that make the search results clearer because it may have more model pixels to match within the frame.

  Note that there may be a significant amount of computation to be performed in each frame. According to some implementations, the mobile device may be configured to perform smoothing / feature extraction for each target, Hausdorff matching (eg, one matching per model), and affine background estimation. Each of these operations can be very computationally expensive individually. In order to achieve real-time performance on mobile devices, the design can be configured to use parallel processing whenever possible.

  At least the following paragraphs to the following paragraphs (source paragraphs [0085] to [0086]), FIG. 5, FIG. 6, FIG. 7, and their corresponding descriptions are a plurality of maps of a scene at a location from at least one mobile device Providing means for receiving, means for identifying a plurality of keyframes of a plurality of maps of the scene, and means for merging the plurality of keyframes to generate a global map of the scene Please note that.

  The methodologies and mobile devices described herein can be implemented by various means depending on the application. For example, these methodologies can be implemented in hardware, firmware, software, or a combination thereof. For a hardware implementation, the processing unit is one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), designed to perform the functions described herein. Implementation within a digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, electronic device, other electronic unit, or a combination thereof Can do. As used herein, the term “control logic circuit” encompasses a logic circuit implemented by software, hardware, firmware, or a combination thereof.

  For firmware and / or software implementations, the methodologies may be performed by modules (eg, procedures, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly embodies the instructions may be used in the implementation of the methodologies described herein. For example, the software code can be stored in memory and executed by a processing unit. The memory may be implemented within the processing unit or external to the processing unit. As used herein, the term “memory” refers to any type of long-term, short-term, volatile, non-volatile, or other storage device, and any particular type of memory or memory There is no limit to the number of media or the type of media on which the memory is stored.

  If implemented in firmware and / or software, the functions may be stored as one or more instructions or code on a computer-readable medium. Examples include computer readable media encoded with a data structure and computer readable media encoded with a computer program. The computer readable medium may take the form of an article of manufacture. Computer-readable media includes physical computer storage media. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, or instructions or data structures with desired program code. Any other media that can be used to store and be accessed by a computer can be included, and the discs used herein include a compact disc (CD), Includes laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, which are usually data Is reproduced magnetically, but the disc optically reproduces the data with a laser. Combinations of the above should also be included within the scope of computer-readable media.

  In addition to being stored on a computer readable medium, the instructions and / or data may be provided as a signal on a transmission medium included in the communication device. For example, the communication device may include a transceiver having signals indicative of instructions and data. The instructions and data are configured to cause at least one processor to perform the functions outlined in the claims. In other words, the communication device includes a transmission medium having a signal indicating information for executing the disclosed function. The transmission medium included in the communication device at the first time may include a first portion of information for performing the disclosed functions, while the transmission included in the communication device at the second time. The medium may include a second portion of information for performing the disclosed function.

  The present disclosure may be implemented with various wireless communication networks such as a wireless wide area network (WWAN), a wireless local area network (WLAN), a wireless personal area network (WPAN), and so on. The terms “network” and “system” are often used interchangeably. The terms “location” and “location” are often used interchangeably. WWAN includes code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, frequency division multiple access (FDMA) networks, orthogonal frequency division multiple access (OFDMA) networks, single carrier frequency division multiple access (SC-FDMA). ) Network, Long Term Evolution (LTE) network, WiMAX (IEEE802.16) network, etc. A CDMA network may implement one or more radio access technologies (RAT) such as cdma2000, wideband CDMA (W-CDMA), and so on. cdma2000 includes IS-95, IS-2000, and IS-856 standards. A TDMA network may implement Global System for Mobile Communications (GSM), Digital Advanced Mobile Phone System (D-AMPS), or some other RAT. GSM® and W-CDMA are described in documents from an organization named “3rd Generation Partnership Project” (3GPP). cdma2000 is described in documents from an organization named “3rd Generation Partnership Project 2” (3GPP2). 3GPP and 3GPP2 documents are publicly available. The WLAN may be an IEEE 802.11x network and the WPAN may be a Bluetooth network, IEEE 802.11x, or some other type of network. The techniques can also be implemented with any combination of WWAN, WLAN, and / or WPAN.

  A mobile station can be a cellular or other wireless communication device, a personal communication system (PCS) device, a personal navigation device (PND), a personal information manager (PIM), a personal digital assistant (PDA), a laptop, or a wireless communication and Refers to devices such as other suitable mobile devices capable of receiving navigation signals. The term “mobile station” also refers to a device that communicates with a personal navigation device (PND), such as a short-range wireless connection, an infrared connection, a wireline connection, or other connection, to receive satellite signals, receive assistance data, and / or Or is intended to include whether location related processing is performed at the device or at the PND. A “mobile station” also receives satellite signals from all devices, including wireless communication devices, computers, laptops, etc., that can communicate with the server via the Internet, Wi-Fi, or other networks. It is intended to include whether assistance data reception and / or location related processing is performed at the device, at the server, or at another device associated with the network. Any operable combination of the above is also considered a “mobile station”.

  Any indication that something is "optimized", "required" or other indications apply this disclosure only to systems that are optimized or have "required" elements (Or other limitations due to other indications). These representations refer only to the particular described implementation. Of course, many implementations are possible. The technique can be used with protocols other than those discussed herein, including protocols that are under development or will be developed in the future.

  Those skilled in the art will recognize that many possible variations and combinations of the disclosed embodiments can be used while still using the same underlying underlying mechanisms and methods. The above description has been written with reference to specific embodiments for purposes of explanation. However, the above discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The present disclosure and various embodiments are best utilized by other persons skilled in the art to explain the principles of the present disclosure and its practical application, and with various modifications to suit the particular application contemplated. Embodiments have been selected and described as possible.

102 places
106 transceiver
108 Camera
110 User interface
112 display
114 keypad
116 sensors
117 Microphone
118 Speaker
120 Control unit
122 processor
124 Memory / Storage device
126 software
128 hardware
130 Firmware
132 Keyframe processing module
134 Map generation module

Claims (36)

A method of merging multiple maps for computer vision based tracking,
Receiving a plurality of maps of a scene at a location from at least one mobile device;
Identifying a plurality of keyframes of the plurality of maps of the scene;
Merging the plurality of key frames to generate a global map of the scene. The step of receiving multiple maps of a scene at a location is
The method of claim 1, comprising receiving the plurality of maps of the scene from a plurality of mobile devices. Evaluating the redundancy of the plurality of key frames from the plurality of maps of the scene from a plurality of mobile devices;
3. The method of claim 2, further comprising the step of deleting redundant key frames. Generating a self location and simultaneous map construction (SLAM) map using the plurality of maps of the scene from a plurality of mobile devices;
3. The method of claim 2, further comprising sharing the SLAM map between the plurality of mobile devices. Identifying the plurality of keyframes of the plurality of maps of the scene comprises:
Identifying the plurality of key frames representing a plurality of viewpoints of the scene from a plurality of angles;
Identifying at least one of the plurality of key frames representing a plurality of viewpoints of the scene at a plurality of scales,
The plurality of key frames include common features, position coordinates, and appearance of the plurality of maps.
The method of claim 1. Multiple scales
A plurality of distances between the at least one mobile device and the scene;
Comprising at least one of a plurality of aspect ratios of the scene from the at least one mobile device;
6. The method according to claim 5. Merging the plurality of keyframes to generate a global map of the scene comprises
For each key frame of the plurality of key frames,
Determining a contribution score based at least in part on the angular diversity of the key frame and the scale diversity of the key frame;
2. The method of claim 1, comprising: deleting the key frame in response to the contribution score less than a predetermined threshold. Determining a contribution score based at least in part on the angular diversity and scale diversity of the keyframes;
Calculating an angle diversity scoring reference value by applying a first weighting factor to the angle diversity of the keyframe;
Calculating a scale diversity scoring reference value by applying a second weighting factor to the scale diversity of the keyframe;
8. The method of claim 7, comprising combining the angle diversity scoring reference value and the scale diversity scoring reference value to generate the contribution score. Merging the plurality of keyframes to generate a global map of the scene comprises
For each key frame of the plurality of key frames,
Determining a contribution score based at least in part on the angular diversity of the key frame;
2. The method of claim 1, comprising: deleting the key frame in response to the contribution score less than a predetermined threshold. Merging the plurality of keyframes to generate a global map of the scene comprises
For each key frame of the plurality of key frames,
Determining a contribution score based at least in part on scale diversity of the key frame;
2. The method of claim 1, comprising: deleting the key frame in response to the contribution score less than a predetermined threshold. Merging the plurality of keyframes to generate a global map of the scene comprises
Counting the plurality of key frames in an angle histogram, wherein the angle histogram includes a plurality of bins representing different viewpoint angles of the scene, each bin of the plurality of bins having a different viewpoint scale of the scene. Including a plurality of sections representing
For each keyframe of the plurality of keyframes in the angle histogram,
Determining a contribution score based at least in part on the angular diversity of the key frame and the scale diversity of the key frame;
2. The method of claim 1, comprising: deleting the key frame in response to the contribution score less than a predetermined threshold. Determining a contribution score based at least in part on the angular diversity and scale diversity of the keyframes;
Calculating an angle diversity scoring reference value by applying to the angle diversity of the key frame a first weighting factor based at least in part on the number of key frames in the bin in which the key frame is included;
Calculating a scale diversity scoring reference value by applying a second weighting factor based at least in part on the number of keyframes in the partition in which the keyframe is included to the scale diversity of the keyframe;
12. The method of claim 11, comprising combining the angle diversity scoring reference value and the scale diversity scoring reference value to generate the contribution score. The method of claim 1, further comprising tracking the location of the at least one mobile device at the location. Logic configured to receive multiple maps of a scene at a location from at least one mobile device;
Logic configured to identify a plurality of keyframes of the plurality of maps of the scene;
And logic configured to merge the plurality of key frames to generate a global map of the scene. Logic configured to receive multiple maps of a scene at a location is
The apparatus of claim 14, comprising logic configured to receive the plurality of maps of the scene from a plurality of mobile devices. Logic configured to evaluate redundancy of the plurality of key frames from the plurality of maps of the scene from a plurality of mobile devices;
16. The apparatus of claim 15, further comprising logic configured to delete redundant key frames. Logic configured to generate a self-localization and simultaneous map construction (SLAM) map using the plurality of maps of the scene from a plurality of mobile devices;
16. The apparatus of claim 15, further comprising logic configured to share the SLAM map between the plurality of mobile devices. Logic configured to identify the plurality of key frames of the plurality of maps of the scene comprises:
Logic configured to identify the plurality of key frames representing a plurality of viewpoints of the scene from a plurality of angles;
And at least one of logic configured to identify the plurality of key frames representing a plurality of viewpoints of the scene at a plurality of scales,
The plurality of key frames include common features, position coordinates, and appearance of the plurality of maps.
15. An apparatus according to claim 14. Multiple scales
A plurality of distances between the at least one mobile device and the scene;
Comprising at least one of a plurality of aspect ratios of the scene from the at least one mobile device;
The apparatus according to claim 18. Logic configured to merge the plurality of keyframes to generate a global map of the scene is:
For each key frame of the plurality of key frames,
Logic configured to determine a contribution score based at least in part on the angular diversity of the keyframe and the scale diversity of the keyframe;
15. The apparatus of claim 14, comprising logic configured to delete the key frame in response to the contribution score below a predetermined threshold. Logic configured to determine a contribution score based at least in part on the angular diversity and scale diversity of the keyframe is:
Logic configured to calculate an angle diversity scoring reference value by applying a first weighting factor to the angle diversity of the keyframe;
Logic configured to calculate a scale diversity scoring criterion by applying a second weighting factor to the scale diversity of the keyframe;
21. The apparatus of claim 20, comprising logic configured to combine the angle diversity scoring reference value and the scale diversity scoring reference value to generate the contribution score. Logic configured to merge the plurality of keyframes to generate a global map of the scene is:
For each key frame of the plurality of key frames,
Logic configured to determine a contribution score based at least in part on the angular diversity of the keyframe;
15. The apparatus of claim 14, comprising logic configured to delete the key frame in response to the contribution score below a predetermined threshold. Logic configured to merge the plurality of keyframes to generate a global map of the scene is:
For each key frame of the plurality of key frames,
Logic configured to determine a contribution score based at least in part on scale diversity of the key frame;
15. The apparatus of claim 14, comprising logic configured to delete the key frame in response to the contribution score below a predetermined threshold. Logic configured to merge the plurality of keyframes to generate a global map of the scene is:
Logic configured to count the plurality of key frames in an angle histogram, the angle histogram including a plurality of bins representing different viewpoint angles of the scene, wherein each bin of the plurality of bins is the Logic, including multiple sections representing different perspective scales of the scene;
For each keyframe of the plurality of keyframes in the angle histogram,
Logic configured to determine a contribution score based at least in part on the angular diversity of the keyframe and the scale diversity of the keyframe;
15. The apparatus of claim 14, comprising logic configured to delete the key frame in response to the contribution score below a predetermined threshold. Logic configured to determine a contribution score based at least in part on the angular diversity and scale diversity of the keyframe is:
Configured to calculate an angle diversity scoring reference value by applying to the angle diversity of the key frame a first weighting factor based at least in part on the number of key frames in the bin in which the key frame is included. Logic
Configured to calculate a scale diversity scoring reference value by applying a second weighting factor based at least in part on the number of key frames in a partition including the key frame to the scale diversity of the key frame Logic
25. The apparatus of claim 24, comprising logic configured to combine the angular diversity scoring reference value and the scale diversity scoring reference value to generate the contribution score. The apparatus of claim 14, further comprising logic configured to track a position of the at least one mobile device at the location. A computer program residing in a non-transitory medium storing instructions to be executed by one or more computer systems, the instructions comprising:
Instructions for receiving multiple maps of a scene at a location from at least one mobile device;
Instructions for identifying a plurality of keyframes of the plurality of maps of the scene;
A computer program comprising: instructions for merging the plurality of key frames to generate a global map of the scene. The instruction to receive multiple maps of a scene at a location is
28. The computer program of claim 27, comprising instructions for receiving the plurality of maps of the scene from a plurality of mobile devices. Instructions for evaluating the redundancy of the plurality of key frames from the plurality of maps of the scene from a plurality of mobile devices;
30. The computer program of claim 28, further comprising instructions for deleting redundant key frames. Instructions for generating a self location and simultaneous map construction (SLAM) map using the plurality of maps of the scene from a plurality of mobile devices;
30. The computer program of claim 28, further comprising instructions for sharing the SLAM map among the plurality of mobile devices. Instructions for identifying the plurality of key frames of the plurality of maps of the scene are:
Instructions for identifying the plurality of key frames representing a plurality of viewpoints of the scene from a plurality of angles;
And at least one of instructions for identifying the plurality of key frames representing a plurality of viewpoints of the scene at a plurality of scales,
The plurality of key frames include common features, position coordinates, and appearance of the plurality of maps.
28. The computer program according to claim 27. Multiple scales
A plurality of distances between the at least one mobile device and the scene;
Comprising at least one of a plurality of aspect ratios of the scene from the at least one mobile device;
28. The computer program according to claim 27. Instructions for merging the plurality of keyframes to generate a global map of the scene are:
For each key frame of the plurality of key frames,
Instructions for determining a contribution score based at least in part on the angular diversity of the key frame and the scale diversity of the key frame;
28. A computer program according to claim 27, comprising: an instruction for deleting the key frame in response to the contribution score less than a predetermined threshold. Instructions for determining a contribution score based at least in part on the angular diversity and scale diversity of the key frame are:
Instructions for calculating an angle diversity scoring reference value by applying a first weighting factor to the angle diversity of the key frame;
Instructions for calculating a scale diversity scoring reference value by applying a second weighting factor to the scale diversity of the keyframe;
34. The computer program of claim 33, comprising instructions for combining the angle diversity scoring reference value and the scale diversity scoring reference value to generate the contribution score. 28. The computer program of claim 27, further comprising instructions for tracking the location of the at least one mobile device at the location. Means for receiving a plurality of maps of a scene at a location from at least one mobile device;
Means for identifying a plurality of keyframes of the plurality of maps of the scene;
Means for merging the plurality of key frames to generate a global map of the scene.

Images