Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
  • G06F3/1407—General aspects irrespective of display type, e.g. determination of decimal point position, display with fixed or driving decimal point, suppression of non-significant zeros
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
  • G02B27/0172—Head mounted characterised by optical features
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/163—Wearable computers, e.g. on a belt
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
  • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
  • G06F1/1686—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
  • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
  • G06F1/1688—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being integrated loudspeakers
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/16—Constructional details or arrangements
  • G06F1/1613—Constructional details or arrangements for portable computers
  • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
  • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
  • G06F1/1694—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/012—Head tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/013—Eye tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T15/00—Three-dimensional [3D] image rendering
  • G06T15/005—General purpose rendering architectures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/006—Mixed reality
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0101—Head-up displays characterised by optical features
  • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera

Definitions

• MR systems may generate and display color data, which increases the realism of MR scenarios.
• Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or“fields” (e.g., Red, Green, and Blue) corresponding to a color image in rapid succession.
• Projecting color sub-images at sufficiently high rates e.g., 60 Hz, 120 Hz, etc. may deliver a smooth color MR scenario in a user’s mind.
• the at least one of the cells also has a pre-determ ined height.
• the camera system comprises a plurality of cameras.
• the plurality of cameras comprises a second side facing camera.
• the processing unit is configured to determine a desired viewing direction of the camera system for improving a value of the metric.
• the data represents a ray or a line that is associated with an image from the camera system and a reference point
• the processing unit is configured to perform the sanitization by disregarding the ray or the line that is associated with the image.
• the processing unit is configured to perform the sanitization as a part of a local optimization.
• the processing unit is configured to determine the score based on a number of reference points detected in the image.
• the processing unit is configured to perform data sanitization based on the score.
• the processing unit is configured to remove a constraint of the image, or to remove the image, when performing the data sanitization.
• the processing unit is configured to determine respective scores of the images.
• the processing unit is configured to: obtain an additional image from the camera system, determine a score for the additional image, and start a second map segment of the map segments in response to the score of the additional image from the camera system meeting a criterion.
• the processing unit is configured to start the second map segment when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.
• An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.
• the processing unit is configured to determine the score based on a number of reference points detected in the image.
• the processing unit is configured to perform a sanitization to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• the camera system comprises a plurality of cameras, wherein the data comprises a set of images generated by the respective cameras, and wherein the processing unit is configured to perform the sanitization by removing or disregarding the set of images.
• the data comprises an identification of a reference point in the image from the camera system, and wherein the processing unit is configured to perform the sanitization by disregarding the identification of the reference point.
• the data represents a ray or a line that is associated with the image from the camera system and a reference point
• the processing unit is configured to perform the sanitization by disregarding the ray or the line that is associated with the image.
• the processing unit is configured to perform the bundle adjustment as a part of a global optimization.
• the processing unit is configured to perform the global optimization based on the one or more images from the camera system and three- dimensional reference points,
• the processing unit of the apparatus is also configured to determine a metric indicating a likelihood of success to localize the user using the map.
• the processing unit is configured to determine the metric based on a co-visibility of a point of interest that is associated with different camera positions.
• the metric indicates the likelihood of success to localize the user in one or more viewing directions.
• the metric has a value that is based on directionality.
• the metric has a first value associated with a first directionality, and a second value associated with a second directionality.
• the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, and wherein the metric has a value that is based on a position within the one of the plurality of cells.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the total number of images is associated with a certain viewing direction for the cell.
• the processing unit is also configured to determine an average score by dividing the total score by a number of the images in the subset of the images.
• the at least one of the cells also has a pre-determ ined height.
• the processing unit is configured to determine the metric by: obtaining a plurality of images from the camera system, the plurality of images including the image for which the score is determined; and determining co-visibility values, wherein each of the co-visibility values indicating a number of reference points detected in a corresponding one of the plurality of images.
• the camera system comprises a plurality of cameras.
• the plurality of cameras comprises a second forward facing camera.
• the plurality of cameras comprises a second side facing camera.
• the processing unit is configured to determine a desired viewing direction of the camera system for improving a value of the metric.
• the camera system is configured to obtain an additional image after the desired viewing direction of the camera system has been achieved.
• the processing unit is configured to update the metric based on the updated map.
• the processing unit is configured to determine the metric before allowing the apparatus to share content with another apparatus.
• the processing unit is configured to determine the metric during a map construction session in which the processing unit determines the map.
• the processing unit is configured to determine the metric retroactively by accessing the map that was previously determined from a non- transitory medium.
• the metric is determined based on a co-visibility of a point of interest that is associated with different camera positions.
• the metric indicates the likelihood of success to localize the user in one or more viewing directions.
• the metric is determined by the processing unit without determining any convex hull.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the act of determining the metric comprises determining a total number of images from the camera system that are associated with the one of the plurality of cells.
• the total number of images is associated with multiple viewing directions for the cell.
• the camera system is configured to obtain multiple images, and wherein the metric is determined for one of the plurality of cells by: identifying a subset of the images that belong to a same range of viewing directions; determining respective scores for the images in the subset of the images; and summing the scores to obtain a total score.
• the metric is determined by dividing the total score by a number of the images in the subset of the images to obtain an average score.
• the co-visibility graph indicates which of the reference points is visible in which of the multiple images.
• the area score is determined based on a spatial distribution of data points of the map.
• the at least one of the cells also has a pre-determ ined height.
• the plurality of cameras comprises a first side facing camera.
• the method further includes determining by the processing unit a desired viewing direction of the camera system for improving a value of the metric.
• the method further includes obtaining an image of the environment from the camera system after the desired viewing direction of the camera system has been achieved.
• the method further includes updating the metric based on the updated map.
• the metric is determined before the map is used to localize the user with respect to the environment.
• the metric is determined before the apparatus shares content with another apparatus.
• the metric is determined retroactively by accessing the map that was previously determined from a non-transitory medium.
• the method further includes performing a sanitization to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• the data comprises an image from the camera system, and wherein the sanitization is performed by removing or disregarding the image.
• the sanitization is performed as a part of a local optimization.
• the method further includes performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed to remove the data.
• the bundle adjustment is performed as a part of a global optimization.
• the global optimization is performed based on images from the camera system and three-dimensional reference points,
• the method further includes determining, by the processing unit, a score for an image obtained from the camera system.
• the score is a constraint score.
• the score indicates how well the image is constrained.
• the score is determined based on a Jacobian of reference points measurements.
• the score is determined based on an information matrix that is a diagonal matrix.
• the score is determined based on a number of reference points detected in the image.
• the method further includes performing data sanitization based on the score.
• the constraint of the image, or the image is removed when the score is below a threshold.
• the map is determined by: determining multiple map segments; and connecting the map segments.
• the act of determining the multiple map segments comprises determining a first map segment of the map segments by obtaining images from the camera system, and linking the images, wherein the images are generated in sequence by the camera system.
• the method further includes determining respective scores of the images.
• the output(s) comprises one or more images from the camera system.
• a method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit includes: obtaining, by the processing unit, output(s) from the camera system; determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and determining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map.
• the act of determining the score comprises computing the score.
• the score is determined based on a Jacobian of reference points measurements.
• the score is determined based on a number of reference points detected in the image.
• the sanitization is performed to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• camera system comprises a plurality of cameras, wherein the data comprises a set of images generated by the respective cameras, and wherein the sanitization is performed to remove or disregard the set of images.
• the data comprises an identification of a reference point in the image from the camera system, and wherein the sanitization is performed to disregard the identification of the reference point.
• the data represents a ray or a line that is associated with the image from the camera system and a reference point, and wherein the sanitization is performed to disregard the ray or the line that is associated with the image.
• the sanitization is performed as a part of a local optimization.
• the method further includes performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed, wherein the image for which the score is determined is one of the one or more images, or is different from the one or more images.
• the global optimization is performed also based on a relative orientation between cameras of the camera system.
• the map is determined by: determining multiple map segments, wherein the multiple map segment comprise the map segment; and connecting the map segments; wherein the portion of the map comprises one of the map segments.
• the camera system is configured to provide additional images, the additional images generated by the camera system before the image for which the score is determined is generated, wherein the act of determining the map comprises determining a first map segment of the map segments by linking the additional images, and wherein the additional images are generated in sequence by the camera system.
• the method further includes starting, by the processing unit, a second map segment of the map segments in response to the score of the image from the camera system meeting a criterion.
• the camera positions comprise a first camera position of a camera of the camera system, and a second camera position of the camera of the camera system.
• the camera positions comprise a first camera position of a first camera of the camera system, and a second camera position of a second camera position of the camera system.
• the metric indicates a number of reference points that are useable to localize the user with respect to the environment.
• the metric indicates the likelihood of success to localize the user in one or more viewing directions.
• the metric is determined based on a number of times a point of interest is detected from different camera positions.
• the metric is determined by the processing unit without determining any convex hull.
• the directionality is with respect to one or more vertical axes, and/or one or more horizontal axes.
• the directionality comprises a turn direction.
• the directionality comprises a tilt angle.
• the directionality comprises a roll angle.
• the metric has a first value associated with a first directionality, and a second value associated with a second directionality.
• the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, and wherein the metric has a value that is based on a position within the one of the plurality of cells.
• the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, wherein the metric has a first value associated with a first position within the one of the plurality of cells, and a second value associated with a second position within the one of the plurality of cells.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the act of determining the metric comprises determining a total number of images from the camera system that are associated with the one of the plurality of cells.
• the total number of images is associated with multiple viewing directions for the cell.
• the average score is the metric.
• the average score represents an average expected number of co-visibility points for the range of viewing directions for the one of the plurality of cells.
• the respective scores are determined by accessing a co- visibility graph that associates reference points with the multiple images.
• the co-visibility graph indicates which of the reference points is visible in which of the multiple images.
• the area score is based on a spatial distribution of data points of the map.
• At least one of the cells has a footprint area that is 2m by 2m.
• the metric is determined by: obtaining a plurality of images from the camera system, the plurality of images including the image for which the score is determined; and determining co-visibility values, wherein each of the co-visibility values indicating a number of reference points detected in a corresponding one of the plurality of images.
• the camera system comprises a plurality of cameras.
• the plurality of images comprises a first subset of images generated by the plurality of cameras when the camera system is at a first position.
• the plurality of images comprises a second subset of images generated by the plurality of cameras when the camera system is at a second position.
• the plurality of cameras comprises a second forward facing camera.
• the plurality of cameras comprises a first side facing camera.
• the method further includes determining, by the processing unit, a desired viewing direction of the camera system for improving a value of the metric.
• the method further includes obtaining an additional image from the camera system after the desired viewing direction of the camera system has been achieved.
• the method further includes updating the map based on the additional image.
• the method further includes updating the metric based on the updated map.
• the metric is determined before the processing unit uses the map to localize the user with respect to the environment.
• the metric is determined before the apparatus shares content with another apparatus.
• the metric is determined during a map construction session in which the processing unit determines the map.
• a method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit includes: obtaining, by the processing unit, output(s) from the camera system; determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and obtaining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map, wherein the act of obtaining the score comprises computing the score or receiving the score.
• FIG. 1 illustrates another image display system having an image display device in accordance with some embodiments.
• FIG. 3 illustrates another image display system having an image display device in accordance with other embodiments.
• FIG. 4 illustrates another image display system having an image display device in accordance with other embodiments.
• FIG. 5 illustrates an image display device displaying frames in multiple depth planes.
• FIG. 6 illustrates a method for determining a map for allowing an image display device to localize a user of the image display device, and/or to perform other function(s).
• FIG. 7 illustrates an example of an environment being divided into multiple cells.
• FIG. 8B illustrates a graphical representation of the method of FIG. 8A.
• FIG. 9 illustrates an example of a co-visibility graph.
• FIG. 10 illustrates a map-and-localization management method.
• FIG. 11 illustrates a method of sharing content between users of image display devices.
• FIG. 12 illustrates a technique for determining a map for allowing an image display device to localize a user of the image display device, and/or to perform other function(s).
• FIG. 13 illustrates a method for determining a map for allowing an image display device to localize a user of the image display device, and/or to perform other function(s).
• FIG. 14 illustrates a method performed by an image display device in accordance with some embodiments.
• FIG. 15 illustrates another method performed by an image display device in accordance with some embodiments.
• FIG. 16 illustrates a specialized processing system in accordance with some embodiments.
• a localizing map of the environment is obtained. Then real-time tracking image from the camera system of the image display device is then matched against the localizing map to localize the user.
• the success of the localization depends on the quality of the localizing map. Accordingly, it would be advantageous to determine a metric for indicating a quality of the map, which indicates a likelihood of success for using the map for localization.
• Various techniques may be employed to determine the metric. In one implementation, the metric is determined based on co-visibility of reference points captured in different images.
• the localizing map may be created using camera system of the image display device.
• the user of the image display device performs different head poses (e.g., turning the head) to“scan” the environment. While doing so, the camera system captures images of the environment.
• the processing unit of the image display device then processes the images to create the map.
• undesirable data may be removed and/or adjusted during the map creation process.
• undesirable data may be an image that is not well- constrained with respect to a map segment. As images are generated in a sequence for creating a map, the image are linked together to form a map segment. Each image may capture a certain number of reference point (e.g., map points).
• images that are not well-constrained may be removed, and map segments with well-constrained images may be connected together to form a localizing map.
• the image display device 101 includes a frame structure 102 worn by an end user 50, a display subsystem 1 10 carried by the frame structure 102, such that the display subsystem 1 10 is positioned in front of the eyes of the end user 50, and a speaker 106 carried by the frame structure 102, such that the speaker 106 is positioned adjacent the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent the other ear canal of the end user 50 to provide for stereo/shapeable sound control).
• the display subsystem 1 10 is designed to present the eyes of the end user 50 with light patterns that can be comfortably perceived as augmentations to physical reality, with high-levels of image quality and three-dimensional perception, as well as being capable of presenting two- dimensional content.
• the display subsystem 1 10 presents a sequence of frames at high frequency that provides the perception of a single coherent scene.
• the display subsystem 1 10 employs “optical see-through” display through which the user can directly view light from real objects via transparent (or semi-transparent) elements.
• the transparent element often referred to as a“combiner,” superimposes light from the display over the user’s view of the real world.
• the display subsystem 1 10 comprises a partially transparent display. The display is positioned in the end user’s 50 field of view between the eyes of the end user 50 and an ambient environment, such that direct light from the ambient environment is transmitted through the display to the eyes of the end user 50.
• an image projection assembly provides light to the partially transparent display, thereby combining with the direct light from the ambient environment, and being transmitted from the display to the eyes of the user 50.
• the projection subsystem may be an optical fiber scan-based projection device, and the display may be a waveguide-based display into which the scanned light from the projection subsystem is injected to produce, e.g., images at a single optical viewing distance closer than infinity (e.g., arm’s length), images at multiple, discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects.
• These layers in the light field may be stacked closely enough together to appear continuous to the human visual subsystem (i.e.
• one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, picture elements may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of an adjacent layer).
• the display subsystem 1 10 may be monocular or binocular.
• the image display device 101 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and/or the eye position and inter-ocular distance of the end user 50.
• sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros), or any combination of the foregoing. Many of these sensors operate on the assumption that the frame 102 on which they are affixed is in turn substantially fixed to the user’s head, eyes, and ears.
• the image display device 101 may also include a user orientation detection module.
• the user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via sensors coupled to the frame 102) and may predict the position of the head 54 of the end user 50 based on position data received from the sensors. Detecting the instantaneous position of the head 54 of the end user 50 facilitates determination of the specific actual object that the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that actual object and further providing an indication of the position in which the virtual object is to be displayed.
• the user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensors.
• the image display device 101 may also include a control subsystem that may take any of a large variety of forms.
• the control subsystem includes a number of controllers, for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/or programmable logic controllers (PLUs).
• controllers for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/or programmable logic controllers (PLUs).
• the control subsystem of the image display device 101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional data base for storing three-dimensional scene data.
• the CPU may control overall operation, while the GPU may render frames (i.e. , translating a three-dimensional scene into a two-dimensional image) from the three- dimensional data stored in the three-dimensional data base and store these frames in the frame buffers.
• One or more additional integrated circuits may control the reading into and/or reading out of frames from the frame buffers and operation of the image projection assembly of the display subsystem 1 10.
• the image display device 101 may also include a remote processing module 132 and remote data repository 134 operatively coupled, such as by a wired lead or wireless connectivity 138, 140, to the local processing and data module 130, such that these remote modules 132, 134 are operatively coupled to each other and available as resources to the local processing and data module 130.
• a remote processing module 132 and remote data repository 134 operatively coupled, such as by a wired lead or wireless connectivity 138, 140, to the local processing and data module 130, such that these remote modules 132, 134 are operatively coupled to each other and available as resources to the local processing and data module 130.
• the couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wires or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR for providing wireless communications.
• all communications may be wired, while in other implementations all communications may be wireless.
• the choice of wired and wireless communications may be different from that illustrated in FIGS. 1- 4. Thus, the particular choice of wired or wireless communications should not be considered limiting.
• the user orientation module is contained in the local processing and data module 130, while CPU and GPU are contained in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained in the local processing and data module 130.
• the 3D database can be associated with the remote data repository 134 or disposed locally.
• Some image display systems use a plurality of volume phase holograms, surface-relief holograms, or light guiding optical elements that are embedded with depth plane information to generate images that appear to originate from respective depth planes.
• a diffraction pattern, or diffractive optical element (“DOE”) may be embedded within or imprinted/embossed upon a light guiding optical element (“LOE”; e.g., a planar waveguide) such that as collimated light (light beams with substantially planar wavefronts) is substantially totally internally reflected along the LOE, it intersects the diffraction pattern at multiple locations and exits toward the user’s eye.
• the DOEs are configured so that light exiting therethrough from an LOE are verged so that they appear to originate from a particular depth plane.
• the collimated light may be generated using an optical condensing lens (a“condenser”).
• a first LOE may be configured to deliver collimated light to the eye that appears to originate from the optical infinity depth plane (0 diopters).
• Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1/2 diopter).
• Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter).
• a stacked LOE assembly it can be appreciated that multiple depth planes may be created, with each LOE configured to display images that appear to originate from a particular depth plane. It should be appreciated that the stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Further, N, 2N or 3N stacked LOEs may be used to generate RGB colored images at N depth planes.
• Multiple-plane focus systems create a perception of variable depth by projecting images on some or all of a plurality of depth planes located at respective fixed distances in the Z direction from the user’s eye.
• multiple-plane focus systems may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 5).
• MR systems can include any number of depth planes 150
• one exemplary multiple-plane focus system has six fixed depth planes 150 in the Z direction.
• 3-D perception is created such that the user perceives one or more virtual objects at varying distances from the user’s eye.
• depth planes 150 are generated closer to the eye, as shown in FIG. 5.
• the depth planes 150 may be placed at equal distances away from each other.
• Depth plane positions 150 may be measured in diopters, which is a unit of optical power equal to the inverse of the focal length measured in meters.
• depth plane 1 may be 1/3 diopters away
• depth plane 2 may be 0.3 diopters away
• depth plane 3 may be 0.2 diopters away
• depth plane 4 may be 0.15 diopters away
• depth plane 5 may be 0.1 diopters away
• depth plane 6 may represent infinity (i.e., 0 diopters away). It should be appreciated that other embodiments may generate depth planes 150 at other distances/diopters.
• the user is able to perceive virtual objects in three dimensions.
• the user may perceive a first virtual object as being close to him when displayed in depth plane 1 , while another virtual object appears at infinity at depth plane 6.
• the virtual object may first be displayed at depth plane 6, then depth plane 5, and so on until the virtual object appears very close to the user.
• all six depth planes may be concentrated on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes could be generated at various cross-sections of the coffee cup, giving the user a highly granulated 3-D view of the coffee cup.
• the image display system 100 may work as a multiple-plane focus system.
• all six LOEs may be illuminated simultaneously, such that images appearing to originate from six fixed depth planes are generated in rapid succession with the light sources rapidly conveying image information to LOE 1 , then LOE 2, then LOE 3 and so on.
• a portion of the desired image, comprising an image of the sky at optical infinity may be injected at time 1 and the LOE retaining collimation of light (e.g., depth plane 6 from FIG. 5) may be utilized.
• an image of a closer tree branch may be injected at time 2 and an LOE configured to create an image appearing to originate from a depth plane 10 meters away (e.g., depth plane 5 from FIG. 5) may be utilized; then an image of a pen may be injected at time 3 and an LOE configured to create an image appearing to originate from a depth plane 1 meter away may be utilized.
• This type of paradigm can be repeated in rapid time sequential (e.g., at 360 Hz) fashion such that the user’s eye and brain (e.g., visual cortex) perceives the input to be all part of the same image.
• the camera of the image display device 101 obtains an image of the environment based on a certain position and orientation of the user 50.
• Such camera image serves as a tracking image (one or more images may be used to create a tracking map) for allowing the processing unit 130 of the image display device 101 to track a position and/or pose and/or orientation of the user 50.
• the processing unit 130 of the image display device 101 processes the image from the camera to determine if features in the image match with certain features in the map 220. If a match is found, the processing unit 130 may then determine the position and orientation of the user 50 based on the matched features.
• the processing unit 130 has been described with reference to providing a target object for display on the screen of the image display device 101 , and instructing the user 50 to follow the target object for improving a value of the metric.
• the processing unit 130 may instruct the user 50 to move to a certain location and/or to change a viewing direction by presenting textual instruction for display on the screen, and/or by presenting an audio instruction using a speaker.
• the processing unit 130 may instruct the user 50 to turn to the right by 30°, etc.
• the desired viewing orientation may be determined by the processing unit 130 in order to maximize the metric value for a particular cell.
• the processing unit 130 may consider the shortest path and/or the quickest time to get a good map.
• the metric may also be stored in association with a certain reference location (e.g., (x, z)).
• the metric for a particular cell 300 may have different values, depending on the reference location. For example, if the reference location is selected to be at location 350 (the centroid of the cell 300) in the illustrated example, there are only three images that are to the east of the reference location 350. Accordingly, only those three images are selected for calculating the metric for the east direction and for the location 350. On the other hand, if the reference location is selected to be at location 352 in the illustrated example, there are five images that are to the east of the reference location 352. Accordingly, the five images are selected for calculating the metric for the east direction and for the location 352.
• the metric may be used by the processing unit 130 of the image display device 101 to select the best map (from among a plurality of available maps) for localization of the user 50. For example, in some embodiments, there may be multiple maps created for the same environment (e.g., for a particular cell 300). In such cases, the processing unit 130 may select the map with the highest value of the metric for use to localize the user 50.
• the map creation process has been described with reference to a camera system that captures images of the environment, and the processing unit is configured to determine the metric based on co-visibility of point(s) of interest that is associated with different camera positions.
• the camera system may include only a single camera (e.g., a forward facing camera) to capture images for map construction.
• the camera system may include two cameras (e.g., two forward facing cameras) to capture images for map construction.
• the camera system of the image display device 101 may include four cameras (e.g., two forward facing cameras, a first side facing camera (e.g., facing left), and a second side facing camera (e.g., facing right)).
• FIG. 10 illustrates an example of a map-and-localization management method 500, for managing creation of localizing map(s) and use of such map(s).
• localizing maps may be tracking maps tied to headpose and specific to a single session, and/or may be a canonical map that the tracking map can localize into.
• a tracking map may become a canonical map if a quality metric associated with the tracking map is above a threshold.
• the method 500 begins at item 502.
• the processing unit 130 of the image display device 101 attempts to localize the user 50 of the image display device 101 using an already created map while the user 50 is in an environment that corresponds with the created map (item 504). If the processing unit 130 successfully localizes the user 50 with respect to the environment using the map, the processing unit 130 may generate a signal to notify the user 50 that the localization is successful (item 506). If the processing unit 130 can successfully localize the user 50 with respect to the environment using the map, that means the user 50 can place virtual content with respect to the environment using the map, and/or may recover content that was previously placed with respect to the environment.
• content that was in the space from the previous session would be presented in the screen of the image display device 101 for presentation to the user in response to a successful localization.
• the processing unit 130 may also inform the user 50 that the recovery of content is successful. For example, the processing unit 130 may generate a signal to operate the screen and/or a speaker to provide the notification to the user 50.
• localization may be a coordinate system transform between the tracking map coordinate system and the canonical map 220 coordinate system, such that the tracking and canonical maps are aligned (e.g. the same) with each other after the coordinate system transform is applied.
• localization may fail if the user 50 is in a new space so that a map has not yet been built before, if there is a lighting change, if there is a change in environment (e.g., a furniture has been moved), if there is a dynamic change (e.g., people moving), if the user 50 is not in same view point as that for the previously built map, if a correct map cannot be identified, if a user’s pose cannot be identified, or any combination of the foregoing.
• the processing unit 130 then starts a map creation session to generate the map (item 510).
• the user 50 performs different head poses to place camera(s) of the image display device 101 in different viewing direction to view different parts of an environment.
• the camera(s) captures images of the environment while the image display device 101 is at the different head poses.
• the processing unit 130 then processes these images to create a map for localization purpose.
• the processing unit 130 may calculate value(s) of the metric to indicate the quality of the map using examples of the techniques described herein.
• the processing unit 130 may calculate value(s) of the metric to indicate the quality of the map when it is a tracking map, and/or may be calculated when the map is a canonical map.
• the processing unit 130 may then ask the user 50 (via a user interface) whether to restore the content from the previous session without placing the content (item 520). If the user 50 decides to restore the content without placement of the content, the processing unit 130 may then provide a user interface for allowing the user 50 to manually restore the content (item 528). On the other hand, if the user 50 decides not to restore the previous content from the previous session, the processing unit 130 may then start a new session (item 524).
• the user 50 may operate the image display device 101 to retrieve another map from a non-transitory medium 526, and/or may create a new map.
• the obtained map may then be used for localization of the user (item 512).
• the maps stored in the non-transitory medium 526 have respective metric values associated with them.
• the processing unit 130 selects one of the maps for localization, the processing unit 130 takes the metric values into consideration.
• the processing unit 130 may be configured to select a map that has a metric value above a prescribed threshold.
• the processing unit 130 may retrieve one or more maps from a map database 530 for storage in the non-transitory medium 526.
• the processing unit 130 also calculates a metric for the map to determine whether the map has a sufficiently good quality (item 606).
• the determination of the metric may be performed using examples of the techniques described herein. If a value of the metric is above a certain threshold, the processing unit 130 may then determine that the quality of the map is sufficient for allowing the user to share content. In such cases, the processing unit 130 of the first image display device 101 may send information regarding the map to a recipient with a second image display device 101 (item 608).
• the information regarding the map may be an identification of the map, a storage location of the map, an invitation to use the map, data regarding the map, data of the map itself, etc., or any combination of the foregoing.
• the recipient may manually place the content with respect to the environment in which the recipient is in.
• the recipient of the content e.g., an image of an apple
• the recipient of the content may place such content on the table in the room.
• localization may occur when the first user’s tracking map localizes into a first canonical map and the second user’s tracking map localizes into a second canonical map, where both canonical maps share the same coordinate system, resulting in both systems accessing virtual content relative to the same coordinate system. Any other method of localization may be used as long as the first user and second user are able to access a shared coordinate system through one or more maps.
• the maps e.g. tracking maps, canonical maps
• the maps may be stored on the local user device, or in a shared cloud, for example.
• Map sanitization As discussed, the map for localization of the user 50 may be created using images captured by the camera system of the image display device 101.
• the processing unit 130 of the image display device 101 may be configured to perform map sanitization to remove undesirable data, and/or to mitigate an effect of undesirable data, that may negatively contribute to the map being created.
• undesirable data may include incorrect observations of reference points, poorly constrained image(s), undesirable map point(s), or any combination of the foregoing.
• the first row shows three images obtained in a sequence that are well-constrained.
• the three images are well-constrained in the sense that each image captures multiple map points (reference points) that are also captured by one or more other images. Accordingly, the three images are well- connected to each other based on the commonly observed map points.
• a map point may be any feature of interest captured in an image that can be used for tracking purpose.
• a map point may be a feature associated with a corner of an object that can be used to identify the same object in different images.
• the processing unit 130 may be configured to determine a quality score for each image, wherein the quality score represents, indicates, or is based on, a number of map points captured in the image.
• the second row in the illustrated figure shows a fourth image being acquired.
• the fourth image only sees a few map points, and is poorly constrained with respect to the first segment.
• the poorly constrained fourth image may have a corresponding quality score that is low due to the poorly constrained nature of the fourth image.
• the fourth image is assigned to a new segment (a second segment) due to the poor quality score arising from the few map point observations.
• the fourth image is poorly constrained due to rapid rotation of the user’s head, which results in blurring of the fourth image. Because the fourth image is blurred, only a few map points can be detected. In other embodiments, the blur in the image may be due to other factors that are different from head rotation.
• the third row shows a fifth image being acquired.
• the fifth image only sees a few map points, and is poorly constrained with respect to the second segment.
• the poorly constrained fifth image may have a corresponding quality score that is low due to the poorly constrained nature of the fifth image. Accordingly, the fifth image is assigned to a new segment (a third segment).
• the fifth image is poorly constrained also due to the continued rapid rotation of the user’s head. In other embodiments, the blur in the image may be due to other factors.
• the act of performing global optimization in item 904 comprises performing bundle adjustment, in which one or more rays from images are adjusted so they have consensus.
• the global optimization may be performed based on images obtained by the camera system of the image display device 101 , orientation of image planes associated with the images, and 3D reference points. In further embodiments, the global optimization may be performed also based on relative orientation of cameras with respect to each other.
• performing the local optimization includes performing a sanitization by the processing unit 130 to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• the processing unit 130 may remove a constraint of the image, or to remove the image, when performing the data sanitization.
• the processing unit 130 may determine a first map segment of the map segments by obtaining images from the camera system, and linking the images, wherein the images are generated in sequence by the camera system.
• the processing unit 130 may start the second map segment when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.
• FIG. 14 illustrates a method 1000 in accordance with some embodiments.
• the method 1000 may be performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit.
• the method 1000 may be performed by any of the image display devices 101 shown in FIGS. 1-4.
• the method 1000 may be performed by the processing unit 130 of the image display device 101.
• the method 1000 includes: obtaining, by the processing unit, output(s) from the camera system (item 1002); determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment (item 1004); and determining, by the processing unit, a metric indicating a likelihood of success to localize the user using the map (item 1006).
• the processing unit may determine the metric by performing computation to obtain the metric. In other embodiments, the processing unit may determine the metric by receiving the metric from another component or device to obtain the metric.
• the other component or device providing the metric may be a module in the image display device 101 , or an external device that is in communication with the image display device, wherein the external device may be worn by the user or may be physically decoupled from the user.
• the external device may be a wireless transmitter, a computer, a handheld or body-worn device, a database, a server, a base station, etc.
• the act of determining the metric comprises computing the metric by the processing unit.
• the metric is determined based on a co- visibility of a point of interest that is associated with different camera positions.
• the metric indicates the likelihood of success to localize the user in one or more viewing directions.
• the metric is determined based on a number of times a point of interest is detected from different camera positions.
• the metric has a value that is based on directionality.
• the directionality is with respect to one or more vertical axes, and/or one or more horizontal axes.
• the directionality comprises a tilt angle.
• the directionality comprises a roll angle.
• the metric has a first value associated with a first directionality, and a second value associated with a second directionality.
• the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, and wherein the metric has a value that is based on a position within the one of the plurality of cells.
• the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, wherein the metric has a first value associated with a first position within the one of the plurality of cells, and a second value associated with a second position within the one of the plurality of cells.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the act of determining the metric comprises determining a total number of images from the camera system that are associated with the one of the plurality of cells.
• the total number of images is associated with a certain viewing direction for the cell.
• the total number of images is associated with multiple viewing directions for the cell.
• the camera system is configured to obtain multiple images, and wherein the metric is determined for one of the plurality of cells by: identifying a subset of the images that belong to a same range of viewing directions; determining respective scores for the images in the subset of the images; and summing the scores to obtain a total score.
• the metric is determined by dividing the total score by a number of the images in the subset of the images to obtain an average score.
• the average score represents an average expected number of co-visibility points for the range of viewing directions for the one of the plurality of cells.
• the respective scores are determined by accessing a co-visibility graph that associates reference points with the multiple images.
• each of the respective scores is determined by determining a number of reference point(s) that is detected in the corresponding one of the images in the subset of images.
• the area score is determined based on a spatial distribution of data points of the map.
• At least one of the cells has a footprint area that is 2m by 2m.
• the at least one of the cells also has a pre- determined height.
• the metric is determined by: obtaining a plurality of images from the camera system; and determining co-visibility values, wherein each of the co-visibility values indicating a number of reference points detected in a corresponding one of the plurality of images.
• the camera system comprises a plurality of cameras.
• the plurality of images comprises a first subset of images generated by the plurality of cameras when the camera system is at a first position.
• the plurality of images comprises a second subset of images generated by the plurality of cameras when the camera system is at a second position.
• the plurality of cameras comprises a first forward facing camera.
• the plurality of cameras comprises a second forward facing camera.
• the plurality of cameras comprises a first side facing camera.
• the plurality of cameras comprises a second side facing camera.
• the method 1000 further includes generating the graphics based on the determined desired viewing direction, the graphics configured to instruct the user to change a current viewing direction of the camera system to the desired viewing direction.
• the method 1000 further includes obtaining an image of the environment from the camera system after the desired viewing direction of the camera system has been achieved.
• the method 1000 further includes updating the map based on the image.
• the method 1000 further includes updating the metric based on the updated map.
• the metric is determined before the map is used to localize the user with respect to the environment.
• the metric is determined before the apparatus shares content with another apparatus.
• the metric is determined during a map construction session in which the processing unit determines the map.
• the metric is determined retroactively by accessing the map that was previously determined from a non-transitory medium.
• the method 1000 further includes performing a sanitization to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• the data comprises an image from the camera system, and wherein the sanitization is performed by removing or disregarding the image.
• camera system comprises a plurality of cameras, wherein the data comprises a set of images generated by the respective cameras, and wherein the sanitization is performed by removing or disregarding the set of images.
• the data comprises an identification of a reference point in an image from the camera system, and wherein the sanitization is performed by disregarding the identification of the reference point.
• the data represents a ray or a line that is associated with an image from the camera system and a reference point, and wherein the sanitization is performed by disregarding the ray or the line that is associated with the image.
• the sanitization is performed as a part of a local optimization.
• the method 1000 further includes performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed to remove the data.
• the bundle adjustment is performed as a part of a global optimization.
• the global optimization is performed based on images from the camera system and three-dimensional reference points
• the global optimization is performed also based on a relative orientation between cameras of the camera system.
• the method 1000 further includes determining, by the processing unit, a score for an image obtained from the camera system.
• the score is a constraint score.
• the score indicates how well the image is constrained.
• the score is determined based on a Jacobian of reference points measurements.
• the score is determined based on an information matrix that is a diagonal matrix.
• the score is determined based on a number of reference points detected in the image.
• the method 1000 further includes performing data sanitization based on the score.
• the act of performing the data sanitization comprises removing a constraint of the image, or removing the image.
• the constraint of the image, or the image is removed when the score is below a threshold.
• the map is determined by: determining multiple map segments; and connecting the map segments.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the camera system is configured to obtain multiple images, and wherein the processing unit is configured to determine the metric for one of the plurality of cells by: identifying a subset of the images that belong to a same range of viewing directions; determining respective scores for the images in the subset of the images; and summing the scores to obtain a total score.
• the processing unit is configured to generate the graphics based on the determined desired viewing direction, the graphics configured to instruct the user to change a current viewing direction of the camera system to the desired viewing direction.
• the data comprises an image from the camera system, and wherein the processing unit is configured to perform the sanitization by (1) removing or disregarding the image, (2) disregarding an identification of a reference point in the image, and/or (3) disregarding a ray or a line that is associated with the image.
• the processing unit is configured to determine a first map segment of the map segments by obtaining images from the camera system, and linking the images, wherein the images are generated in sequence by the camera system.
• the metric indicates a number of reference points that are useable to localize the user with respect to the environment.
• the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.
• the camera system is configured to obtain multiple images, and wherein the metric is determined for one of the plurality of cells by: identifying a subset of the images that belong to a same range of viewing directions; determining respective scores for the images in the subset of the images; and summing the scores to obtain a total score.
• the metric is determined by dividing the total score by a number of the images in the subset of the images to obtain an average score.
• the respective scores are determined by accessing a co-visibility graph that associates reference points with the multiple images.
• each of the respective scores is determined by determining a number of reference point(s) that is detected in the corresponding one of the images in the subset of images.
• the method also includes determining an area score indicating a degree of coverage by the map.
• the metric is determined by: obtaining a plurality of images from the camera system; and determining co-visibility values, wherein each of the co-visibility values indicating a number of reference points detected in a corresponding one of the plurality of images.
• the method also includes generating the graphics based on the determined desired viewing direction, the graphics configured to instruct the user to change a current viewing direction of the camera system to the desired viewing direction.
• the method also includes performing a sanitization to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.
• the method also includes performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed to remove the data.
• the score indicates how well the image is constrained.
• the score is determined based on a Jacobian of reference points measurements.
• the method also includes performing data sanitization based on the score; and wherein the data sanitization is performed to remove a constraint of the image, or to remove the image.
• the act of determining the multiple map segments comprises determining a first map segment of the map segments by obtaining images from the camera system, and linking the images, wherein the images are generated in sequence by the camera system.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Computer Hardware Design (AREA)
• Human Computer Interaction (AREA)
• Computer Graphics (AREA)
• Multimedia (AREA)
• Software Systems (AREA)
• Optics & Photonics (AREA)
• Architecture (AREA)
• Processing Or Creating Images (AREA)
• Image Processing (AREA)
• User Interface Of Digital Computer (AREA)
• Position Input By Displaying (AREA)
• Image Analysis (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=69178530

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (28)

Citations (7)

Family Cites Families (53)

• 2019
  • 2019-07-24 US US16/520,582 patent/US11182614B2/en active Active
  • 2019-07-24 EP EP24193469.4A patent/EP4435736B1/en active Active
  • 2019-07-24 WO PCT/US2019/043154 patent/WO2020023582A1/en not_active Ceased
  • 2019-07-24 EP EP19841887.3A patent/EP3827302B1/en active Active
  • 2019-07-24 CN CN202310122842.3A patent/CN116088783A/zh active Pending
  • 2019-07-24 CN CN201980049093.8A patent/CN112639664B/zh active Active
  • 2019-07-24 JP JP2021503757A patent/JP7309849B2/ja active Active
• 2021
  • 2021-10-13 US US17/500,798 patent/US11687151B2/en active Active
• 2023
  • 2023-05-09 US US18/314,289 patent/US12079382B2/en active Active
  • 2023-07-05 JP JP2023110450A patent/JP7625650B2/ja active Active

Patent Citations (7)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events