BR102013027358A2 - AUTOMATED REFERENCE REFERENCE CALIBRATION STRUCTURE FOR INCREASED REALITY - Google Patents

Abstract

ABSTRACT Patent of Invention: "AUTOMATED REFERENCE CALIBRATION STRUCTURE FOR AUGMENTED REALITY". The present invention relates to one or more systems, methods, routines and / or techniques for automated reference calibration structure for augmented reality (100). One or more systems, methods, routines and / or techniques can allow the calibration of an Augmented Reality (AR) system (100), for example, automatically calibrating the reference structures of virtual objects and / or a camera (102). An example of a calibration routine and / or technique can determine and / or calculate a mapping or transformation from a reference structure of a virtual object (112) (for example, a CAD model) to a camera structure (204) associated with the tracking system (108). Another example of a routine and / or calibration technique can determine and / or calculate a mapping or transformation from a camera lens reference structure (102) to a camera reference structure (102) all as determined by a calibration system. tracking (108). These routines or techniques can calibrate an AR system (100) to provide fast and accurate alignment between virtual content and a live view on the camera (102) of a real scene.

Description

Report of the Invention Patent for "AUTOMATED REFERENCE REFERENCE CALIBRATION STRUCTURE".

FIELD

The present description refers to augmented reality and, more

particularly one or more systems, methods, routines and / or techniques for augmented reality reference calibration enhanced structure.

BACKGROUND

Augmented Reality (AR) increases or additions to the perception of a real world view, for example, a live video feed, superimposing virtual objects or media on the real world view. Augmented Reality enables artificial or simulated information related to the real world and its goals to be superimposed on the real world view. Augmented reality is related to, but different from, virtual reality (VR), which replaces a real-world view with an artificial or simulated view. Augmented Reality has been used in applications such as entertainment, video games, sports and mobile phone applications.

Additional limitations and disadvantages of conventional and traditional approaches will become apparent to one skilled in the art by comparing such systems with some aspects of the present invention as set forth in the remainder of the present application and with reference to the drawings.

SUMMARY

The present disclosure discloses one or more systems, methods, routines and / or techniques for automated augmented reality reference calibration framework. One or more systems, methods, routines and / or techniques may enable simple and rapid calibration of an Augmented Reality (AR) system, for example by automatically calibrating virtual object reference structures and / or a camera.

One or more embodiments of the present disclosure describe a

augmented reality method, for example, performed by a data processing system having at least one processor. The method may include receiving or establishing a coordinate tracking system structure associated with an object tracking system. The coordinate structure of the tracking system can be aligned with real 3D space. The tracking system can track the position and orientation in real 3D space of a real object and a camera. The method may include receiving from the tracking system a first real object reference structure for the real object. The first real object reference structure can indicate a position and orientation of the real object with respect to the coordinate structure of the tracking system. The method may include determining a second real object reference structure for the real object. The second real object reference structure may indicate a position and orientation of the real object with respect to the coordinate structure of the tracking system. The method may include receiving a first virtual object reference structure for a virtual object. The virtual object can be modeled after the real object. The first virtual object reference structure may be unrelated to the coordinate structure of the tracking system. The method may include determining a second virtual object reference structure for the virtual object. The second virtual object reference structure may indicate a position and orientation of the virtual object with respect to the coordinate structure of the tracking system. The method may include determining a virtual object mapping between the first virtual object reference structure and the coordinate structure of the tracking system. The method may include showing an enlarged scene including a 3D real space view, a real object view, and one or more overlapping virtual items. Virtual object mapping can be used to place one or more overlapping virtual items in the enlarged scene so that one or more virtual items align with the actual object.

One or more embodiments of the present disclosure describe an augmented reality method, for example, performed by a data processing system having at least one processor. The method may include or establish a coordinate tracking system structure associated with an object tracking system. The coordinate structure of the tracking system can track the position and orientation in real 3D space of a camera that captures real 3D space and a printed marker. The method may include receiving from the tracking system a frame of reference from camera to camera. The camera reference frame may indicate a portion and orientation of the camera with respect to the coordinate structure of the tracking system. The method may include receiving or establishing a coordinated printed marker structure associated with the printed marker. The coordinate structure of the printed marker can be aligned with real 3D space. The coordinate structure of the printed marker can be aligned with the coordinate structure of the tracking system. The method may include determining a camera lens reference structure for the camera lens. The camera lens reference frame can indicate a camera position and orientation relative to the printed marker coordinate frame. The method may include determining a camera lens mapping between the camera reference frame and the camera reference frame. The method may include showing an enlarged scene including a real 3D space view and one or more virtual items. Camera lens mapping can be used to alter or distort one or more virtual items in the enlarged scene.

One or more embodiments of the present disclosure describe a system. The system may include a camera that captures a view of a real 3D space including a real object. The system may include a tracking system that tracks the position and orientation in real 3D space of the actual object and camera. The tracking system can be configured to establish a coordinate structure of the tracking system associated with the tracking system, and the coordinate structure of the tracking system can be aligned with actual 3D space. The system may include a computer attached to the camera and tracking system, and the computer may include one or more memory units. The computer can be configured with a virtual modeler. The virtual modeler can be configured to receive from the tracking system a first real object reference structure for the real object, where the first real object reference structure can indicate a position and orientation of the real object with respect to the sis5 coordinate structure. Tracking theme. The virtual modeler may be further configured to compute a second real object reference structure for the real object, where the second real object reference structure may indicate a position and orientation of the real object with respect to the coordinate structure of the tracking system. The virtual modeler may further be configured to receive from one or more memory units a first virtual object reference structure of a virtual object, where the virtual object may be modeled after the real object, and where the first reference structure of virtual object may not be related to the coordinate structure of the tracking system. The virtual modeler may further configure to compute a second virtual object reference structure for the virtual object, where the second virtual object reference structure may indicate a position and orientation of the virtual object with respect to the coordinate structure of the tracking system. The virtual modeler may be further configured to compute a virtual object mapping between the first virtual object reference structure and the coordinate structure of the tracking system. The virtual modeler may further be configured to generate and store in one or more memory units an enlarged scene including a 3D real space view, a real object view and one or more overlapping virtual items. Virtual object mapping can be used to place one or more overlapping virtual items in the enlarged scene so that one or more virtual items are aligned with the actual object.

One or more embodiments of the present disclosure describe a data processing system that includes one or more memory units that store computer code and one or more processor units coupled to one or more memory units. One or more processor units may execute computer code stored in one or more memory units to receive or establish a coordinate tracking system structure associated with an object tracking system. The coordinate structure of the tracking system can be aligned with real 3D space. The tracking system 5 can track the position and orientation in real 3D space of a camera that captures real 3D space and a printed marker. One or more processor units may execute computer code stored in one or more memory units to receive from the tracking system a frame of reference from camera 10 to camera. The camera reference frame may indicate a camera position and orientation with respect to the coordinate structure of the tracking system. One or more processor units may execute computer code stored in one or more memory units to receive or establish a coordinate printed marker structure associated with the printed marker. The coordinate structure of the printed marker can be aligned with the actual 3D space, and the coordinate structure of the printed marker can be aligned with the coordinate structure of the tracking system. One or more processor units may execute computer code stored in one or more memory units to determine a camera lens reference structure for the camera lens. The camera lens reference frame may indicate a position and orientation of the camera lens relative to the printed marker coordinate frame. One or more processor units may execute computer code stored in one or more 25 memory units to determine a camera lens mapping between the camera reference frame and the camera lens reference frame. One or more processor units may execute computer code stored in one or more memory units to show an enlarged scene including a view of real 3D space and one or more virtual items. Camera lens mapping can be used to alter or distort one or more virtual items in the enlarged scene. These and other advantages, aspects and novel features of the present disclosure, as well as details of an illustrated embodiment thereof, will be more fully understood from the following description and drawings. It is to be understood that the aforementioned general descriptions 5 are exemplary and explanatory only and are not restrictive of the description as claimed.

BRIEF DESCRIPTION OF DRAWINGS

Several features and advantages are described in the following description, in which various embodiments are explained, using the following drawings as examples.

FIG. 1 depicts a block diagram showing exemplary devices, components, software, and interactions of an augmented reality (AR) system according to one or more embodiments of the present disclosure, where the automated structure of reference calibration techniques discussed herein may be useful in such an RA system.

FIG. 2 depicts a block diagram showing an example of calibration technique according to one or more embodiments of the present disclosure.

FIG. 3 depicts an illustration of a tool or stick used for various reasons by a tracking system according to one or more embodiments of the present disclosure.

FIG. 4A depicts an illustration of an example real object with a number of tracking markers attached to or placed on the real object according to one or more embodiments of the present disclosure.

FIG. 4B depicts an illustration of how a tracking system can create and place a representation of a real object according to one or more embodiments of the present disclosure.

FIG. 5 depicts an illustration of how virtual modeling software can establish a new frame of reference for a real object according to one or more embodiments of the present disclosure.

FIG. 6 depicts an illustration of how virtual modeling software can establish a new reference structure for a virtual object according to one or more embodiments of the present disclosure.

FIG. 7 depicts a block diagram showing an example of calibration technique according to one or more embodiments of the present disclosure.

FIG. 8A depicts an illustration of an example camera and camera structure according to one or more embodiments of the present disclosure.

FIG. 8B describes an illustration of how a ras system

You may create and place a representation of a camera according to one or more embodiments of the present disclosure.

FIG. 8C depicts an illustration of a personal computer with an integrated camera according to one or more embodiments of the present disclosure.

FIGURES 9A and 9B describe illustrations of how a printed marker can make it possible to determine a reference frame of a camera lens.

FIG. 10A depicts an illustration of an example enlarged scene that can be produced according to one or more embodiments of the present disclosure.

FIG. 10B depicts an illustration of an example enlarged scene that can be produced according to one or more embodiments of the present disclosure.

FIG. 11 depicts a flowchart showing an example of eta

in a method for automated augmented reality reference calibration structure according to one or more embodiments of the present disclosure.

FIG. 12 depicts a flowchart showing example steps in a method for automated augmented reality reference calibration structure according to one or more embodiments of the present disclosure. FIG. 13 depicts a block diagram of an example data processing system that may be used to implement one or more embodiments of the present disclosure.

DETAILED DESCRIPTION In many RA systems a tracking system may be

Used to track the position and orientation of a camera and various real world objects in a 3D space. For example, a tracking system may track a camera and a piece of machinery that the camera is viewing / capturing. Various RA systems may attempt to create an enlarged scene that includes a real-world scene as captured by the camera (including various real-world objects) and overlapping media and / or virtual objects. To create the enlarged scene, the tracking system can establish a virtual coordinate structure and can track or "place" representations of real world objects in that coordinate structure. Several RA systems may attempt to "place" multiple virtual objects (for example, CAD models / objects) into the coordinate structure to create an enlarged scene. Virtual objects / models can have their own standard or arbitrary reference structure (eg, 3D position and orientation) and therefore, placing a virtual object in the coordinate structure of the tracking system, a mapping or transformation must be determined between the coordinate structure of the tracking system and the reference structure of the virtual object. Additionally, if the camera (for example, the camera captures the real world scene) moves, an AR system may attempt to change the view of virtual objects. In order to do this accurately, an RA system may need to track the position and orientation of the camera lens. However, a tracking system can only track the position and orientation of the entire camera. Several software programs (for example, in conjunction with other parts) may be used to determine a reference frame for the camera lens in a coordinated frame, but these lens software programs may track the lens into an established coordinate frame. by the lens software program. Therefore, to place the camera lens in the coordinate structure of the tracking system, a mapping or transformation must be determined between the coordinate structure of the tracking system and the reference frame of the lens. Determining these mappings and / or transformations (for example, for virtual objects and / or 5 for the camera lens) can be referred to as the AR calibration system or calibration reference frameworks.

It should be understood that the terms "coordinate structure", "reference structure" and "pose" may be used throughout this description and may be closely related. The term "coordinate structure" may refer to a 3D representation of a 3D space, where the coordinate structure includes three planes or axes (for example, an X axis, a Y axis, a Z axis) and an origin (for example). , a point where the three axes intersect). The term "reference structure" may refer to a 3D location and 3D orientation of an object or point, for example, in a coordinate structure. An object's reference structure can include an origin (for example, an approximate center of mass) for the object and an object orientation (for example, 3 axes established relative to the object). The term "pose" is short for "position and orientation" and can refer to a 3D position (eg, X, Y, Z coordinates) and a 3D orientation (eg, cylinder, pitch, yaw) of a 3D space object.

Several RA systems may perform RA system calibration through a manual or trial and error process, for example by approximating the virtual model reference frame and / or the camera lens 25 with respect to the coordinate structure of the tracking system. and then testing the enlarged scene to determine if the approach was a good one. For example, a technician can simply look at the overlapping virtual objects in the enlarged scene and make a determination as to whether they appear to be in their correct location in various camera locations and orientations. This manual calibration process may require manipulating twelve parameters, for example, six parameters for a virtual object (for example, X, Y, Z and cylinder coordinates, pitch, yaw) and six parameters for a camera lens (for example, X, Y, Z coordinates and cylinder, pitch, yaw). This process can be expensive and / or time intensive, for example, taking many hours (eg, more than 8 hours) to complete. Even when the manual calibration process is complete, it may still not result in an accurate solution / calibration. For example, a virtual object that appears to be appropriately from one camera pose may not appear to be properly placed from different poses. Minor errors in virtual object placement can lead to large errors in larger real-world objects. Additionally, every 10 minutes of the AR system is adjusted in a new environment or by a new real object or camera, the AR system must be manually calibrated.

The present disclosure describes one or more systems, methods, routines and / or techniques for automated augmented reality reference calibration framework. One or more systems, methods, routines and / or techniques may enable simple and rapid calibration of an Augmented Reality (AR) system, for example by automatically calibrating the reference structures of virtual objects and / or a camera. One or more systems, methods, routines and / or techniques may enable the AR system to be adjusted in a new environment or new real object (eg, a piece of machinery) in a relatively short amount of time (eg. less than 15 minutes) and can enable precise alignment of overlapping virtual content with a real-world scene. Accurate alignment of virtual content can be critical if the RA system is being used to instruct a technician to perform a precise task, such as drilling a small hole in a precise location. One or more systems, methods, routines, and / or techniques may determine and / or compute mappings or transformations between various reference structures (for example, the coordinate structure of the tracking system, the reference structure of a virtual object, and the structure of camera lens reference 30). The present disclosure may describe two or more calibration and / or technical routines. The first routine and / or calibration technique may determine and / or calculate a mapping or transformation between a reference structure of a virtual object (for example, a CAD model) and a coordinate structure associated with the tracking system. The second routine and / or calibration technique may determine and / or calculate a mapping or transformation between a camera lens reference frame and an entire camera reference frame by a tracking system. These routines and / or techniques can calibrate an AR system to provide fast, accurate alignment between virtual content and a live camera view of a real scene.

FIG. 1 depicts a block diagram showing exemplary devices, components, software, and interactions of an augmented reality (AR) system 100 according to one or more embodiments of the present disclosure, where the automated structure of reference calibration techniques discussed herein. may be useful in such an RA system. The AR 100 system may include a camera 102 that can capture and / or transmit a live video feed of a real world scene. The real world scene can include one or more real objects, for example real object (RO) 104. RO 104 can be one of several objects, for example a tool, a piece of machinery, a large satellite, a box control panel, control panel, or various other objects. Camera 102 may be in communication with a computer 106, where the computer may interpret and / or process information (e.g., live broadcast video) sent from the camera related to real world scenes and / or objects captured by the camera.

The AR 100 system may include a tracking system 108 108. The tracking system 108 can track the "pose" (position and orientation in 3D space) of the actual object 104 and camera 102, and can transmit this information (for example). example, in real time) to a computer (for example, computer 106) or another component. The tracking system 108 may include various components, for example, a number of tracking markers, a number of detection devices for detecting tracking markers, and a base computing device that may operate in association with the tracking system software. . In one example, each marker may be a small sphere (for example, a 10 mm sphere) with a reflective coating that is designed to reflect certain wavelengths of light. In this example, markers may be placed in various places and / or on various objects in real world space so that tracking system 108 can track the position and / or orientation of certain points and / or objects in 3D space. . For example, a number (for example, three or more) of tracking markers may be placed on the actual object 104, and a number (for example, three or more) of tracking markers may be placed on camera 10 102.

Tracking system detection devices 108 may be cameras that are designed to detect the 3D space location of tracking markers. For example, each camera may be an infrared camera that is designed to detect flexions of various tracking markers (for example, tracking markers placed on camera 102 and real object 104). The various detection devices (eg infrared cameras) can be placed and / or mounted at various locations around 3D space, for example, a number (eg eight or more) cameras can be mounted on walls of an area or laboratory, for example, mounted in an arrangement so that the 3D space of interest is largely covered by the viewing ranges of the various cameras. The various detection devices of the tracking system 108 may be in communication (for example, over a real time communication link such as Ethernet, Wi-Fi or the like) with a base computing device, where the scanning device Basic computing can operate associated with tracking system software. Tracking system software can process data from various detection devices. Tracking system 108 may be in communication (e.g., by a real time communication link 30 such as Ethernet, Wi-Fi or the like) with a computer 106. The computer may be the computer that is in communication with the camera 102. In some embodiments, the base computing device of the tracking system 108 may be the same computing device as the computer 106.

In some embodiments, camera 102 may be integrated into computer 106. In some examples, computer 106 may be a mobile device, for example, a personal computer, smartphone, PDA or the like. As a specific example, a computer 106 may be a personal computer (see FIG. 7C, for example) with an integrated camera. A mobile device with an integrated camera can provide flexibility and freedom of movement for a user. For example, a user might see an enlarged scene that includes a real object (for example, a real piece of machinery), and the user could walk around the real object, viewing different parts or angles of the real object. Additionally, the user can view virtual content on the desk screen that assists the user in performing a task, for example, virtual content may include instructions, arrows, hardware, tools or the like that may instruct the user how to work on or with the real object. The personal computer in this example (for example, a computer 106) may include virtual modeling software 110. The reference calibration in this example may be in communication (for example, by a real time communication link such as Ethernet, Wi-Fi). -Fi or the like) with the tracking system 108 (for example, the base computing device of the tracking system).

Computer 106 may include virtual modeling software 110. Virtual modeling software may access or load several 25 virtual objects, for example, virtual object (VO) 112. Virtual objects (for example, VO 112) may be designed in one of several known ways to create virtual objects and / or computer aided design (CAD) and / or models. Virtual objects / CAD / models can be created using CAD software, for example, software that uses graph-based vector or the like to describe an object, for example, an object modeled after a real-world object. Virtual objects / CAD / models can be 3D objects that are responsible for, in detail, the various 3D features of the real world object. Virtual object 112 may be a virtual representation of real object 104. Computer 106 can access or load various other virtual objects in addition to exactly virtual objects representing real objects in real world space. As an example, the actual object 104 may be a piece of machinery, and the virtual object 112 may be a virtual representation of the same piece of machinery. Additionally, other virtual objects cannot have real-world counterparts, for example, virtual objects could represent hypothetical screws, tools, wires, and the like 10 that show a technician how to interact with the real object 104.

Virtual modeling software 110 may receive data (e.g., simultaneous real time data) from tracking system 108, for example, the coordinate system established by the tracking system, camera reference frame 102 and reference frame 15 of a real object 104. Virtual modeling software 110 can perform various routines, techniques, and the like described herein to create an enlarged scene (e.g., enlarged scene 114), for example, a real-time view of real world space. as captured by camera 102 enlarged and / or overlaid with virtual objects. Virtual modeling software 110 may perform various routines and / or calibration techniques as described herein to align coordinate structures and a camera lens with the coordinate structure associated with the tracking system. Once calibration is achieved, virtual modeling software 110 can maintain correlation and / or alignment between multiple virtual objects and a live real-world scene. The computer 106 may include or be in communication with a display 116 which may show the enlarged scene 114 to a user. Virtual modeling software 110 can produce an enlarged scene 114 (shown on display 116) showing virtual objects placed on a live video feed. Virtual modeling software 110 may appropriately deform (e.g., change 3D location, 3D orientation, and / or 3D size) virtual objects in the enlarged scene, for example, depending on the relative camera pose 102 and / or the pose of the real object 104. For example, if camera 102 moves further away from the real object 104, one or more virtual objects in the enlarged scene may shrink. As another example, if camera 102 moves closer to the actual object 104, one or more virtual objects may widen. As another example, if camera 102 moves at an angle relative to the actual object 104, one or more virtual objects would rotate appropriately.

FIG. 2 depicts a block diagram showing an example of calibration technique according to one or more embodiments of the present disclosure. More specifically, FIG. 2 shows an automated calibration technique reference structure that can be used to determine and / or calculate a mapping or transform between virtual object reference structures (for example, virtual objects added to an enlarged scene) and the reference structure. associated with the tracking system. As can be seen from FIG. 2, a tracking system 202 (e.g., similar to tracking system 108 of FIG. 1) may establish a coordinate structure 204, for example as part of a tracking system adjustment performed by a technician. The coordinate structure of the tracking system may include three axes (eg, X axis, Y axis, Z axis) and an origin where the three axes intersect. The tracking system can "place" or associate the origin of the coordinate structure with a particular point in a real world 3D space and can orient the coordinate structure with respect to the real world space. The tracking system may utilize a tool or stick to establish its coordinate structure, for example, a stick similar to stick 300 shown in FIG. 3. With reference to FIG. 3, rod 300 may resemble the letter "T" and may include a first extended member (e.g., member 302) which may designate a first axis (e.g., X axis) and may include a second extended member (e.g. , member 304) which may designate a second axis (e.g., Z axis). Rod 300 may also designate an origin at point 306 where first member 302 / axis and second member 304 / axis intersect. A third imaginary axis (for example, Y axis) may operate through the origin point 300. As an example, the rod 300 may be placed on the floor of an area or laboratory, and the tracking machine may establish its coordinate structure by detecting the tracking stick and / or markers (e.g., tracking markers 308, 310, 312, 314, 316) attached to the stick. The tracking system can establish a virtual origin and three virtual axes that associate with the origin and axes as designated by stick 300. Once the coordinate system of the tracking system is established, the tracking system can track an actual object ( for example, equipped with three or more tracking markers) in the area or laboratory and determine its pose within the coordinate structure and determine the orientation of the object with respect to the three axes.

Referring again to FIG. 2, tracking system 202 may determine a frame of reference 206 for a real object. In other words, tracking system 202 can track the actual object. The actual object may be similar to the actual object 104 of FIG. 1, for example. The actual object (RO) can be one of several objects, for example, a tool, a machinery part, a large satellite, a control box, a control panel, or various other objects. FIG. 4A shows an example of a real object 400 - a punch sharpening tool. In order for the tracking system to track (i.e. determine a frame of reference) the actual object 400, a number (e.g. three or more) of tracking markers 402, 404, 406) may be attached to or placed on the object. For proper tracking, tracking markers may have to be placed properly on the actual object 400, for example, in a non-collinear, non-symmetrical arrangement. Three or more points are said to be collinear if they are in a single straight line. Therefore, a non-collinear arrangement of tracking markers means that tracking markers are arrangements so that markers are not all in a straight line. For proper tracking, at least three non-collinear tracking markers can be placed on the actual object. More than three tracking markers can be placed on the real object, for example, to improve the reliability and / or accuracy of calculations, for example if the view of one of the tracking markers is obstructed.

The tracking system (for example, including a number of infrared cameras) can detect tracking markers and create and place a representation of the actual object in the coordinate structure stabilized by the tracking system. FIG. 4B shows an illustration of how a tracking system can create and place a representation of a real object. The tracking system can detect the location of tracking markers (e.g. markers 402, 404, 408 and perhaps more markers not shown in FIG. 4A) and can create and place points (e.g., points 452, 454, 456 and 458), respectively associated with the tracking markers, in the coordinate structure of the tracking system. From these points (for example, points 452, 454, 456, and 458), the tracking system can determine an origin (for example, point 460) and an orientation (see cube and guidance lines surrounding point 460) for the representation of the real object. The origin can be determined by calculating a centroid (for example, a center of mass) of points 452, 454, 456, and 458. The orientation can be adjusted to match (or relate to) the orientation of the tracking system coordinate system. Since the tracking system determines a frame of reference (for example, an origin / location and orientation associated with the tracking system's coordinate structure) for the actual object (RO), the tracking system can transmit pose information. from the real object to the virtual modeling software. Transmission of pose information about the real object can update in real time as the real object can move and / or rotate.

Referring again to FIG. 2, virtual modeling software 210 may establish a new reference structure 212 for the real object (RO). Virtual modeling software 210 may be similar to virtual modeling software 110 of FIG. 1, for example. Virtual modeling software 210 may use the same coordinate structure as one associated with the tracking system. The new RO 212 referenced structure can specify different (when compared to the RO 206 reference structure) reference points in the actual object and can determine and / or calculate a different origin. Establishing a new RO reference framework can allow virtual modeling software to choose reference points in the real object that are the same (or very close to) reference points in a virtual object (for example, a CAD model). which is associated with the actual object.

FIG. 5 shows an illustration of how virtual modeling software can establish a new reference structure for a real object, for example real object 500. A number of reference points (for example points 502, 504, 506) can be indicated. on the actual object 500. These reference points may be indicated, for example, by a technician using a tool or stick, for example a stick similar to stick 300 of FIG. 3. The tool or stick may be traceable by the tracking system, for example, the position between the tracking markers attached to the stick with respect to each other may be determined by the tracking system, enabling accurate point collection. Virtual modeling software can use tracking system position data to record landmarks on the real object. As an example, virtual modeling software may recognize a point on the stick as a "pointer" (for example, a tip of a stick extension member). A technician can touch the pointer at various points on the actual object (points 502, 504, 506), and via the stick and tracking system, virtual modeling software can capture or record these points and place them in the associated coordinate structure. with the tracking system. To determine the new reference structure, the reference points may have to be placed properly on the real object 500, for example, in a non-collinear arrangement. At least three non-collinear reference points can be placed on the real object. More than three reference points can be placed on the real object, for example, to improve the reliability and / or accuracy of calculations. From these reference points (eg points 502, 504, 506), virtual modeling software can determine an origin (eg point 508) and an orientation (see axis lines extending from point 508) to the real object. . The origin can be determined by calculating a centroid (for example, a 5 center of mass) of points 502, 504, 506. An orientation to the real object can be determined by placing two axes (for example, X axis, axis). Z) extending from the origin in the plane created by points 502, 504, 506.

Referring again to FIG. 2, once the new RO reference frame 212 is established, virtual modeling software 210 can calculate translational and / or rotational differences between the new RO reference frame 212 and the system-determined RO reference frame 206 Tracking

With reference to FIG. 2, virtual modeling software 210 can access or load various virtual objects, for example, pre-designed CAD models. Virtual modeling software 210 may place the virtual objects in the coordinate structure associated with the tracking system, but the RA system may need to be calibrated before proper placement operates. A virtual object (for example, a CAD model) can have its own reference structure (for example, an origin and three orientation axes), for example, as specified when the virtual object was assigned. It may be the case (for example, for a particular environment) that all virtual objects referenced by the virtual modeling software may share the same reference structure (eg VO 214 reference structure). To place virtual objects in the coordinate structure associated with the tracking system, virtual modeling software 210 can determine a mapping or transformation between the VO 214 reference frame of the virtual objects and the coordinate structure 204 associated with the tracking system. To calibrate the RA system (e.g., determine mapping or transformation), virtual modeling software 210 may use a virtual object (e.g. virtual object 112 of FIG. 1) that corresponds to a real object (e.g., real object 104 of FIG. 1) that the camera (e.g., camera 102 of FIG. 1) of the AR system is capturing. This virtual object can have a VO 214 reference structure.

Virtual modeling software 210 can establish a new VO 216 reference structure for the actual virtual object. Virtual modeling software 210 may use the same coordinate structure as one associated with the tracking system. The new VO 216 reference frame may have a different origin and orientation when compared to the VO 214 reference frame. Establishing a new VO reference frame may allow virtual modeling software to choose reference points in the virtual object that same (or very close to) reference points as indicated (as explained above) with respect to the corresponding real object, and may enable alignment (see generally paragraph 218) between the new reference structure of RO 212 and the new VO 216 reference frame. Alignment between the new RO 212 reference frame (associated with the actual object) and the new VO 216 reference frame (associated with the virtual object) can be achieved, for example, by choosing the same reference points in both the real object and the virtual object and performing the same source and orientation calculation for each.

FIG. 6 shows an illustration of how virtual modeling software can establish a new VO reference structure for the virtual object, for example virtual object 600. Note that for calibration purposes, virtual object 600 can be a modeled virtual object. 25 because a real object associated with the AR system camera is capturing, for example, the real object 500 of FIG. 5. A number of waypoints (for example, points 602, 604, 606) can be selected in virtual object 600. These waypoints can correspond (for example, be in the same relative position as) to waypoints that are 30 were chosen, nominated and / or registered to create the new RO reference framework (see FIG. 5 and related discussion). As with determining the RO reference structure, the reference points for determining the new VO reference structure can be non-collinear arrays, and at least three collinear reference points can be selected in the virtual object. From these reference points (e.g., points 602, 604, 606), virtual modeling software can terminate an origin (e.g., point 608) and an orientation (see extending axis lines from point 608) to the virtual object. The origin can be determined by calculating a centroid (for example, a center of mass) of points 602, 604, 606. An orientation for the virtual object can be determined by placing two axes (for example, X axis, Z axis) that are 10 they tend to originate from the plane created by points 602, 604, 606.

Referring again to FIG. 2, once the new VO 216 reference framework is established, virtual modeling software 210 can calculate translational and / or rotational differences between the new VO 216 reference framework and the VO 214 reference framework associated with the virtual object

As explained above, a virtual object, which is modeled after an associated real object in 3D space, may be required to calibrate the AR system, for example to determine a new VO 216 reference structure that can be aligned with However, once calibration is complete, it should be understood that several other virtual objects can be placed (for example, by virtual modeling software) in the coordinate structure associated with the system. tracking. With reference to FIG. 2, can be seen why this positioning works. To place a virtual object in the camera frame associated with the tracking system, a mapping or transformation (for example, Transformation M4 shown in FIG. 2) must be determined between the reference frame of virtual object 214 (for example, origin and orientation) and the coordinate structure 204 associated with the tracking system. Transformation M4 may not be known before the calibration process is complete. The calibration process, as explained above, can determine several other mappings or transformations that are related to the M4 transformation. As shown in FIG. 2, the calibration process can determine Transformation M1 (ie where the tracking system places the actual object tracked in its coordinate structure), Transformation M2 (ie the translational and rotational differences between the RO reference structure). 5 206 and the new RO 212 reference structure) and the M3 Transformation (i.e., translational and rotational differences between the VO 214 reference structure and the new VO 216 reference structure). Once the transformations M1, M2 and M3 are known, the transformation M4 can be calculated. Once the M4 transformation is known, several virtual objects can be placed in the coordinate structure associated with the tracking system. Additionally, since the information (for example, real object pose) of the tracking system is transmitted to the virtual modeling software, if the transformation M1 changes (ie, the real object pose in 3D space), the M4 transformation can update. , for example, at time re15 al. In this respect, virtual objects may overlap in a real-world scene, and the appearance of virtual objects may change appropriately, for example, as an associated real object in scene movements.

M4 transformation as shown in FIG. 2. The various transformations (M1, 20 M2, M3, M4) as shown in FIG. 2 may each be represented as a transformation matrix, for example, a 4x4 transformation matrix as commonly used in 3D computer graphics. The transformation M1 can be represented as the transformation M4 shown in Eq. 1 below. The transformation M2 can be represented as the transformation matrix shown in Eq. 2 below. Transformation M3 can be represented as the transformation matrix shown in Eq. 3 below.

The following explains an example of the technique for computing M. M.

DCBi1 V1]

. OOO li

'DCM ^ v ^}

, DO0 ~ 11

(Eq. 1)

(Eq. 2)

(Eq. 3) Each transformation matrix can include either a rotational or orientation component (DCMn or "Direction Cosine Array") and a translational or location component (Vn). For example, DCMn represents the rotary matrix for the Mn transformation, and Vn represents the translational vector 5 for the Mn transformation. The rotational component (DCMn) can be a 3x3 matrix that represents a change in orientation between two objects. The DCMn component can represent three values - a change in the cylinder (for example, rotation about a Y axis), and a change in yaw (for example, rotation around a Z axis). These three 10 values can be expanded into a 3x3 DCMn matrix to fit properly into the 4x4 Mn transformation matrix. A familiar person with matrix multiplication and transformation matrix will realize that a transformation matrix must be populous in an appropriate manner so that multiplication of one matrix by the others results in the desired transformation. The translational component (Vn) can be a 1x3 matrix (that is, 3 numbers in a vertical column) that represents the change in the location of two objects (for example, the change in location between two object origins). The translational component (Vn) can include three values - a change in 3D location relative to the X axis, a change in 3D location relative to the Z axis. When the rotational component (DCMn) and the translational component (Vn) are added to the transformation matrix (Mn) and filled with "0001" in the background series (to do the matrix multiplication work), the transformation matrix is complete.

The M4 transformation can then be calculated as shown.

in Eq. 4 below, resulting in the M4 transformation shown in Eq. 5 below.

In some embodiments, since the M4 transformation is

PCM1 V1

Lõõo í

SCM4 v *

000 1

DCM2 ΐ ?,] pCM3 v.

L 000 li and SOO Ü

(Eq. 4) (Eq. 5) calculated, remains the same. As can be seen from FIG. 2, since the M4 transformation is calculated, it can represent the transformation from a virtual object reference structure to the TS-determined RO Reference Structure. The M4 transformation can be used to place multiple virtual objects in a coordinate structure associated with the tracking system, for example, placed with a pose that is related to the TS Reference Structure RO. As the information (for example, real object pose) of the tracking system is transmitted to the virtual modeling software, if the M1 transformation changes 10 (ie the real object pose in 3D space), the multiple object pose can update, for example, in real time.

Again with reference to FIG. 1, camera 102 may also need to be calibrated before the AR system can accurately deform, alter or align virtual objects as camera 102 moves. The challenges and solutions associated with camera 102 calibration can be similar to those associated with virtual object calibration with respect to the coordinate structure associated with the tracking system, as described above. To achieve accurate alignment between virtual objects and a real-world scene (as captured by camera 102), virtual modeling software may need to track the camera lens pose 102, not exactly the camera body as a whole. Several methods for calibrating cameras involved a lengthy trial and error process (eg taking several hours) to manually manipulate six parameters (eg X, Y, Z, cylinder, pitch, yaw) associated with the camera lens. With these manual methods, camera placement is not assured even after calibration is complete.

FIG. 7 depicts a block diagram showing an example of calibration technique according to one or more embodiments of the present disclosure. More specifically, FIG. 7 shows an automated frame of reference calibration technique that can be used to determine and / or calculate a mapping and / or transformation between a camera's reference frame as tracked by a tracking system and the lens reference frame. from the camera. As can be seen from FIG. 7, a tracking system 702 (for example, similar to tracking system tracking system 108 of FIG. 1) may establish a coordinate structure 704, for example as part of a tracking system adjustment performed by a technician ( explained in detail above). Tracking system 702 may determine a frame of reference 706 for a camera. In other words, tracking system 702 can track the camera. The camera may be similar to camera 102 of FIG. 1, for example. The camera may be a standalone camera or may be incorporated into a computer, for example, the computer operating virtual modeling software. FIG. 8A shows an illustration of an example of camera 800. For the tracking system to track (i.e. determine a frame of reference for) camera 800, a number (for example, three or more) of tracking markers 15 (e.g. (eg tracking markers 802, 804, 806) can be attached or replaced on camera 800. In some embodiments, tracking markers may be attached to the camera body itself. In other embodiments, tracking markers may be attached to a frame 801 containing or supporting camera 800, as shown in the example of FIG. 8A. For proper tracking, tracking markers may have to be placed properly on camera 800, for example, in a non-collinear arrangement. For proper tracking, at least three non-collinear tracking markers can be placed on the camera (or camera frame). More than three tracking markers may be placed in the camera, for example, to improve the reliability and / or accuracy of calculations, for example, if vision of one of the tracking markers is obstructed.

The tracking system (for example, including a number of infrared cameras) can detect tracking markers on the camera (or camera frame) and can create and place a camera representation on the coordinate frame established by the tracking system. FIG. 8B shows an illustration of how a tracking system can create and place a representation of a camera. The tracking system can detect the location of tracking markers (for example, 802, 804, 806 markers, and perhaps several other markers) and can create and place points (for example, points 852, 854, 856, and perhaps several others), associated with the tracking markers, in the coordinate structure of the tracking system. From these points (for example, points 852, 854, 856, and perhaps several others), the tracking system can determine an origin (for example, point 860) and an orientation (see cube and guidance lines surrounding the point). 860) for the representation of the real object. The origin can be determined by calculating a centroid (for example, a center of mass) of points 852, 854, 856 and perhaps several others. The orientation may be determined to match (or be related to) the orientation of the tracking system coordinate system. Since the tracking system determines a frame of reference (for example, a source / location and orientation associated with the tracking system's coordinate structure) to the camera, the tracking system can transmit information about the software camera pose. of virtual modeling. Pose transmission information about the camera can update in real time as the camera moves and / or rotates.

In some embodiments of the present disclosure, the camera may be incorporated into a computer, for example, the computer operating virtual modeling software. As a specific example, the computer may be a personal computer with an integrated camera. FIG. 8C 25 shows an illustration of an example 870 personal computer with an integrated camera. For example, a first side 874 of the personal computer 870 can face a user 872, and a second side 876 can face away from user 872. The camera can be mounted on the second side 876 so that the camera can capture a real object. (e.g. 30 real object 880). If the AR system is properly calibrated, user 872 can see a real world scene (including a view 881 of the real object 880) on the personal computer screen 870. The screen can also show virtual content (for example, virtual content 882 ), overlaid on top of the real world / real object scene. For a tracking system to track the camera (for example, embedded in personal computer 870), a number of tracking markers (for example, tracking markers 5,884, 886, 888) can be mounted on personal computer 870. Then, the Camera tracking can be done in a similar method to that explained above.

Referring again to FIG. 7, virtual modeling software 710 may receive transmission information from tracking system 702 about the camera's pose / frame of reference. However, virtual modeling software 710 may need to track the location of the camera lens rather than the camera body (or a camera frame, or a personal computer) as a whole. To determine a frame of reference for the camera lens, a special printed marker and related software may be used. FIGURES 9A and 9B depict illustrations showing how the printed marker can enable determination of the lens reference structure. As shown in FIG. 9A, a 902 camera can capture or record real world items in the camera's 904 field of view. A special printed marker 906 may be placed 20 in the camera's field of view 904. It should be understood that the 906 printed marker can only be used to calibrate the camera and the AR system. Since virtual modeling software is able to track the camera lens, the printed marker 906 can be removed.

To calibrate the RA system, the printed marker 906 may be placed in the camera's 904 field of view, for example, somewhere in the 3D space of an area or laboratory (for example, on the floor). The printed marker 906 may include various markers (e.g., markers 908) which may include a coordinate structure (e.g., an origin and orientation) for the printed marker. Camera 902 can then capture printed marker 906 (including the various markers) and can transmit this information to a computer 910 (for example, similar to computer 106 of FIG. 1). The 910 computer may be the same computer that includes the virtual modeling software. The computer 910 may include software 912 that is associated with the printed marker 906. The printed marker software 912 may receive information from camera 902, including how the camera "sees" the printed marker 906, for example 5 as the printed marker. appears to be located and oriented in the camera's field of view. The printed marker software 912 can then process this information to determine a reference structure (e.g., an origin and orientation) for the camera lens 902, for example, how the lens is placed in the coordinate structure established by printed marker 906. (and the various markers 908). As an example and reference to FIG. 9B, for calibration purposes, camera 902 and printed marker 906 may be oriented relative to each other so that the axes (e.g., X, Y, Z) are aligned with the vertical, horizontal and depth axes of the camera, specifically the camera lens.

Referring again to FIG. 7, the modeling software come

(eg via software related to a special printed marker) can determine a camera lens reference structure 714, for example as it relates to a coordinate structure 712 established by the printed marker. In order to relate the camera lens 20 pose to the pose of various virtual objects, the virtual modeling software may place the camera lens in a coordinate structure associated with the tracking system, for example by relating the reference frame 714 of the camera. camera lens to the camera's frame reference 706 as a whole as tracked by the tracking system. However, virtual modeling software 710 may not be able to relate camera lens reference frame 714 to camera reference frame 706 as a whole until the AR system has been calibrated, for example, because the frame The coordinate 712 established by the printed marker may differ from the coordinate structure 704 established by the tracking system. Therefore, the calibration process may include aligning (generally shown by number 716) the coordinate structure 712 established by the printed marker and the coordinate structure 704 established by the tracking system. Such alignment may include placing the printed marker (for example, the origin of the printed marker) at the same location (for example, the same 3D space location on the floor of an area or laboratory) as the coordinate structure origin of the system. tracking. Alignment may also include aligning the axes (for example, X, Y, Z) of the printed marker with the axes of the tracking system coordinate structure. In this regard, since the two coordinate structures are aligned, the virtual modeling software 710 can treat them as the same coordinate structure.

To relate the camera lens reference structure 714

In reference frame 706 of the camera as a whole, virtual modeling software 710 can determine / calculate a mapping or transformation (e.g., Transformation C3 shown in FIG. 7). The C3 transformation may not be known before the calibration process is complete. The calibration process, as explained above, can determine several other mappings or transformations that are related to the C3 transformation. As shown in FIG. 7, the calibration process can determine Transformation C1 (ie where the tracking system places the tracked camera in its coordinate structure) and Transformation 20 C2 (ie the translational and rotational differences between the printed marker coordinate structure). 712 and the new camera lens reference frame 714 as determined by the software associated with the printed marker). Once transformations C1 and C2 are shown, transformation C3 can be calculated. Once the C3 transform is known, the camera can be moved around, and virtual modeling software can track the camera lens in the coordinate structure of the tracking system even if the printed marker no longer appears in the field of view. from the camera. Information (for example, camera pose) of the tracking system is transmitted to the virtual modeling software, if the transformer C1 changes (ie the camera pose in 3D space), the transformer C3 can be updated, for example. example in real time. In this respect, virtual objects may overlap in a real-world scene, and the appearance of virtual objects may change appropriately, for example as the camera moves.

The following explains an example technique for computing the C3 transformation as shown in FIG. 7. The various transformations (C1, 5 C2, C3) as shown in FIG. 7 may each be represented as a transformation matrix, for example a 4x4 transformation matrix as is commonly used in computer graphics in 3D. Transformation C1 can be represented as the transformation matrix in Eq. 6 below. Transformation C2 can be represented as the transformation matrix 10 shown in Eq. 7 below.

Cl =

c. =

'DCM2. 000 i

(Eq. 7)

15

Similar to the Mn transformation matrices described above, each Cn transformation matrix can include a rotational and orientation component (DCMn) and a translational or location component (Vn). The C3 transformation can then be calculated as shown in Eq. 8 below, resulting in the C3 transformation shown in Eq. 9 below.

fnrsi? λ

^

1,000 Ii

v.

D00 ~ IJ

■ 2? CAf,

(Eq. 8) (Eq. 9)

Then, with reference again to FIG. 7, transformation C3 may be used to place the camera lens in a coordinate structure associated with the tracking system, for example by relating the camera lens pose to the camera pose as tracked by the tracking system 20. Since the information (for example, camera pose) of the tracking system is transferred to the virtual modeling software, if the C1 transformation changes (ie, the camera pose in 3D space), the C3 transformation can update, for example. example in real time. In operation, the C3 transform update can operate as follows: Tracking system 702 can detect a change in pose of a camera (tracking system updates Cl). Tracking system 702 can transmit reference frame information 706 (for example, in the form of a transformation matrix) from the camera to virtual modeling software 710. Virtual modeling software can multiply this reference structure / transformation matrix by transformation matrix C3 to perform transformation C3. The virtual modeling software can then update the pose of various virtual objects in the coordinate structure associated with the tracking system based on the changing camera pose.

Referring again to FIG. 1, virtual modeling software 110 may perform various routines, techniques, and the like described herein to create a dynamically enlarged scene (e.g., augmented scene 114), for example, a real-time view of the real world scene as captured by camera 102 enlarged and / or overlaid with dynamically changing virtual objects. Once RA system calibration is achieved, as described above, virtual modeling software 110 can dynamically maintain correlation and / or alignment between multiple virtual objects and a live real world scene, including one or more real objects ( e.g. real object 104). Virtual modeling software 110 can maintain this alignment even though camera 102 can be moved and rotated around an object 104, and even if the actual object 104 can be moved and rotated.

Virtual modeling software 110 can produce a dynamically enlarged scene 114 (for example, shown on display 116) showing virtual objects placed on a live video feed. Virtual modeling software 110 may appropriately deform (e.g., change 3D location, 3D orientation, and / or 3D size) virtual objects in the enlarged scene, for example, depending on camera pose 102 and / or camera pose. real object 104. For example, if camera 102 moves further away from real object 104, one or more virtual objects in the enlarged scene may shrink. As another example, if camera 102 moves closer to the actual object 104, one or more virtual objects would widen. As another example, if camera 102 moves at an angle relative to the actual object 104, one or more virtual objects would rotate appropriately. The enlarged scene 114 may be stored (for example, momentarily) in memory (for example, a volatile or nonvolatile memory unit) before the enlarged scene is shown on display 116. The augmented or virtual content that is shown on display 116 and / or kept in the enlarged scene 114 may be useful to a user using the AR system. For example, a user may interact with virtual content and / or receive beneficial information from augmented content. As a specific example, virtual objects / content can provide valuable instructional information to a technician regarding a piece of machinery during a manufacturing process.

FIG. 10A depicts an illustration of an example enlarged scene that can be produced according to one or more embodiments of the present disclosure. The enlarged scene may include a real world scene / environment as captured by a camera, for example, a portion of an area 1002, with a table 1004, and a real object 1006 (for example, a piece of machinery) on the table. The enlarged scene may include one or more virtual objects as added by the AR system described herein, for example, a virtual object 1010 that is related to (for example, a CAD model based off) real object 1006. FIG. 25 10A shows only part (for example, a clipping) of virtual object 1010. This clipping view can assist in showing how virtual object 1010 can align with actual object 1006. It should be understood, however, that in some embodiments, The full virtual object can be shown in the enlarged scene. If the camera moves, the real world scene and virtual objects can move in a similar manner. If the real object moves, any virtual objects that are related to the real object can move in a similar manner. The enlarged scene example of FIG. 10A is an example for showing how virtual objects modeled outside the real object can be aligned with the real object, for example to calibrate the AR system. In some examples, after calibration is complete, the virtual object modeled outside the real object may not appear in the augmented scene. Instead, various other virtual objects may appear, for example, tools, hardware (eg screws), wiring, instructions, and the like that are related to the actual object. For example, these virtual objects can provide valuable instructional information to a technician regarding a piece of machinery, for example, instructions on how to install an item or perform a task (for example, such as drilling a well).

FIG. 10B depicts an illustration of an example enlarged scene that can be produced according to one or more embodiments of the present disclosure. The enlarged scene may include a real world scene / environment as captured by a camera, for example, a part of an area 1052, with a real object 1056 (for example, a panel). The enlarged scene may include one or more virtual objects as added by the AR system described herein, for example, a virtual object 1060, which may be a box or unit, and various associated wires, conduits and / or wire networks. If the camera moves, the real world scene and virtual objects can move in a similar way. If the real object 1056 moves (for example, the panel), any virtual objects (for example, virtual object 1060) that are related to the real object can move in a similar manner. For example, the enlarged scene example of FIG. 10B can instruct a technician how to install a 1060 virtual object on a 1056 panel.

Certain embodiments of the present disclosure may be found in one or more methods for automated augmented reality reference calibration framework. With respect to various methods described herein and depicted in associated figures, it should be understood that in some embodiments, one or more of the described and / or depicted steps may be performed in a different order. Additionally, in some embodiments, a method may include more or less steps than described and / or depicted.

FIG. 11 depicts a flowchart 1100 showing example steps in a method for automated augmented reality reference calibration structure in accordance with one or more embodiments of the present disclosure. More specifically, FIG. 11 shows example steps in a method for automated reference calibration structure that can be used to determine and / or calculate a mapping or transformation between the reference frame of virtual objects (for example, virtual objects added to an enlarged scene). and the reference structure associated with a tracking system. At step 1102, a coordinate system for the tracking system is established, for example, during an adjustment process for the tracking system. At step 1104, the tracking system can track or determine a frame of reference for an actual object (RO). The tracking system may also determine Transformation M1 at step 1104. In order for the tracking system to track an actual object, the actual object may need to be equipped with a number of tracking markers.

At step 1106, virtual modeling software can determine a new reference structure for the real object, for example by indicating a number of reference points (for example, using a stick) and computing a source. At step 1108, a virtual modeling software can compute the M2 Transformation (for example, the difference in pose between the new RO reference structure and the real object reference structure as determined by the tracking system). At step 1110, virtual modeling software can access or load a virtual object (for example, a virtual object modeled outside the real object) and can determine the VO reference structure. At step 1112, a virtual modeling software can determine a new VO reference structure, for example by indicating in the virtual model the same reference points that were indicated in the real object to create a new RO reference structure. The origin of the points can be computed. At step 1114, the virtual modeling software can compute the M3 Transformation (for example, the difference in pose between the new VO reference structure and the original VO reference structure). At step 1116, the virtual modeling software can compute Transformation M4 (for example, by multiplying with matrices M1, M2, and M3).

FIG. 12 depicts a flowchart 1200 showing example steps in a method for automated augmented reality reference calibration structure according to one or more embodiments of the present disclosure. More specifically, FIG. 12 shows an example of 10 steps in a method for automated reference calibration structure that can be used to determine and / or calculate a mapping or transformation between a camera reference structure as tracked by the tracking system and the camera reference structure. Camera lens. At step 1202, a coordinate system for the tracking system is established, for example, during an adjustment process for the tracking system. At step 1204, the tracking system can track or determine a frame of reference for a camera. The tracking system may also determine Transformation C1 at step 1204. For the tracking system to track a camera, camera 20 (or camera frame, or personal computer) may need to be equipped with a number of tracking markers. At step 1206, a coordinate structure associated with a printed marker can be established, for example, using a printed marker and related software. The coordinate structure associated with the printed marker may be aligned with the coordinate structure of the tracking system. At step 1208, virtual modeling software (e.g. via the software associated with the printed marker) can determine the camera lens reference structure, for example with respect to the coordinate structure of the printed marker. At step 1210, the virtual modeling software can compute Transformation C2 (for example, the difference in pose between the camera lens reference frame and the printed marker coordinate frame). Such computation may be (at least partially) performed by the software associated with the printed marker. At step 1212, the virtual modeling software can compute Transformation C3 (for example, dividing the Transformation C1 matrix by the C2 transformation matrix).

Any of the systems and methods described herein also contain

There are variations which include an augmented reality method 100 performed by a data processing system 100 having at least one processor. The alternative method may include or establish a coordinate structure of the tracking system 204 associated with an object tracking system 108. The coordinate structure of the tracking system 204 is aligned with a real 3D space, and tracks a position and orientation in a real 3D space of a real object 104 and a camera 102. In this arrangement, the data processing system 100 also receives from the tracking system 108 a first real object reference frame 212 for the real object 104. first real object reference frame 212 indicates a position and orientation of the real object 104 with respect to the coordinate structure of tracking system 204.

Next, the data processing system 100 determines a second real object reference frame 212 for real object 20 104, wherein the second real object reference structure 212 indicates a position and orientation of the real object 104 with respect to the coordinate structure of tracking system 204. Data processing system 100 then receives a first virtual object reference structure 216 for a virtual object 112, wherein virtual object 112 is modeled after actual object 104, and wherein first virtual object reference frame 216 is not related to the coordinate structure of tracking system 204.

The data processing system 100 also determines a second virtual object reference frame 216 for virtual object 112, wherein the second virtual object reference frame 216 indicates a position and orientation of virtual object 112 with respect to the coordinate structure of the virtual object 112. tracking system 204. A virtual object mapping 112 is also determined between the first virtual object reference frame 216 and the coordinate structure of tracking system 204.

An enlarged scene 114 is shown by the data processing system 100 and includes a 3D real space view, a real object view 5, and one or more overlapping virtual items. Here, virtual object mapping 112 is used to place one or more overlapping virtual items in the augmented scene 114 so that one or more virtual items are aligned with the actual object 104.

In additional optional arrangements, data processing system 100 is configured to also determine second real object reference structure 212, which can receive or detect three or more non-collinear real object points on real object 104. The location of three or more non-collinear real object points is defined with respect to the coordinate structure of tracking system 204. A real object origin 104 is determined by calculating a centroid of three or more non-collinear real object points. A real object orientation is then determined that is related to the orientation of the first real object reference structure 212. It may be preferred that the second virtual object reference structure 216 is determined to receive or indicate three or more non-collinear object points. where the location of three or more non-collinear virtual object points is defined with respect to the coordinate structure of tracking system 204. Thereafter, a virtual object origin can be determined by calculating a centroid of the three or more non-collinear virtual object points, which may additionally make it possible to determine a virtual object orientation.

In further modifications to any of the described arrangements, the second real object reference frame 212 and the second virtual object reference frame 216 are aligned wherein three or more 30 non-collinear virtual object points and the three or more points are not aligned. The collinear real object directions are located at approximately the same location with respect to the coordinate structure of the tracking system 204. In addition, the real object orientation 104 and the virtual object orientation 112 are approximately the same since each orientation refers to the coordinate structure of the tracking system 204. tracking system 204.

In still other embodiments, determining virtual object mapping 112 may include receiving or determining a first transformation matrix representing the first real object reference structure 212. The second transformation matrix may also be determined which represents the difference in location and orientation between the first real object reference frame 212 and the second real object reference frame 212. In addition, a third transformation matrix can be determined that represents the difference in location and orientation between the first real object reference frame 212. virtual object 216 and the second virtual object reference structure 216. A fourth transformation matrix can also be computed, which represents the mapping of virtual object 112. Here, the fourth transformation matrix represents the difference in location and orientation between the first reference structure of the virtual object 216 and the coordinate structure of the tracking system 204. The fourth transformation matrix is also computed by performing matrix multiplication between the first, second and third transformation matrices.

In another arrangement, using virtual object mapping 112 to place one or more overlapping virtual items in augmented scene 114 may include receiving a first virtual object reference frame 216 for a first overlapping virtual item. In this variation, the first virtual object reference frame 216 is not related to the coordinate structure of tracking system 204. Virtual object mapping 112 may be referred to transforming the first virtual object reference frame 216 to a reference structure that relates to the coordinate structure of tracking system 204. The first overlapping virtual item can be placed in the enlarged scene 114 using the transformation reference structure.

The data processing system 100 may further be configured to receive from the tracking system 108 real-time information about the location and orientation of the real object 104. In this modification, the virtual object mapping 112 is updated based on the real-time information. and augmented scene 114 is updated by updating 5 by replacing one or more overlapping virtual items so that one or more virtual items remain aligned with the actual object 104.

The methods, routines and techniques of the present disclosure, including exemplary methods and routines illustrated in the flowcharts and block diagrams of the different embodiments described may be implemented as software executed by one or more data processing systems which are programmed such that data processing systems are adapted to perform and / or perform part or all of the methods, routines and / or techniques described herein. Each block or symbol in a block diagram or flowchart diagram referenced herein may represent a usable or computer readable program code module, segment, or portion comprising one or more executable instructions for implementing through one or more processing systems. specified data, function or functions. In some alternative implementations of the present disclosure, the function or functions illustrated in the blocks or symbols of a block diagram or flowchart may occur outside the order perceived in the figures. For example, in some cases two blocks or symbols shown in succession may be executed substantially concurrently or the blocks may be executed in reverse order depending on the functionality involved. Part or all of the computer code may be loaded into the memory of a data processing system before the data processing system executes the code.

FIG. 13 depicts a block diagram of an example data processing system 1300 that may be used to implement one or more embodiments of the present disclosure. For example, with reference also to FIG. 1 momentarily, computer 106 may take the form of a data processing system similar to data processing system 1300 of FIG. 13. As a further example, tracking system software related to tracking system 108 may be run on a data processing system similar to data processing system 1300 of FIG. 13. With reference to FIG. 13, data processing system 1300 may be used to partially or fully execute one or more of the methods, routines and / or solutions of the present disclosure. In some embodiments of the present disclosure, more than one data processing system may be used to implement the methods, routines, techniques and / or solutions described herein.

In the example of FIG. 13, the data processing system

1300 may include a communication frame 1302 that provides communications between components, for example, a processor unit 1304, a memory 1306, a persistent storage 1308, a communication unit 1310, an input / output (I / O) unit 1312 and a display 1314. A bus system may be used to implement a communications frame 1302 and may be comprised of one or more buses such as a system bus or an input / output bus. The bus system may be implemented using any suitable architecture that provides data transfer between different components or devices attached to the bus system.

Processor unit 1304 may be used to execute instructions (for example, a software program) that may be loaded into data processing system 1300, for example in memory 1306. Processor unit 1304 may be a set of or more processors or may be a multiprocessor core depending on the particular implementation. Processor unit 1304 may be implemented using one or more heterogeneous processor systems wherein a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit 1304 may be a symmetric multiprocessor system containing multiple processors of the same type. Memory 1306 may be, for example, a random access memory or any other suitable volatile or nonvolatile storage device. Memory 1306 may include one or more layers of cache memory. Persistent storage 1308 may take various forms 5 depending upon particular implementation. For example, persistent storage 1308 may contain one or more components or devices. For example, persistent storage 1308 may be a hard disk, solid state drive, flash memory, or some combination of the above.

Instructions for an operating system may be localized.

in persistent storage 1308. In a specific embodiment, the operating system may be some version of a number of known operating systems. Instructions for applications and / or programs may also be located in persistent storage 1308. These instructions may be loaded into 1306 memory for execution by processor unit 1304. For example, methods and / or processes of the different embodiments described in this description may be executed by processor unit 1304 using computer-implemented instructions that can be loaded into memory such as memory 1306. 20 These instructions are referred to as program code, computer usable program code, or computer readable program code. read and executed by a processor in processor unit 1304.

Display 1314 may provide a mechanism for displaying information to a user, for example via an LCD or LED screen or monitor or other type of display. It should be understood throughout this description that the term "display" may be used in a flexible manner to refer to both a physical display such as a physical screen and the image a user sees on a physical device screen. . The input / output (I / O) unit 1312 provides data input and output with other devices that may be connected to the 1300 data processing system. Input / output devices may be coupled to the system either directly or via intervention I / O controllers.

Communications unit 1310 may provide communications with other data processing systems or devices, for example, via one or more networks. Communications unit 1310 may be a network interface card. The 1310 communications unit can provide communications through the use of wired and / or wireless communications links. In some embodiments, the communications unit may include circuits that are designed and / or adapted to communicate according to various wireless communication standards, for example, Wi-Fi standards, 10 Bluetooth standards, and the like.

The different components illustrated for data processing system 1300 are not meant to provide architectural limitations to the manner in which different modalities may be implemented. The different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those illustrated for the data processing system 1300. Other components shown in FIG. 13 may be varied from the illustrative examples shown.

The data processing system 1300 may also be arranged with one or more memory units that store computer code, and one or more processor units 1304 coupled to one or more memory units 1306, wherein one or more processor units 1304 execute computer code stored in one or more memory units 1306. Running computer code receives or establishes a coordinate structure of the tracking system 204 associated with an object tracking system 108, 202. The coordinate structure of the system Tracking 204 is aligned with real 3D space, and tracking system 108 tracks the position and orientation in real 3D space of a camera 102 that captures real 3D space and a printed marker 906.

Tracking system 108 receives a camera reference frame for camera 102, and the camera reference frame indicates a position and orientation of camera 102 with respect to the coordinate frame of tracking system 204. Also received or established is a frame. printed marker coordinate 204 associated with the printed marker, wherein the coordinate structure of the printed marker 5 204 is aligned with actual 3D space. The coordinate structure of the printed marker 204 is aligned with the coordinate structure of the tracking system 204, and a camera lens reference frame is determined for camera lens 102. The camera lens reference frame indicates a position and orientation. of the camera lens 102 with respect to the coordinate structure of the printed marker 204.

A camera lens mapping 102 is determined between the camera reference frame 102 and the camera lens reference frame 102, and an enlarged scene 114 is shown including a 3D real space view and one or more virtual items, wherein camera lens mapping 102 is used to alter or distort one or more virtual items in the enlarged scene 114.

Additionally, the invention may comprise embodiments according to the following clauses:

Clause 1. A method for augmented reality by a data processing system having at least one processor, the method comprising: receiving or establishing a coordinate structure of the tracking system associated with an object tracking system, wherein the coordinate structure of the tracking system is aligned with a real 3D space, and the tracking system tracks the position and orientation in a real 3D space of a real object and a camera; receiving from the tracking system the first real object reference structure to the real object, wherein the first real object reference structure indicates a position and orientation of the real object with respect to the coordinate structure of the tracking system; determining a second real object reference structure for the real object, wherein the second real object reference structure indicates a position and orientation of the real object with respect to the coordinate structure of the tracking system; receiving a first virtual object reference structure for a virtual object, wherein the virtual object is modeled after the actual object, and wherein the first virtual object reference structure is not related to the coordinate structure of the tracking system; determining a second from the virtual object reference structure to the virtual object, wherein the second virtual object reference structure indicates a position and orientation of the virtual object with respect to the coordinate structure of the tracking system; determining a virtual object mapping between the first virtual object reference structure and the coordinate structure of the tracking system 10; and show an enlarged scene including a 3D real space view, a real object view, and one or more overlapping virtual items, where virtual object mapping is used to place one or more overlapping virtual items in the enlarged scene so one or more virtual items are aligned with the actual object.

Clause 2. The method of clause 1, in which it determines whether

A second real object reference structure includes: receiving or detecting three or more non-collinear real object points on the real object, where the location of three or more non-collinear real object points is defined with respect to the tracking system's coordinate structure ; determining a real object origin by calculating a centroid of three or more non-collinear real object points; and determining a real object orientation that is related to the orientation of the first real object reference structure.

Clause 3. The method of clause 2, wherein determining the second virtual object reference structure includes: receiving or indicating three 25 or more non-collinear virtual object points in the virtual object, where the location of three or more non-collinear points virtual object definition is defined to the coordinate structure of the tracking system; determine a virtual object origin by calculating a centroid of three or more non-collinear virtual object points, and determine a virtual object orientation.

Clause 4. The method of clause 3, wherein the second structure

object reference frame and the second virtual object reference frame are aligned ensuring that: the three or more non-collinear virtual object points and the three or more non-collinear real object points are located at approximately the same location with respect to the structure coordinate tracking system; and the real object orientation and the virtual object orientation are approximately the same since each orientation refers to the coordinate structure of the tracking system.

Clause 5. The method of clause 1, wherein determining virtual object mapping includes: receiving or determining a first transformation matrix representing the first real object reference structure; determining a second transformation matrix representing the difference in location and orientation between the first real object reference frame and the second real object reference frame; determining a third transformation matrix representing the difference in location and orientation between the first virtual object reference structure and the second virtual object reference structure; and computing a fourth transformation matrix representing the virtual object mapping, wherein the fourth transformation matrix represents the difference in location and orientation between the first virtual object reference structure and the coordinate structure of the tracking system and where the The fourth transformation matrix is computed by performing matrix multiplication between the first, second, and third matrices.

Clause 6. The method of clause 1, wherein using virtual object mapping to place the one or more overlapping virtual items in the augmented scene includes: receiving a first virtual item 25 reference structure for a first overlapping virtual item, where The first virtual item reference structure is not related to the coordinate structure of the tracking system; referencing the virtual object mapping to transform the first virtual item reference structure to a transformed reference structure that refers to the coordinate structure of the tracking system; and place the first overlapping virtual item in the enlarged scene using the transform reference frame.

Clause 7. The method of clause 1 further comprising: receiving from the tracking system real-time information about the location and orientation of the actual object; update virtual object mapping based on real-time information; and updating the scene increases by updating the placement of one or more overlapping virtual items so that one or more virtual items remain aligned with the actual object.

Clause 8. An augmented reality method performed by a data processing system having at least one processor, the method comprising: receiving or establishing a coordinate structure of the tracking system associated with an object tracking system, wherein the The coordinate structure of the tracking system is aligned with real 3D space, and the tracking system tracks the position and orientation in real 3D space of a camera that captures real 3D space and a printed marker; receiving from the tracking system a camera reference frame for the camera, wherein the camera reference frame indicates a camera position and orientation with respect to the coordinate structure of the tracking system; receive or establish a printed marker coordinate structure associated with the printed marker, wherein the printed marker coordinate structure is aligned with actual 3D space, and in which the printed marker coordinate structure is aligned with the system coordinate structure tracking; determining a camera lens reference frame to the camera lens, wherein the camera lens reference frame indicates a position and orientation of the camera lens relative to the printed marker coordinate structure; determining a camera lens mapping between the camera reference frame and the camera lens reference frame; and show an enlarged scene including a 3D real-space view and one or more virtual items, where camera lens mapping is used to alter or distort one or more virtual items in the enlarged scene.

Clause 9. The method of clause 8, in which it determines the structure

Camera lens reference frame includes receiving position information and lens orientation from the software program camera associated with the printed marker.

Clause 10. The method of clause 8, in which aligning the coordinate structure of the printed marker with the coordinate structure of the tracking system includes placing the printed marker at a location and orientation in real 3D space so that: an origin associated with the printed marker is located at approximately the same location in 3D space as a source associated with the tracking system; and three axes associated with the printed marker are approximately aligned in 3D space with three axes associated with the tracking system.

Clause 11.0 The method of clause 8, wherein determining the camera lens mapping includes: receiving or determining a first transformation matrix representing the first real object reference structure; determining a second transformation matrix representing the difference in location and orientation between the camera lens reference frame and the printed marker coordinate frame; compute a third transformation matrix representing camera lens mapping, wherein the third transformation matrix represents the difference in location and orientation between camera reference frame 20 and camera lens reference frame, and wherein The third transformation matrix is computed by performing matrix division between both the first and second transformation matrices and between the second and first transformation matrices.

Clause 12. The method of clause 8, wherein using camera lens mapping to alter or distort one or more overlapping virtual items in the enlarged scene includes: placing a first overlapping virtual item in the enlarged scene using a first frame reference frame. virtual item, where the first virtual item reference structure refers to the coordinate structure of the tracking system; receive from the tracking system real-time information about the camera's location and orientation; reference virtual object mapping for transformation to update camera lens reference structure based on real-time information; and update the enlarged scene by changing or distorting one or more overlapping virtual items based on the updated camera lens reference frame.

Clause 13. The method of clause 8, wherein altering or distorting one or more overlapping virtual items in the enlarged scene includes shrinking overlapping virtual items in response to camera movement additionally away from a real object in 3D space.

Clause 14. The method of clause 8, wherein altering or distorting one or more overlapping virtual items in the enlarged scene includes widening overlapping virtual items in response to camera movement closer to a real object in 3D space.

Clause 15. The method of clause 8, wherein altering or distorting one or more overlapping virtual items in the enlarged scene includes rotating overlapping virtual items in response to camera movement in an anguish relative to a real object in 3D space.

Clause 16. A system comprising: a camera that captures a view of a real 3D space including a real object; a tracking system that tracks the position and orientation in real 3D space of the actual object and camera, wherein the tracking system is configured to establish a coordinated tracking system structure associated with the tracking system, wherein the coordinate structure of the tracking system is aligned with real 3D space; and a computer coupled to the camera and tracking system, the computer having one or more memory units, the computer being configured with a virtual modeler, wherein the virtual modeler is configured to receive from the tracking system a first frame of reference. real object to real object, wherein the first real object reference structure indicates a position and orientation of the real object relative to the coordinate structure of the tracking system; wherein the virtual mode30 is further configured to compute a second real object reference structure to the real object, wherein the second real object reference structure indicates a position and orientation of the real object with respect to the coordinate structure of the system. tracking; wherein the virtual modeler is further configured to receive from one or more memory units a first virtual object reference structure for a virtual object, wherein the virtual object is modeled after the real object, and wherein the first virtual object reference is not related to the coordinate structure of the tracking system; wherein the virtual modeler is further configured to compute a second virtual object reference structure for the virtual object, wherein the second virtual object reference structure indicates a position and orientation of the virtual object with respect to the coordinate structure of the system. tracking; wherein the virtual modeler is further configured to compute a virtual object mapping between the first virtual object reference structure and the coordinate structure of the tracking system; and wherein the virtual modeler is further configured to generate and store in one or more memory units an enlarged scene including a 3D real space view, a real object view and one or more overlapping virtual items, wherein the mapping The virtual object is used to place one or more overlapping virtual items in the enlarged scene so that one or more virtual items are aligned with the actual object.

Clause 17. The system of clause 16 additionally comprises

comprising a computer-coupled display, wherein the virtual modeler is further configured to communicate the enlarged scene to the display, and wherein the display is configured to show the enlarged scene to a user.

Clause 18. The system of clause 16, wherein the mapping

of the virtual object to place one or more overlapping virtual items in the enlarged scene includes: receiving a first virtual item reference structure for a first overlapping virtual item, where the first virtual item reference structure is not related to the sis30 coordinate structure tracking theme; referencing the virtual object mapping to transform the first virtual item reference structure to a transformed reference structure that refers to the coordinate structure of the tracking system; and place the first overlapping virtual item in the enlarged scene using the transform reference frame.

Clause 19. The method of clause 16, wherein the virtual modeler is further configured to: receive from the tracking system real-time information about the location and orientation of the real object; update virtual object mapping based on real-time information; and update the enlarged scene by updating the placement of one or more overlapping virtual items so that one or more virtual items remain aligned with the actual object.

Clause 20. A data processing system, buy

address: one or more memory units that store computer code; and one or more processor units coupled to one or more memory units, wherein one or more processor units execute computer code stored in one or more memory units 15 to: receive or establish a coordinate structure of the memory system. tracking system associated with an object tracking system, wherein the coordinate structure of the tracking system is aligned with a real 3D space, and the tracking system tracks the position and orientation in a real 3D space of a camera that captures the actual 3D space and a printed marker; receiving from the tracking system a camera reference frame to the camera, wherein the camera reference frame indicates a camera position and orientation with respect to the coordinate structure of the tracking system; receiving or establishing a printed marker coordinate structure associated with the printed marker, wherein the printed marker coordinate structure is aligned with actual 3D space, and wherein the printed marker coordinate structure is aligned with the system coordinate structure tracking; determining a camera lens reference frame to the camera lens, wherein the camera lens reference frame indicates a position and orientation of the camera lens relative to the printed marker coordinate structure; determine a camera lens mapping between the camera reference frame and the camera lens reference frame; and show an enlarged scene including a 3D real space view and one or more virtual items, where camera lens mapping is used to alter or distort the one or more virtual items in the enlarged scene.

The description of the different advantageous embodiments has been

intended for illustration and description purposes only and is not intended to be exhaustive or limited to the embodiments in the foregoing form. Many modifications and variations will be apparent to those of ordinary versatility in the art. Additional different advantageous embodiments may provide different advantages as compared to other advantageous embodiments. The selected embodiment or embodiments are chosen and described so as to better explain the principles of the embodiments of practical application and enable others of ordinary skill in the art to understand the description by various embodiments with various modifications as are suitable for the particular use contemplated.

Claims (12)

An augmented reality method (100) performed by a data processing system having at least one processor, the method comprising: receiving or establishing a coordinate structure of the tracking system (204) associated with an object tracking system (108) ), wherein the coordinate structure of the tracking system (204) is aligned with the real 3D space, and wherein the tracking system (108) tracks the position and orientation in a real 3D space of a camera (102) capturing real 3D space and a printed marker (906); receiving from the tracking system (108) a camera reference frame to the camera (102), wherein the camera reference frame indicates a camera position and orientation (102) relative to the tracking system coordinate structure (204) ); receiving or establishing a printed marker coordinate structure (204) associated with the printed marker (906), wherein the printed marker coordinate structure (204) is aligned with actual 3D space, and wherein the printed marker coordinate structure (204) is aligned with the coordinate structure (204) of the tracking system (108); determine a camera lens reference frame (102) for the camera lens, wherein the camera lens reference frame indicates a position and orientation of the camera lens (102) relative to the printed marker coordinate structure (204). ); determining a camera lens mapping between the camera reference frame (102) and the camera lens reference frame (102); and show an enlarged scene (114) including a 3D real space view and one or more virtual items, where camera lens mapping (102) is used to alter or distort one or more virtual items in the enlarged scene (114). ). The method of claim 1, wherein determining the camera lens reference structure (102) includes receiving camera lens position and orientation information (102) from a software program associated with the printed marker ( 906). The method of claim 1 or 2, wherein aligning the coordinate structure of the printed marker (204) with the coordinate structure of the tracking system (204) includes placing the printed marker (906) in a location and orientation on the real 3D space such that: an origin associated with the printed marker (906) is located at approximately the same location as an origin associated with the tracking system (108); and three axes associated with the printed marker (906) are approximately aligned in 3D space with three axes associated with the tracking system (108). A method according to any one of claims 1 to 3, wherein determining the camera lens mapping (102) includes: receiving or determining a first transformation matrix representing the first real object reference structure (212); determining a second transformation matrix representing the difference in location and orientation between the camera lens reference structure (102) and the printed marker coordinate structure (204); computing a third transformation matrix representing camera lens mapping (102), wherein the third transformation matrix represents the difference in location and orientation between the camera reference frame and the camera lens reference frame, and wherein the third transformation matrix is computed by performing matrix division both between the first and second transformation matrices and between the second and first transformation matrices. A method according to any one of claims 1 to 4, wherein using camera lens mapping (102) to alter or distort one or more overlapping virtual items in the enlarged scene (114) includes: placing a first overlapping virtual item in the enlarged scene (114) using a first virtual item reference structure (216), wherein the first virtual item reference structure (216) refers to the coordinate structure of the tracking system (204); receiving from the tracking system (108) real-time information about the location and orientation of the camera (102); reference virtual object mapping (112) for transformation to update camera lens reference structure (102) based on real-time information; and updating the enlarged scene (114) by changing or distorting one or more overlapping virtual items based on the updated camera lens reference frame (102). A method according to any one of claims 1 to 5, wherein altering or distorting one or more overlapping virtual items in the enlarged scene (114) includes shrinking the overlapping virtual items in response to camera movement (102) further away of a real object (104) in 3D space. A method according to any one of claims 1 to 6, wherein altering or distorting one or more overlapping virtual items in the enlarged scene (114) includes widening the overlapping virtual items in response to camera movement (102) closer to a real object (104) in 3D space. A method according to any one of claims 1 to 7, wherein altering or distorting one or more overlapping virtual items in the enlarged scene (114) includes rotating the overlapping virtual items in response to camera movement (102) at an angle. with respect to a real object (104) in 3D space. A system comprising: a camera (102) that captures a view of a real 3D space including a real object (104); tracking system that tracks the position and orientation in real 3D space of the actual object (104) and camera (102), wherein the tracking system (108) is configured to establish a coordinated structure of the tracking system (204). ) associated with the tracking system (108), wherein the coordinate structure of the tracking system (204) is aligned with real 3D space; and a computer (106) coupled to the camera (102) and tracking system (108), the computer (106) having one or more memory units, the computer (106) being configured with a virtual modeler, wherein the modeler The virtual system is configured to receive from the tracking system (108) a first real object reference structure to the real object (104), wherein the first real object reference structure (212) indicates a position and orientation of the real object ( 104) with respect to the coordinate structure of the tracking system (204); wherein the virtual modeler is further configured to compute a second real object reference structure (212) to the real object (104), wherein the second real object reference structure (212) indicates a position and orientation of the real object with respect to the coordinate structure of the tracking system (204); wherein the virtual modeler is further configured to receive from one or more memory units a first virtual object reference structure (216) to a virtual object (112), wherein the virtual object (112) is modeled after the real object (104), and wherein the first virtual object reference frame (216) is unrelated to the coordinate structure of the tracking system (204); wherein the virtual modeler is further configured to compute a second virtual object reference structure (216) to the virtual object (112), wherein the second virtual object reference structure (216) indicates a position and orientation of the virtual object (112) with respect to the coordinate structure of the tracking system (204); wherein the virtual modeler is further configured to compute a virtual object mapping (112) between the first virtual object reference structure (216) and the tracking system coordinate structure (204); and wherein the virtual modeler is further configured to generate and store in one or more memory units an enlarged scene (114) including a 3D real space view, a real object view (104) and one or more overlapping virtual items wherein the virtual object mapping (112) is used to place the one or more overlapping virtual items in the enlarged scene (114) so that one or more virtual items are aligned with the actual object (114). The system of claim 9, further comprising a computer-coupled display (106), wherein the virtual modeler is further configured to communicate the enlarged scene (114) to the display, and wherein the display is configured to display the enlarged scene (114) to a user. The system of claim 9 or 10, wherein using virtual object mapping (112) to place one or more overlapping virtual items in the enlarged scene (114) includes: receiving a first virtual object reference structure (216) for a first overlapping virtual item, wherein the first virtual object reference structure (216) is not related to the coordinate structure of the tracking system (204); referencing the virtual object mapping (112) to transform the first virtual item reference structure (first virtual object reference structure 216) to a transformed reference structure that refers to the coordinate structure of the tracking system (204); and placing the first overlapping virtual item in the enlarged scene (114) using the transform reference frame. A system according to any one of claims 9 to 11, wherein the virtual modeler is further configured to: receive from the tracking system (108) real-time information about the location and orientation of the real object (104); update virtual object mapping (112) based on real-time information; and updating the enlarged scene (114) by updating the placement of one or more overlapping virtual items so that one or more virtual items remain aligned with the actual object.