JP6272687B2 - Construction of augmented reality environment with pre-calculated lighting - Google Patents

Description

  [0001] Adding realistic lighting and shadows to a virtual environment such as a virtual video game environment may be costly in terms of computational processing. Therefore, the rendering time for lighting effects can be unacceptably long for use during video game play. For example, creating realistic lighting (eg, global lighting) and texture maps (“light maps”) that encode shadows in a virtual environment can take hours or even days to calculate. Thus, such lighting effects for a virtual environment are generally precomputed during the development of the virtual environment, rather than being calculated in real time during game play.

  [0002] Dynamic lighting and shadow processing can be calculated at higher speed. However, the visual quality of dynamic lighting may be much lower than the visual quality of pre-calculated lighting effects. Furthermore, dynamic lighting can use significant resources at runtime.

  [0003] Various embodiments are disclosed that relate to efficiently building an augmented reality environment with global lighting effects. For example, one disclosed embodiment provides a method for displaying an augmented reality image via a display device. The method includes receiving image data taken of an image of a local environment of the display device, and identifying physical features of the local environment via the image data. The method further includes the step of constructing an augmented reality image of a virtual structure for display on the physical feature in a state spatially aligned with the physical feature as seen from the user's viewpoint. The augmented reality image includes a plurality of modular virtual structure segments arranged in adjacent positions so as to form a feature of the virtual structure, each modular virtual structure segment having a pre-computed lighting effect And outputting the augmented reality image to the display device.

  [0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Absent. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

FIG. 1 illustrates an exemplary implementation of a see-through display device in an exemplary use environment. FIG. 2 illustrates one embodiment of an augmented reality image in the exemplary use environment of FIG. FIG. 3 illustrates an exemplary implementation of a set of modular virtual structure segments. FIG. 4 shows a schematic depiction of pre-calculated lighting effects applied to a portion of a modular virtual structure segment. FIG. 5A shows the addition of a dynamic point lighting effect to the augmented reality image of FIG. FIG. 5B shows an example of a dynamic point lighting effect on a portion of a modular virtual structure segment having a pre-calculated lighting effect. FIG. 6 shows a flow diagram representing one embodiment of a method for building a virtual environment that fits a detected physical environment. FIG. 7 shows a block diagram of an exemplary embodiment of a see-through display device. FIG. 8 shows a block diagram of an exemplary implementation of a computing system.

  [0013] As described above, realistic lighting effects for a virtual environment are generally pre-calculated after the virtual environment is constructed and then stored, for example, as a light map of the virtual environment. Such a virtual environment is usually created by a fixed geometric structure that is not adapted around the user.

  [0014] In contrast, an augmented reality display system can be configured to adapt a virtual image around the user. For example, an augmented reality video game can fit a virtual structure in the game to a corresponding physical structure in the user's physical environment. In this way, the geometrical arrangement of augmented reality image objects can vary based on the physical environment of the user.

  [0015] Since the fitting of the augmented reality environment to the physical environment is performed during real-time use, if a high quality lighting effect is applied to the environment after the environment is created, the lighting calculation process Will be done at that time. However, if such a lighting effect for an augmented reality environment is calculated after fitting the augmented reality image to the physical environment, the user is computationally involved in applying a realistic lighting effect. Due to the cost, depending on the particular computing system used to calculate the lighting effects, it may be necessary to wait hours to days to play the augmented reality experience. This can lead to an unacceptably slow user experience. In addition, the physical environment may change during such long time delays. This can lead to inconsistencies between the real world and the virtual world, which can have a significant impact on the augmented reality experience.

  [0016] As one possible solution, dynamic lighting can be used instead of pre-calculated lighting effects for augmented reality environments. However, as described above, dynamic lighting may be of lower quality than pre-calculated lighting and therefore may not provide a much better user experience. In addition, dynamic lighting may be computationally expensive at runtime, which may reduce the computational budget for other aspects of the experience, such as other videos and gameplay.

  [0017] Accordingly, embodiments relating to the efficient construction of augmented reality environments fitted with local physical environment geometries having high quality pre-computed lighting effects are described herein. Disclosed. Briefly, the disclosed embodiments utilize modular virtual structure segments that can be arranged adjacent to each other to form a virtual structure for augmented reality images, where the modular virtual structure The structural segment comprises a high quality pre-calculated lighting effect. Since the lighting effect is pre-calculated for each modular virtual structure segment, the virtual structure constructed by the modular virtual structure segment will include the lighting effect. Further, in some embodiments, local illumination characteristics can be detected and used to adjust the appearance of the modular virtual structure segment. Examples of such local lighting characteristics may include, but are not limited to, color characteristics and light source positions in the local physical environment.

  [0018] FIG. 1 illustrates an exemplary implementation in the form of a living room of a use environment 100 for an augmented reality display system. A user 102 looking at a living room through a see-through display device 104 is shown. FIG. 1 also represents the user's field of view 103, which is expanded by a portion of the environment that is viewable through the see-through display device 104, and thus the image displayed through the see-through display device 104. Represents a portion of the environment that can. In some implementations, the user's field of view 103 can be substantially coextensive with the user's actual field of view, while in other embodiments, the user's field of view 103 can be the user's actual field of view. Can occupy a smaller part of

  [0019] As described in more detail below, the see-through display device 104 provides image data (eg, color / grayscale images, depth images / point cloud data / mesh data, etc.) representing the usage environment 100 to the user. Can include one or more outward-facing image sensors (e.g., a two-dimensional camera and / or a depth camera) configured to acquire as the user moves through the environment. This image data can be used to obtain information about the layout of the environment and structural features of the environment, such as the ceiling 106 and walls 108, as well as other features.

  [0020] The see-through display device 104 is further configured to overlay the displayed virtual object on top of a physical object visible through the device to create an augmented reality image. For example, referring to FIG. 2, an exemplary augmented reality image is depicted in which a virtual room framework structure 200, such as a virtual stud 202, lintel 204, etc., is displayed as an overlay on the user's wall. Images of infrastructure such as pipes 206, conduits / cables can also be displayed in addition to any other suitable virtual structure. A similar structure (not shown) can be similarly displayed for the ceiling. Furthermore, images corresponding to furniture and other unstructured objects in the room can be displayed to occupy an empty space in the room. To more fully illustrate the augmented reality environment, it will be appreciated that the augmented reality image depicted in FIG. 2 is not limited to the user's field of view illustrated in FIG.

  [0021] The virtual wall framing structure of FIG. 2 is geometrically fitted to the underlying physical structure (eg, wall 108). Since each user's local physical environment will be different, the overall virtual structure for each player's local physical environment is not pre-designed, but rather the local physical environment. Built when environmental image data (eg, stereo depth image data, structured illumination image data, time-of-flight image data, or other depth image data) is acquired. Thus, if a global lighting effect is applied to the virtual structure after building the virtual structure, the player may be forced to wait an undesirably long time before the activity can be played, Also, changing the environment during the game (for example, walking to a different room) may be restricted. This is because the creation and application of lighting to a new environment may take an inappropriately long time period.

  [0022] Accordingly, as described above, the virtual structure 200 is assembled from a set of modular virtual structure segments having a precomputed lighting effect, and instances of the modular virtual structure segments are arranged adjacent to each other. Can be processed (eg, rotated, scaled, etc.) to shape the appearance of an integral virtual structure. FIG. 3 illustrates a set 300 of modular virtual structure segments including a stud column segment 302, a stud column 304 with joint piping, a stud column 306 with horizontal piping, a pair of door frame segments 308, 310, and a pair of window frame segments 312, 314. An exemplary embodiment of is shown. Referring to FIG. 2, the virtual structure 200 is a virtual structure selected from the set 300 and scaled, rotated, split, deformed, and / or otherwise appropriately processed based on the placement of each particular instance. It will be noted that it can be built entirely from instances of wall segments.

  [0023] Although FIG. 3 depicts a relatively simple set of modular virtual structure segments, the set of modular virtual structure segments can be any suitable number and choice with any desired complexity. It will be understood that there may be a number of segments. In addition, the set of modular virtual structure segments with pre-calculated lighting can be applied to any other desired physical feature, including but not limited to a virtual structure that fits the ceiling, other than a virtual structure that fits the wall. May be used to create any other suitable structure to fit, plus non-structural features such as furniture, wall hangings, plants, exterior articles, counter tops, bar tops, etc. Will be understood. For example, pre-illuminated modular virtual structure segments are commonly found in pre-illuminated virtual sofas, desks, televisions, mirrors, and physical environments but differ in shape and / or appearance in different physical environments. It can include other objects that it may have. Such a virtual structure segment of such an object in an instance may consist of a single “segment” so that one virtual structure element is not necessarily combined with the adjacent segment, but the desired physical It will be understood that it will be resized, rotated and otherwise processed to fit the structure. In addition, an empty space in the physical environment may be considered a physical feature of the environment, and is modular so that virtual objects are built in unoccupied parts of the space in a room or other use environment. It will be appreciated that virtual structure segments can be arranged.

  [0024] Some modular virtual structure segments can include connectivity constraints that restrict the set of other segments that can be coupled to the segment. For example, in FIG. 3, each window segment and door segment can be coupled on one side (eg, the window / door side) to another window segment or door segment instead of the stud segment 302, otherwise. Segments will fit incorrectly. Further, the segments of the joint pipe 304 and the horizontal pipe 306 may be limited to a state where the joint pipe 304 and the horizontal pipe 306 are coupled to a segment having a complementary pipe portion. It will be appreciated that these connectivity constraints are described for exemplary purposes and that any other suitable connectivity constraints may be used.

  [0025] Any suitable pre-computed lighting effect can be applied to the modular virtual structure segment. For example, in some implementations, the set of modular virtual structure segments can be intended for use in any local lighting environment, regardless of the location of physical lighting in the environment. In such an embodiment, a directional lighting effect can be utilized. An example of this is shown in FIG. 4 as directional illumination incident on a portion 400 of a virtual stud segment. The applied virtual illumination may have any suitable direction. For example, for horizontally laid pieces, the illumination may be perpendicular to the horizontal axis, while for vertically laid pieces, the illumination may be perpendicular to the vertical axis. In addition, for module pieces laid both horizontally and vertically, the illumination may be perpendicular to both axes. This ensures that the pre-computed shadows and lighting for that segment always have a common lighting characteristic for each segment, and thus the parallax and / or other It is possible to prevent the discontinuity from appearing. In the embodiment of FIG. 4, the directional illumination is shown as being illuminated at an angle of approximately 45 degrees relative to the vertical plane, but this is presented for illustrative purposes. It will be appreciated that any other suitable angle may be used.

  [0026] In other implementations, the set of modular virtual structure segments can be configured to be used with specific lighting characteristics, such as a single overhead point light source, a wall lamp, and the like. In such embodiments, any suitable type of virtual lighting can be used to pre-calculate the lighting effects. In any case, after pre-calculating the lighting effect, the calculated light map has a modular virtual structure so that the image of the virtual structure assembled by its associated modular virtual structure segment has a realistic lighting effect. It can be stored with a high level of information along with the segment.

  [0027] Any suitable type of lighting information may be stored for the modular virtual structure segment. For example, pre-calculated lighting effects can be stored as light maps, cube maps, spherical harmonics (eg, pre-calculated radiance transfer functions), and / or in any other suitable form. The use of the pre-calculated radiance transfer function may be based on physical illumination in the physical environment as depicted by the virtual point light source 500 in FIG. 5A, for example, based on the detected location of the physical illumination in the environment of use. Applying virtual point lighting to the location of the object will allow realistic lighting and shadows to be generated on the virtual object. FIG. 5B shows an example of how the appearance of a portion of the stud shown in FIG. 4 can be adjusted based on the virtual point light source of FIG. 5A. In addition, procedural or dynamic lighting may be applied in real time (eg, light arising from dynamic virtual objects displayed in augmented reality images).

  [0028] Local physical lighting characteristics can also be used to adjust the appearance of the modular virtual structure segment in other ways. For example, the pre-calculated lighting effect of the modular virtual structure segment can be calculated based on white light illumination. Then, in creating a virtual image for a particular physical environment, the color characteristics of the physical lighting in the physical environment can be analyzed from the image data acquired by the see-through display device, The determined color characteristics (eg, hue, saturation, reflectance) can be applied to the virtual lighting effect so that the virtual lighting more closely matches the local physical lighting. In this way, displayed instances of pre-illuminated virtual wall / ceiling segments, pre-illuminated virtual furniture, and any other suitable pre-illuminated virtual objects are represented in the physical environment. You can fit more closely with the appearance of.

  [0029] FIG. 6 illustrates one embodiment of a method 600 for building an augmented reality environment by applying an instance of a modular virtual structure segment to a detected physical structure. The method 600 includes, at 602, receiving image data that captures an image of a local environment of a see-through display device. The image data may include any suitable data including, but not limited to, a depth image 604 and / or a two-dimensional image, and is received from an image sensor on or external to the see-through display device. Can do. Depth image data can be received from any suitable depth imaging system, including but not limited to stereo imaging systems, time-of-flight imaging systems, and structured illumination imaging systems.

  [0030] The method 600 then includes, at 606, identifying physical features of the local environment from image data. The physical feature can be identified in any suitable manner. For example, in some embodiments, at 608, a mesh representation of the physical environment is determined from the depth image data and mesh analysis is performed, and at 610, a major surface within the physical environment is identified. The Examples include, but are not limited to, wall and ceiling features such as doors, windows, skylights, pillars, and other protrusions / notches in the room, in addition to walls 612 and ceiling 614. included. Furthermore, free space within the geometric structure may be identified, for example, a desired virtual structure can be fitted to the identified free space.

  [0031] The method 600 may also include, at 616, identifying one or more local lighting characteristics of the physical environment. Examples of local illumination characteristics can include, but are not limited to, color characteristics 618 and local light source positions 620.

  [0032] The method 600 further includes, at 622, an augmented reality image including virtual structural features for display on the detected physical features in spatial alignment with the physical features. Including the steps of: As described above and shown at 624, the virtual structure can be constructed by arranging a plurality of modular virtual structure segments, each with a pre-computed lighting effect. Virtual structure segments may be arranged in any suitable manner, including but not limited to rotating, scaling, deforming, splitting, etc., parts to fit the physical geometry of interest. Can do. Similarly, the modular virtual structure segment may include any suitable pre-calculated information regarding the pre-calculated lighting effects. Examples include, but are not limited to, light map 626 and / or radiance transfer function 628. Further, as described above, in selecting and arranging modular virtual structure segments, the selected module type is selected at 630 to ensure that complementary features are properly coupled to the adjacent segment. Connectivity constraints can be applied that restrict the set of other modular virtual structure segments that can be coupled to the virtual structure segment.

  [0033] In addition, as described above, local illumination characteristics may be utilized when constructing an augmented reality image. For example, as shown at 632, in some implementations, the appearance of the modular virtual structure segment can be adjusted based on local lighting characteristics. Its appearance can be adjusted in any suitable way. For example, as shown at 634, the color of the local lighting environment may be attached to a pre-calculated lighting effect. Similarly, as shown at 636, a virtual light source, such as a virtual point light source, may be applied to the position of the physical light source in the environment.

  [0034] In other embodiments, instead of adjusting the appearance of the modular virtual structure segment, multiple different sets of modular virtual structure segments with different lighting characteristics may be available. For example, one set of modular virtual structure segments can include pre-calculated lighting effects corresponding to overhead point light sources, while another set corresponds to pre-directional lighting corresponding to directional lighting coming from the side window. Calculated lighting effects can be included. In this example, as shown at 638, local lighting characteristics can be utilized to select a set of modular virtual structure segments with corresponding lighting characteristics, resulting in the result. The virtual structure will have lighting characteristics similar to the physical lighting in its environment.

  [0035] Once the augmented reality image is constructed, the method 600 includes outputting the augmented reality image to a see-through display device, as shown at 640. Sensor data from see-through display devices (eg inward and outward image sensors) is used to detect the user's eye position and gaze direction, as well as to detect physical objects in the user's field of view, The virtual structure can be displayed on the corresponding physical feature in spatial alignment with the physical feature to give the user an augmented reality view of the physical environment.

  [0036] As described above, the methods described above can be implemented by any suitable display device. Examples include, but are not limited to, see-through display devices such as the head-mounted see-through display device 104 of FIG. 1 and other displays with one or more image sensors such as smartphones and notepad computers. Includes devices. FIG. 7 shows a block diagram of an exemplary configuration of see-through display device 104.

  [0037] The see-through display device 104 may include one or more lenses 702 that form part of a near-eye see-through display subsystem 704. The see-through display device 104 can further comprise one or more outward-facing image sensors 706 configured to acquire an image of the background scene that the user is looking at, and can also provide sound, such as voice commands from the user. One or more microphones 708 configured to detect may be included. Outward image sensor 706 can include one or more depth sensors (including but not limited to a stereo depth imaging arrangement) and / or one or more two-dimensional image sensors.

  [0038] The see-through display device 104 further comprises a gaze detection subsystem 710 configured to detect the gaze direction of each eye of the user as described above. The gaze detection subsystem 710 can be configured to determine the gaze direction of each of the user's eyes in any suitable manner. For example, in the depicted embodiment, the gaze detection subsystem 710 includes one or more flash light sources 712, such as an infrared light source configured to reflect a flash of light from the cornea of each of the user's eyes, and the user And one or more image sensors 714 configured to capture one or more eye images. Flash and pupil images determined from image data collected via the image sensor 714 can be used to determine the optical axis of each eye. It will be appreciated that the line-of-sight detection subsystem 710 may include any suitable number and arrangement of light sources and image sensors.

  [0039] The see-through display device 104 may further comprise additional sensors. For example, the see-through display device 104 can include a global positioning (GPS) subsystem 716 to allow the position of the see-through display device 104 to be determined.

  [0040] The see-through display device 104 may further include one or more motion sensors 718 for detecting movement of the user's head when the user is wearing the see-through display device 104. Motion data can be used, for example, for image stabilization, to help correct image blur from the outward image sensor 706. Similarly, motion sensor 718 can be utilized as a user input device with microphone 708 and line-of-sight detection subsystem 710 so that in addition to verbal commands, the user can have eyes, neck and / or head. It will be possible to interact with the see-through display subsystem 704 via gestures. The sensor depicted in FIG. 7 is shown for illustrative purposes and is limited in any way, as any other suitable sensor and / or combination of sensors may be utilized. It will be understood that it is not intended to be.

  [0041] The see-through display device 104 further comprises a computing device 720 having a logical subsystem 722 and a storage subsystem 724 that can communicate with the sensors, the gaze detection subsystem 710, and the see-through display subsystem 704. . The storage subsystem 724 includes instructions stored on the storage subsystem 724 that may, for example, receive image data from an outward-facing image sensor 706 that captures images of the local environment of the see-through display device. It can be implemented by the logical subsystem 722 to receive and identify physical features of the local environment via the image data. The instructions also construct an augmented reality image of the virtual structure by arranging multiple modular virtual structure segments, each with a pre-calculated global lighting effect, at adjacent locations, and then physicalizing the augmented reality image. It may be feasible to display the object in a state of being spatially aligned with the physical feature from the user's viewpoint. The instructions further detect local illumination characteristics, adjust the augmented reality image based on the local illumination characteristics, and convert the augmented reality image of the physical feature via the see-through display subsystem 704. It may be feasible to display in a state of being spatially aligned with the physical feature.

  [0042] Further information regarding exemplary hardware of the logical subsystem 722, storage subsystem 724, and other above-described components is described below with reference to FIG.

  [0043] It will be appreciated that the depicted see-through display device 104 is provided as an example and is therefore not intended to be limiting. Accordingly, a display device may include additional and / or alternative sensors, cameras, microphones, input devices, output devices, etc. other than those shown without departing from the scope of this disclosure. Must be understood. The physical configuration of the display device and its various sensors and subcomponents can take a variety of different forms without departing from the scope of this disclosure.

  [0044] Further, computing systems configured to display augmented reality images via a see-through display device include, but are not limited to, mainframe computers, server computers, desktop computers, laptop computers, tablet computers, Can take any suitable form other than a head mounted display device, including home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (eg, smart phones), other wearable computers, etc. Will be understood. It will be further appreciated that the methods and processes described above can be implemented as a computer application program or service, application programming interface (API), library, and / or other computer program product.

  [0045] FIG. 8 schematically illustrates a non-limiting implementation of a computing system 800 that can implement one or more of the methods and processes described above. The computing system 800 is shown in a simplified form and, as mentioned above, any suitable device and / or including but not limited to those described above with reference to FIGS. 1-9. Or it can represent a combination of devices.

  [0046] Computing system 800 includes a logical subsystem 802 and a storage subsystem 804. The computing system 800 may optionally include a display subsystem 806, an input device subsystem 808, a communication subsystem 810, and / or other components not shown in FIG. The computing system 800 may also include one or more users, such as the eye tracking system described above, as well as, for example, a keyboard, mouse, game controller, camera (depth and / or 2D), microphone, and / or touch screen. Input devices may optionally be included or interfaced therewith. Such user input devices can form part of the input device subsystem 808 or can interface with the input device subsystem 808.

  [0047] The logical subsystem 802 includes one or more physical devices configured to execute instructions. For example, a logical subsystem is configured to execute machine-readable instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Can. Such instructions can be implemented to perform tasks, implement data types, convert the state of one or more components, or otherwise reach a desired result.

  [0048] The logical subsystem 802 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logical subsystem 802 can include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logical subsystem 802 may be single-core or multi-core, and a program executed on the processor can be configured for sequential processing, parallel processing, or distributed processing. The logical subsystem 802 may optionally include discrete components distributed across two or more devices that may be remotely located and / or configured for collaborative processing. The aspects of the logical subsystem can be virtualized and executed by a remotely accessible network computing device configured in a cloud computing configuration.

  [0049] The storage subsystem 804 is one or more physical units configured to hold data and / or instructions executable by the logical subsystem to implement the methods and processes described herein. And non-transitory computer readable storage devices. When such methods and processes are implemented, the state of the storage subsystem 804 can be converted to hold different data, for example.

  [0050] The storage subsystem 804 may include removable media and / or embedded devices. The storage subsystem 804 can be an optical memory device (eg, CD, DVD, HD-DVD, Blu-Ray® disk, etc.), a semiconductor memory device (eg, RAM, EPROM, EEPROM, etc.), and / or magnetic, among others. Memory devices (eg, hard disk drive, floppy disk drive, tape drive, MRAM, etc.) can be included. Storage subsystem 804 includes volatile devices, non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, random access devices, sequential access devices, location addressable devices, file addressable devices And / or content addressable devices. In some implementations, the logical subsystem 802 and storage subsystem 804 may be integrated into one or more single devices, such as an application specific integrated circuit (ASIC) or system on chip.

  [0051] It will be appreciated that the storage subsystem 804 includes one or more physical and non-transitory devices. However, in some implementations, the instructional aspects described herein may be temporary by a pure signal (eg, an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite period of time. May be propagated in a manner. Moreover, data and / or other types of information related to the present disclosure may be propagated by pure signals.

  [0052] The term "program" can be used to describe one aspect of a computing system 800 that is implemented to perform a particular function. In some cases, the program may be instantiated by a logical subsystem 802 that executes instructions held by the storage subsystem 804. It will be appreciated that different programs can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term “program” can encompass an individual or collection of executable files, data files, libraries, drivers, scripts, database records, and the like.

  It will be appreciated that a “service” as used herein is an application program that can be executed across multiple user sessions. A service may be available to one or more system components, programs, and / or other services. In some examples, the service can run on one or more server computing devices.

  [0054] When included, the display subsystem 806 may be used to present a visual representation of the data maintained by the storage subsystem 804. This visual representation can take the form of a graphical user interface (GUI). When the methods and processes described herein change the data held by the storage subsystem and consequently convert the state of the storage subsystem, the state of the display subsystem 806 changes the underlying data. Can be similarly transformed to visually represent Display subsystem 806 may include one or more display devices that utilize almost any type of technology. Such a display device may be combined with the logical subsystem 802 and / or the storage subsystem 804 in a common enclosure, or may be a display device as a peripheral device.

  [0055] When included, the communication subsystem 810 may be configured to communicatively couple the computing system 800 to one or more other computing devices. The communication subsystem 810 can include wired and / or wireless communication devices that are compatible with one or more different communication protocols. By way of non-limiting example, the communication subsystem can be configured for communication over a wireless telephone network or a wired or wireless local area network or wide area network. In some implementations, the communication subsystem may allow the computing system 800 to send and / or receive messages from other devices over a network such as the Internet. .

  [0056] The configurations and / or techniques described herein are illustrative in nature and many variations are possible, and these specific embodiments or examples should be understood in a limiting sense. It will be understood that it must not be done. The specific routines or methods described herein may represent one or more of many processing strategies. Thus, the various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Similarly, the order of the processes described above can be changed.

  [0057] The subject matter of this disclosure is not only various processes, systems and configurations disclosed herein, and other features, functions, operations, and / or characteristics, but any and all equivalents thereof. Includes all new and non-obvious combinations and sub-combinations of objects.

Claims (11)

A method of displaying an augmented reality image with a lighting effect in a display device,
Receiving image data taken of an image of a local environment of the display device;
Identifying the physical surface of the local environment via the image data;
Constructing an augmented reality image of a virtual structure for display on the physical surface in spatial alignment with the physical surface, the construction comprising the first module of the virtual structure Fitting the first modular augmented reality image segment to the surface based on the geometry of the surface by one or more of scaling, rotating, fragmenting, and deforming the augmented reality image segment; Each of the modular extensions is arranged by placing a second modular augmented reality image segment of the virtual structure adjacent to the first modular augmented reality image segment to shape the appearance of the virtual structure. A real image segment having a pre-computed lighting effect and having a look of a portion of the virtual structure;
Outputting the augmented reality image to the display device;
Including methods.   The method of claim 1, wherein the step of identifying a physical surface of the local environment includes performing a mesh analysis of the local environment.   The method of claim 1, wherein the physical surface comprises one or more of a wall and a ceiling.   The method of claim 1, wherein the pre-calculated lighting effect comprises a pre-calculated directional lighting effect.   The method of claim 1, wherein the pre-calculated lighting effect comprises a pre-calculated radiance transfer function.   Identifying lighting characteristics of the local environment via the image data and adjusting the appearance of the first and second modular augmented reality image segments based on the lighting characteristics of the local environment The method of claim 1, further comprising:   The lighting characteristic includes a color characteristic of the local environment, and the step of adjusting the appearance of the first and second modular augmented reality image segments includes the first and second modular augmented reality images. The method of claim 6, comprising attaching the color characteristic to a segment.   The lighting characteristic includes a position of physical illumination in the local environment, and the step of adjusting the appearance of the first and second modular augmented reality image segments includes the position of the physical illumination. The method of claim 6, comprising calculating a lighting effect resulting from a virtual point illumination.   The step of constructing the augmented reality image of the virtual structure comprises identifying a set of modular augmented reality image segments determined to have similar precomputed lighting effects with identifying local lighting characteristics. Selecting the method.   The method of claim 1, wherein the step of constructing the augmented reality image of the virtual structure comprises applying connectivity constraints to select adjacent modular augmented reality image segments. A computing device,
See-through display devices,
An image sensor;
A logical subsystem;
A storage subsystem in which instructions are stored;
The instructions are executed by the logical subsystem,
Obtaining image data of an image of a local environment of the computing device via the image sensor;
Identifying the physical surface of the local environment via the image data;
Constructing an augmented reality image of a virtual structure for display by the see-through display device on the physical surface in spatial alignment with the physical surface, the construction comprising: Based on the geometric structure of the surface, the first modular augmented reality image segment by one or more of scaling, rotation, fragmentation, and deformation of the first modular augmented reality image segment of virtual structure Fitting the surface and then placing a second modular augmented reality image segment of the virtual structure adjacent to the first modular augmented reality image segment to shape the appearance of the virtual structure. Each modular augmented reality image segment has a precomputed lighting effect and said Has the appearance of a portion of a virtual structure, and the step,
Outputting the augmented reality image to the see-through display device;
A computing device that is executable to implement.

Images