CN118020043A

CN118020043A - Augmented reality and screen image rendering coordination

Info

Publication number: CN118020043A
Application number: CN202280064470.7A
Authority: CN
Inventors: A·尼南
Original assignee: Dolby Laboratories Licensing Corp
Current assignee: Dolby Laboratories Licensing Corp
Priority date: 2021-09-24
Filing date: 2022-09-22
Publication date: 2024-05-10

Abstract

A first image is received for rendering on a first image display in a combination of a still image display and a non-still image display. A visual object depicted in the first image is identified. A corresponding image portion in a second image is generated for rendering on a second image display in a combination of the still image display and the non-still image display. The corresponding image portion in the second image as rendered on the second image display overlaps the visual object as depicted in the second image as rendered on the first image display in the field of view of the viewer to modify one or more visual characteristics of the visual object. The second image is rendered simultaneously on the second image while the first image is being rendered on the second image display.

Description

Augmented reality and screen image rendering coordination

Cross Reference to Related Applications

The present application claims the benefit of priority from European patent application number 21,198,907.4 filed 24 at 9 of 2021 and U.S. provisional patent application number 63/248,011 filed 24 at 9 of 2021, each of which is incorporated herein by reference in its entirety.

Technical Field

The present invention relates generally to video images, and in particular to augmented reality and screen image rendering coordination.

Background

User devices such as wearable devices or handheld devices may be used in a wide variety of video or media consumption applications including, but not limited to, virtual Reality (VR) and Augmented Reality (AR), mixed Reality (MR), telepresence, telemedicine, 3D video, omnidirectional video, and the like. To add sense and interactivity to the viewer's experience, such applications may monitor the time-varying spatial position and orientation of the user device from physical movements and render images based on the spatial position and orientation of the user device.

However, visual objects rendered by the user device, as well as other objects not rendered by the user device, can be readily discerned by the user as belonging to two distinct worlds. For example, in a scenario where the user device is an AR device, the user can view both AR objects rendered by the user device and other non-AR objects rendered by the screen image display (e.g., through 3D glasses, etc.), the AR objects may appear as manually floating visual objects or information blocks, not belonging to the same physical world as other non-AR objects that the user will see, and may even inadvertently obstruct and interfere with other non-AR objects that the user will see.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, any approaches described in this section are not to be construed so as to qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, questions identified with respect to one or more methods should not be deemed to be recognized in any prior art based on this section.

US2021/120317 A1 discloses a method for geometrically matching a Virtual Reality (VR) or Augmented Reality (AR) output simultaneously with a video output formatted for display on a 2D screen, the method comprising determining sets of values that, when used in image processing, cause an off-screen angular field of view of at least one of the AR output object or the VR output object to have a fixed relationship with at least one of an angular field of view of an on-screen object or an angular field of view of the 2D screen. The AR/VR output object is output to the AR/VR display device and improves the user experience by geometric matching between objects observed on the AR/VR display device and corresponding objects appearing on the 2D screen.

US10 078 917 B1 discloses a method comprising rendering a first view of a three-dimensional (3-D) virtual scene comprising a first content view displayed on a virtual display device from a position in the 3-D virtual scene. The method may also include rendering a second view that includes one or more content objects. The second view may be rendered from the location in the 3-D virtual scene, and the second scene may include a display device view displaying one or more content objects as would be seen through a pair of augmented reality glasses. The method may additionally include generating a composite view by combining the first view with the second view. The method may further include causing the composite view to be displayed on the virtual reality headset.

Benko Hrvoje et al ："FoveAR Combining an Optically See-Through Near-Eye Display with Projector-Based Spatial Augmented Reality[ combine an optical see-through near-eye display with projector-based spatial augmented reality FoveAR ] ", user Interface Software and Technology [ user interface software and technology ], ACM,2015, 11/month 5, pages 129 to 135, XP 058525302, disclose optical see-through (OST) augmented reality glasses that overlay spatially registered computer-generated content to the real world. Optical designs and weight considerations known in the art limit their diagonal field of view to less than 40 degrees, making it difficult to create immersion or to give the viewer an overview of the augmented reality space. The combination of OST glasses with projection-based spatial augmented reality displays enables a novel display mixture named FoveAR that enables an interesting combination of field of view greater than 100 degrees, view dependent graphics, extended brightness and color, and public and personal data displays.

Disclosure of Invention

The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention.

Drawings

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIGS. 1A and 1B illustrate an example 3D object rendered jointly and simultaneously by an AR and a screen image display; FIG. 1C illustrates an example media system including an AR and a screen image display;

FIGS. 2A and 2B illustrate example perspective corrections to AR and screen images;

3A-3C illustrate example compensated image rendering operations by an AR and a screen image display;

FIG. 4 illustrates an example process flow; and

FIG. 5 illustrates an example hardware platform on which a computer or computing device as described herein may be implemented.

Detailed Description

Example embodiments are described herein that relate to augmented reality and screen image rendering coordination. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It may be evident, however, that the subject invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, or confusing the present invention.

Example embodiments are described herein in terms of the following summary:

1. General overview

2. Augmented reality and screen image display

3. Perspective correction

4. Compensation image rendering

5. Dynamic image rendering transitions between image displays

6. Supplementary information display

7. Example image rendering System

8. Example Process flow

9. Implementation mechanism-hardware overview

10. Equivalents, extensions, alternatives and others

1. General overview

This summary presents a basic description of some aspects of example embodiments of the invention. It should be noted that this summary is not an extensive or exhaustive overview of aspects of the example embodiments. Moreover, it should be noted that this summary is not intended to be understood as identifying any particularly important aspects or elements of the example embodiments, nor is it intended to be construed as specifically depicting any scope of the example embodiments, nor is it intended to be a general depiction of the present invention. This summary merely presents some concepts related to the example embodiments in a compressed and simplified form and is understood to be merely a conceptual prelude to the more detailed description of the example embodiments that follow. Note that although individual embodiments are discussed herein, any combination of the embodiments and/or portions of the embodiments discussed herein can be combined to form further embodiments.

Example embodiments described herein relate to compensated image rendering between two or more different image displays. A first image is received for rendering on a first image display in a combination of a still image display and a non-still image display. A visual object depicted in the first image is identified. A corresponding image portion in a second image is generated for rendering on a second image display in a combination of the still image display and the non-still image display. The corresponding image portion in the second image as rendered on the second image display overlaps the visual object as depicted in the second image as rendered on the first image display in the field of view of the viewer to modify one or more visual characteristics of the visual object. The second image is rendered simultaneously on the second image while the first image is being rendered on the second image display.

Example embodiments described herein relate to a multi-display system implementing compensated image rendering. A display system, comprising: a non-still image display rendering a non-screen display image; a still image display rendering a screen display image; an image rendering controller that controls the non-still image display and the still image display to perform the compensated image rendering as described herein. Additionally, optionally or alternatively, the display system controls one or both of the non-still image display or the still image display to perform perspective correction with respect to the rendered image.

In some example embodiments, mechanisms as described herein form part of a media processing system, including, but not limited to, any of the following: AR devices, VR devices, MR devices, cloud-based servers, mobile devices, virtual reality systems, augmented reality systems, heads-up display devices, head-mounted display devices, CAVE-type systems, wall-sized displays, video gaming devices, display devices, media players, media servers, media production systems, camera systems, home-based systems, communication devices, video processing systems, video codec systems, production room systems, streaming media servers, cloud-based content service systems, handheld devices, gaming machines, televisions, cinema displays, laptop computers, notebook computers, tablet computers, cellular radiotelephones, electronic book readers, point-of-sale terminals, desktop computers, computer workstations, computer servers, computer kiosks, or various other types of terminals and media processing units.

Various modifications to the preferred embodiments and the generic principles herein described will be readily apparent to those skilled in the art. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

2. Augmented reality and screen image display

Fig. 1A illustrates example 3D (visual) objects (e.g., 120-1, 120-2, 118-1, 118-2, etc.) that are jointly and simultaneously rendered to a viewer (or user) 112 with a user-coupled (e.g., user-attached, user-movable, user-operated, user-worn, head-worn, wearable, etc.) image display device 102 and a user-separated image display 104.

The user-coupled image display device (102) may be a wearable device such as a head-mounted device or a handheld device such as a mobile phone or tablet device. The user-coupled image display device (102) is physically coupled or attached to the viewer (112).

For simplicity, the user device (102) may be referred to herein as an Augmented Reality (AR) image display. However, it should be noted that in various embodiments, the user device (102) may be any image display device that is physically coupled or attached (e.g., handheld, wearable, stationary, etc.) to the viewer (112) by hand-held, wearable, head-worn, etc. It should be further noted that in various embodiments, the user device (102) may be used in VR, MR or other interactive image/video display applications or non-interactive image/video display applications.

The image display (104) separate from the user may be a computer screen, a television display, a home image display, a movie theater, an image display in a venue such as a store, park, bar, office, etc. For simplicity, the image display device (104) separate from the user may be referred to as a screen image display. However, it should be noted that in various embodiments, the image display device (104) separate from the user may be any image display device that is not physically attached (or worn) with the viewer (112). It should be further noted that in various embodiments, the image display device (104) separate from the user may be used in VR, MR or other interactive image/video display applications or non-interactive image/video display applications along with the AR image display (102).

While the screen image display (104) may be stationary in the physical 3D space 196 in which the viewer (112) is located, the AR image display (102) may be moved (e.g., translated, rotated, etc.) from time to time in the physical 3D space (196) due to physical actions or movements of the viewer. In some operational scenarios, a movement position and/or spatial position 192-1 and/or spatial orientation 192-2 of an AR image display (102) in a physical 3D space (196) is tracked by one or more tracking devices. Because of device tracking, the movement position and/or spatial position 192-1 and/or spatial orientation 192-2 of the AR image display (102) in the physical 3D space (196) is estimated, predicted, or otherwise determined at any given time, such as by the points in time at which the AR image display (102) and the screen image display (104) are rendering 3D objects (e.g., 120-1, 120-2, 118-1, 118-2, etc.).

Example device tracking may be found in U.S. provisional patent application No. 62/484,131, entitled "PASSIVE MULTI-WEARABLE-DEVICES TRACKING [ PASSIVE tracking of multiple wearable devices ]" filed by Ajit Ninan and NEIL MAMMEN at 11, 2017, the entire contents of which are incorporated herein by reference as if fully set forth herein.

FIG. 1B illustrates an example image space where 3D objects (e.g., 120-1, 120-2, 118-1, 118-2, etc.) are rendered by an AR image display (102) and a screen image display (104). The rendered 3D objects (e.g., 120-1, 120-2, 118-1, 118-2, etc.) reside or are depicted in an overall image space that includes a first image space 196-1 and a second image space 196-2. The first image space (196-1) may be referred to simply as an AR image space, and the second image space (196-2) may be referred to simply as a screen image space. The AR image space may additionally, alternatively or alternatively be referred to as VR image space, MR image space, etc., depending on what specific display applications are supported by the image displays (102 and 104) as described herein.

As shown in fig. 1B, 3D objects (120-1 and 120-2) are rendered by AR image display (102) in AR image space (196-1), while 3D objects (118-1 and 118-2) are rendered by screen image display (104) in screen image space (196-2).

The screen image display (104) may render a time series of (continuous) screen images depicting 2D or 3D objects at a (continuous) time point series using image rendering techniques including, but not necessarily limited to, one of: time multiplexed 3D, spatially multiplexed 3D, spectral multiplexed 3D, dolby 3D, realD, linear polarization based 3D, circular polarization based 3D, spectral spatial separation based 3D, and the like.

In some operational scenarios, a viewer (112) may see or visually perceive a 3D object rendered in screen image space (196-2) using 3D glasses or another 3D viewing method. The 3D glasses may or may not be integrated with a single end user device including the AR image display (102).

In some operational scenarios, the screen image display (104) is a single user image display used by a single viewer (or user) to view an image rendered on the screen image display (104). Additionally, optionally or alternatively, in some operational scenarios, the screen image display (104) is a shared image display that is used by one or more viewers (or users) simultaneously to view images rendered on the screen image display (104).

The AR image display (102) may render a time series of (continuous) AR images depicting 2D or 3D objects at a (continuous) time point series using image rendering techniques including, but not necessarily limited to, one of: AR displays, holonens displays, MAGIC LEAP displays, mixed Reality (MR) displays, tensor displays, volumetric displays, light Field (LF) displays, immy displays, meta displays, a relatively simple pair of AR glasses, and the like.

In some operational scenarios, 3D objects in AR image space (196-1), or a subset of their 3D objects, may be rendered with an image display of an AR, MR, VR, or wearable device (e.g., a head-mounted device). Additionally, optionally or alternatively, in some operational scenarios, the 3D objects in the AR image space (196-1), or a subset of the 3D objects thereof, may be rendered with an image display of a handheld user device, such as a mobile phone or tablet computing device. The viewer (112) may see or visually perceive the 3D image rendered in the AR image space (196-1) using 3D glasses or another 3D image viewing method. Example ARs, MRs, VRs, or wearable devices may be found in: U.S. provisional patent application Ser. No. 62/484,157, entitled "AUGMENTED 3DENTERTAINMENT SYSTEMS [ enhanced 3D entertainment System ]", filed by Ajit Ninan and NEIL MAMMEN at 11 of 2017; U.S. provisional patent application No. 63/060,783, entitled "SYSTEM FOR OPTIMIZED SELECTIVE LIGHT PASS threugh [ system for optimizing selective light passage ]" filed by Ajit Ninan, titus Devine, and Chun Chi Wan at month 8 and 4 of 2020, the entire contents of which are incorporated herein by reference as if fully set forth herein.

The screen image sequence and the AR image sequence may be generated or divided from an overall 3D image sequence covering a time interval or duration (e.g. a user session represented by a plurality of time points). Example image generation and partitioning can be found in U.S. provisional patent application No. 62/484,121, entitled "LAYERED AUGMENTED ENTERTAINMENT EXPERIENCES [ hierarchically enhanced entertainment experience ]" filed on date 4/11 at 2017 by Ajit Ninan, NEIL MAMMEN, and Tyrome Brown, the entire contents of which are incorporated herein by reference as if fully set forth herein.

For example, 3D objects (120-1 and 120-2) rendered by AR image display (102) in AR image space (196-1) may be represented in one or more AR images in an AR image sequence, while 3D objects (118-1 and 118-2) rendered by screen image display (104) in screen image space (196-2) may be represented in one or more corresponding screen images of the screen image sequence. One or more AR images and one or more corresponding screen images may be generated or divided, respectively, from one or more whole 3D images in the whole 3D image sequence.

The AR image may be used to provide a differentiated media consumption experience or an enhanced media consumption experience to a viewer (112) in a single viewer operational scene or multiple viewer operational scenes.

A 2D or 3D screen image may be rendered or displayed on the screen image display (104) and a 2D or 3D AR image may be rendered or displayed on the AR image display (102). A viewer (112) may view the screen image and see (or visually perceive) additional 3D objects or additional 3D depth information in the AR image rendered with the AR image display (102) at the same time. Some additional 3D objects depicted in an AR image rendered with the AR image display (102) may appear to pop up from the screen image display (104) in front of the viewer (112).

The viewer (112) may track any of these 3D objects as if such 3D objects were actually present in the 3D physical space (196) in which the viewer (112) was located. As the viewer (112) moves around the 3D object, the viewer (112) may see previously obscured visual details of the 3D object around the 3D object with the sense of realism of increasing/enhancing dimensions or depth. As the viewer (112) moves, some or all of the depicted objects in the rendered image may move with the viewer (112), although this may vary depending on the spatial relationship (e.g., depth, angle, distance, etc.) between each of the objects and the viewer (112) and/or depending on the spatial trajectory formed by the movement of the viewer (112). Thus, by combining the AR image with the screen image, the viewer (112) may obtain a psychovisual perception that the 3D object floats around by seeing different parallaxes to these objects in response to the head and/or body movements of the viewer (112).

3. Perspective correction

In some operational scenarios, perspective correction may be performed on the screen image and/or the AR image while the viewer (112) is performing a time-varying head movement or body movement that changes the spatial position and/or spatial orientation of the AR image display (102). Example perspective corrections to a 2D or 3D image or 2D or 3D object depicted therein may include, but are not necessarily limited to, one or more of the following: relative or global translation, relative or global rotation, relative or global size scaling (e.g., zooming in, zooming out, etc.), etc.

For example, the viewer (112) may view 3D objects depicted in An (AR) image rendered on a phone representing an AR image display (102), as well as other 3D objects depicted in a screen image rendered on a screen image display (104). When a viewer (112) moves the phone, perspective correction may be made to the AR image rendered on the phone so that the AR image space (196-1) and the screen image space (196-2) depicted (in part) on the phone are one continuous (image) space between the phone and the screen image display (104). The viewer (112) may visually perceive 3D objects that are visible on the phone as being in the same overall image space as other 3D objects depicted on the screen image display (104), even if the viewer (112) is moving around the phone to see a different 3D object among the 3D objects to be depicted on the phone.

Additionally, optionally or alternatively, in some operational scenarios, perspective correction may not be performed on the screen image and/or the AR image while the viewer (112) is performing a time-varying head movement or body movement that changes the spatial position and/or spatial orientation of the AR image display (102).

Fig. 2A illustrates an example perspective correction to an AR image. As shown, the spatial position (192-1) and/or spatial orientation (192-2) of the AR image display (102) may change from time to time in the 3D space (196) as the viewer (112) performs head movements and/or body movements.

In response to detecting a change in the spatial position (192-1) and/or the spatial direction (192-2) of the AR image display (102) (e.g., in real-time, near real-time, within a strict time budget, etc.), the image rendering controller (e.g., 110 of fig. 1C, etc.) may generate one or more perspective corrected AR images from the one or more pre-perspective corrected (e.g., original, non-perspective corrected, etc.) AR images, respectively, based on the change in the spatial position (192-1) and/or the spatial direction (192-2) of the AR image display (102). The AR image before perspective correction may be generated or divided from one or more corresponding overall 3D images to be viewed by a viewer (112) located at a reference spatial position and/or reference spatial direction of the AR image display (102). Perspective corrected AR images may be generated, adjusted, and rendered by the AR image display (102) to enable the viewer (112) to view or visually inspect/examine the 3D objects from different viewing perspectives at the (currently changing) spatial position (192-1) and/or the (currently changing) spatial direction (192-2) of the AR image display (102).

For example, assuming that 3D objects (120-1 and 120-2) rendered with the AR image display (102) have not changed with reference to the 3D physical space (196) while the viewer (112) is moving head and/or body, perspective corrected AR images may enable the viewer (112) to view or visually inspect/verify image details of these 3D objects that are visible to different viewing perspectives at the (currently changing) spatial position (192-1) and/or the (currently changing) spatial direction (192-2) of the AR image display (102).

Fig. 2B illustrates an example perspective correction to a screen image. As shown, the spatial position (192-1) and/or spatial orientation (192-2) of the 3D glasses (which may or may not be integrated with the AR image display (102) in a single device) may change from time to time in the 3D space (196) as the viewer (112) performs head movements and/or body movements.

In response to detecting a change in the spatial position (192-1) and/or the spatial direction (192-2) of the 3D glasses (e.g., in real time, near real time, within a strict time budget, etc.), the image rendering controller (e.g., 110 of fig. 1C, etc.) may generate one or more perspective corrected screen images from one or more pre-perspective corrected (e.g., original, non-perspective corrected, etc.) screen images, respectively, based on the change in the spatial position (192-1) and/or the spatial direction (192-2) of the 3D glasses. The screen image before perspective correction may be generated or divided from one or more corresponding overall 3D images to be viewed by a viewer (112) located at a reference spatial position and/or reference spatial direction of the 3D glasses. Perspective corrected screen images may be generated, adjusted, and rendered by the screen image display (104) to enable the viewer (112) to view or visually inspect/examine the 3D objects from different viewing perspectives at the (currently changing) spatial position (192-1) and/or the (currently changing) spatial direction (192-2) of the 3D glasses.

For example, assuming that the 3D objects (118-1 and 118-2) have not changed with reference to the 3D physical space (196) while the viewer (112) is performing head movements and/or body movements, the perspective corrected image may enable the viewer (112) to view or visually inspect/verify the image details of these 3D objects, which are visible to different viewing perspectives at the (currently changing) spatial position (192-1) and/or the (currently changing) spatial direction (192-2) of the 3D glasses.

In a multiple viewer operation scene, for example in a movie theater, the same screen image may be rendered to multiple viewers. Perspective correction with respect to a screen image rendered on the screen image display (104) may not be performed while the viewer is making a head movement or body movement that changes the spatial position and/or spatial orientation of their AR image display. For example, in these operational scenarios, the same pre-perspective (e.g., original, non-perspective corrected, etc.) screen image may be rendered on the screen image display (104) as the spatial position (192-1) and/or spatial direction (192-2) of the 3D glasses change from time to time in the 3D space (196) due to head movement and/or body movement of the viewer (112).

In these operational scenarios, perspective correction may or may not be performed with respect to AR images rendered on the AR image display (102) as the viewer (112) makes head movements or body movements that change the spatial position (192-1) and/or spatial orientation (192-2) of the AR image display (102).

In an example, perspective correction with respect to an AR image rendered on the AR image display (102) may not be performed while the viewer (112) is making a head movement or body movement that changes the spatial position (192-1) and/or spatial orientation (192-2) of the AR image display (102). Thus, as the spatial position (192-1) and/or spatial direction (192-2) of the AR image display (102) changes from time to time in the 3D space (196) due to head movement and/or body movement of the viewer (112), the same pre-perspective (e.g., original, non-perspective corrected, etc.) AR image may be rendered on the AR image display (102).

In another example, perspective correction is performed with respect to an AR image rendered on the AR image display (102) while the viewer (112) is making a head movement or body movement that changes the spatial position (192-1) and/or spatial orientation (192-2) of the AR image display (102). Thus, as illustrated in fig. 2A, perspective corrected AR images may be rendered on the AR image display (102) in response to changes in the spatial position (192-1) and/or spatial direction (192-2) of the AR image display (102) due to head movement and/or body movement of the viewer (112).

In a single viewer operation scene, the AR image display (102) has a one-to-one correspondence with the screen image display (104). As the viewer (112) makes head movement or body movement that causes the AR image display (102) to change spatial position (192-1) and/or spatial orientation (192-2), perspective correction may or may not be performed with respect to the AR image rendered on the AR image display (102) as illustrated in fig. 2A, or the screen image rendered on the screen image display (104) as illustrated in fig. 2B, or both the AR image and the screen image.

In these operational scenarios, perspective correction may be performed with different references. In an example, perspective correction may be performed with reference to the 3D physical space (196) -e.g., both the AR-rendered object and the screen-rendered object may move as the viewer (112) moves. Perspective correction may render 3D objects as if they were real objects physically present in 3D physical space (196). In another example, perspective correction may be performed with reference to the screen image space (196-2) -e.g., an object depicted by AR may move as the viewer (112) moves and an object depicted by the screen may appear stationary. The AR image perspective may be corrected to the same perspective that the viewer (112) employed when viewing the 3D object in the screen image depicted in screen image space (196-2) (e.g., perspective adjusted, non-perspective adjusted, etc.). In yet another example, perspective correction may be performed with reference to the AR image space (196-1) -e.g., the screen-rendered object may move as the viewer (112) moves and the AR-rendered object may appear stationary. The screen image perspective may be corrected to the same perspective that the viewer (112) employed in viewing the 3D object in the AR image depicted in AR image space (196-1) (e.g., perspective adjusted, non-perspective adjusted, etc.).

Additionally, optionally or alternatively, neither the AR image nor the screen image is perspective corrected when the viewer (112) makes a head movement or body movement that causes the AR image display (102) and/or the 3D glasses to change spatial position (192-1) and/or spatial orientation (192-2).

For illustration purposes only, it has been described that an AR image display (102) by which a viewer (112) may view AR images or 3D glasses by which a viewer (112) may view screen images may be spatially characterized or represented by a spatial position (192-1) and/or a spatial orientation (192-2). It should be noted that in various embodiments, different spatial representations of viewing perspectives related to AR images and/or screen images may be used. For example, naked eye (e.g., autostereoscopic, etc.) 3D viewing may be supported under the techniques described herein. Instead of 3D glasses, the viewing angle of the viewer (112) may be represented and tracked by means of a viewer's head grid or a logical face grid. In some embodiments, 3D glasses for viewing screen images may not be integrated into a single (e.g., wearable, head-mounted, etc.) device with AR image display (102) -e.g., AR image display (102) is a handheld device. In these embodiments, the spatial position and/or spatial orientation of the 3D glasses and the spatial position and/or spatial orientation of the AR image display (102) may be tracked separately and simultaneously and used in respective perspective corrections (if performed) to the screen image and the AR image.

4. Compensation image rendering

For example, the AR and the screen image display may be controlled or coordinated by an image rendering controller to perform a compensation image rendering that increases dynamic range, enlarges color gamut, improves color accuracy, enhances color saturation, sets forth (e.g., appropriate, specific, etc.) a background for foreground objects, etc. in a combination of the AR and the screen image.

FIG. 3A illustrates an example compensated image rendering for providing or setting forth a background for foreground objects rendered in an AR image. As shown, the object (120-1) may be depicted or rendered as a foreground object by the AR image display (102).

The AR image display (102) may render the AR object with a transparent or semi-transparent background without implementing other methods of the technology as described herein. Thus, the viewer (112) may not be able to perceive this object (120-1) as a real or physical entity object.

An image rendering controller (e.g., 110 of fig. 1C, etc.) as described herein may receive user tracking information from one or more tracking devices (e.g., 124 of fig. 1C, etc.) that indicates a spatial position (192-1) and/or a spatial direction (192-2) of an AR image display (102) and/or 3D glasses, a geometric relationship between a viewer (112) and a screen image space (192), a geometric relationship between the viewer (112) and a screen image display (104), etc. Example tracking devices as described herein may include, but are not necessarily limited to, any of the following: an external device tracker, an internal device tracker, an outside-in device tracker, an inside-out device tracker, etc.

Additionally, optionally or alternatively, the image rendering controller (110) may access and/or use real-time or non-real-time AR and/or screen image display device configuration and/or operational information (e.g., manufacturer, model, device operational settings, device operational status, etc.) to determine a geometric relationship between the viewer (112) and each AR image space (196-1), a geometric relationship between the viewer (112) and the AR image display (102), etc.

The image rendering controller (110) decodes, parses, or identifies geometric information of the object (120-1) rendered in the AR image based at least in part on image data and/or image metadata in the received image/video signal (e.g., 114 of fig. 1C, etc.). In an operational scenario in which perspective correction is performed in an AR and/or screen image, geometric information (of the object (120-1)) as derived from received image data and/or received image metadata may be perspective corrected accordingly.

Using some or all of the geometric information collected from device tracking, device configuration and/or operational information, geometric information of objects depicted from received image data and image metadata, the image rendering controller (110) may calculate or determine a portion 126 of the screen image corresponding to the AR image (e.g., to be rendered concurrently with the AR image, etc.). The image rendering controller (110) may make or cause a change to be made to a portion (126) of the screen image to generate a background (e.g., dark, black, etc.) object 128 to be rendered in the screen image with the change to the portion (126). In addition to default background, such as transparent or translucent background, the background object (128) may also provide the object (120-1) depicted by the AR with background as visually perceived by the viewer (112). In some operational scenarios, a background object (128) accompanying a foreground object (120-1), as visually perceived by a viewer (112), may be completely identical to or larger or smaller than the foreground object (120-1). In some operational scenarios, a background object (128) accompanying a foreground object (120-1), as visually perceived by a viewer (112), may or may not coincide with the foreground object (120-1) at the same depth from the viewer (112).

In some operational scenarios, for the purpose of providing a background to a foreground object depicted in an AR image (e.g., local, localized, changing only the background behind the foreground object, etc.), a similar compensation image rendering operation may be performed for some or all other foreground objects depicted in the AR image to make localized changes or local adjustments in the corresponding screen image to be rendered concurrently with the AR image.

In some operational scenarios, only a single object or a limited number of objects in the AR image may be compensated with the background object or background image portion rendered on the corresponding screen image. For example, the object (120-1) may be selected or identified based on information collected from eye tracking and device tracking. The collected information may be used to determine which objects are being focused, followed, or tracked by the gaze of the viewer (112) at the time the AR image and corresponding screen image are to be rendered. Only objects that are gazed by the viewer (112) may be compensated with the background objects or background image portions rendered on the corresponding screen image. Additionally, optionally or alternatively, only objects that are gazed by the viewer (112) and objects that are within a limited (e.g., 3-angle, 5-angle, 10-angle, etc.) view of the viewer (112) around the objects that are gazed by the viewer (112) may be compensated along with the background objects or background image portions that are rendered on the corresponding screen images.

Additionally, alternatively or in addition to or instead of background compensated image rendering, other compensated image rendering may be implemented or performed under the techniques as described herein.

In a first example, the compensated image rendering may be implemented or performed to increase the effective or combined dynamic range of the combination of the AR image display (102) and the screen image display (104). Peak brightness of one or more pixels (e.g., corresponding to a depicted object, etc., as viewed by a viewer (112)) rendered in AR image space (102) and/or screen image space (104) may be increased by implementing or performing a compensation image rendering from a combination of AR image display (102) and screen image display (104) to increase image rendering light level/transmittance. Conversely, the darkest black of one or more pixels rendered in the AR image space (102) and/or the screen image space (104) may be reduced by implementing or performing a compensating image rendering with a combination of the AR image display (102) and the screen image display (104) to reduce the image rendering light level/transmittance, for example, according to an artistic intent for the rendered overall image including those pixels. For example, when looking at or gazing at a depicted object by a viewer (112), image details of the depicted object may be rendered by using a combination of an AR image display (102) and a screen image display (104) with a compensation image rendering using a light/brightness level up to a relatively high peak light brightness and/or a dark level down to a darkest black.

In a second example, compensation image rendering may be implemented or performed to enlarge the effective or combined color gamut of a combination of the AR image display (102) and the screen image display (104). Color saturation of one or more pixels (e.g., corresponding to a depicted object, etc., as viewed by a viewer (112)) rendered in the AR image space (102) and/or the screen image space (104) may be increased by implementing or performing a compensation image rendering from a combination of the AR image display (102) and the screen image display (104) to use the relatively saturated primary colors. For example, when looking at or gazing at a depicted object by a viewer (112), image details of the depicted object may be rendered by using the enlarged color gamut for compensation image rendering with a combination of the AR image display (102) and the screen image display (104).

In a third example, compensation image rendering may be implemented or performed to enhance the effective or combined color accuracy of a combination of the AR image display (102) and the screen image display (104). Color accuracy of one or more pixels (e.g., corresponding to a depicted object as viewed by a viewer (112), etc.) rendered in the AR image space (102) and/or the screen image space (104) may be improved by implementing or performing a compensated image rendering from a combination of the AR image display (102) and the screen image display (104) to use relatively accurate colors with less quantization or coding errors. For example, image details of a depicted object may be rendered by compensating image rendering with enhanced color accuracy using a combination of an AR image display (102) and a screen image display (104) while looking at or gazing at the depicted object by a viewer (112).

In a fourth example, the compensated image rendering may be implemented or performed to optimize the effective or combined spatial resolution of the combination of the AR image display (102) and the screen image display (104). The spatial resolution of one or more image details rendered in the AR image space (102) and/or the screen image space (104) (e.g., corresponding to a depicted object being gazed by a viewer (112), etc.) may be increased by implementing or performing a compensated image rendering from a combination of the AR image display (102) and the screen image display (104) to use relatively high spatial resolution pixels with less physical animation or errors. For example, upon looking at or gazing at the depicted object from different viewing perspectives of the viewer (112), previously occluded image details may be revealed and the combination of AR image display (102) and screen image display (104) utilized by the compensated image rendering to show the revealed image details at a relatively high or optimized spatial resolution.

The AR image display (102) may render AR objects with discontinuous dynamic range, different color gamuts, different color saturation, different color precision, different spatial resolution, and the like without implementing other methods of the techniques as described herein. The same semantic object as represented in the overall image may be depicted or rendered with abrupt and visually noticeable changes or discontinuities in peak light intensity, dark level, color gamut or color range, color saturation, color accuracy, spatial resolution, etc., as the depicted object moves through the boundary separating the AR image space (196-1) and the screen image space (196-2) or the boundary between the AR image space and the screen image space. Thus, the viewer (112) may not be able to perceive this object (120-1) as a real or physical entity object.

As described, the image rendering controller (e.g., 110 of fig. 1C, etc.) may use and access some or all of the user tracking information, real-time or non-real-time AR and/or screen image display device configuration and/or operation information, received image data, and/or image metadata to determine geometric information and/or geometric relationships related to the depicted objects in the AR image space (196-1) and/or the screen image space (196-2).

Using some or all of the determined geometric information and/or geometric relationships related to the depicted objects, the image rendering controller (110) may divide the depicted objects represented in the one or more global images into one or more AR images and corresponding mutually compensated image details in the one or more corresponding screen images, both of which are generated from the one or more global images. The 3D dimension of the depicted object may be implemented or rendered in one or both of the AR image and the screen image.

In some operational scenarios, rendering the depicted object with the AR image display (102) and rendering the depicted object through the screen image display (104) may be time synchronized (e.g., within a time difference that the human visual system or HVS may temporarily resolve, etc.).

Additionally, optionally or alternatively, the depicted object rendered with the AR image display (102) as seen or visually perceived by the viewer (112) may be completely identical to the depicted object rendered by the screen image display (104). In some operational scenarios, a depicted object rendered with the AR image display (102) as seen or visually perceived by a viewer (112) and a depicted object rendered by the screen image display (104) may or may not coincide with each other at the same depth from the viewer (112).

In some operational scenarios, only a single object or a limited number of objects in an AR image and/or screen image may be compensated or cooperatively rendered using a combination of an AR image display (102) and a screen image display (104) as controlled by an image rendering controller (110). For example, such objects may be selected or identified based on information collected from eye tracking and device tracking. The collected information may be used to determine which objects are being focused, followed, or tracked by the gaze of the viewer (112) at the time the AR image and corresponding screen image are to be rendered. Identifying one or more objects (i.e., a single object or a limited number of objects) includes detecting edges of the identified one or more objects to separate the background from the identified one or more objects. Although the term "focus" refers to all objects located in a focus area that includes a background, the identified one or more objects relate to a subset of the focus area that has separated the background from the identified one or more objects. Only objects that are gazed by the viewer (112) may be compensated and cooperatively rendered by the combination of the AR image display (102) and the screen image display (104), while other portions of the image related to the background in the focus area remain uncompensated. Additionally, alternatively or alternatively, only objects that are gazed by the viewer (112) and objects that are within a limited (e.g., 3-angle, 5-angle, 10-angle, etc.) view of the viewer (112) around the objects that are gazed by the viewer (112) may be compensated for and cooperatively rendered by a combination of the AR image display (102) and the screen image display (104).

In some operational scenarios, the screen image display (104) may be implemented or configured to have a higher display capability than the AR image display (102). For example, the screen image display (104) may be capable of rendering a higher dynamic range, a larger color gamut, higher color accuracy, finer spatial resolution, etc., as compared to the AR image display (102). For example, the AR image display (102) may be a black and white image display or an LED (light emitting diode) image display with smaller color gamut, lower dynamic range, lower spatial resolution, etc., while the screen image display (104) may be a laser color master image display with larger color gamut, higher dynamic range, higher spatial resolution, etc. The screen image display (104) may be controlled or driven by an image rendering controller (110) to perform a compensation image rendering to enhance the depicted object or image details rendered with the AR image display (102). For example, the image rendering controller (110) may modify the screen image for the depicted object or add to the screen image with one or more compensated image portions for the purpose of causing the screen image display (104) to compensate for the depicted object in one or more visual characteristics, such as one or more of dynamic range, color gamut, spatial resolution, color saturation, color accuracy, and the like.

Additionally, optionally or alternatively, even though the screen image display (104) may be implemented or configured to have lower or comparable display capabilities as compared to the AR image display (102), the screen image display (104) may be controlled or driven by the image rendering controller (110) to perform a compensating image rendering to enhance the depicted object or image detail rendered with the AR image display (102), albeit to a relatively limited extent. For example, the image rendering controller (110) may still modify the screen image for the depicted object or add to the screen image with one or more compensated image portions for the purpose of causing the screen image display (104) to compensate for the depicted object in one or more visual characteristics, such as one or more of dynamic range, color gamut, spatial resolution, color saturation, color accuracy, etc.

Additionally, optionally or alternatively, in some operational scenarios, the AR image display (102) may be implemented or configured to have a higher display capability than the screen image display (104). For example, the AR image display (102) may be capable of rendering a higher dynamic range, a larger color gamut, higher color accuracy, finer spatial resolution, etc., as compared to the screen image display (104). The AR image display (102) may be controlled or driven by an image rendering controller (110) to perform a compensation image rendering to enhance depicted objects or image details rendered with the screen image display (104). For example, the image rendering controller (110) may modify the AR image for the depicted object or add one or more compensated image portions to the AR image for the purpose of causing the AR image display (102) to compensate for the depicted object in one or more visual characteristics, such as one or more of dynamic range, color gamut, spatial resolution, color saturation, color accuracy, etc.

Additionally, optionally or alternatively, even though the AR image display (102) may be implemented or configured to have lower or comparable display capabilities as compared to the screen image display (104), the AR image display (102) may be controlled or driven by the image rendering controller (110) to perform a compensating image rendering to enhance depicted objects or image details rendered with the screen image display (104), albeit to a relatively limited extent. For example, the image rendering controller (110) may still modify the AR image for the depicted object or add portions of the one or more compensated images to the AR image for the purpose of causing the AR image display (102) to compensate for the depicted object in one or more visual characteristics, such as one or more of dynamic range, color gamut, spatial resolution, color saturation, color accuracy, etc.

The brain processes performed by the HVS may blend, mix or integrate AR and screen vision information, or image-rendered light generated from a compensated image rendering for depicted objects. Thus, even though the AR image display (102) renders the depicted object as black and white, the color or chromaticity information rendered from the screen image display (104) may be used by the brain process to allow the HVS, such as the viewer (112), to perceive the overall depicted object as a color object.

In addition, the compensation image rendering may be performed using any of one or more different color spaces, such as YCbCr, RGB, etc., using a combination of the AR image display (102) and the screen image display (104) to achieve optimal or best possible image rendering quality. In an example, the AR image may be represented as a screen image in the same color space (e.g., YCbCr, RGB, etc.). In another example, a color space (e.g., one of YCbCr and RGB, etc.) in which the AR image is represented may be different from a color space (the other of YCbCr and RGB in this example) in which the screen image is represented. For example, the compensated image rendering may be performed based on modified pixel values in a single color component/channel (e.g., luminance, chrominance, red, green, blue, etc.) or based on modified pixel values in a plurality of different color components/channels.

5. Dynamic image rendering transitions between image displays

The compensated image rendering may be performed dynamically and cooperatively by the AR image display (102) and the screen image display (104) in a time-varying manner for spatially moving 3D objects.

FIG. 3B illustrates an example compensated image rendering for a moving 3D object (e.g., 118-2). The screen image display (104) may be configured or controlled by the image rendering controller (110) to render 3D objects within the screen image space (196-2). The AR image display (102) may be configured or controlled by the image rendering controller (110) to render the 3D object within the AR image space (196-1). As shown, the screen image space (196-2) partially overlaps with the AR image space (196-1). In this partially overlapping image subspace (196-3) between the screen image space (196-2) and the AR image space (196-1), 3D objects may be rendered or depicted by both the AR image display (102) and the screen image display (104). The partially overlapping image subspaces (196-3) may be pre-specified or may be dynamically adapted by the image rendering controller (110). Additionally, alternatively or in addition, the partially overlapping image subspace (196-3) may be individually defined, determined, or set by the image rendering controller (110) for each 3D object using a plurality of determining factors including, but not limited to, the respective direction and velocity magnitudes of the 3D objects, the respective sizes of the 3D objects, whether any of these 3D objects are being gazed by the viewer (110), and the like. For illustration purposes only, the partially overlapping image subspace (196-3) is depicted with reference to the viewer (112) as being in front of the screen image display (104). It should be noted that in various embodiments, a partially overlapping image subspace as described herein may or may not be depicted as being in front of a screen image display (104).

As indicated by the start of the dashed arrow sequence in fig. 3B, the 3D object (118-2) may be depicted or rendered in the first screen image initially at the first spatial position at the first point in time. For illustration purposes only, the first spatial location is depicted behind the screen image display (104) with reference to the viewer (112). Since the first spatial location is outside of the partially overlapping image subspace (196-3) and entirely within the screen image space (196-2), the 3D object (118-2) is depicted or rendered with the first screen image at the first point in time. All user perceived visual characteristics of the 3D object (118-2) at the first spatial location-light brightness, chromaticity, saturation, hue, RGB, spatial resolution, image details, image refresh rate, etc. -are conveyed or generated by means of the first screen image rendered by the screen image display (104).

Subsequently, the 3D object (118-2) may be gradually or continuously moved to a second spatial position at a second point in time, thereby entering a partially overlapping image subspace (196-3) between the AR image space (196-1) and the screen image space (196-2). As shown in fig. 3B, the second spatial position is indicated by the end point of the first arrow in the sequence of dashed arrow points. Since the second spatial location is within the partially overlapping image subspace (196-3), the 3D object (118-2) is jointly and cooperatively depicted or rendered in the first AR image generated with the AR image display (102) and in the second screen image generated with the screen image display (104). The AR image display (102) and the screen image display (104) may be configured and controlled by the image rendering controller (110) to perform a compensated image rendering to generate and communicate all user-perceived visual characteristics of the 3D object (118-2) at the second spatial location-light brightness, chromaticity, saturation, hue, RGB, spatial resolution, image details, image refresh rate, and the like.

As indicated by the dashed arrow sequence in fig. 3B, the 3D object (118-2) may continue to move to a third spatial position at a third point in time, reaching near the end of the partially overlapping image subspace (196-3). The third spatial position is indicated by the end point of the second arrow in the sequence of dashed arrow points. Since the second spatial location remains within the partially overlapping image subspace (196-3), the 3D object (118-2) is jointly and cooperatively, continuously depicted or rendered in the first AR image generated with the AR image display (102) and in the second screen image generated with the screen image display (104). As at the second spatial location, the AR image display (102) and the screen image display (104) may be configured and controlled by the image rendering controller (110) to continuously perform the compensated image rendering to generate and communicate all user-perceived visual characteristics of the 3D object (118-2) at the third spatial location—light brightness, chromaticity, saturation, hue, RGB, spatial resolution, image details, image refresh rate, and the like.

Finally, as indicated by the end points of the dashed arrow sequence in fig. 3B, the 3D object (118-2) may continue to move to a fourth spatial position at a fourth point in time, exiting the partially overlapping image subspace (196-3) and entering into a non-overlapping image subspace of the AR image space (196-1), which is relatively close to the viewer (112). Since the fourth spatial location is outside of the partially overlapping image subspace (196-3) and entirely within the AR image space (196-1), the 3D object (118-2) is rendered or rendered with the second AR image at a fourth point in time. All user perceived visual characteristics of the 3D object (118-2) at the fourth spatial location-light brightness, chromaticity, saturation, hue, RGB, spatial resolution, image details, image refresh rate, etc. -are conveyed or generated by means of the second AR image rendered by the AR image display (102).

In the spatial trajectory formed by the motion of the 3D object (118-2), rendering the 3D object (118-2) in the overall image space formed by the AR image space (196-1) and the screen image space (196-2) is switching from one image display (or screen image display (104) in this example) to a different image display (or AR image display (102) in this example) in a gentle manner. For example, in a partially overlapping image subspace (196-3), both the AR image display (102) and the screen image display (104) may render the same object, such as the 3D object (118-2), in their respective AR and/or screen images, separately and simultaneously. The individually rendered objects in the respective AR and/or screen images may be integrated by the HVS or viewer (112) into a single unitary rendered object having the visual characteristics of a smooth transition.

In response to determining that the depicted object is at a spatial location closer to the non-overlapping image subspace of the AR image space (196-1), the AR contribution to the visual characteristics from the AR image may be set by the image rendering controller (110) to a relatively large weight factor value, while the screen contribution to the visual characteristics from the screen image may be set by the image rendering controller (110) to a relatively small weight factor value.

Conversely, in response to determining that the depicted object is at a spatial location closer to the non-overlapping image subspace of the screen image space (196-2), the AR contribution to the visual characteristics from the AR image may be set by the image rendering controller (110) to a relatively small weight factor value, while the screen contribution to the visual characteristics from the screen image may be set by the image rendering controller (110) to a relatively large weight factor value.

Under the compensated image rendering techniques as described herein, user-perceived visual characteristics of the 3D object (118-2), such as light brightness, chromaticity, saturation, hue, RGB, spatial resolution, image details, image refresh rate, etc., may be continuously and smoothly communicated or generated by the AR image display (102) and/or the screen image display (104) as the 3D object (118-2) moves along its spatial trajectory without visually noticeable discontinuities or discontinuities in these user-perceived visual characteristics. In an example, as a 3D object (118-2), such as a deer, moves toward a viewer (118-2), the 3D object (118-2) may be rendered as an object that becomes increasingly brighter. The light/brightness level of the 3D object (118-2) may be communicated or generated by the AR image display (102) and/or the screen image display (104) to increase continuously and smoothly without visually noticeable discontinuities or discontinuities in these user-perceived visual characteristics. In another example, as the 3D object (118-2) or deer moves toward the viewer (118-2), the 3D object (118-2) may be rendered with increasingly more image detail at progressively higher spatial resolutions. The spatial resolution or image details of the 3D object (118-2) may be communicated or generated by the AR image display (102) and/or the screen image display (104) to increase continuously and smoothly without visually noticeable discontinuities or discontinuities in these user-perceived visual characteristics. Thus, even though the AR image display (102) and the screen image display (104) may render AR and screen images depicting 3D objects in the respective AR and screen image spaces (196-1 and 196-2), the viewer (112) visually perceives these 3D objects as if rendered continuously and smoothly in a single unitary image space by a single unitary image display without visual artifacts, discontinuities or discontinuities.

6. Supplementary information display

As illustrated in fig. 1B and 3B, in some operational scenarios, AR image space (196-1) may be an image space that is at least partially in front of screen image space (196-2) with reference to viewer (112). In some operational scenarios, the reference viewer (112) AR image space (196-1) may form a different spatial relationship with the screen image space (196-2). Additionally, optionally or alternatively, the AR image space (196-1) may be divided into different AR image subspaces that may perform different types of compensated image rendering.

Fig. 3C illustrates an example image subspace in which different types of compensated image rendering may be performed between an AR image display (102) and a screen image display (104), for example, in an interactive image/video display application, such as a computer (video) game. As shown, the AR image space (196-1) is partitioned or divided into AR image subspaces 196-1-1, 196-1-2, and 196-1-3.

The AR image subspaces (196-1-1 and 196-1-3) may be used by the AR image display (102) to function as a heads-up display (HUD) in this application. The viewer may be a gamer focusing on gaming activity in a visual scene rendered in screen image space (196-2) on screen image display (104). The visual scene rendered by the screen image display (104) in the screen image space (196-2) may include 2D or 3D objects that may be viewed by a viewer (112) through the AR image subspace (196-1-2) with or without 3D glasses.

While the viewer (112) is playing or focusing on the video game, a first type of compensation image rendering that decorates or enhances visual characteristics of a 2D or 3D object rendered by the screen image display (104) may be performed by the AR image display (102) using the AR image subspace (196-1-2). Additionally, optionally or alternatively, while the viewer (112) is playing or focusing on a video game, the AR image display (102) may not directly perform a compensation image rendering for decorating or enhancing visual characteristics of the 2D or 3D object rendered by the screen image display (104).

In some operational scenarios, while a viewer (112) is playing or focusing on a video game, AR image subspaces (196-1-1 and 196-1-3) may be used by an AR image display (102) to act as a heads-up display (HUD) for displaying 2D or 3D objects such as supplemental game information. Supplemental game information, including but not limited to available ammunition, other player messages, etc., may be presented or rendered by the AR image display (102) in the AR image subspaces (196-1-1 and 196-1-3) without disturbing, obscuring, impeding or confusing the visual scene or primary game region (or key region of relative interest) rendered by the screen image display in the screen image space (196-2). Some or all of the supplemental game information may be collected or generated by a gaming device operating with the screen image display (104), received by or pushed to the image rendering controller (110) by the gaming device, and used by the image rendering controller (110) to generate or modify corresponding AR images to be simultaneously rendered in the AR image subspaces (196-1-1 and 196-1-3) with the AR image display (102). Thus, some or all of the supplemental game information may not be displayed or rendered by the screen image display (104) so as not to disrupt the visual scene depicting the ongoing gaming activity in the screen image space (196-2).

Additionally, optionally or alternatively, the non-game information may be collected or generated by a computing device operating with the screen image display (104), received by or pushed to the image rendering controller (110) by the computing device, and used by the image rendering controller (110) to generate or modify a corresponding AR image to be concurrently rendered in the AR image space (196-1) with the AR image display (102) along with the screen image rendered by the screen image display (104).

For example, in addition to or in lieu of a device that tracks the spatial position (192-1) and orientation (192-2) of the AR image display (102) or 3D glasses, the AR image display (102) may also be part of a user device that is configured to have eye tracking capabilities that track gaze or viewing directions of a viewer (112) at relatively low latency at multiple (consecutive) points in time intervals or user sessions (e.g., real-time, near real-time, within a strict latency budget, etc.).

Eye tracking information indicative of a gaze or viewing direction of a viewer (112) (e.g., real-time, near real-time, within a strict latency budget, etc.), as generated or collected by a user device including or operating with an AR image display (102), may be used to modify a screen image to be rendered by the screen image display (104).

For example, one or more of the screen image portions of the screen image corresponding to a foveal or focused field of view portion (e.g., real-time, near real-time, within a strict delay budget, etc.) of the viewer (112) may be streamed with relatively high quality video/image data and rendered on the screen image display (104) with relatively high dynamic range, relatively wide color gamut, relatively high spatial resolution, relatively high saturation, relatively accurate color accuracy, etc.

At the same time, non-foveal visual image portions of the screen image corresponding to non-foveal or non-focused field-of-view portions of the viewer (112) (e.g., real-time, near real-time, within a strict latency budget, etc.) may be streamed with relatively low quality video/image data and rendered on the screen image display (104) with relatively low dynamic range, relatively narrow color gamut, relatively low spatial resolution, relatively low saturation, relatively inaccurate color accuracy, etc.

Example eye tracking and foveal visual image rendering may be found in U.S. provisional patent application No. 62/699,583, entitled "FOVIATION AND HDR [ fovea and HDR ]" filed by Ajit Ninan at 7/17 of 2018, the entire contents of which are incorporated herein by reference as if fully set forth herein.

Additionally, optionally or alternatively, the non-eye tracking information may be collected or generated by a computing device operating with the AR image display (102), received by or pushed to the image rendering controller (110) by the computing device, and used by the image rendering controller (110) to generate or modify a corresponding screen image to be simultaneously rendered in the screen image space (196-2) with the screen image display (104).

7. Example image rendering System

FIG. 1C illustrates an example media system 100 that includes an image rendering controller 110, a screen image renderer 106, an AR image renderer 108, an AR image display 102, a screen display 104, a device tracker 122, a tracking sensor assembly 124, and the like. Some or all of the components/devices as depicted in fig. 1C may be implemented in software, hardware, a combination of software and hardware, etc., through one or more mechanical components, one or more electro-optical components, one or more computing devices, modules, units, etc. Some or all of the components/devices as depicted in fig. 1C may be communicatively coupled (e.g., wirelessly, through a wired connection, etc.) with some other components/devices as depicted in fig. 1C, or other components/devices not depicted in fig. 1C.

In some embodiments, the device tracker (122) in conjunction with the tracking sensor assembly (124) monitors the spatial position and orientation of the AR image display (102) or 3D glasses used by the viewer (112) to view the screen image rendered on the screen image display (104) at a relatively low latency at a plurality of (consecutive) points in time within a time interval or user session.

Image content 114 may be received, composed, or generated by an image rendering controller (110). In some operational scenarios, the image content (114) may include or represent a sequence of images (e.g., continuous, integral, input, raw, streamed, etc.) to be divided into a corresponding (continuous) AR image sequence and/or a corresponding (continuous) screen image sequence.

An image rendering controller (110) uses image content (114) to generate a screen image to be provided or streamed to a screen image display (104) for rendering. Additionally, optionally or alternatively, the image rendering controller (110) uses the image content (114) to generate an AR image to be provided or streamed to the AR image display (102) for rendering.

The screen display image may represent a 2D image, or a 3D/stereoscopic image including a left view image and a right view image. Likewise, the AR display image may represent a 2D image, or a 3D/stereoscopic image including a left view image and a right view image.

The image rendering controller (110) receives device tracking information from the device tracker (122) that indicates a spatial location (e.g., 192-1, etc.) and a spatial direction (e.g., 192-2, etc.) of the AR image display (102) or the 3D glasses over time (e.g., over a time interval, over the entire duration of a 3D movie, etc.). Additionally, optionally or alternatively, some or all of the tracking information or spatial location (e.g., 192-1, etc.) and spatial direction (e.g., 192-2, etc.) of the AR image display (102) or 3D glasses over time may be sent by the device tracker (122) or the image rendering controller (110) to one or both of: (a) A user device comprising an AR image renderer (108) and an AR image display (102); and (b) a computing device comprising a screen image renderer (106) and a screen image display (104). Additionally, optionally or alternatively, eye tracking information indicating gaze or viewing direction of the eyes of the viewer (112) over time may be collected by a user device comprising an AR image renderer (108) and an AR image display (102) and sent to an image rendering controller (110) or to a computing device comprising a screen image renderer (106) and a screen image display (104).

Based at least in part on the spatial position and spatial orientation of the AR image display (102) or the 3D glasses, the image rendering controller (110) or the AR image renderer (108) may determine a spatial relationship between the AR image display (102) (or the 3D glasses) and the screen image display (104) and use the spatial relationship to modify or adjust the AR image to be rendered by the AR image display (102). Additionally, optionally or alternatively, based at least in part on the spatial position and spatial orientation of the AR image display (102) or the 3D glasses, the image rendering controller (110) or the screen image renderer (106) may determine a spatial relationship between the AR image display (102) (or the 3D glasses) and the screen image display (104) and use the spatial relationship to modify or adjust the screen image to be rendered by the screen image display (104). Additionally, optionally or alternatively, based at least in part on the viewing direction of the viewer (112), the image rendering controller (110) or the screen image renderer (106) may modify or adjust the screen image to be rendered by the screen image display (104) to improve the rendering quality of the image portion corresponding to the foveal view portion of the viewer (112).

Examples of the screen image display (104) may be an image display in a movie theater, an image display in a home entertainment system, and the like. The screen display (104) may be stationary in the 3D physical space.

In some embodiments, the AR image display (102) may not be a physical display, but a virtual image display generated by light emitted by the imager(s) in the AR image display (102).

The media system (100) may be used to support real-time video applications, near-real-time video applications, non-real-time video applications, virtual Reality (VR) applications, augmented Reality (AR) applications, telepresence applications, automotive entertainment applications, head-mounted display applications, heads-up display applications, games, 2D display applications, 3D display applications, multi-view display applications, and the like. For example, some or all of the input image content (114) may be generated or accessed by the image rendering controller (110) in real-time, near real-time, non-real-time, or the like.

8. Example Process flow

FIG. 4 illustrates an example process flow. In some example embodiments, one or more computing devices or components may perform this process flow. In block 402, an image rendering system (e.g., an image rendering controller, a user device including a non-still image display such as an AR image display, a wearable device, a PC with a screen image display, a laptop with a screen image display, etc.) receives a first image for rendering on a first image display in a combination of the still image display and the non-still image display.

As used herein, a still image display may refer to one of the following: an external screen image display external to a user device coupled or attached to a user, an external screen image display external to a user device rendering an image but without a physical screen, a portable or attached image display with a physical screen, an image display that is relatively static with respect to or with reference to a 3D physical space, an image display that is relatively static and unaffected by head movement or body movement of a viewer, a more static image display among two or more image displays, and the like. The non-still image display may refer to one of the following: an internal screen image display within or attached to a portion of a user device of a user, an image display without a physical screen, a movable image display without a physical screen, an image display that is relatively non-stationary with respect to or with reference to a 3D physical space, an image display that is relatively dynamic and susceptible to head movement or body movement of a viewer, an image display that is less stationary among two or more image displays, etc.

In block 404, the image rendering system identifies a visual object depicted in the first image.

In block 406, the image rendering system generates a corresponding image portion in a second image for rendering on a second image display in a combination of the still image display and the non-still image display. The corresponding image portion in the second image as rendered on the second image display overlaps the visual object as depicted in the second image as rendered on the first image display in the field of view of the viewer to modify one or more visual characteristics of the visual object.

In block 408, the image rendering system causes the second image to be simultaneously rendered on the second image while the first image is being rendered on the second image display.

In an embodiment, the still image display represents a screen image display that is stationary in a physical 3D space in which the still image display and the non-still image display are located.

In an embodiment, the non-still image display is included in a device coupled to a user of the viewer.

In an embodiment, the gaze of the viewer is tracked in real time; the gaze of the viewer is used to identify the visual object in the first image.

In an embodiment, a viewing direction of a user device including a non-still image display is tracked in real time; the viewing direction of the user device is used to perform perspective correction adjustment on at least one of the first image or the second image.

In an embodiment, the viewing direction of the viewer is represented by one or both of a spatial position of the user device or a spatial direction of the user device.

In an embodiment, the first image represents a stereoscopic image; the first image depicts one or more 3D images comprising a visual object; a viewer visually perceives one or more 3D objects depicted in the first image through the 3D viewing device.

In an embodiment, the one or more visual characteristics include at least one of: dynamic range, peak light intensity, darkest black level, object background, color saturation, or another visual perception characteristic.

In an embodiment, the first image and the second image relate to one of: augmented reality applications, virtual reality applications, mixed reality applications, computer game applications, interactive video applications, non-interactive video applications, and the like.

In an embodiment, an overall image space in which the first image display and the second image display render the visual object is divided into a first image subspace, a second image subspace, and a third image subspace; the particular visual characteristics of any visual object in the first image subspace are fully rendered by the first image display; the particular visual characteristics of any visual object in the second image subspace are jointly rendered by the first image display and the second image display; the specific visual characteristics of any visual object in the third image subspace are fully rendered by the second image display.

In an embodiment, the first image display is a still image display; the visual object is a portion of a primary visual scene depicted in a sequence of images rendered on a still image display; the supplemental information is rendered by the non-still image display in an image space portion that does not interfere with the primary visual scene depicted in the sequence of images rendered on the still image display.

In an embodiment, eye tracking information is collected with a user device comprising a non-still image display; the viewer's gaze, as derived from the eye tracking information, is used to render one or more relatively high quality image portions corresponding to the viewer's foveal field of view portion and other relatively low quality image portions not corresponding to the foveal field of view portion on a still image display.

In an embodiment, a display system includes: a non-still image display rendering a non-screen display image; a still image display rendering a screen display image; an image rendering controller that performs at least a portion of the foregoing methods or operations.

In various example embodiments, an apparatus, system, device, or one or more other computing devices performs any of the foregoing methods or portions thereof as described. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause performance of a method as described herein.

Note that although individual embodiments are discussed herein, any combination of the embodiments and/or portions of the embodiments discussed herein can be combined to form further embodiments.

9. Implementation mechanism-hardware overview

According to one embodiment, the techniques described herein are implemented by one or more special purpose computing devices. The special purpose computing device may be hardwired for performing the techniques, or may include a digital electronic device, such as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs), that are permanently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques in accordance with program instructions in firmware, memory, other storage, or a combination. Such special purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement the techniques. The special purpose computing device may be a desktop computer system, portable computer system, handheld device, networking device, or any other device that incorporates hardwired and/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram illustrating a computer system 500 upon which an example embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled with bus 502 for processing information. The hardware processor 504 may be, for example, a general purpose microprocessor.

Computer system 500 also includes a main memory 506, such as a Random Access Memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to the processor 504, cause the computer system 500 to become a special-purpose machine that is customized to perform the operations specified in the instructions.

Computer system 500 further includes a Read Only Memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504.

A storage device 510, such as a magnetic disk or optical disk, solid state RAM, or the like, is provided and coupled to bus 502 for storing information and instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer viewer. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of viewer input device is cursor control device 516, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 504 and for controlling cursor movement on display 512. Typically, this input device has two degrees of freedom in two axes, a first axis (e.g., x-axis) and a second axis (e.g., y-axis), allowing the device to specify positions in a plane.

Computer system 500 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and/or program logic, in combination with the computer system, to make computer system 500 a or program a special purpose machine. According to one embodiment, the techniques herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

The term "storage medium" as used herein refers to any non-transitory medium that stores data and/or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 510. Volatile media includes dynamic memory, such as main memory 506. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a flash EPROM, an NVRAM, any other memory chip or cartridge.

Storage media are different from, but may be used in conjunction with, transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 500 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 502. Bus 502 carries the data to main memory 506, from which processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, communication interface 518 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 528. Local network 522 and internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received, and/or stored in storage device 510, or other non-volatile storage for later execution.

10. Equivalents, extensions, alternatives and others

In the foregoing specification, example embodiments of the application have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the application, and is the application, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Thus, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method, comprising:

Receiving a first image for rendering on a first image display in a combination of a still image display and a non-still image display;

Tracking a viewer's gaze in real time by tracking a viewing direction of a user device comprising the non-still image display;

Identifying a visual object depicted in the first image by analyzing the tracked viewing direction to determine a visual object gazed by the viewer;

Generating a corresponding image portion in a second image for rendering on a second image display in a combination of the still image display and the non-still image display, wherein the corresponding image portion in the second image as rendered on the second image display overlaps the visual object as depicted in the second image as rendered on the first image display in a field of view of a viewer to perform a compensated image rendering to enhance only the identified visual object in terms of one or more visual characteristics such as one or more of dynamic range, color gamut, spatial resolution, color saturation, and color accuracy;

The second image is rendered simultaneously on the second image display while the first image is being rendered on the first image display.

2. The method of claim 1, wherein the still image display represents a screen image display that is stationary in a physical 3D space in which the still image display and the non-still image display are located.

3. The method of claim 1 or 2, wherein the non-still image display is included in a device coupled to a user of the viewer.

4. The method of any of claims 1 to 4, wherein a viewing direction of the user device is used to perform perspective correction adjustment on at least one of the first image or the second image.

5. The method of any of claims 1 to 4, wherein the viewing direction of the viewer is represented by one or both of a spatial position of the user device or a spatial direction of the user device.

6. The method of any one of claims 1 to 5, wherein the first image represents a stereoscopic image; wherein the first image depicts one or more 3D images comprising the visual object; wherein the viewer visually perceives the one or more 3D objects depicted in the first image through a 3D viewing device.

7. The method of any of claims 1 to 6, wherein the first image and the second image relate to one of: augmented reality applications, virtual reality applications, mixed reality applications, computer game applications, interactive video applications, or non-interactive video applications.

8. The method of any of claims 1 to 7, wherein an overall image space in which the first and second image displays render visual objects is divided into a first image subspace, a second image subspace, and a third image subspace; wherein the particular visual characteristics of any visual object in the first image subspace are fully rendered by the first image display; wherein the particular visual characteristics of any visual object in the second image subspace are jointly rendered by the first image display and the second image display; wherein the specific visual characteristics of any visual object in the third image subspace are fully rendered by the second image display.

9. The method of any one of claims 1 to 8, wherein the first image display is the still image display; wherein the visual object is a portion of a primary visual scene depicted in a sequence of images rendered on the still image display; wherein supplemental information is rendered by the non-still image display in an image space portion that does not interfere with a primary visual scene depicted in the sequence of images rendered on the still image display.

10. The method of any of claims 1 to 9, wherein the viewer's gaze is used to render one or more relatively high quality image portions corresponding to the viewer's foveal field portion and other relatively low quality image portions not corresponding to the foveal field portion on the still image display.

11. A display system, comprising:

A non-still image display rendering a non-screen display image;

A still image display rendering a screen display image;

An image rendering controller performing at least a portion of the method of any one of claims 1 to 10.

12. A non-tangible computer-readable storage medium storing software instructions that, when executed by one or more computer processors, cause performance of at least a portion of the method of any one of claims 1 to 10.

13. An apparatus comprising one or more computer processors and one or more storage media storing a set of instructions that, when executed by the one or more computer processors, cause performance of at least a portion of the method of any of claims 1-10.