WO2022051931A1

WO2022051931A1 - Ar/vr image display method, ar/vr image display apparatus, and computer-program product

Info

Publication number: WO2022051931A1
Application number: PCT/CN2020/114194
Authority: WO
Inventors: Wenyu Li; Jinghua Miao; Lili Chen; Hao Zhang
Original assignee: Boe Technology Group Co., Ltd.; Beijing Boe Optoelectronics Technology Co., Ltd.
Priority date: 2020-09-09
Filing date: 2020-09-09
Publication date: 2022-03-17
Also published as: CN114531904A

Abstract

An Augmented Reality/Virtual Reality (AR/VR) image display method is provided. The AR/VR method includes determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

Description

AR/VR IMAGE DISPLAY METHOD, AR/VR IMAGE DISPLAY APPARATUS, AND COMPUTER-PROGRAM PRODUCT

TECHNICAL FIELD

The present invention relates to display technology, more particularly, to an Augmented Reality/Virtual Reality (AR/VR) image display method, an AR/VR image display apparatus, and a computer-program product.

BACKGROUND

With the continuous development of display technologies, ultra-high resolution small screens are increasingly favored by major manufacturers and various types of intelligent display products. In particular, ultra-high resolution image display are in demand for AR/VR image display.

SUMMARY

In one aspect, the present disclosure provides an Augmented Reality/Virtual Reality (AR/VR) image display method, comprising determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

Optionally, the method is for AR image display; the method further comprises determining a size and a location of a first virtual object to be displayed in a previous adjacent frame of image; determining a size and a location of a second virtual object to be displayed in a present frame of image, the previous adjacent frame of image and the present frame of image being two sequential frames of image; determining a first regular polygonal display area in which the first virtual object in the previous adjacent frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area; determining a second regular polygonal display area in which the second virtual object in the present frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area; determining a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area; and in the present frame of image, displaying a composite image in an area limited to the composite regular polygonal display area.

Optionally, the second regular polygonal display area is used as the reduced display area for determining the second field of view; the reduced image resolution is used as a present image resolution for the composite image in the present frame of image; and the intermediate image is rendered based on the present image resolution and the second field of view.

Optionally, displaying the composite image in the present frame of image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel.

Optionally, the composite image comprises the intermediate image in a first region of the composite regular polygonal display area, the first region corresponding to the second regular polygonal display area, and a background image in a second region of the composite regular polygonal display area, the second region being non-overlapping with the first region; the limited subset of subpixels corresponds to the composite regular polygonal display area; the data signals of the intermediate image are provided to subpixels corresponding to the first region; and background data signals are provided to subpixels corresponding to the second region.

Optionally, the limited subset of subpixels comprises subpixels having grayscales greater than a threshold value; and the limited subset of subpixels corresponds to a limited region of the composite regular polygonal display area.

Optionally, rendering the intermediate image comprises generating dummy data signals comprising information of the size and the location of a second virtual object, and the present image resolution; and the method further comprises transmitting the dummy data signals together with the data signals of the intermediate image to the integrated circuit; and caching the dummy data signals in the integrated circuit for a next adjacent frame of image.

Optionally, the first regular polygonal display area is a minimal polygonal display area in which the first virtual object in the previous adjacent frame of image can be displayed; the second regular polygonal display area is a minimal polygonal display area in which the second virtual object in the present frame of image can be displayed; and the composite regular polygonal display area is a minimal polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area.

Optionally, the first regular polygonal display area and the second regular polygonal display area overlap with each other.

Optionally, the first regular polygonal display area, the second regular polygonal display area, and the composite regular polygonal display area are of a same type of polygonal shape.

Optionally, the first regular polygonal display area is a first rectangular display area; the second regular polygonal display area is a second rectangular display area; and the composite regular polygonal display area is a composite rectangular display area.

Optionally, the method is for VR image display; the method further comprises displaying the intermediate image in the reduced display area; and displaying a background image in a region of the total display area outside of the reduced display area.

Optionally, displaying the intermediate image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel; the limited subset of subpixels corresponds to the reduced display area; displaying the background image comprises directly assigning, by the integrated circuit, a grayscale value to subpixels in the region of the total display area outside of the reduced display area.

Optionally, the reduced display area comprises a high-definition region and a low-definition region; and the limited subset of subpixels comprises all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region.

Optionally, the AR/VR image display method further comprises obtaining coordinates of a gaze point; based on the coordinates of a gaze point, rendering a high-definition image for displaying in the subpixels corresponding to the high-definition region; rendering a low-definition image for displaying in the alternate rows of subpixels of the subpixels corresponding to the low-definition region; processing the high-definition image and the low-definition image using an image processing algorithm to generated a processed high-definition image and a processed low-definition image; and transmitting the processed high-definition image and the processed low-definition image to the integrated circuit; wherein the intermediate image comprising the processed high-definition image and the processed low-definition image.

Optionally, the AR/VR image display method further comprises stretching, by the integrated circuit, the processed low-definition image to generate a stretched low-definition image; stitching, by the integrated circuit, the stretched low-definition image and the processed high-definition image together to generate a stitched image; and displaying the stitched image.

Optionally, the image processing algorithm is one selected from a group consisting of an image enhancing algorithm and a local dimming algorithm.

Optionally, rendering the intermediate image comprises generating dummy data signals comprising information of a size and a location of the reduced display area, and the reduced image resolution; and the method further comprises transmitting the dummy data signals together with the data signals of the intermediate image to the integrated circuit.

In another aspect, the present disclosure provides an Augmented Reality/Virtual Reality (AR/VR) image display apparatus, comprising a display panel; an integrated circuit connected to the display panel; a memory; and one or more processors; wherein the memory and the one or more processors are connected with each other; and the memory stores computer-executable instructions for controlling the one or more processors to determine a first field of view corresponding to a total display area; determine a second field of view corresponding to a reduced display area; obtain a screen resolution corresponding to the first field of view; determine a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the reduced image resolution and the second field of view; and transmit data signals of the intermediate image to an integrated circuit connected to a display panel.

In another aspect, the present disclosure provides a computer-program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

BRIEF DESCRIPTION OF THE FIGURES

The following drawings are merely examples for illustrative purposes according to various disclosed embodiments and are not intended to limit the scope of the present invention.

FIG. 1 shows Augmented Reality/Virtual Reality (AR/VR) image display method in some embodiments according to the present disclosure.

FIG. 2 illustrates a field of view in an AR/VR image display method in some embodiments according to the present disclosure.

FIG. 3 illustrates an AR image display method in some embodiments according to the present disclosure.

FIG. 4 illustrates a composite regular polygonal display area in some embodiments according to the present disclosure.

FIG. 5 illustrates an AR image display method in some embodiments according to the present disclosure.

FIG. 6 illustrates a composite image in some embodiments according to the present disclosure.

FIG. 7 illustrates a composite image in some embodiments according to the present disclosure.

FIG. 8 illustrates a VR image display method in some embodiments according to the present disclosure.

FIG. 9 illustrates a VR image display method in some embodiments according to the present disclosure.

FIG. 10 illustrates a VR image display method in some embodiments according to the present disclosure.

FIG. 11 illustrates an intermediate image and a background image in a VR image display method in some embodiments according to the present disclosure.

FIG. 12 illustrates a VR image display method in some embodiments according to the present disclosure.

FIG. 13 is a schematic diagram of a structure of an apparatus in some embodiments according to the present disclosure.

FIG. 14 is a schematic diagram illustrating the structure of an AR/VR image display apparatus in some embodiments according to the present disclosure.

DETAILED DESCRIPTION

The disclosure will now be described more specifically with reference to the following embodiments. It is to be noted that the following descriptions of some embodiments are presented herein for purpose of illustration and description only. It is not intended to be exhaustive or to be limited to the precise form disclosed.

The present disclosure provides, inter alia, an Augmented Reality/Virtual Reality (AR/VR) image display method, an AR/VR image display apparatus, and a computer-program product that substantially obviate one or more of the problems due to limitations and disadvantages of the related art. In one aspect, the present disclosure provides an AR/VR image display method. In some embodiments, the AR/VR image display method includes determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

FIG. 1 shows Augmented Reality/Virtual Reality (AR/VR) image display method in some embodiments according to the present disclosure. Referring to FIG. 1, the AR/VR image display method in some embodiments includes determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel. As used herein, the term ‘Augmented Reality’ (AR) is defined as a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) , by a computer. As used herein, the term ‘Virtual Reality’ or VR is defined as a computer-simulated environment that can simulate physical presence in places in the real world or imagined worlds. Virtual reality could recreate sensory experiences, including virtual taste, sight, smell, sound, touch, and the like. Many traditional VR systems use a near eye display for presenting a 3D virtual environment.

FIG. 2 illustrates a field of view in an AR/VR image display method in some embodiments according to the present disclosure. FIG. 2 shows a viewing frustum VF, or a visible region of three-dimensional space. A real world settings may involve a camera, whereas a virtual world setting may involve a viewpoint VP (often referred to as a virtual camera) . A field of view FOV is often discussed in association with the viewing frustum VF. In FIG. 2, a rectangular viewport is illustrated, thus the viewing frustum in FIG. 2 is a truncated four-sided (e.g., rectangular) pyramid. The viewport may have various other appropriate shapes such as a circular shape, and accordingly the viewing frustum VF may have other appropriate shapes such as a cone shape. The field of view FOV includes a vertical field of view VFOV and a horizontal field of view HFOV. The boundaries of the viewing frustum VF are defined by the vertical field of view VFOV, the horizontal field of view HFOV, a near limit, and a far limit. The near limit is defined by a near clip plane CPn of the viewing frustum VF, and the far limit is defined by a far clip plane CPf of the viewing frustum VF. The viewing frustum VF may be further defined by other parameters such as position and orientation. The viewing frustum VF in the virtual world generally corresponds with a real camera's field of view in the real world.

In some embodiments, the present method is used in an AR image display. FIG. 3 illustrates an AR image display method in some embodiments according to the present disclosure. Referring to FIG. 3, in some embodiments, the AR/VR image display method further includes determining a size and a location of a first virtual object to be displayed in a previous adjacent frame of image; determining a size and a location of a second virtual object to be displayed in a present frame of image, the previous adjacent frame of image and the present frame of image being two sequential frames of image; determining a first regular polygonal display area in which the first virtual object in the previous adjacent frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area; determining a second regular polygonal display area in which the second virtual object in the present frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area; determining a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area; and in the present frame of image, displaying a composite image in an area limited to the composite regular polygonal display area.

In some embodiments, the AR image display method includes displaying a plurality of frames of image successively. The plurality of frames of image includes the previous adjacent frame of image, the present frame of image, and a next adjacent frame of image. In one example, the AR image display method includes displaying N number of frames of images, N being an integer greater than 1. Optionally, the AR image display method includes determining a size and a location of a first virtual object to be displayed in an (n-1) th frame of image, n is an integer, and 1 < n < N; determining a size and a location of a second virtual object to be displayed in an (n) -th frame of image, the (n-1) -th frame of image and the (n) -th frame of image being two sequential frames of image; determining a first regular polygonal display area in which the first virtual object in the (n-1) -th frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area; determining a second regular polygonal display area in which the second virtual object in the (n) -th frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area; determining a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area; and in the (n) -th frame of image, displaying a composite image in an area limited to the composite regular polygonal display area.

In each frame of image (e.g., the present frame of image, the (n) -th frame of image, the previous adjacent frame of image, the (n-1) -th frame of image, the next adjacent frame of image, the (n+1) -th frame of image) , the AR image display method includes calculating a position of a virtual object (e.g., a three-dimensional virtual object) and a position of the viewpoint (e.g., a virtual camera) . Based on the position of the virtual object (e.g., the first virtual object in the previous adjacent frame of image, or the second virtual object in the present frame of image) , the method further includes calculating a size of the virtual object, and rendering the virtual object based on the size of the virtual object.

FIG. 4 illustrates a composite regular polygonal display area in some embodiments according to the present disclosure. Referring to FIG. 4, the method in some embodiments includes determining a first regular polygonal display area PDA1 in which the first virtual object VO1 in the previous adjacent frame of image is to be displayed, the first regular polygonal display area PDA1 is smaller than a total display area TDA; determining a second regular polygonal display area PDA2 in which the second virtual object VO2 in the present frame of image is to be displayed, the second regular polygonal display area PDA2 is smaller than the total display area TDA; determining a composite regular polygonal display area CPDA comprising the first regular polygonal display area PDA1 and the second regular polygonal display area PDA2, wherein the composite regular polygonal display area CPDA is smaller than the total display area TDA, equal to or greater than the first regular polygonal display area PDA1, and equal to or greater than the second regular polygonal display area PDA2. In the present frame of image, a composite image is displayed in an area limited to the composite regular polygonal display area CPDA.

In some embodiments, the first regular polygonal display area PDA1 is a minimal polygonal display area in which the first virtual object VO1 in the first frame of image can be displayed. The second regular polygonal display area PDA2 is a minimal polygonal display area in which the second virtual object VO2 in the second frame of image can be displayed. The composite regular polygonal display area CPDA is a minimal polygonal display area comprising the first regular polygonal display area PDA1 and the second regular polygonal display area PDA2.

In some embodiments, the first regular polygonal display area PDA1 is a minimal polygonal display area in which the first virtual object VO1 in the first frame of image can be displayed plus a first buffer region surrounding the minimal polygonal display area. The second regular polygonal display area PDA2 is a minimal polygonal display area in which the second virtual object VO2 in the second frame of image can be displayed plus a second buffer region surrounding the minimal polygonal display area. The composite regular polygonal display area CPDA is a polygonal display area comprising the first regular polygonal display area PDA1 and the second regular polygonal display area PDA2, with or without an additional buffer region.

Optionally, the first regular polygonal display area PDA1 and the second regular polygonal display area PDA2 overlap with each other.

Optionally, the first regular polygonal display area PDA1 and the second regular polygonal display area PDA2 are substantially non-overlapping with each other.

In some embodiments, the first regular polygonal display area PDA1, the second regular polygonal display area PDA2, and the composite regular polygonal display area CPDA are of a same type of polygonal shape. In one example, as shown in FIG. 4, the first regular polygonal display area PDA1 is a first rectangular display area; the second regular polygonal display area PDA2 is a second rectangular display area; and the composite regular polygonal display area CPDA is a composite rectangular display area.

FIG. 5 illustrates an AR image display method in some embodiments according to the present disclosure. Referring to FIG. 5, in some embodiments, prior to displaying the composite image in the present frame of image, the AR image display method further includes determining a first field of view corresponding to the total display area; determining a second field of view corresponding to the second regular polygonal display area; obtaining a screen resolution corresponding to the first field of view; determining a present image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the present image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel for displaying the composite image. In some embodiments, the second regular polygonal display area is used as the reduced display area for determining the second field of view. The reduced image resolution is used as the present image resolution for the composite image in the present frame of image. The intermediate image is rendered based on the present image resolution and the second field of view.

In the present AR image display method, rather than rendering an image based on the first field of view corresponding to the total display area of the display panel, only the intermediate image based on the present image resolution and the second field of view corresponding to the second regular polygonal display area is rendered. In the present AR image display method, computation resources required for rendering an image for displaying the virtual object can be much reduced. Moreover, transmission bandwidth required for transmitting the rendered image to the integrated circuit of the display panel can be significantly reduced. The inventors of the present disclosure discovers that AR image display can be dramatically improved by the present method, particularly for a high-resolution AR image display.

In some embodiments, the present image resolution (e.g., the reduced image resolution) is determined based on the first field of view, the second field of view, and the screen resolution. In one example, the reduced image resolution is calculated based on Equation (1) and Equation (2) :

res_me. x = res_1. x *fov_me. x /fov_1. x (1) ; and

res_me. y = res_1. y *fov_me. x /fov_1. y (2) ;

wherein res_me. x stands for a reduced horizontal image resolution for the present frame of image, res_me. x stands for a reduced vertical image resolution for the present frame of image, fov_me. x stands for a horizontal field of view of the second field of view, fov_me. y stands for a vertical field of view of the second field of view, fov_1. x stands for a horizontal field view of the first field of view, and fov_1. y stands for a vertical field of view of the first field of view.

In some embodiments, when rending the intermediate image, the method further includes generating dummy data signals comprising information of the size and the location of a second virtual object, and the present image resolution. The dummy data signals are transmitted, together with the data signals of the intermediate image, to the integrated circuit of the display panel. In one example, the intermediate image includes data signals corresponding to a row of dummy subpixels for containing the information of the size and the location of a second virtual object, and the present image resolution. The dummy data signals for the present frame of image are cached in the integrated circuit, these information are then used for displaying image in a next adjacent frame of image (e.g., a (n+1) -th frame of image) . As discussed above, the amount of computation resources and transmission bandwidth required for the present AR image display method are much reduced.

In some embodiments, displaying the composite image in the present frame of image includes providing gate scanning signals and data signals limited to a limited subset of subpixels of the plurality of subpixels of the display panel. In subpixels other than the limited subset of subpixels, the method does not involve scanning the subpixels. For example, only in the limited subset of subpixels, gate scanning signals are provided to scan the subpixels (turning on or off transistors therein) row by row to allow input of data signals. Because only the limited subset of subpixels are scanned in the present AR image display method, the refresh rate of the display panel can be much enhanced, achieving a superior AR display effect.

FIG. 6 illustrates a composite image in some embodiments according to the present disclosure. Referring to FIG. 6, the composite image in some embodiments includes the intermediate image in a first region R1 of the composite regular polygonal display area CPDA, the first region R1 corresponding to the first regular polygonal display area PDA2, and a background image in a second region R2 of the composite regular polygonal display area CPDA, the second region R2 being non-overlapping with the first region R1. In one example, a dark image is displayed in the second region R2, the subpixels corresponding to the second region R2 have a grayscale value equal to or close to zero. Images of various other colors may be displayed in the second region R2, for example, a gray image, a blue image, a red image, and so on. Optionally, all subpixels corresponding to the second region R2 have a substantially uniform grayscale. By displaying a background image in the second region R2, the previous adjacent frame of image can be reset.

In some embodiments, the limited subset of subpixels that are scanned corresponds to the composite regular polygonal display area CPDA. The data signals of the intermediate image are provided to subpixels corresponding to the first region R1. Optionally, background data signals (e.g., data signals of a substantially uniform grayscale) are provided to subpixels corresponding to the second region R2.

FIG. 7 illustrates a composite image in some embodiments according to the present disclosure. Referring to FIG. 7, in some embodiments, only subpixels in a white line sub-region of the composite regular polygonal display area CPDA are scanned, and subpixels other than the subpixels in the white line sub-region are not scanned (e.g., not receiving gate scanning signals to turning on or off transistors therein) . Accordingly, the AR image display method in some embodiments includes displaying the composite image in the present frame of image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of the plurality of subpixels of the display panel, wherein the limited subset of subpixels includes subpixels having grayscales greater than a threshold value (e.g., greater than 5, greater than 10, greater than 15, or greater than 20) . The limited subset of subpixels corresponds to a limited region RL of the composite regular polygonal display area CPDA. By having this more accurate subpixel scanning control, refresh rate of the display panel can be further improved and power consumption lowered.

In some embodiments, the present method is used in a VR image display. Referring to FIG. 1, the VR image display in some embodiments includes determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

FIG. 8 illustrates a VR image display method in some embodiments according to the present disclosure. Referring to FIG. 8, the VR image display method in some embodiments includes obtaining coordinates of a gaze point (e.g., by a gaze tracking device) ; based on the coordinates of a gaze point, rendering (e.g., by a processor) a high-definition image for displaying in the subpixels corresponding to the high-definition region; rendering (e.g., by a processor) a low-definition image for displaying in the alternate rows of subpixels of the subpixels corresponding to the low-definition region; processing (e.g., by a processor) the high-definition image and the low-definition image using an image processing algorithm to generated a processed high-definition image and a processed low-definition image; and transmitting the processed high-definition image and the processed low-definition image to the integrated circuit. The intermediate image includes the processed high-definition image and the processed low-definition image. The integrated circuit, upon receiving the intermediate image, is configured to stretch the processed low-definition image to generate a stretched low-definition image; and stitch the stretched low-definition image and the processed high-definition image together to generate a stitched image. The display panel is configured to display the stitched image. Various appropriate image processing algorithms may be used for processing the high-definition image and the low-definition image. Examples of image processing algorithms include an image enhancing algorithm and a local dimming algorithm.

FIG. 9 illustrates a VR image display method in some embodiments according to the present disclosure. Referring to FIG. 9, in some embodiments, once the coordinates of the gaze point are obtained, the high-definition image and the low-definition image can be rendered. In one example, densities of subpixels configured to emit light respectively in the high-definition image and the low-definition image are substantially the same. However, the low-definition image in a VR image display is not the focus of the VR experience. Thus, computation resources and transmission bandwidth are wasted by configured subpixels in the low-definition image to emit light to have the same density as those in the high-definition image.

FIG. 10 illustrates a VR image display method in some embodiments according to the present disclosure. Referring to FIG. 10, to save computation resources and transmission bandwidth, only alternate rows of subpixels of subpixels in the low-definition image are scanned (e.g., are provided with gate scanning signals and data signals) . Accordingly, in some embodiments, displaying the intermediate image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of the plurality of subpixels of the display panel. As shown in FIG. 10, the limited subset of subpixels include all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region. Other rows of subpixels in the low-definition region are not scanned (e.g., are not provided with gate scanning signals and data signals) . By scanning less (e.g., almost half) of the subpixels in the low-definition region, refresh rate of the display panel can be much enhanced.

FIG. 11 illustrates an intermediate image and a background image in a VR image display method in some embodiments according to the present disclosure. Referring to FIG. 11, to further reduce the computation resources and transmission bandwidth demand in the VR image display, in some embodiments, the VR image display method includes displaying the intermediate image IM in a reduced display area; and displaying a background image BI in a region of the total display area outside of the reduced display area. In one example, the VR image display as illustrated in FIG. 11 is a VR image display in a cinema mode.

FIG. 12 illustrates a VR image display method in some embodiments according to the present disclosure. Referring to FIG. 12, in the cinema mode, the method includes displaying the intermediate image in the reduced display area (encircled by dotted lines in FIG. 12) ; and displaying a background image in a region of the total display area outside of the reduced display area. In some embodiments, displaying the intermediate image includes providing gate scanning signals and data signals limited to a limited subset of subpixels of the plurality of subpixels of the display panel. Optionally, the limited subset of subpixels corresponds to the reduced display area. Optionally, displaying the background image comprises directly assigning, by the integrated circuit, a grayscale value to subpixels in the region of the total display area outside of the reduced display area. In one example, a grayscale value equal to or close to zero is assigned to the subpixels in the region of the total display area outside of the reduced display area, forming the background image. In another example, images of various other colors may be displayed in the region of the total display area outside of the reduced display area, for example, a gray image, a blue image, a red image, and so on. Optionally, all subpixels corresponding to the region of the total display area outside of the reduced display area are assigned a substantially uniform grayscale value. By displaying a background image in the region of the total display area outside of the reduced display area, the cinema mode can be achieved.

Referring to FIG. 12 again, in some embodiments, the reduced display area includes a high-definition region and a low-definition region. To save computation resources and transmission bandwidth, only alternate rows of subpixels of subpixels in the low-definition region are scanned (e.g., are provided with gate scanning signals and data signals) . Accordingly, in some embodiments, displaying the intermediate image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of the plurality of subpixels corresponding to the reduced display area. As shown in FIG. 12, the limited subset of subpixels include all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region. Other rows of subpixels in the low-definition region are not scanned (e.g., are not provided with gate scanning signals and data signals) . By scanning less (e.g., almost half) of the subpixels in the low-definition region, refresh rate of the display panel can be much enhanced.

In another aspect, the present disclosure provides an AR/VR image display apparatus. FIG. 13 is a schematic diagram of a structure of an apparatus in some embodiments according to the present disclosure. Referring to FIG. 13, in some embodiments, the apparatus includes the central processing unit (CPU) configured to perform actions according to the computer-executable instructions stored in a ROM or in a RAM. Optionally, data and programs required for a computer system are stored in RAM. Optionally, the CPU, the ROM, and the RAM are electrically connected to each other via bus. Optionally, an input/output interface is electrically connected to the bus.

FIG. 14 is a schematic diagram illustrating the structure of an AR/VR image display apparatus in some embodiments according to the present disclosure. Referring to FIG. 14, in some embodiments, the AR/VR image display apparatus includes a display panel DP; an integrated circuit IC connected to the display panel DP; a memory M; and one or more processors P. The memory M and the one or more processors P are connected with each other. In some embodiments, the memory M stores computer-executable instructions for controlling the one or more processors P to determine a first field of view corresponding to a total display area; determine a second field of view corresponding to a reduced display area; obtain a screen resolution corresponding to the first field of view; determine a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the reduced image resolution and the second field of view; and transmit data signals of the intermediate image to an integrated circuit connected to a display panel DP.

In some embodiments, the AR/VR image display apparatus is an AR image display apparatus. The memory M stores computer-executable instructions for controlling the one or more processors P to determine a size and a location of a first virtual object to be displayed in a previous adjacent frame of image; determine a size and a location of a second virtual object to be displayed in a present frame of image, the previous adjacent frame of image and the present frame of image being two sequential frames of image; determine a first regular polygonal display area in which the first virtual object in the previous adjacent frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area; determine a second regular polygonal display area in which the second virtual object in the present frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area; and determine a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area. The display panel DP is configured to display, in the present frame of image, a composite image in an area limited to the composite regular polygonal display area.

In some embodiments, the memory M further stores computer-executable instructions for controlling the one or more processors P to determine a first field of view corresponding to the total display area; determining a second field of view corresponding to the second regular polygonal display area; obtaining a screen resolution corresponding to the first field of view; determine a present image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the present image resolution and the second field of view; and transmit data signals of the intermediate image to an integrated circuit connected to a display panel for displaying the composite image. In some embodiments, the second regular polygonal display area is used as the reduced display area for determining the second field of view. The reduced image resolution is used as the present image resolution for the composite image in the present frame of image. The intermediate image is rendered based on the present image resolution and the second field of view.

In some embodiments, the integrated circuit IC is configured to provide gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel DP. The composite image includes the intermediate image in a first region of the composite regular polygonal display area, the first region corresponding to the second regular polygonal display area, and a background image in a second region of the composite regular polygonal display area, the second region being non-overlapping with the first region. The limited subset of subpixels corresponds to the composite regular polygonal display area. Optionally, the integrated circuit IC is configured to provide the data signals of the intermediate image to subpixels corresponding to the first region; provide background data signals are provided to subpixels corresponding to the second region.

In some embodiments, the integrated circuit IC is configured to only provide gate scanning signals and data signals limited to subpixels having grayscales greater than a threshold value. Thus, the limited subset of subpixels includes subpixels having grayscales greater than a threshold value, and the limited subset of subpixels corresponds to a limited region of the composite regular polygonal display area.

In some embodiments, the memory M further stores computer-executable instructions for controlling the one or more processors P to generate dummy data signals including information of the size and the location of a second virtual object, and the present image resolution, when rendering the intermediate image; and transmit the dummy data signals together with the data signals of the intermediate image to the integrated circuit. The integrated circuit IC is configured to cache the dummy data signals in the integrated circuit for a next adjacent frame of image.

In some embodiments, the first regular polygonal display area is a minimal polygonal display area in which the first virtual object in the previous adjacent frame of image can be displayed; the second regular polygonal display area is a minimal polygonal display area in which the second virtual object in the present frame of image can be displayed; and the composite regular polygonal display area is a minimal polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area. Optionally, the first regular polygonal display area and the second regular polygonal display area overlap with each other.

In some embodiments, the first regular polygonal display area, the second regular polygonal display area, and the composite regular polygonal display area are of a same type of polygonal shape. Optionally, the first regular polygonal display area is a first rectangular display area; the second regular polygonal display area is a second rectangular display area; and the composite regular polygonal display area is a composite rectangular display area.

In some embodiments, the AR/VR image display apparatus is a VR image display apparatus. The display panel DP is configured to display the intermediate image in the reduced display area; and display a background image in a region of the total display area outside of the reduced display area.

In some embodiments, the integrated circuit IC is configured to provide gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel. The limited subset of subpixels corresponds to the reduced display area. the integrated circuit IC is further configured to directly assign a grayscale value to subpixels in the region of the total display area outside of the reduced display area, thereby displaying the background image.

In some embodiments, the reduced display area includes a high-definition region and a low-definition region. The limited subset of subpixels includes all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region. The integrated circuit IC is configured to provide gate scanning signals and data signals limited to all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region.

In some embodiments, the memory M further stores computer-executable instructions for controlling the one or more processors P to obtain coordinates of a gaze point; based on the coordinates of a gaze point, render a high-definition image for displaying in the subpixels corresponding to the high-definition region; render a low-definition image for displaying in the alternate rows of subpixels of the subpixels corresponding to the low-definition region; process the high-definition image and the low-definition image using an image processing algorithm to generated a processed high-definition image and a processed low-definition image; and transmit the processed high-definition image and the processed low-definition image to the integrated circuit. The intermediate image comprising the processed high-definition image and the processed low-definition image. Optionally, the image processing algorithm is one selected from a group consisting of an image enhancing algorithm and a local dimming algorithm.

In some embodiments, the VR image display apparatus further includes a gaze tracker for tracking a gaze of a user.

In some embodiments, the integrated circuit IC is further configured to stretch the processed low-definition image to generate a stretched low-definition image; and stitch the stretched low-definition image and the processed high-definition image together to generate a stitched image. The display panel DP is configured to display the stitched image.

In some embodiments, the memory M further stores computer-executable instructions for controlling the one or more processors P to generate dummy data signals including information of a size and a location of the reduced display area, and the reduced image resolution, when rendering the intermediate image; and transmit the dummy data signals together with the data signals of the intermediate image to the integrated circuit IC.

In another aspect, the present disclosure provides a computer-program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions being executable by a processor to cause the processor to perform determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.

In some embodiments, the computer-readable instructions being executable by a processor to cause the processor to perform determining a size and a location of a first virtual object to be displayed in a previous adjacent frame of image; determining a size and a location of a second virtual object to be displayed in a present frame of image, the previous adjacent frame of image and the present frame of image being two sequential frames of image; determining a first regular polygonal display area in which the first virtual object in the previous adjacent frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area; determining a second regular polygonal display area in which the second virtual object in the present frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area; and determining a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area.

In some embodiments, the second regular polygonal display area is used as the reduced display area for determining the second field of view; the reduced image resolution is used as a present image resolution for the composite image in the present frame of image; and the intermediate image is rendered based on the present image resolution and the second field of view.

In some embodiments, the computer-readable instructions being executable by a processor to cause the processor to perform generating dummy data signals including information of the size and the location of a second virtual object, and the present image resolution, when rendering the intermediate image.

In some embodiments, the first regular polygonal display area is a minimal polygonal display area in which the first virtual object in the previous adjacent frame of image can be displayed; the second regular polygonal display area is a minimal polygonal display area in which the second virtual object in the present frame of image can be displayed; and the composite regular polygonal display area is a minimal polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area. Optionally, the first regular polygonal display area and the second regular polygonal display area overlap with each other. Optionally, the first regular polygonal display area, the second regular polygonal display area, and the composite regular polygonal display area are of a same type of polygonal shape. Optionally, the first regular polygonal display area is a first rectangular display area; the second regular polygonal display area is a second rectangular display area; and the composite regular polygonal display area is a composite rectangular display area.

In some embodiments, the computer-readable instructions being executable by a processor to cause the processor to perform, based on coordinates of a gaze point, rendering a high-definition image for displaying in the subpixels corresponding to the high-definition region; rendering a low-definition image for displaying in the alternate rows of subpixels of the subpixels corresponding to the low-definition region; and processing the high-definition image and the low-definition image using an image processing algorithm to generated a processed high-definition image and a processed low-definition image. The intermediate image comprising the processed high-definition image and the processed low-definition image. Optionally, the image processing algorithm is one selected from a group consisting of an image enhancing algorithm and a local dimming algorithm.

In some embodiments, the computer-readable instructions being executable by a processor to cause the processor to perform generating dummy data signals comprising a size and a location of the reduced display area, and the reduced image resolution, when rendering the intermediate image.

The foregoing description of the embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form or to exemplary embodiments disclosed. Accordingly, the foregoing description should be regarded as illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to practitioners skilled in this art. The embodiments are chosen and described in order to explain the principles of the invention and its best mode practical application, thereby to enable persons skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use or implementation contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents in which all terms are meant in their broadest reasonable sense unless otherwise indicated. Therefore, the term “the invention” , “the present invention” or the like does not necessarily limit the claim scope to a specific embodiment, and the reference to exemplary embodiments of the invention does not imply a limitation on the invention, and no such limitation is to be inferred. The invention is limited only by the spirit and scope of the appended claims. Moreover, these claims may refer to use “first” , “second” , etc. following with noun or element. Such terms should be understood as a nomenclature and should not be construed as giving the limitation on the number of the elements modified by such nomenclature unless specific number has been given. Any advantages and benefits described may not apply to all embodiments of the invention. It should be appreciated that variations may be made in the embodiments described by persons skilled in the art without departing from the scope of the present invention as defined by the following claims. Moreover, no element and component in the present disclosure is intended to be dedicated to the public regardless of whether the element or component is explicitly recited in the following claims.

Claims

An Augmented Reality/Virtual Reality (AR/VR) image display method, comprising:

determining a first field of view corresponding to a total display area;

determining a second field of view corresponding to a reduced display area;

obtaining a screen resolution corresponding to the first field of view;

determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution;

rendering an intermediate image based on the reduced image resolution and the second field of view; and

transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.
The AR/VR image display method of claim 1, wherein the method is for AR image display;

the method further comprises:

determining a size and a location of a first virtual object to be displayed in a previous adjacent frame of image;

determining a size and a location of a second virtual object to be displayed in a present frame of image, the previous adjacent frame of image and the present frame of image being two sequential frames of image;

determining a first regular polygonal display area in which the first virtual object in the previous adjacent frame of image is to be displayed, the first regular polygonal display area is smaller than a total display area;

determining a second regular polygonal display area in which the second virtual object in the present frame of image is to be displayed, the second regular polygonal display area is smaller than the total display area;

determining a composite regular polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area, wherein the composite regular polygonal display area is smaller than the total display area, equal to or greater than the first regular polygonal display area, and equal to or greater than the second regular polygonal display area; and

in the present frame of image, displaying a composite image in an area limited to the composite regular polygonal display area.
The AR/VR image display method of claim 2, wherein the second regular polygonal display area is used as the reduced display area for determining the second field of view;

the reduced image resolution is used as a present image resolution for the composite image in the present frame of image; and

the intermediate image is rendered based on the present image resolution and the second field of view.
The AR/VR image display method of claim 3, wherein displaying the composite image in the present frame of image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel.
The AR/VR image display method of claim 4, wherein the composite image comprises the intermediate image in a first region of the composite regular polygonal display area, the first region corresponding to the second regular polygonal display area, and a background image in a second region of the composite regular polygonal display area, the second region being non-overlapping with the first region;

the limited subset of subpixels corresponds to the composite regular polygonal display area;

the data signals of the intermediate image are provided to subpixels corresponding to the first region; and

background data signals are provided to subpixels corresponding to the second region.
The AR/VR image display method of claim 4, wherein the limited subset of subpixels comprises subpixels having grayscales greater than a threshold value; and

the limited subset of subpixels corresponds to a limited region of the composite regular polygonal display area.
The AR/VR image display method of any one of claims 3 to 6, wherein rendering the intermediate image comprises generating dummy data signals comprising information of the size and the location of a second virtual object, and the present image resolution; and

the method further comprises transmitting the dummy data signals together with the data signals of the intermediate image to the integrated circuit; and

caching the dummy data signals in the integrated circuit for a next adjacent frame of image.
The AR/VR image display method of any one of claims 2 to 7, wherein the first regular polygonal display area is a minimal polygonal display area in which the first virtual object in the previous adjacent frame of image can be displayed;

the second regular polygonal display area is a minimal polygonal display area in which the second virtual object in the present frame of image can be displayed; and

the composite regular polygonal display area is a minimal polygonal display area comprising the first regular polygonal display area and the second regular polygonal display area.
The AR/VR image display method of claim 8, wherein the first regular polygonal display area and the second regular polygonal display area overlap with each other.
The AR/VR image display method of any one of claims 2 to 9, wherein the first regular polygonal display area, the second regular polygonal display area, and the composite regular polygonal display area are of a same type of polygonal shape.
The AR/VR image display method of claim 10, wherein the first regular polygonal display area is a first rectangular display area;

the second regular polygonal display area is a second rectangular display area; and

the composite regular polygonal display area is a composite rectangular display area.
The AR/VR image display method of claim 1, wherein the method is for VR image display;

the method further comprises:

displaying the intermediate image in the reduced display area; and

displaying a background image in a region of the total display area outside of the reduced display area.
The AR/VR image display method of claim 12, wherein displaying the intermediate image comprises providing gate scanning signals and data signals limited to a limited subset of subpixels of a plurality of subpixels of the display panel;

the limited subset of subpixels corresponds to the reduced display area;

displaying the background image comprises directly assigning, by the integrated circuit, a grayscale value to subpixels in the region of the total display area outside of the reduced display area.
The AR/VR image display method of claim 13, wherein the reduced display area comprises a high-definition region and a low-definition region; and

the limited subset of subpixels comprises all subpixels corresponding to the high-definition region and subpixels corresponding to only alternate rows of subpixels of subpixels corresponding to the low-definition region.
The AR/VR image display method of claim 14, further comprising:

obtaining coordinates of a gaze point;

based on the coordinates of a gaze point, rendering a high-definition image for displaying in the subpixels corresponding to the high-definition region;

rendering a low-definition image for displaying in the alternate rows of subpixels of the subpixels corresponding to the low-definition region;

processing the high-definition image and the low-definition image using an image processing algorithm to generated a processed high-definition image and a processed low-definition image; and

transmitting the processed high-definition image and the processed low-definition image to the integrated circuit;

wherein the intermediate image comprising the processed high-definition image and the processed low-definition image.
The AR/VR image display method of claim 15, further comprising:

stretching, by the integrated circuit, the processed low-definition image to generate a stretched low-definition image;

stitching, by the integrated circuit, the stretched low-definition image and the processed high-definition image together to generate a stitched image; and

displaying the stitched image.
The AR/VR image display method of claim 15, wherein the image processing algorithm is one selected from a group consisting of an image enhancing algorithm and a local dimming algorithm.
The AR/VR image display method of any one of claims 12 to 17, wherein rendering the intermediate image comprises generating dummy data signals comprising information of a size and a location of the reduced display area, and the reduced image resolution; and

the method further comprises transmitting the dummy data signals together with the data signals of the intermediate image to the integrated circuit.
An Augmented Reality/Virtual Reality (AR/VR) image display apparatus, comprising:

a display panel;

an integrated circuit connected to the display panel;

a memory; and

one or more processors;

wherein the memory and the one or more processors are connected with each other; and

the memory stores computer-executable instructions for controlling the one or more processors to:

determine a first field of view corresponding to a total display area;

determine a second field of view corresponding to a reduced display area;

obtain a screen resolution corresponding to the first field of view;

determine a reduced image resolution based on the first field of view, the second field of view, and the screen resolution;

render an intermediate image based on the reduced image resolution and the second field of view; and

transmit data signals of the intermediate image to an integrated circuit connected to a display panel.
A computer-program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform:

determining a first field of view corresponding to a total display area;

determining a second field of view corresponding to a reduced display area;

obtaining a screen resolution corresponding to the first field of view;

determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution;

rendering an intermediate image based on the reduced image resolution and the second field of view; and

transmitting data signals of the intermediate image to an integrated circuit connected to a display panel.