CN109714583B - Augmented reality display method and augmented reality display system - Google Patents

Augmented reality display method and augmented reality display system Download PDF

Info

Publication number
CN109714583B
CN109714583B CN201910058904.2A CN201910058904A CN109714583B CN 109714583 B CN109714583 B CN 109714583B CN 201910058904 A CN201910058904 A CN 201910058904A CN 109714583 B CN109714583 B CN 109714583B
Authority
CN
China
Prior art keywords
image
target sub
display
area
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910058904.2A
Other languages
Chinese (zh)
Other versions
CN109714583A (en
Inventor
陈丽莉
张�浩
陈明
董学
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Original Assignee
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BOE Technology Group Co Ltd, Beijing BOE Optoelectronics Technology Co Ltd filed Critical BOE Technology Group Co Ltd
Priority to CN201910058904.2A priority Critical patent/CN109714583B/en
Publication of CN109714583A publication Critical patent/CN109714583A/en
Priority to US16/745,412 priority patent/US11232602B2/en
Application granted granted Critical
Publication of CN109714583B publication Critical patent/CN109714583B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/38—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory with means for controlling the display position
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/06—Adjustment of display parameters
    • G09G2320/0686—Adjustment of display parameters with two or more screen areas displaying information with different brightness or colours
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0464—Positioning
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels

Abstract

The invention provides an augmented reality display method and an augmented reality display system, relates to the field of augmented reality, and aims to reduce data transmission bandwidth and display power consumption and further improve the display quality of an AR system. The augmented reality display method comprises the following steps: generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: a background and a target sub-image, the content of the target sub-image being associated with the target object; outputting image information and position information of the target sub-image to a display unit; displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information; and the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image. Display for augmented reality display systems.

Description

Augmented reality display method and augmented reality display system
Technical Field
The present invention relates to the field of augmented reality, and in particular, to an augmented reality display method and an augmented reality display system.
Background
The AR (Augmented Reality) technology is a technology that real environment information is acquired by photographing, and then a virtual projection object (such as an image, a scene or system prompt information) is superimposed on the acquired real environment information and displayed to a user, so that the user can see a visual effect that the virtual object exists in a real environment in sense, that is, "Augmented" to the real scene is realized.
For example, the augmented reality process includes: the camera acquires an image of a real scene, the processor generates a whole image for projection according to the image, transmits the whole image to the micro display screen, and projects the whole image into a visual field range of a user through the processing of the optical system so as to be overlapped with the real scene.
However, since only a part of the whole image transmitted by the processor is usually the region associated with the target object in the real scene (hereinafter referred to as the additional image), and the rest of the region except the additional image is the background, the whole image needs to be transmitted and displayed during transmission and display, which results in a large data processing amount and transmission amount, thereby increasing the data transmission bandwidth and display power consumption, and further affecting the display quality of the AR system.
Disclosure of Invention
In view of this, to solve the problems in the prior art, embodiments of the present invention provide an augmented reality display method and an augmented reality display system, which can reduce data transmission bandwidth and display power consumption, thereby improving display quality of an AR system.
In order to achieve the above purpose, the embodiment of the invention adopts the following technical scheme:
an embodiment of the present invention provides an augmented reality display method, where the display method includes: generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: a background and a target sub-image, the content of the target sub-image being associated with the target object; outputting image information and position information of the target sub-image to a display unit; displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information; and the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image.
In some embodiments of the present invention, the displaying the target sub-image in a target sub-area within a display area of the display unit according to the image information and the position information includes: displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information, and not displaying the image in the rest areas in the display area; or according to the image information and the position information, displaying the target sub-image in a target sub-area in a display area of the display unit at a first refreshing frequency, and displaying a preset pure color picture in the rest areas in the display area at a second refreshing frequency; the second refresh frequency is less than the first refresh frequency.
In some embodiments of the present invention, the display method further comprises: projecting the target sub-image displayed in the target sub-area into the visual field range of a user to generate a virtual target sub-image; or projecting the target sub-image and the pure color picture displayed in the display area into a visual field range of a user to respectively generate a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure color picture.
In some embodiments of the present invention, the pure color frame is a black frame, and a transparent background is projected to the field of view of the user and corresponds to the black frame.
In some embodiments of the invention, the projected virtual target sub-image is non-overlapping with the target object.
In some embodiments of the present invention, the generating a preset image according to a target object focused in a field of view of a user includes: determining a target object concerned in the visual field range of a user; acquiring a real image of the target object; comparing the real image with a pre-stored characteristic image to identify the target object; and generating a preset image according to the characteristic image.
In some embodiments of the present invention, the determining the target object of interest in the user's field of view comprises: the target object concerned by the user is determined by tracking the eyeball movement track of the user.
In some embodiments of the present invention, the first refresh frequency is greater than or equal to 60Hz and the second refresh frequency is equal to 1 Hz.
Another aspect of an embodiment of the present invention provides an augmented reality display system, where the display system includes: a processor and a display unit; wherein the processor comprises: a preprocessing unit and an output unit; the preprocessing unit is used for generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: a background and a target sub-image, the content of the target sub-image being associated with the target object; the output unit is used for outputting the image information and the position information of the target sub-image to the display unit; the display unit is used for displaying the target sub-image in a target sub-area in a display area of the display unit according to the received image information and the position information; and the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image.
In some embodiments of the present invention, the display unit is specifically configured to display the target sub-image in a target sub-area in a display area of the display unit according to the received image information and the position information, and not display an image in the remaining area of the display area; or according to the received image information and the position information, displaying the target sub-image in a target sub-area in a display area of the display unit at a first refresh frequency, and displaying a preset pure color picture in the rest areas of the display area at a second refresh frequency; the second refresh frequency is less than the first refresh frequency.
In some embodiments of the present invention, the display system further comprises an optical system; the optical system is used for projecting the target sub-image displayed in the target sub-area into the visual field range of a user to generate a virtual target sub-image; or projecting the target sub-image and the pure color picture displayed in the display area of the display unit into a visual field range of a user to respectively generate a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure color picture.
In some embodiments of the present invention, the display unit is specifically configured to display the target sub-image at a first refresh frequency in a target sub-area in a display area of the display unit according to the received image information and the position information, and display a preset black frame at a second refresh frequency in the remaining area of the display area; the second refresh frequency is less than the first refresh frequency; the optical system is specifically configured to project the target sub-image and the black screen displayed in the display area of the display unit into a visual field of a user, and generate a virtual target sub-image and a transparent background corresponding to the black screen, respectively.
In some embodiments of the present invention, the display system further comprises: the determination unit is used for determining a target object focused in the visual field range of the user; a shooting unit for acquiring a real image of the target object; the processor further comprises a comparison unit, wherein the comparison unit is used for comparing the real image with a pre-stored characteristic image so as to identify the target object; the preprocessing unit is specifically configured to generate a preset image according to the feature image.
In some embodiments of the invention, the determining unit comprises: an eyeball tracker.
In some embodiments of the invention, the display system is augmented reality glasses.
Based on this, according to the augmented reality display method provided by the embodiment of the present invention, the processor processes only the image information and the position information of the target sub-image associated with the target object interested by the user in the generated preset image, and then transmits only the processed image information and position information of the target sub-image to the display unit, so as to intelligently display the interested content, thereby greatly reducing the data processing amount and the data transmission amount, and significantly reducing the transmission bandwidth and the display power consumption.
Moreover, by adopting the augmented reality display method, the transmission bandwidth of the processor and the display power consumption of the whole augmented reality system can be reduced, so that the augmented reality display method is more suitable for the high-end display requirement of ultrahigh resolution and/or large visual angle.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a display method for augmented reality according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of another augmented reality display method according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of another augmented reality display method according to an embodiment of the present invention;
fig. 4 is a schematic configuration diagram of an augmented reality display system according to an embodiment of the present invention;
fig. 5 is a schematic configuration diagram of a display system for augmented reality according to another embodiment of the present invention;
fig. 6 is a schematic configuration diagram of another augmented reality display system according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an example display process of a display method for augmented reality according to an embodiment of the present invention.
Reference numerals:
01-augmented reality display system; 02-user;
10-a processor; 11-a pre-treatment unit; 12-an output unit; 13-an alignment unit;
20-a display unit; 20 a-a display area;
30-a determination unit; 40-a shooting unit; 50-an optical system; 60-a mirror frame;
p1 — target sub-image; p1' -virtual target sub-image; s1-target subregion.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present application, the meaning of "a plurality" is two or more unless otherwise specified.
It is to be noted that, unless otherwise defined, all terms (including technical and scientific terms) used in the embodiments of the present invention have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For example, as used in the specification and claims of the present invention, the term "comprising" or "comprises" and the like means that the element or item presented before the term covers the element or item listed after the term and its equivalents, but does not exclude other elements or items.
In the related art, in the conventional AR display, for projecting to the whole image within the user's visual field, the background portion needs to be processed into a solid color (for example, black), and then the additional image is superimposed to provide the content of interest associated with the target object in the real scene, so as to form the whole image; the whole image is transmitted to the micro display screen for display through a Processor (i.e. an add Processor, abbreviated as AP), and the whole image displayed on the micro display screen is projected to the eyes of the user through an AR optical system (including optical elements such as a reflective optical element and an imaging lens). Wherein, the background in the whole projected image is in a transparent state (when the background is black) or a partially transparent state (when the background is other pure colors) so that the content of the real world can be observed by human eyes, and the content of the region of interest in the whole projected image is normally projected to the visual field range of the human eyes, namely, the normal display is performed in the visual field range of the human eyes.
Since the region of interest associated with the target object in the real scene is only a small region of the whole image, i.e. only occupies a small region of the display area of the whole micro display screen, the processor (AP) still outputs the whole image to the micro display screen, i.e. processes the whole image at the resolution of the whole display area of the micro display screen, thereby increasing the data transmission bandwidth and the display power consumption.
In particular, when the display area of the microdisplay has a higher resolution (e.g., 8K, i.e., 7680 × 4320 pixels) and a larger viewing angle, the background portion of the data to be processed by the processor occupies a larger proportion, which causes a large amount of redundancy in data transmission, thereby affecting the display effect.
Based on this, an aspect of the embodiment of the present invention provides a display method for augmented reality, as shown in fig. 1, the display method includes the following steps S01-S03:
s01: generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: the background and the target sub-image, and the content of the target sub-image is associated with the target object;
s02: outputting image information and position information of the target sub-image to a display unit;
s03: displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information; the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image.
In step S01, the content of the target sub-image in the preset image is associated with the target object, and the content of the target sub-image may be interpretation information describing the target object, or additional information or derivative information related to the target object.
For example, when the target object of interest in the user's visual field range is the eiffel tower, the content of the target sub-image may be five chinese characters "eiffel tower" to explain and explain the target object (i.e., the eiffel tower).
For example, the content of the target sub-image may be additional information such as the height of the eiffel tower, information of a designer, and a build time.
As another example, the content of the target sub-image may be a french flag, that is, information of a flag of the country derived from the target object.
In addition, the background of the preset image, except for the target sub-image, may be a pure color image, for example, processed into black or other pure color images.
In the above-described step S02, the image information of the target sub-image, for example, including RGB data signals, is output to the display unit; and outputting the position information of the target sub-image to a display unit, for example, the starting coordinate of the target sub-image in a preset image.
In subsequent step S03, the display unit may display a target sub-image associated with the target object within a corresponding target sub-area within the display area according to the received image information and position information; since the processor outputs only the image information and the position information of the target sub-image to the display unit, the data processing amount of the processor and the data amount output to the display unit can be greatly reduced, and the transmission bandwidth and the display power consumption are significantly reduced.
Based on this, with the augmented reality display method (hereinafter referred to as AR display method) provided by the embodiment of the present invention, the AP (i.e., the processor) processes only the image information and the position information of the target sub-image associated with the target object that is interested by the user in the generated preset image, and then transmits only the processed image information and position information of the target sub-image to the display unit, so as to intelligently display the interested content, thereby greatly reducing the data processing amount and the data transmission amount, and thus significantly reducing the transmission bandwidth and the display power consumption.
Moreover, by adopting the AR display method, the transmission bandwidth of the AP and the display power consumption of the whole AR system can be reduced, so that the AR display method is more suitable for high-end display requirements of ultra-high resolution (i.e., Pixels Per inc, abbreviated as PPI) and/or large viewing angle (i.e., Field of View, abbreviated as FOV).
The AR display method provided by the embodiment of the present invention is specifically described below.
The step S01 of generating the preset image may include the following sub-steps S11-S14:
s11: determining a target object of interest in a user visual field range;
for example, by tracking the eye movement trajectory of the user, a target object focused by the user is determined.
S12: acquiring a real image of a target object;
s13: comparing the real image with a pre-stored characteristic image to identify a target object;
s14: and generating a preset image according to the characteristic image.
For example, a preset image associated with the feature image is retrieved through a pre-established lookup gallery.
Further, as shown in fig. 2, the above step S03 can be performed by the following two exemplary methods:
in the first embodiment, step S03 includes:
s03 a: displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information, and not displaying the image in the rest areas in the display area; the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image;
that is, when the target sub-image is displayed in the target sub-area, the remaining area in the display area of the display unit is in an off (i.e., non-display) state, so that display power consumption can be significantly reduced.
The second method, step S03 includes:
s03 b: according to the image information and the position information, displaying a target sub-image at a first refreshing frequency in a target sub-area in a display area of the display unit, and displaying a preset pure color picture at a second refreshing frequency in the rest areas in the display area; the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image; the second refresh frequency is less than the first refresh frequency.
That is, when the target sub-image is displayed in the target sub-area, since the target sub-image is associated with the target object of interest focused by the user, the display can be refreshed at a higher first refresh frequency f1 (e.g., f1 ≧ 60Hz) for updating at any time; the remaining area in the display area of the display unit may be controlled by a Driver Integrated Circuit (Driver IC) in the display unit to fixedly display a preset pure color image, which is a background when the target sub-image is displayed, and since the pure color image does not need to be updated at any time, the area may be refreshed at a lower second refresh frequency f2 (e.g., f2 equals 1Hz), thereby reducing the display power consumption to a certain extent.
Further, the AR display method further includes: and projecting at least the target sub-image displayed by the display unit into the visual field range of the user, so that the user can see the visual effect that the virtual object exists in the real environment in sense, namely the 'enhancement' of the real scene is realized.
In addition, depending on the specific display mode of the display unit, the mode of projecting the screen displayed by the display unit into the visual field of the user may be performed in step S04a or step S04b as shown in fig. 3:
mode one, S04 a: projecting the target sub-image displayed in the target sub-area into the visual field range of a user to generate a virtual target sub-image;
mode two, S04 b: and projecting the target sub-image and the pure-color picture displayed in the display area into the visual field range of a user to respectively generate a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure-color picture.
Here, when the pure color screen is a black screen, the black screen displayed in the display area as the background of the target sub-image is processed by the AR optical system and then projected into the field of view of the user as a transparent background, so that the human eye can better observe the real world in the field of view. Of course, when the pure color picture is another pure color picture, the black picture displayed in the display area as the background of the target sub-image is processed by the AR optical system, and then projected into the field of view of the user as a partially transparent (i.e., not completely transparent) background, so that the human eye can continue to observe the real world in the field of view.
It should be noted that, the manner of projecting the image displayed by the display unit into the visual field of the user may adopt related technologies, for example, the image (i.e. light rays) displayed by the display unit is projected into the visual field of the user by using a light conduction manner of a light guide or a prism, which is not limited in the embodiment of the present invention.
Further, for the projection mode adopting the first mode or the second mode, when the step of generating the preset image is performed, the preset image may be processed, so that a virtual target sub-image formed by projecting the target sub-image in the preset image is not overlapped with the target object. Exemplary ways of processing include, but are not limited to, the following:
when the target object focused in the user visual field range is located in the center region of the user visual field range by default, it may be set that the target sub-image is located in an edge region of the preset image, for example, at least one of an upper left corner, an upper right corner, a lower left corner, and a lower right corner of the generated preset image.
Thus, since the position of the target sub-area in the display area of the display unit, which is used for displaying the target sub-image, in the display area is corresponding to the position of the target sub-image in the preset image, the target sub-area in the display area of the display unit, which is used for displaying the target sub-image, is correspondingly at least one of the upper left corner, the upper right corner, the lower left corner and the lower right corner of the whole display area, so that the virtual target sub-image formed by projecting the target sub-image is correspondingly also at least one of the upper left corner, the upper right corner, the lower left corner and the lower right corner of the whole projection range, and the projected virtual target sub-image is not covered on the target object.
Or, the AP may automatically adjust the area of the generated preset image where the target sub-image is located in the preset image according to the approximate form of the target object focused in the user visual field and the approximate position of the target object in the user visual field. For example, when the target object is an eiffel tower and the target object is approximately located at the center of the user's field of view, the eiffel tower is approximately vertical, so that the target sub-image in the generated preset image may be set to be located on the left side or the right side of the center of the preset image, so that the virtual target sub-image formed by projecting the target sub-image is correspondingly located on the left side or the right side of the center of the whole projection range, and thus the virtual target sub-image is not covered on the target object.
For another example, when the target object is an airplane in the sky, because the airplane is located in an area above the user's view field, it may be set that, in the generated preset image, the target sub-image is located at the center of the preset image or below the center of the preset image, so that the virtual target sub-image formed by projecting the target sub-image is correspondingly located at the center of the whole projection range or below the center of the whole projection range, and the projected virtual target sub-image is not covered on the target object.
On the basis of the above, another aspect of the embodiments of the present invention further provides a display system for augmented reality (hereinafter, referred to as an AR display system) for performing the AR display method, as shown in fig. 4, where the AR display system 01 includes: a processor 10 and a display unit 20; wherein, this treater 10 includes: a preprocessing unit 11 and an output unit 12. The display unit 20 is an OLED display (i.e., Organic Light-Emitting Diode).
Referring to fig. 4, the preprocessing unit 11 is configured to generate a preset image according to a target object focused in a user view range. The preprocessing unit 11 may be, for example, a corresponding processing component in an image processing chip, and the generated preset image includes: background and target sub-images, the content of which is associated with the target object.
An output unit 12 for outputting image information and position information of the target sub-image to the display unit 20; the output unit 12 may be, for example, a transmission Interface (e.g., an HDMI, High Definition Multimedia Interface).
Correspondingly, the display unit 20 is configured to display the target sub-image in the target sub-area in the display area of the display unit 20 according to the received image information and the position information; the position of the target sub-area in the display area corresponds to the position of the target sub-image in the preset image.
Further, as shown in fig. 5, the AR display system 01 further includes: a determination unit 30 and a photographing unit 40; the determination unit 30 is configured to determine a target object focused within a field of view of a user, and the shooting unit 40 is configured to acquire a real image of the target object.
Correspondingly, the processor 10 further includes a comparing unit 13, configured to compare the received real image acquired by the shooting unit 40 with a pre-stored characteristic image, so as to identify the target object. The comparison unit 13 may be, for example, a corresponding identification component in an image processing chip.
Further, the preprocessing unit 11 is specifically configured to generate a preset image according to the feature image.
The determining unit 30 may include, for example: the eyeball tracker determines a target object concerned by the user by tracking the moving track of the eyeballs of the user. The photographing unit 40 may include, for example, a camera for photographing a target object to acquire a real image thereof.
Further, the display unit 20 is specifically configured to display the target sub-image in the target sub-area in the display area of the display unit according to the received image information and the position information, and not display the image in the remaining area of the display area.
In this way, when the target sub-image is displayed in the target sub-region, the remaining region in the display region of the display unit is in an off (i.e., non-display) state, which can significantly reduce display power consumption.
Alternatively, the display unit 20 may be specifically configured to display the target sub-image in the target sub-area of the display unit at the first refresh frequency according to the received image information and the position information, and display the preset pure color picture in the remaining area of the display area at the second refresh frequency; the second refresh frequency is less than the first refresh frequency.
In this way, when the target sub-image is displayed in the target sub-area, since the target sub-image is associated with the target object of interest focused by the user, the display can be refreshed with a higher first refresh frequency f1 (e.g., f1 ≧ 60Hz) for updating at any time; the rest of the display area of the display unit can be controlled by the Driver IC in the display unit to fixedly display a preset pure color picture, which is the background when the pure color picture is displayed as the target sub-image, and since the pure color picture does not need to be updated at any time, the area can be refreshed with a lower second refresh frequency f2 (for example, f2 equals to 1Hz), thereby reducing the display power consumption to a certain extent.
Further, as shown in fig. 6, the AR display system 01 further includes an optical system 50; the optical system 50 is configured to project the target sub-image displayed in the target sub-region into the field of view of the user to generate a virtual target sub-image.
Alternatively, the optical system 50 may be further configured to project the target sub-image and the pure color frame displayed in the display area of the display unit into the visual field of the user, so as to generate a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure color frame, respectively.
Here, when the pure color picture is a black picture, the black picture displayed in the display area as the background of the target sub-image is processed by the AR optical system and then projected to the field of view of the user as a transparent background, so that the human eye can better observe the real world in the field of view. Of course, when the pure color picture is another pure color picture, the black picture displayed in the display area as the background of the target sub-image is processed by the AR optical system, and then projected to the visual field range of the user, the black picture is a partially transparent (i.e. not a completely transparent) background, and the human eye can continue to observe the real world in the visual field range.
The optical system 50 may include, for example: the display device comprises a reflective optical element and an imaging lens, wherein an optical image generated by a display unit (such as an LCD display screen or an OLED display screen) is reflected to the imaging lens through the reflective optical element, is magnified and imaged through the imaging lens, and is projected to the visual field range of a user, so that the augmented reality effect is generated.
The composition and imaging principle of the optical system 50 are only exemplary, and the specific composition may follow the structural design of the AR system as long as the image displayed by the display unit 20 can be projected into the visual field of the user.
Illustratively, the display unit 20 is specifically configured to display the target sub-image at a first refresh frequency f1 in a target sub-area within the display area of the display unit 20, and display the preset black screen at a second refresh frequency f2 smaller than the first refresh frequency f1 in the remaining area of the display area, according to the received image information and position information.
Accordingly, the optical system 50 is specifically configured to project the target sub-image and the black frame displayed in the display area of the display unit 20 into the visual field of the user, so as to generate a virtual target sub-image and a transparent background corresponding to the black frame, respectively.
Further, the AR display system 01 further includes a memory, and the memory stores a processing program that is executable on the processor, generates a preset image, processes the preset image, and transmits the processed image to the display unit.
By way of example, the AR display system provided by the embodiment of the present invention may be augmented reality glasses (hereinafter referred to as AR glasses).
The AR glasses may include, for example: the glasses comprise a glasses body (comprising components such as glasses and a glasses frame), an eyeball tracker serving as a determination unit, a camera serving as a shooting unit, a processor AP, a miniature LCD display screen serving as a display unit, an optical system comprising a reflection optical element and an imaging lens and the like.
For example, the processor may be disposed on the frame and coupled to the eye tracker, the camera, and the LCD display via wires and/or wirelessly.
The eyeball tracker may be, for example, a tracker that tracks the movement trajectory of the eyeball according to the change in characteristics of the eyeball and the eyeball periphery; or a tracker for tracking the moving track of the eyeball according to the change of the iris angle; still alternatively, the tracker may be a tracker that actively projects a light beam such as infrared rays onto an iris to extract corresponding features, thereby tracking the movement trajectory of the eyeball. The eyeball tracker is arranged on the glasses body at a position which can receive the eyeball and the characteristic change around the eyeball, or can receive the change of the angle of the iris, or can project a light beam to the iris and receive the emitted light beam.
Furthermore, the camera, the miniature LCD display screen and the optical system can be arranged on one side, far away from the eyeball, of the lens frame in the lens frame so as to shoot a target object, and target sub-images which are displayed on the miniature LCD display screen and are interesting to a user are projected into the visual field range of the user through the optical system, so that the user can see the visual effect that virtual objects exist in the real environment on sense organs at the same time, namely the 'enhancement' of a real scene is realized.
The following takes an example that a target object of interest in the user view field is an eiffel tower, and the AR display method and the AR display system provided by the embodiment of the present invention are specifically described.
As shown in fig. 7, taking the resolution of the display area 20a of the display unit 20 as 1920 × 1080 pixels as an example, when the processor 10 recognizes that the target object (i.e. the object of interest) in the visual field of the user 02 is the eiffel tower, the processor 10 generates a preset image with a resolution of 1920 × 1080 pixels, wherein the starting points of the horizontal and vertical coordinates of the target sub-image P1 in the whole preset image are respectively at the 1200 th pixel and the 600 th pixel, i.e. the coordinate position is (1200,600), the resolution of the target sub-image P1 is 500 × 200 pixels, and the content of the target sub-image P1 is five chinese characters, i.e. "eiffel" characters, for explaining and explaining the target object (i.e. the eiffel tower).
The processor 10 provided in the embodiment of the present invention only needs to transmit the image information and the position information of the target sub-image P1, i.e. the image with the resolution of 500 × 200 pixels and the start coordinate position (1200,600), to the display unit 20 for displaying, instead of using the processor provided in the related art, to transmit the whole image with the resolution of 1920 × 1080 pixels, so as to significantly reduce the transmission bandwidth and the power consumption, and meanwhile, perform intelligent display according to the interested objects of the user.
When the display unit 20 displays the target sub-image P1 in the target sub-area S1 in the display area 20a, the position of the target sub-area S1 in the display area 20a corresponds to the position of the target sub-image P1 in the preset image, that is, the area size of the target sub-area S1 is 1920 × 1080 pixels, and the starting coordinate position of the horizontal direction and the vertical direction is (1200,600); the remaining area within the display area 20a does not display an image, i.e., is in an off state (i.e., not displayed), so that display power consumption can be significantly reduced.
With continuing reference to fig. 7, further, the target sub-image P1 displayed in the target sub-area S1 is projected to the visual field of the user through the optical system 50 to generate a virtual target sub-image P1 ', so that the user can see the visual effect of the virtual target sub-image P1' existing in the real environment through the lens (the lens is not illustrated in fig. 7, and only the frame 60 with the fixed lens is illustrated because the lens is transparent), that is, the "enhancement" of the real scene is realized.
Based on this, according to the AR increasing display method and the AR display system provided by the embodiment of the present invention, the AP processes only the image information and the position information of the target sub-image associated with the target object interested by the user in the generated preset image, and then transmits only the processed image information and the processed position information of the target sub-image to the display unit to intelligently display the interested content, so that the data processing amount and the data transmission amount are greatly reduced, and the transmission bandwidth and the display power consumption are significantly reduced.
Moreover, by adopting the AR display method, the transmission bandwidth of the AP and the display power consumption of the whole AR system can be reduced, so that the AR display method is more suitable for the high-end display requirements of ultra-high PPI and/or large FOV.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present invention, and shall cover the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims (12)

1. A display method of augmented reality, the display method comprising:
generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: a background and a target sub-image, the content of which is associated with the target object, wherein the content of the target sub-image is interpretation information describing the target object, additional information or derived information related to the target object;
outputting only the image information and the position information of the processed target sub-image to a display unit; and
displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information; wherein the position of the target sub-area within the display area corresponds to the position of the target sub-image in the preset image, the position information comprises coordinates of a reference point in the target sub-image in the display area, and wherein the position of the target sub-image in the display area is automatically adjusted according to the form of the target object and its position within the user's field of view,
displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information, including:
displaying the target sub-image in a target sub-area in a display area of the display unit according to the image information and the position information, and not displaying the image in the rest areas in the display area; or according to the image information and the position information, displaying the target sub-image in a target sub-area in a display area of the display unit at a first refreshing frequency, and displaying a preset pure color picture in the rest areas in the display area at a second refreshing frequency; the second refresh frequency is less than the first refresh frequency.
2. The augmented reality display method according to claim 1, further comprising:
projecting the target sub-image displayed in the target sub-area into the visual field range of a user to generate a virtual target sub-image; or projecting the target sub-image and the pure color picture displayed in the display area into a visual field range of a user to respectively generate a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure color picture; and the virtual target sub-image formed by projecting the target sub-image is not overlapped with the target object.
3. The augmented reality display method according to claim 1, wherein the pure color picture is a black picture, and a transparent background is projected to a field of view of the user corresponding to the black picture.
4. The augmented reality display method according to claim 1, wherein the generating a preset image according to a target object focused in a user visual field range includes:
determining a target object of interest in a user visual field range;
acquiring a real image of the target object;
comparing the real image with a pre-stored characteristic image to identify the target object;
and generating a preset image according to the characteristic image.
5. The augmented reality display method according to claim 4, wherein the determining a target object of interest within a field of view of a user comprises:
determining a target object focused by a user by tracking an eyeball movement track of the user, wherein the tracking of the eyeball movement track of the user is realized according to any one of the following steps: the change of features of the eyeball and the eyeball periphery, the change of the iris angle, and the projection of a light beam to the iris to extract the corresponding features.
6. The augmented reality display method of claim 1, wherein the first refresh rate is greater than or equal to 60Hz, and the second refresh rate is equal to 1 Hz.
7. An augmented reality display system, the display system comprising: a processor and a display unit; wherein the processor comprises: a preprocessing unit and an output unit;
the preprocessing unit is used for generating a preset image according to a target object concerned in the visual field range of a user; wherein the preset image includes: a background and a target sub-image, the content of the target sub-image being associated with the target object, wherein the content of the target sub-image is explanatory information describing the target object, additional information or derived information relating to the target object;
the output unit is used for outputting only the image information and the position information of the target sub-image processed by the processor to the display unit;
the display unit is used for displaying the target sub-image in a target sub-area in a display area of the display unit according to the received image information and the position information; wherein the position of the target sub-area within the display area corresponds to the position of the target sub-image in the preset image, the position information comprises coordinates of a reference point in the target sub-image in the display area, and wherein the position of the target sub-image in the display area is automatically adjusted according to the morphology of the target object and its position within the user's field of view,
the display unit is particularly adapted to,
displaying the target sub-image in a target sub-area in a display area of the display unit according to the received image information and the position information, and not displaying the image in the rest areas of the display area; or, according to the received image information and the received position information, displaying the target sub-image in a target sub-area in a display area of the display unit at a first refreshing frequency, and displaying a preset pure color picture in the rest areas of the display area at a second refreshing frequency; the second refresh frequency is less than the first refresh frequency.
8. The augmented reality display system of claim 7, further comprising an optical system; the optical system is used for the purpose of,
projecting the target sub-image displayed in the target sub-area into the visual field range of a user to generate a virtual target sub-image; or projecting the target sub-image and the pure color picture displayed in the display area of the display unit into a visual field range of a user, and respectively generating a virtual target sub-image and a transparent background or a partially transparent background corresponding to the pure color picture, wherein the virtual target sub-image formed by projecting the target sub-image is not overlapped with the target object.
9. Augmented reality display system according to claim 8,
the display unit is specifically configured to display the target sub-image at a first refresh frequency in a target sub-area in a display area of the display unit according to the received image information and the position information, and display a preset black picture at a second refresh frequency in the remaining area of the display area; the second refresh frequency is less than the first refresh frequency;
the optical system is specifically configured to project the target sub-image and the black screen displayed in the display area of the display unit into a visual field of a user, and generate a virtual target sub-image and a transparent background corresponding to the black screen, respectively.
10. The augmented reality display system of claim 7, further comprising:
the determination unit is used for determining a target object focused in the visual field range of the user;
a shooting unit for acquiring a real image of the target object;
the processor further comprises a comparison unit, wherein the comparison unit is used for comparing the real image with a pre-stored characteristic image so as to identify the target object;
the preprocessing unit is specifically configured to generate a preset image according to the feature image.
11. The augmented reality display system according to claim 10, wherein the determination unit includes: an eyeball tracker configured to track a movement trajectory of an eyeball of a user to determine the target object of interest to the user, and wherein a position of the eyeball tracker is configured such that the eyeball tracker can receive a change in characteristics of the eyeball and a periphery of the eyeball, or can receive a change in an angle of an iris, or can project a beam to the iris and receive the beam emitted back.
12. Augmented reality display system according to any one of claims 7 to 11, wherein the display system is augmented reality glasses.
CN201910058904.2A 2019-01-22 2019-01-22 Augmented reality display method and augmented reality display system Active CN109714583B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910058904.2A CN109714583B (en) 2019-01-22 2019-01-22 Augmented reality display method and augmented reality display system
US16/745,412 US11232602B2 (en) 2019-01-22 2020-01-17 Image processing method and computing device for augmented reality device, augmented reality system, augmented reality device as well as computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910058904.2A CN109714583B (en) 2019-01-22 2019-01-22 Augmented reality display method and augmented reality display system

Publications (2)

Publication Number Publication Date
CN109714583A CN109714583A (en) 2019-05-03
CN109714583B true CN109714583B (en) 2022-07-19

Family

ID=66262480

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910058904.2A Active CN109714583B (en) 2019-01-22 2019-01-22 Augmented reality display method and augmented reality display system

Country Status (2)

Country Link
US (1) US11232602B2 (en)
CN (1) CN109714583B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110572632A (en) * 2019-08-15 2019-12-13 中国人民解放军军事科学院国防科技创新研究院 Augmented reality display system, helmet and method based on sight tracking
CN111090104B (en) * 2019-12-26 2022-11-11 维沃移动通信有限公司 Imaging processing method and electronic device
CN111563432A (en) * 2020-04-27 2020-08-21 歌尔科技有限公司 Display method and augmented reality display device
CN111553972B (en) * 2020-04-27 2023-06-30 北京百度网讯科技有限公司 Method, apparatus, device and storage medium for rendering augmented reality data
CN111930236B (en) * 2020-08-17 2023-06-20 Oppo广东移动通信有限公司 Equipment control method and device, storage medium and electronic equipment
CN113465611B (en) * 2021-06-29 2024-03-26 阿波罗智联(北京)科技有限公司 Navigation method, navigation device, electronic equipment and readable storage medium
KR20230118211A (en) * 2022-02-03 2023-08-11 삼성디스플레이 주식회사 Display device and method of driving display device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107765842A (en) * 2016-08-23 2018-03-06 深圳市掌网科技股份有限公司 A kind of augmented reality method and system

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4679661B1 (en) * 2009-12-15 2011-04-27 株式会社東芝 Information presenting apparatus, information presenting method, and program
US20140146394A1 (en) * 2012-11-28 2014-05-29 Nigel David Tout Peripheral display for a near-eye display device
CN103873840B (en) * 2012-12-12 2018-08-31 联想(北京)有限公司 Display methods and display equipment
JP2016519785A (en) * 2013-04-08 2016-07-07 フィリップス ライティング ホールディング ビー ヴィ Large area display system
CN106293043B (en) * 2015-06-29 2023-11-10 北京智谷睿拓技术服务有限公司 Visual content transmission control method, transmission method and device thereof
US10134198B2 (en) * 2016-04-19 2018-11-20 Adobe Systems Incorporated Image compensation for an occluding direct-view augmented reality system
CN106203289A (en) * 2016-06-28 2016-12-07 广东欧珀移动通信有限公司 Acquisition methods, device and the mobile terminal of a kind of targets improvement real content
CN106959759B (en) * 2017-03-31 2020-09-25 联想(北京)有限公司 Data processing method and device
CN107333119A (en) * 2017-06-09 2017-11-07 歌尔股份有限公司 The processing method and equipment of a kind of display data

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107765842A (en) * 2016-08-23 2018-03-06 深圳市掌网科技股份有限公司 A kind of augmented reality method and system

Also Published As

Publication number Publication date
US20200234470A1 (en) 2020-07-23
US11232602B2 (en) 2022-01-25
CN109714583A (en) 2019-05-03

Similar Documents

Publication Publication Date Title
CN109714583B (en) Augmented reality display method and augmented reality display system
US9874932B2 (en) Avoidance of color breakup in late-stage re-projection
CN109633907B (en) Method for automatically adjusting brightness of monocular AR (augmented reality) glasses and storage medium
US11480804B2 (en) Distributed foveated rendering based on user gaze
US9076033B1 (en) Hand-triggered head-mounted photography
US9375639B2 (en) Image display system and head-mounted display device
US9424767B2 (en) Local rendering of text in image
US8957916B1 (en) Display method
US20140176591A1 (en) Low-latency fusing of color image data
JP2002509289A (en) Enhanced imaging using silhouettes to improve contrast
US11567568B2 (en) Display apparatuses and methods incorporating foveated rendering
US10699673B2 (en) Apparatus, systems, and methods for local dimming in brightness-controlled environments
US9766458B2 (en) Image generating system, image generating method, and information storage medium
US20180158390A1 (en) Digital image modification
US11651544B2 (en) Systems and methods employing multiple graphics processing units for producing images
US11366315B2 (en) Image processing apparatus, method for controlling the same, non-transitory computer-readable storage medium, and system
US11749141B2 (en) Information processing apparatus, information processing method, and recording medium
US11347060B2 (en) Device and method of controlling device
EP3655814B1 (en) Display apparatus and method of displaying using means for providing visual cues
US11615767B2 (en) Information processing apparatus, information processing method, and recording medium
US20230379594A1 (en) Image blending
WO2023136073A1 (en) Image display device and image display method
US20240013688A1 (en) Image processing apparatus, display apparatus, image processing method, and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant