CN114371819A

CN114371819A - Augmented reality screen system and augmented reality screen display method

Info

Publication number: CN114371819A
Application number: CN202011102457.5A
Authority: CN
Inventors: 曾惠萍; 杨朝光; 徐文正; 黄志文; 谭驰澔
Original assignee: Acer Inc
Current assignee: Acer Inc
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2022-04-19
Anticipated expiration: 2040-10-15
Also published as: CN114371819B

Abstract

The invention relates to an augmented reality screen system and an augmented reality screen display method, comprising the following steps: shooting a physical mark through a camera of an augmented reality device; receiving the entity mark through a host, judging position information and rotation information of the entity mark, and acquiring a virtual picture from a storage device through a processor of the host; and adjusting the virtual picture and transmitting the adjusted virtual picture to the augmented reality device through the processor according to the position information and the rotation information, so that the augmented reality device projects the adjusted virtual picture onto a display of the augmented reality device, the adjusted virtual picture becomes a virtual extended screen, and the virtual extended screen and the entity mark are simultaneously displayed on the display of the augmented reality device.

Description

Augmented reality screen system and augmented reality screen display method

Technical Field

The invention relates to application of augmented reality, in particular to an augmented reality screen and an augmented reality screen display method.

Background

Extended screens are now widely used in work and daily life, for example, where one user uses two screens, whereby more windows can be expanded. However, a user needs to buy more screens than a corresponding number of screens, for example, a user needs to buy three screens, and the more screens, the higher the cost. In addition, if the user needs to manually adjust the placement position of the screen during the process of configuring the screen.

Therefore, how to conveniently use the extended screen to enable the user to see a wider display range has become one of the problems to be solved in the art.

Disclosure of Invention

In order to solve the above-mentioned problems, an embodiment of the present disclosure provides an augmented reality screen system. The augmented reality screen system includes: an augmented reality device and a host. The augmented reality device is used for shooting a physical mark through a camera. The host is used for receiving the entity mark, judging position information and rotation information of the entity mark and obtaining a virtual picture from a storage device through a processor of the host. The processor adjusts the virtual picture and transmits the adjusted virtual picture to the augmented reality device according to the position information and the rotation information, so that the augmented reality device projects the adjusted virtual picture onto a display of the augmented reality device, the adjusted virtual picture becomes a virtual extended screen, and the virtual extended screen and the entity mark are simultaneously displayed on the display of the augmented reality device.

Another embodiment of the present disclosure provides an augmented reality screen system. The augmented reality screen display method comprises the following steps: shooting a physical mark through a camera of an augmented reality device; receiving the entity mark through a host, judging position information and rotation information of the entity mark, and acquiring a virtual picture from a storage device through a processor of the host; and adjusting the virtual picture and transmitting the adjusted virtual picture to the augmented reality device through the processor according to the position information and the rotation information, so that the augmented reality device projects the adjusted virtual picture onto a display of the augmented reality device, the adjusted virtual picture becomes a virtual extended screen, and the virtual extended screen and the entity mark are simultaneously displayed on the display of the augmented reality device.

The augmented reality screen system and the augmented reality screen display method provided by the invention have the advantages that the user can see the virtual extended screen by wearing the augmented reality device to scan the entity mark placed at any angle, the picture size of the virtual extended screen can be adjusted according to the requirements of the user, and the problems of weight, volume, fixed picture size, space limitation, inconvenience in portability and the like caused by using the entity screen are solved. Therefore, the extended screen is virtualized to be displayed in the augmented reality device, and a new use situation can be created besides replacing the entity extended screen.

Drawings

FIG. 1 is a block diagram illustrating an augmented reality screen system, according to one embodiment of the present invention.

FIG. 2 is a flowchart illustrating an augmented reality screen display method according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating an augmented reality screen display method according to an embodiment of the invention.

Fig. 4A to 4C are schematic diagrams illustrating adjusting a viewing angle range according to an embodiment of the invention.

Fig. 5 is a schematic diagram illustrating a virtual extended screen of a head-mounted display device according to an embodiment of the invention.

FIG. 6 is a schematic diagram illustrating an application of a virtual extended screen, in accordance with one embodiment of the present invention.

Description of reference numerals:

100: augmented reality screen system

10: processor with a memory having a plurality of memory cells

SR: main unit

PM: display device

HDM: head-mounted display device

ST: storage device

CAM: camera with camera head

GYR: gyroscope

IRR: infrared receiver

HDP (high density plasma) is as follows: display device

200,300: augmented reality screen display method

210-230, S1-S5: step (ii) of

MK: entity tagging

VIM: virtual picture

VIM': adjusted virtual picture

And (3) USR: user's hand

CAF: picture frame

R0-R2, R2': reference point

R3: target coordinates

P1, P1': pixel

FOV0, FOV 1: range of viewing angles

FOV 2: a first specific range

FOV 3: virtual picture

FOV 4: second specific range

Detailed Description

The following description is of the preferred embodiments of the invention and is made for the purpose of illustrating the general principles of the invention and not for the purpose of limiting the same. Reference must be made to the following claims for their true scope of the invention.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of further features, integers, steps, operations, elements, components, and/or groups thereof.

The use of the terms first, second, third and the like in the claims is used for modifying elements in the claims and is not intended to distinguish between elements having the same name, priority, or other relationship between elements, whether one element precedes another element, or whether a method step is performed in a chronological order.

Referring to fig. 1 to 3, fig. 1 is a block diagram illustrating an augmented reality screen system 100 according to an embodiment of the invention. FIG. 2 is a flow chart illustrating an augmented reality screen display method 200 according to one embodiment of the invention. FIG. 3 is a diagram illustrating an augmented reality screen display method 300, according to an embodiment of the invention.

As shown in fig. 1, the augmented reality screen system 100 includes a head mounted display device HMD and a host SR. In one embodiment, the host SR and the head mounted display device HMD may establish a communication link in a wired/wireless manner.

In one embodiment, the augmented reality device may be a head mounted display device HMD or other device capable of displaying both virtual images and physical images. For convenience of description, the head mounted display device HMD is taken as an example for description.

In one embodiment, the host SR may be a notebook computer, a desktop computer, or other electronic devices with computing functions.

In one embodiment, the host SR includes a processor 10. In one embodiment, the processor 10 may be implemented by a volume circuit such as a micro controller (mcu), a microprocessor (microprocessor), a digital signal processor (digital signal processor), an Application Specific Integrated Circuit (ASIC), or a logic circuit.

In one embodiment, the host SR may include the display PM or an external display PM. In one embodiment, the display PM is a physical display coupled to the host SR.

In one embodiment, the host SR includes a storage device ST, which can be implemented as a rom, a flash memory, a floppy disk, a hard disk, an optical disk, a usb disk, a magnetic tape, a database accessible through a network, or a storage medium with the same functions as those easily understood by those skilled in the art.

In one embodiment, the user USR may wear a head mounted display device HMD, which includes a camera CAM, a gyroscope GYR, an infrared receiver IRR, and a display HDP. Among them, the gyroscope GYR can know the visual field and direction of the user USR wearing the head mounted display device HMD by measuring the angular velocity of the movement such as yaw and tilt.

In one embodiment, the camera CAM of the head mounted display HMD is used to capture and track the physical marker MK.

In one embodiment, the physical marker MK may be any type of marker, such as a binary square marker (ArUco marker), a QR code, an infrared ray, or a screen appearance.

In one embodiment, when the augmented reality screen system 100 defaults that the entity mark MK is a QR code and the camera CAM captures the QR code, the camera CAM transmits the captured image and the values measured by the gyroscope GYR to the processor 10 of the host SR.

In one embodiment, when the augmented reality screen system 100 defaults that the entity mark MK is infrared and the infrared receiver IRR receives infrared, the infrared receiver IRR transmits the image captured by the camera CAM and the values measured by the gyroscope GYR to the processor 10 of the host SR. Therefore, the head mounted display device HMD can capture the physical marker MK by the camera CAM or the infrared receiver IRR. However, the present disclosure is not limited thereto, and any method may be used as long as the method allows the head mounted display device HMD to know that the physical marker MK is captured.

Referring to fig. 2 to 3, the following describes a flow of the augmented reality screen display method 200, and the augmented reality screen display method 200 can be implemented by using the elements in fig. 1. In one embodiment, as shown in fig. 3, the goal is for the user USR to see through the head mounted display device HMD that the virtual extended screen VIM is displayed next to the display PM (the physical display of the host SR).

In step 210, a camera CAM of a head mounted display device HMD captures a physical marker MK.

In one embodiment, when a camera CAM of the head mounted display device HMD captures the physical mark MK, the captured images CAF and the values measured by the gyroscope GYR are transmitted to the processor 10 of the host SR (i.e., step S2 in fig. 3). In one embodiment, the frame CAF captured by a camera CAM is a two-dimensional image that includes at least the physical marker MK, and in this case, also includes a display PM (physical display).

In an embodiment, a software, such as a game software, may be run in the host SR all the time, but a game picture is not displayed on the display PM (physical display), the game picture is temporarily stored in the storage device ST (the storage device here takes an internal memory as an example), the display PM may be used to display text editing software, and the processor 10 receives the physical mark MK captured by the head mounted display device HMD and then obtains the game picture VIM from the internal memory (i.e., step S1 in fig. 3).

In one embodiment, the processor 10 may obtain the game picture VIM from the memory at regular time (e.g., every 5 seconds), and therefore, the steps S1 and S2 in fig. 3 have no sequential points.

In step 220, a host SR receives the physical mark MK, determines a position information and a rotation information of the physical mark MK, and obtains a virtual frame VIM from a storage device ST through a processor 10 of the host SR.

In one embodiment, the processor 20 of the host SR receives the frame CAF captured by the camera CAM as a two-dimensional image, and thereby obtains the physical marker MK from the two-dimensional image. Processor 20 analyzes the positional information and rotational information of this physical marker MK.

In one embodiment, the position information may be three-axis coordinates (ax, ay, az) relative to the head-mounted display device HMD, and the rotation information may be three-axis rotation angles (rx, ry, rz) relative to the head-mounted display device HMD.

In one embodiment, processor 10 executes a tracking algorithm to detect the position information and the rotation information of physical marker MK. The tracking algorithm may be implemented by a known mark (marker) tracking method, a Visual-Inertial-Odometry (VIO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, or an object tracking (object tracking) algorithm to track the position information and the rotation information of the entity mark MK.

In step 230, the processor 10 adjusts the virtual frame VIM and transmits the adjusted virtual frame VIM to the head mounted display HMD according to the position information and the rotation information, and the head mounted display HMD projects the adjusted virtual frame VIM ' onto a display HDP of the head mounted display HMD, so that the adjusted virtual frame VIM ' becomes a virtual extended screen, and the virtual extended screen VIM ' and the physical mark MK are simultaneously displayed on the display HDP of the head mounted display HMD.

In an embodiment, since the frame CAF captured by the camera CAM is a three-dimensional image, the physical mark MK obtained from the frame CAF is also a three-dimensional image (as shown in fig. 3S 3), and the processor 10 obtains the virtual frame VIM obtained by the fetching operation from the memory as a two-dimensional image, a correction process for projecting the two-dimensional image onto the three-dimensional image is performed, so that the display HDP of the head-mounted display device HMD can correctly display the adjusted virtual frame VIM' (as shown in fig. 3S 4). When the user USR sees both the virtual frame VIM 'and the display PM (physical display) through the display HDP of the head mounted display device HMD, the user USR visually regards the virtual frame VIM' as a virtual extended screen (as shown in fig. 3S 5).

In an embodiment, the processor 1f0 calculates a target coordinate of each pixel of the virtual frame VIM projected to the display HDP of the head mounted display HMD through a position projection algorithm by using a position projector algorithm to adjust the virtual frame VIM, and transmits the adjusted virtual frame VIM 'to the head mounted display HMD, so that the head mounted display HMD projects the adjusted virtual frame VIM' onto the display HDP of the head mounted display HMD.

The following is a method of adjusting the virtual screen VIM so that the adjusted virtual screen VIM' can be projected on the display HDP of the head mounted display device HMD. However, the following method is only an example, and other known conversion methods can be applied thereto.

In one embodiment, the processor 10 reads the physical marker MK information from the head mounted display device HMD, the physical marker MK information includes the position information (ax, ay, az) and the rotation information (rx, ry, rz), and the processor 10 also receives the image data of the camera CAM, where each pixel is represented by (cx, cu, cz). In addition, the processor 10 obtains the virtual frame VIM in the memory at the same time according to the entity mark MK information (position information and rotation information) obtained by the tracking algorithm.

In one embodiment, the processor 10 calibrates the physical mark MK captured by the camera CAM with a light engine (projector) inside the head mounted display device HMD as an origin to obtain relative coordinates with the light engine of the head mounted display device HMD as the origin. The calibration (calibration) function is as follows:

wherein the symbol M_RotateThe relative rotation matrix between the camera CAM and the optical machine of the head-mounted display device HMD is obtained; symbol D_x–axis、D_y-axisAnd D_z-axisThe triaxial position of the relative translation of the camera CAM and the optical engine of the head mounted display device HMD (with the origin of the camera CAM and the optical engine of the head mounted display device HMD as a new origin) is shown.

In one embodiment, the processor 10 substitutes the calibrated data Xcalib, Ycalib, and Zcalib into a Homography matrix (homograph) for operation. Homography matrices are a concept in projective geometry, also known as projective transformations, that maps points on one projective plane onto another projective plane and maps straight lines as straight lines, with line-preserving properties. The homography matrix function is as follows:

thereby, the projective transformed coordinate data can be obtained.

In one embodiment, the display HDP may be implemented by a projector that displays an imaged (or "display in AR world") display in an augmented reality environment.

In one embodiment, since the field of view (FOV) and resolution of the camera CAM are different from those of the display HDP (or light engine) in the wearable display device HMD, the following calculations (circle selected view range (Corp) and adjusted resolution (Resize)) are required for final imaging position adjustment:

wherein

Wherein, the symbol W_cam、H_camResolution, respectively width and height of the camera CAM, denoted W_des、H_desThe resolution of the width and height of the virtual frame VIM' (i.e., the virtual extended screen), respectively. In an embodiment, the processor 10 displays the calculated virtual extended screen position (dx, dy) on the display HDP of the head mounted display device HMD, so that the user can see the virtual extended screen with the same rotation angle as the physical mark MK in an augmented reality, and the processor 10 performs Crop and Resize operations on the virtual extended screen position (dx, dy) to perform Crop and Resize adjustments on the virtual extended screen.

Referring to fig. 4A to 4C, 5 and 6, fig. 4A to 4C are schematic diagrams illustrating adjusting a viewing angle range according to an embodiment of the invention. Fig. 5 is a schematic diagram illustrating a virtual extended screen of the head mounted display device HMD according to an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating an application of a virtual extended screen, in accordance with one embodiment of the present invention.

In one embodiment, as shown in fig. 4A, the processor 10 obtains a viewing angle range FOV0 of the frame CAF captured by the camera CAM, selects a reference point R0 from the viewing angle range FOV0, and moves the reference point R0 to a boundary of the frame CAF (e.g., fig. 4B, the reference point R0 is moved to overlap with the boundary R1, which is regarded as R1 and the viewing angle range as FOV 1).

Then, the processor 10 obtains a first specific range of the viewing angle range FOV1 within the frame CAF (e.g. the first specific range FOV2 within the frame CAF, at which time the designated reference point is R2), enlarges the first specific range FOV2 to be consistent with the resolution of a real display PM coupled to the host SR (as shown in fig. 4C, enlarges the first specific range FOV2 to the state of the virtual frame FOV3, makes the enlarged resolution of the first specific range FOV2 consistent with the resolution of the real display PM, at which time the designated reference point is shifted from R2 to the position of R2 '), moves the reference point R2 ' to a target coordinate R3 on the display HDP of the head mounted display HMD (according to the position projection algorithm, the processor 10 may calculate the target position R3 of the display HDP corresponding to the reference point R0 of the camera CAM, and at which time the target position R2 ' is shifted to the target coordinate R3), a second specific range FOV4 of the viewing angle range within the virtual picture FOV3 is acquired, a designated pixel P1 (e.g., designated upper left pixel P1) of the second specific range FOV4 is mapped to a designated position of the physical mark MK (the designated position is, for example, designated upper left pixel P1 'of the MK, and the designated pixel P1 overlaps the designated position P1'), so that the second specific range FOV4 of the picture CAF becomes a virtual extended screen (i.e., an adjusted virtual picture VIM '), and the virtual extended screen VIM' and the physical mark MK are simultaneously displayed on the display HDP of the head mounted display HMD.

In one embodiment, the position of the designated pixel P1 of the second specific range FOV4 and the designated position P1' of the entity mark MK may be defined in advance.

In one embodiment, when the physical mark MK is not captured temporarily (e.g. within 5 seconds) by the head mounted display device HMD, the virtual extended screen VIM' may be displayed on the display HDP of the head mounted display device HMD (e.g. temporarily for 3 seconds).

In an embodiment, the processor 10 adjusts the second specific range FOV4 according to the position information and the rotation information, and regards the adjusted second specific range FOV4 as the virtual extended screen VIM ', the processor 10 knows an extending direction from the physical marker MK (for example, in fig. 5, the physical marker MK is located right on the real display PM, and the extending direction of the virtual extended screen VIM' extends right), the processor 10 transmits the extending direction to the head mounted display device HMD, and the head mounted display device HMD displays the virtual extended screen VIM 'on the display HDP according to the physical marker MK and the extending direction, so that the virtual extended screen VIM' is parallel to the real display PM.

In one embodiment, the size and resolution of the virtual extended screen VIM 'and the real display PM are the same, and the virtual extended screen VIM' can dynamically display different frames.

In an embodiment, the processor 10 obtains (crop) a viewing angle range FOV0 in the virtual frame VIM, adjusts the viewing angle range FOV0 to be consistent with the size and resolution of the real display PM, and regards the adjusted viewing angle range FOV0 as a virtual extended screen VIM ', and transmits the virtual extended screen VIM' to the head mounted display HMD, which displays the virtual extended screen VIM 'on the display HDP according to the physical marker MK and the extending direction, so that the virtual extended screen VIM' is parallel to the real display PM.

As shown in fig. 6, by the augmented reality screen display method 200, as long as the camera CAM of the head mounted display device HMD captures the physical mark MK, the head mounted display device HMD transmits an image including the physical mark MK to the host SR, the host SR may receive the physical mark MK, generate the virtual extended screen VIM ' from the physical mark MK, and transmit the virtual extended screen VIM ' to the display HDP of the head mounted display device HMD, and the user USR may view the physical mark MK and the virtual extended screen VIM ' through the display HDP. In an embodiment, as shown in fig. 6, the user USR can see the plurality of physical markers MK and the virtual extended screen VIM 'corresponding to each physical marker MK through the display HDP, so that the extended screen VIM' can be seen beside the desktop, the wall, and the physical display PM.

The methods of the present invention, or certain aspects or portions thereof, may take the form of program code. The program code may be embodied in tangible media, such as floppy diskettes, cd-roms, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine thereby becomes an apparatus for practicing the invention. The program code may also be transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission, wherein, when the program code is received and loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. When implemented in a general-purpose processing unit, the program code combines with the processing unit to provide a unique apparatus that operates analogously to specific logic circuits.

Although the present invention has been described with reference to the above embodiments, it should be understood that various changes and modifications can be made therein by those skilled in the art without departing from the spirit and scope of the invention.

Claims

1. An augmented reality screen system, comprising:

an augmented reality device for photographing a physical mark by a camera; and

a host computer, which is used for receiving the entity mark, judging a position information and a rotation information of the entity mark, and obtaining a virtual picture from a storage device through a processor of the host computer;

the processor adjusts the virtual image and transmits the adjusted virtual image to the augmented reality device according to the position information and the rotation information, the augmented reality device projects the adjusted virtual image onto a display of the augmented reality device, the adjusted virtual image becomes a virtual extended screen, and the virtual extended screen and the entity mark are simultaneously displayed on the display of the augmented reality device.

2. The augmented reality screen system of claim 1, wherein the processor executes a tracking algorithm to detect the position information and the rotation information of the physical marker.

3. The augmented reality screen system of claim 1, wherein the processor calculates a target coordinate of each pixel of the virtual frame projected to the display of the augmented reality device through a position projection algorithm to adjust the virtual frame, and projects the adjusted virtual frame to the display of the augmented reality device.

4. The augmented reality screen system of claim 1, wherein the processor acquires a viewing angle range in a frame captured by the camera, selects a reference point from the viewing angle range, moves the reference point to a boundary in the frame, acquires a first specific range of the viewing angle range within the frame, enlarges the first specific range to be consistent with a resolution of a real display coupled to the host, moves the reference point to a target coordinate on the display of the augmented reality device, acquires a second specific range of the viewing angle range within the frame, corresponds a designated pixel of the second specific range to a designated position of the real mark, and makes the second specific range of the frame the virtual extended screen, the virtual extended screen and the physical marker are simultaneously displayed on the display of the augmented reality device.

5. The augmented reality screen system of claim 4, wherein the processor adjusts the second specific range according to the position information and the rotation information, and regards the adjusted second specific range as the virtual extended screen, the processor knows an extending direction from the physical mark and transmits the extending direction to the augmented reality device, and the augmented reality device displays the virtual extended screen on the display of the augmented reality device according to the physical mark and the extending direction.

6. An augmented reality screen display method, comprising:

shooting a physical mark through a camera of an augmented reality device;

receiving the entity mark through a host, judging position information and rotation information of the entity mark, and obtaining a virtual picture from a storage device through a processor of the host; and

the virtual picture is adjusted and then transmitted to the augmented reality device through the processor according to the position information and the rotation information, the augmented reality device projects the adjusted virtual picture onto a display of the augmented reality device, the adjusted virtual picture becomes a virtual extension screen, and the virtual extension screen and the entity mark are simultaneously displayed on the display of the augmented reality device.

7. The augmented reality screen display method of claim 6, further comprising:

executing a tracking algorithm to detect the position information and the rotation information of the physical mark.

8. The augmented reality screen display method of claim 6, further comprising:

and calculating a target coordinate of each pixel of the virtual picture projected to the display of the augmented reality device through a position projection algorithm and the position projector algorithm so as to adjust the virtual picture, and projecting the adjusted virtual picture to the display of the augmented reality device.

9. The augmented reality screen display method of claim 6, further comprising:

acquiring a visual angle range in a picture shot by the camera;

selecting a reference point from the view angle range;

moving the reference point to a boundary in the picture;

acquiring a first specific range of the visual angle range in the picture;

enlarging the first specific range to be consistent with the resolution of a real display coupled with the host;

moving the reference point to a target coordinate on the display of the augmented reality device;

acquiring a second specific range of the visual angle range in the picture;

corresponding a designated pixel of the second specific range to a designated position of the physical mark, so that the second specific range of the picture becomes the virtual extended screen; and

displaying the virtual extended screen and the physical marker on the display of the augmented reality device simultaneously.

10. The augmented reality screen display method of claim 9, further comprising:

adjusting the second specific range according to the position information and the rotation information, and regarding the adjusted second specific range as the virtual extended screen;

obtaining an extending direction from the physical mark; and

and displaying the virtual extended screen on the display of the augmented reality device according to the physical mark and the extension direction.