WO2020140905A1 - Virtual content interaction system and method - Google Patents

Abstract

A virtual content interaction system, comprising a terminal device and an external head-mounted display device. The terminal device establishes a communicational connection with the head-mounted display device; the terminal device is configured to obtain first relative space position information of the head-mounted display device and the terminal device, determine a display position of virtual content according to the first relative space position information, render the virtual content according to the display position, obtain display data of the virtual content, and transmit the display data to the head-mounted display device; the head-mounted display device is configured to receive the display data sent by the terminal device and display the virtual content according to the display data.

Description

Virtual content interaction system and method Technical field

This application relates to the field of augmented reality, in particular to a virtual content interaction system and method.

Background technique

With the development of science and technology, machine intelligence and information intelligence are becoming more and more popular. The technology of identifying user images through machine vision or virtual vision and other image acquisition devices to realize human-computer interaction is becoming more and more important. Augmented reality technology uses computer graphics technology and visualization technology to construct virtual content that does not exist in the real environment, and accurately integrates the virtual content into the real environment through image recognition and positioning technology, and integrates the virtual content and the real environment with the help of display devices. And show the user a real sensory experience. The primary technical problem to be solved by augmented reality technology is how to accurately integrate the virtual content into the real world, that is, to make the virtual content appear in the correct position of the real scene at the correct angle posture, so as to generate a strong visual reality. Therefore, the interaction with virtual content is an important research direction of augmented reality technology.

Summary of the invention

The embodiments of the present application propose a virtual content interaction system and method.

In a first aspect, an embodiment of the present application provides a virtual content interaction system, including a terminal device and an external head-mounted display device, the terminal device establishing a communication connection with the head-mounted display device, wherein the terminal device For acquiring first relative spatial position information of the head-mounted display device and the terminal device, determining a display position of virtual content according to the first relative spatial position information, and rendering the virtual content according to the display position , And obtain the display data of the virtual content, and transmit the display data to the head-mounted display device; the head-mounted display device is used to receive the display data sent by the terminal device, and according to the The display data displays the virtual content.

In a second aspect, an embodiment of the present application provides a virtual content interaction method, which is applied to a terminal device, and the terminal device is in communication connection with an external head-mounted display device. The method includes: acquiring the head-mounted display device and The first relative spatial position of the terminal device; determining the display position of the virtual content according to the first relative spatial position; rendering the virtual content according to the display position, and obtaining display data of the virtual content; and The display data is transmitted to the head-mounted display device, and the display data is used to instruct the head-mounted display device to display the virtual content.

In a third aspect, an embodiment of the present application provides a virtual content interaction method, which is applied to a head-mounted display device, and the head-mounted display device establishes a communication connection with a terminal device, including: collecting an image containing a marker through a camera. The marker is provided on the terminal device; the marker image is transmitted to the terminal device, and the image is used to instruct the terminal device to recognize the marker contained in the image and obtain the A first relative spatial position of the head-mounted display device and the terminal device; and receiving display data sent by the terminal device, and displaying virtual content according to the display data, the display data is determined by the terminal device according to the first The virtual content is obtained by rendering the virtual content at a relative spatial position.

According to a fourth aspect, an embodiment of the present application provides a virtual content display method, which is applied to a first terminal, and the first terminal is in communication connection with a second terminal. The method includes: acquiring the first terminal and the first terminal First relative spatial position information between the two terminals; obtaining display content data from the second terminal, wherein the display content data includes at least the data of the display content currently displayed by the second terminal; and according to the first A relative spatial position information and the display content data to display virtual content, wherein the virtual content includes the display content displayed by the second terminal and the extended content corresponding to the display content.

According to a fifth aspect, an embodiment of the present application provides an image processing method, which is applied to a terminal device, and the terminal device is in communication connection with a head-mounted display device. The method includes: acquiring the terminal device and the head-mounted display device Relative spatial position relationship between them; according to the relative spatial position relationship, acquiring a projection area of the virtual content displayed by the head-mounted display device on the screen of the terminal device, the projection area being displayed by the head-mounted display The projection area of the virtual content on the screen observed by the device; obtaining the image content corresponding to the projection area in the screen content according to the screen content to be displayed on the screen; and performing on the image content Specify the process and display the screen content containing the image content after the specified process, and the difference in tone between the first tone of the specified image content and the second tone of the virtual content is greater than the first threshold.

In a sixth aspect, an embodiment of the present application provides a terminal device, including: one or more processors; a memory; wherein the memory stores one or more application programs, and the one or more application programs are configured to be configured by the one Or executed by multiple processors, and the one or more programs are configured to execute the method provided above.

According to a seventh aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a program code, and the program code can be called by a processor to execute the method provided above.

BRIEF DESCRIPTION

FIG. 1 is a schematic diagram of an application scenario according to an embodiment of the present application.

FIG. 2 is a schematic diagram of an augmented reality system according to an embodiment of this application.

FIG. 3 is a schematic diagram of another application scenario according to an embodiment of the present application.

FIG. 4 is a flowchart of a virtual content interaction method in an embodiment.

FIG. 5 is a schematic diagram showing virtual content in an embodiment.

FIG. 6 is a schematic diagram showing extended content on a screen of a terminal device in an embodiment.

FIG. 7 is a schematic diagram showing extended content on a screen of a terminal device in an embodiment.

FIG. 8 is a flowchart of a virtual content interaction method in an embodiment.

FIG. 9 is a flowchart of determining the display position of virtual content in one embodiment.

FIG. 10 is a schematic diagram showing virtual content in one embodiment.

FIG. 11 is a schematic diagram showing that the virtual content displayed in one embodiment overlaps with the screen content of the terminal device.

FIG. 12 is a schematic diagram showing virtual content in one embodiment.

13A-13B are schematic diagrams of updating the displayed virtual content according to the change of the screen content of the terminal device in one embodiment.

14A-14C are schematic diagrams of the sliding of the control area of the terminal device in an embodiment.

15 is a schematic diagram of controlling virtual content according to a finger sliding trajectory in one embodiment.

FIG. 16 is a schematic diagram of controlling virtual content according to a finger sliding trajectory in another embodiment.

17A-17B are schematic diagrams of controlling virtual content according to a finger sliding trajectory in yet another embodiment.

FIG. 18 is a schematic diagram of controlling virtual content according to a sliding track of a finger in still another embodiment.

FIG. 19 is a flowchart of a virtual content interaction method in yet another embodiment.

Figure 20 is a block diagram of an augmented reality system in one embodiment.

21 is a flowchart of an image processing method in an embodiment.

22 is a schematic diagram of a projection area of virtual content on a screen of a terminal device in an embodiment.

23 is a schematic diagram showing virtual content and processed screen content in an embodiment.

24 is a flowchart of an image processing method in another embodiment.

FIG. 25 is a flowchart of acquiring a projection area of virtual content on a screen of a terminal device in an embodiment.

26 is a schematic diagram of a projection area of virtual content on a screen of a terminal device in another embodiment.

FIG. 27 is a flowchart of acquiring the first projection area of the left-eye image on the screen in one embodiment.

FIG. 28 is a schematic diagram showing virtual content and processed screen content in another embodiment.

FIG. 29 is a flowchart of an image processing method in still another embodiment.

30 is a block diagram of a head-mounted display device in one embodiment.

detailed description

Referring to FIGS. 1-2, the augmented reality system 10 provided by the embodiment of the present application includes: a terminal device 100 and a head-mounted display device 200 connected to the terminal device 100.

In an embodiment, the terminal device 100 may be held and controlled by a user, and may be an electronic device capable of running application programs, such as a mobile phone, a smart watch, a tablet computer, an e-reader, and a notebook computer.

In some embodiments, the head-mounted display device 200 may be an external head-mounted display device, that is, the head-mounted display device 200 includes a display module, a communication module, a camera, etc. for display. The processor, memory, etc. of the terminal device 100 control the displayed virtual content. The display module may include a display screen (or a projection device) and a display lens to display virtual content.

Among them, the head-mounted display device 200 can transmit information and instructions with the terminal device 100. Among them, the transmitted information may include virtual content displayed by the head-mounted display device 200. The terminal device 100 and the head-mounted display device 200 can be connected via wireless communication methods such as Bluetooth, WiFi, ZigBee, etc., and can also be connected by wired communication through an interface such as a USB, as shown in FIG. Wired communication connections such as terminals.

In some embodiments, the marker 101 is provided on the terminal device 100. The marker 101 can be any graphic or object with identifiable features, which can be collected and tracked by the head-mounted display device 200. For example, the marker 101 can be a pattern with a topology structure, which refers to the marker The connected relationship between the sub-markers and the feature points. When the marker 101 is within the visual range of the camera of the head-mounted display device 200, the camera can collect an image containing the marker 101 in real time and transmit the image to the terminal device 100, and the terminal device 100 can obtain the marker according to the image The relative position, orientation, and other spatial position information between the object 101 and the head-mounted display device 200 to obtain relative spatial position information between the terminal device 100 and the head-mounted display device 200. The terminal device 100 may render a corresponding virtual object based on the spatial position information between the head-mounted display device 200 and transmit the display screen of the virtual object to the head-mounted display device 200, and the head-mounted display device 200 may display the virtual object through the display module Object.

In some embodiments, the head-mounted display device 200 can also track the shape of the terminal device 100 to determine the relative spatial position relationship between the terminal device 100 and the head-mounted display device 200.

As shown in FIG. 1, the terminal device 100 is wirelessly connected to the head-mounted display device 200, and the user can scan the marker 101 on the terminal device 100 through the head-mounted display device 200 worn to watch a virtual space scene containing multiple virtual planets Superimposed on the real space, where the virtual space scene corresponds to the space scene displayed by the terminal device 100.

As shown in FIG. 3, the terminal device 100 and the head-mounted display device 200 are connected by wire communication, and the user scans the marker 101 on the terminal device 100 through the head-mounted display device 200 worn, and can view the superimposed display of the medical human model including the virtual reality in reality On the surface of the terminal device 100 in the space.

Referring to FIG. 4, an embodiment of the present application provides a virtual content interaction method, which is applied to a terminal device. The terminal device is in communication connection with an external head-mounted display device. The method includes the following steps.

Step 410: Obtain the first relative spatial position of the head-mounted display device and the terminal device.

The first relative spatial position may include relative position and posture information between the head-mounted display device and the terminal device, and the posture information may be the orientation and rotation angle of the terminal device relative to the head-mounted display device.

In some embodiments, the terminal device includes an inertial measurement unit (IMU), which may first acquire the measurement data of the IMU, and determine the first relative spatial position between the terminal device and the head-mounted display device based on the measurement data.

In some embodiments, a light spot may also be provided on the terminal device, and the head-mounted display device collects an image containing the light spot through the camera and sends the image to the terminal device. The terminal device may identify the light spot in the image to determine the first relative spatial position of the head-mounted display device and the terminal device. The light spot set on the terminal device may be a visible light spot or an infrared light spot. When the light spot is an infrared light spot, an infrared camera may be provided on the head-mounted display device to collect an image of the infrared light spot. The light spot set on the terminal device may be one, or may be a light spot sequence composed of multiple light spots.

In one embodiment, the light spot may be arranged on the casing of the terminal device, for example, may be arranged around the screen. The light point can also be set on the protective cover of the terminal device. When the terminal device is used, the protective cover containing the light point can be put on, so that the terminal device can be tracked. For example, different light spots can be set around the screen of the terminal device, such as different numbers of light spots or different color spots around the terminal device, so that the terminal device can determine The relative spatial position of the head-mounted display device.

Step 420: Determine the display position of the virtual content according to the first relative spatial position.

The terminal device can convert the first relative spatial position from the real space to the position coordinates in the virtual space, and according to the positional relationship between the virtual content to be displayed and the terminal device in the virtual space and the position coordinates, refer to the head-mounted display device as a reference Calculate the spatial position of the virtual content to be displayed in the virtual space relative to the head-mounted display device, that is, obtain the display position of the virtual content in the virtual space. The display position refers to the three-dimensional space coordinates of the virtual content in the virtual space with the head-mounted display device as the origin (which can also be regarded as the origin of the human eye).

Step 430: Render the virtual content according to the display position and obtain the display data of the virtual content.

The terminal device may acquire the data of the virtual content to be displayed, construct the virtual content according to the data of the virtual content, and render the virtual content according to the display position. The data of the virtual content to be displayed may include model data of the virtual content, which is used to construct a three-dimensional model of the virtual content. For example, the model data may include data such as the color, vertex coordinates, and outline of the model.

In some embodiments, the data of the virtual content may also be downloaded from the server by the terminal device, or may be obtained from other terminals for the terminal device.

Step 440: Transmit the display data to the head-mounted display device to instruct the head-mounted display device to display virtual content.

The terminal device may obtain display data of the rendered virtual content, and the display data may include RGB values of respective pixels in the display screen and corresponding pixel coordinates, and the like. The terminal device can transmit the display data to the head-mounted display device, and the head-mounted display device can generate a display screen according to the display data, and project the display screen onto the optical lens to display the virtual content. By wearing the optical lens of the display device, the user can see that the virtual content is superimposed and displayed in the real world, realizing the effect of augmented reality.

The terminal device obtains the data of the virtual content for rendering. The data of the virtual content may be the content data displayed on the current screen of the terminal device, or it may be the extended content data corresponding to the content displayed on the current screen, or it may be the data on the current screen. The complete content data corresponding to the content of. The complete content may include the content displayed on the current screen and the extended content corresponding to the displayed content. The content displayed on the current screen may be a part of the complete content, and the extended content is a part of the complete content except the displayed content.

As shown in FIG. 5, the content currently displayed by the terminal device 100 is a part of the China map, the terminal device 100 transmits the data of the part of the China map, and the virtual content 300 a displayed by the head-mounted display device 200 is the part of the China map. As shown in FIG. 6, the terminal device 100 transmits data of a complete China map, and the virtual content 300b displayed by the head-mounted display device 200 is a complete China map. As shown in FIG. 7, the content currently displayed by the terminal device 100 is a game map, and the terminal device 100 transmits the data of the extended content (game character) corresponding to the game map to the head-mounted display device, and the game character 300c that the user can see through the head-mounted display device Superimposed on the real space to solve the problem that the display content is limited by the screen of the mobile terminal.

In some embodiments, different buttons and other interface elements can be displayed on the screen of the terminal device, and the user can enter different modes by selecting and clicking different buttons, for example, entering or exiting the augmented reality mode; in the augmented reality mode, Select different display modes, for example, display the same content as the screen of the terminal device, display extended content corresponding to the screen of the terminal device, or display preset content irrelevant to the screen content of the terminal device, etc. The terminal device may transmit the display data corresponding to the selected display mode to the head-mounted display device, so that the head-mounted display device displays virtual content according to the display data.

In one embodiment, the terminal device may only serve as the processor and memory of the head-mounted display device to determine the virtual content displayed by the head-mounted display device, and the virtual content is not related to the content currently displayed on the screen of the terminal device. The data of the virtual content may be generated in real time according to the actual environment, or may be stored in advance. When the virtual content is displayed on the head-mounted display device, the terminal device may not display any content.

Referring to FIG. 8, a virtual content interaction method provided by an embodiment of the present application is applied to a terminal device, and the terminal device is in communication connection with an external head-mounted display device, including the following steps.

Step 810: Acquire a first relative spatial position of the head-mounted display device and the terminal device.

In some embodiments, the terminal device is provided with markers. The markers may be provided on the casing of the terminal device, or may be displayed on the screen of the terminal device in the form of an image. The marker may also be an external marker. When in use, the terminal device can be inserted through the USB or the earphone hole, etc., so as to track the terminal device.

Step 820: Determine the display position of the virtual content according to the first relative spatial position.

In some embodiments, as shown in FIG. 9, determining the display position of the virtual content according to the first relative spatial position includes steps 822 and 824.

Step 822: Acquire the second relative spatial position of the virtual content and the terminal device.

The second relative spatial position may include the position and posture information of the virtual content in the virtual space relative to the terminal device, and may also be understood as the relative spatial position relationship between the superimposed position of the virtual content in the real space and the terminal device.

The superimposed position of the virtual content in the real space can be superimposed on the terminal device, such as on the screen area of the terminal device, as shown in FIG. 12, the user can see that the virtual medical mannequin is superimposed on the screen area of the terminal device through the head-mounted display device ; Virtual content can also be located on the edge or around the terminal device, as shown in Figure 6, China map 300b is superimposed on the upper right of the terminal device 100; the virtual content can also be in a different plane from the terminal device, such as the plane where the terminal device is located and the virtual content The plane is vertical, and the superimposition position of the virtual content is not limited.

In some embodiments, the second relative spatial position may be stored in the terminal device in advance, or may be set according to the content displayed on the current screen of the terminal device. The content displayed on the current screen of the terminal device is different, and the second relative spatial position may be different. As shown in FIG. 6, the current screen of the terminal device 100 displays a part of the China map, and the virtual content 300b including the complete China map is in the superimposed position of the terminal device in the real space. Top right; as shown in FIG. 10, the current screen of the terminal device 100 displays a photo, and the virtual content 300d containing multiple photos is located directly above the terminal device 100 in the superimposed position of the real space.

In some embodiments, the second relative spatial position may also be set according to the inclusion relationship between the content displayed on the current screen of the terminal device and the virtual content. When the virtual content includes the content displayed on the screen of the terminal device, the second relative spatial position of the virtual content and the terminal device may be determined according to the position of the content displayed on the screen in the virtual content, so that the displayed content included in the virtual content It overlaps with the content displayed on the terminal device screen.

As shown in FIG. 11, the content currently displayed by the terminal device 100 is the surrounding map 110 of Hebei Province, and the virtual content may be the map 300b of China. According to the location of the surrounding map 110 of Hebei Province in the map 300b of China, the map 300b of China and the mobile terminal can be set The second relative spatial position enables the user to see the surrounding map of Hebei Province in the China map 300b through the head-mounted display device, which overlaps with the surrounding map 110 of Hebei Province currently displayed on the terminal device.

Step 824: Determine the display position of the virtual content according to the first relative spatial position and the second relative spatial position.

The terminal device may determine the spatial coordinates of the terminal device in the virtual space according to the first relative spatial position, and based on the spatial coordinates and the second relative spatial position, use the head-mounted display device as a reference to determine the space of the virtual content relative to the head-mounted display device Position to get the display position of the virtual content.

In some embodiments, the terminal device may determine whether the display position of the virtual content overlaps with the terminal device. The display position of the virtual content may be a superimposition position of the virtual content in the real space, or a space position of the virtual content in the virtual space with the head-mounted display device as the origin. When the display position of the virtual content overlaps with the terminal device, the corresponding display area of the content displayed on the current screen of the terminal device in the virtual content can be determined, and the display area is the overlapping area between the display position of the virtual content and the terminal device. The content of the display area is the same as the content displayed on the current screen of the terminal device.

In some embodiments, the terminal device may perform a specified process on the display area, may adjust the content color of the display area to the specified color, and may also adjust the content transparency of the display area to the specified transparency. Wherein, the brightness value of each color component of the specified color is lower than the first threshold, and the specified transparency is lower than the second threshold.

The first threshold is the maximum brightness value of each color component when the virtual content cannot be normally superimposed and displayed. For example, the first threshold may be set to 13 brightness, which is 95% black, or may be set to 0 brightness, which is black. The second threshold is the maximum transparency when the virtual content cannot be observed normally. For example, the second threshold may be set to 1, which is 90% transparent, or may be set to 0, which is 100% transparent. After performing the designated display processing on the first display area, the user cannot observe the virtual content in the first display area, so that the user can see that the content currently displayed on the terminal device and the virtual content displayed on the head-mounted display device are accurately connected together.

There is a corresponding relationship between the content displayed on the current screen of the terminal device and the display position of the virtual content, the content displayed on the screen is different, and the display position corresponding to the virtual content is different. The corresponding relationship can be stored in the terminal device. When the content displayed on the screen of the terminal device changes, the display position of the virtual content can be updated according to the changed content.

As an embodiment, the terminal device may receive the user's sliding operation on the screen, and change the content displayed on the screen according to the sliding operation. The terminal device may re-determine the corresponding display area of the changed content displayed on the screen in the virtual content. Based on the newly determined display area, the updated display position of the virtual content is calculated, and the virtual content is re-rendered based on the updated display position, so that the user can observe that the position of the virtual content changes as the content displayed on the terminal device changes. As shown in FIGS. 13A and 13B, the user slides to the right on the screen of the terminal device 100, and the content displayed by the terminal device 100 changes from the surrounding map 110 of Hebei to the surrounding map 120 of Ningxia. The user can see the China map 300 through the head-mounted display device 200 Has moved, and the map 300 of China is accurately connected with the content displayed on the mobile terminal.

Step 830: Render the virtual content according to the display position and obtain the display data of the virtual content.

Step 840: Transmit the display data to the head-mounted display device, and the display data is used to instruct the head-mounted display device to display the virtual content.

Step 850: When receiving the control operation, generate a control instruction according to the control operation.

The terminal device may receive the user's control operation, generate a control instruction according to the control operation, and control the virtual content displayed by the head-mounted display device. As an implementation manner, the terminal device includes a manipulation area, which can receive a user's control operation in the manipulation area. The manipulation area may include at least one of a touch screen and keys, where the user's control operations include, but are not limited to, single-finger sliding, clicking, pressing, multi-finger coordinated sliding, etc., acting on the manipulation area of the terminal device. The control instructions generated according to the control operation may include a movement instruction, a zoom-in instruction, a zoom-out instruction, a rotation instruction, a selection instruction, etc., so as to realize the display effect of controlling the movement, zooming, rotation, and selection of virtual content.

In some embodiments, the terminal device may generate a control instruction according to one or more of the detected number of fingers when the control operation is performed, the gesture action when the control operation is performed, and the finger sliding trajectory when the control operation is performed. As an embodiment, the number of fingers when performing control operations in the manipulation area can be detected in real time, and different control instructions can be generated according to different numbers of fingers. As shown in FIG. 14A, when it is detected that the user performs a single-finger slide control operation in the manipulation area of the terminal device, a control instruction for moving virtual content is generated. As shown in FIGS. 13A-13B, the control instruction is to control the head-mounted display device to move the currently displayed virtual map to the right relative to the user’s perspective. As shown in FIG. 14B, when it is detected that the user performs a control operation of relatively narrowing and merging the distance between two fingers in the manipulation area of the terminal device, a control instruction for reducing the virtual content is generated. As shown in FIGS. 6 and 16, the control instruction is to control the head-mounted display device to reduce the currently displayed virtual map relative to the user's viewing angle.

As an embodiment, the finger sliding trajectory of the control operation may be detected in real time, and different control instructions may be generated according to different finger sliding trajectories. As shown in FIG. 15, when the user performs a sliding operation on the touch screen area of the terminal device 100 with a finger to the right, a control instruction to rotate the virtual content 300 (virtual medical human body) to the right is generated.

In some embodiments, the control operation may also be a gesture control of the user. The head-mounted display device can collect the user's gesture image through the camera and recognize the gesture in the gesture image. When the recognized gesture is a preset gesture, a control instruction is generated. The preset gestures can be ascending, descending, waving left and right gestures.

In some embodiments, the terminal device may also be connected to the controller, and may receive user control operations in the control area of the controller. The user's control operations include, but are not limited to, single-finger slide, click, press, multi-finger cooperative slide acting on the control area of the controller. In some embodiments, for different virtual content, the same control operation may correspond to different control instructions. The terminal device may generate a control instruction corresponding to the control operation according to the virtual content and the received control operation.

Step 860: Control the displayed virtual content according to the control instruction.

In some embodiments, the terminal device may adjust the display state of the virtual content according to the generated control instruction, and transmit the adjusted virtual content to the head-mounted display device, so that the head-mounted display device displays the adjusted virtual content. The user controls the virtual content by operating the terminal device, and can also view the display effect of the virtual content while operating on the terminal device, thereby realizing the interaction between the terminal device and the head-mounted display device.

In some embodiments, if the virtual content corresponds to the currently displayed display content of the terminal device, the content displayed on the screen of the terminal device may also be controlled according to the control instruction. As shown in Figure 17A, the content displayed by the terminal device is a map of Beijing, Shanxi Province and other cities. The virtual content displayed by the head-mounted display device is a map of China. If the control command is an enlarged display command, as shown in Figure 17B, the user displays through the head-mounted display The enlarged map of China seen by the device, and at the same time, the maps of Beijing, Shanxi Province and other cities displayed on the terminal device are enlarged and displayed to the map of Beijing.

In some embodiments, controlling virtual content may include selecting specified virtual content, moving virtual content, zooming in on virtual content, and so on. As shown in FIG. 18, when the area selected by the user on the virtual content 300 is a map of Beijing, the user can see that the map of Beijing is displayed on the terminal device 100 simultaneously.

In some embodiments, in addition to being a processor and a memory of the head-mounted display device, the terminal device can also provide power to the head-mounted display device, keep the head-mounted display device light, and reduce the manufacturing cost of the head-mounted display device.

In some embodiments, the virtual content interaction method provided in the above embodiments may also be independently implemented in a head-mounted display device, and the head-mounted display device may have a processor and a memory to generate and display virtual content.

In the virtual content interaction method provided in the above embodiment, the content currently displayed by the terminal device and the virtual content displayed by the head-mounted display device are accurately connected to improve the display effect; and the displayed virtual content can be controlled to improve the terminal device and the head-mounted device. Show interactivity between devices.

Referring to FIG. 19, an embodiment of the present application provides a virtual content interaction method, which is applied to a head-mounted display device. The head-mounted display device and a terminal device communicate and connect, including the following steps.

Step 1910: Acquire an image containing a marker, which is set on the terminal device.

Step 1920: Transmit the collected image to the terminal device. The image is used to instruct the terminal device to identify the marker in the image, and obtain the first relative spatial position of the head-mounted display device and the terminal device.

Step 1930: Receive the display data sent by the terminal device, and display the virtual content according to the display data. The display data is rendered by the terminal device according to the first relative spatial position.

Referring to FIG. 20, an augmented reality system provided by an embodiment of the present application includes a terminal device 11 and a head-mounted display device 12 communicatively connected to the terminal device 11. The terminal device 11 is used to obtain first relative spatial position information of the head-mounted display device 12 and the terminal device 11, determine the display position of the virtual content according to the first relative spatial position information, render the virtual content according to the display position, and obtain the virtual content The display data is transmitted to the head-mounted display device 12, and the display data is used to instruct the head-mounted display device 12 to display virtual content. The head-mounted display device 12 is used to receive the display data sent by the terminal device 11 and display the virtual content according to the display data.

Referring to FIG. 21, an embodiment of the present application provides an image processing method, which can be applied to a terminal device. The terminal device and a head-mounted display device are communicatively connected. The method includes the following steps.

Step 2110: Obtain the relative spatial position relationship between the terminal device and the head-mounted display device.

Step 2120: Acquire the projection area of the virtual content displayed on the head-mounted display device on the screen of the terminal device according to the relative spatial position relationship, and the projection area is the projection area of the virtual content observed on the screen by the human eye through the head-mounted display device.

The projection area can be understood as, when the shape of the virtual content is projected on the plane where the screen of the terminal device is based on the human eye observing the virtual content as a reference point, between the projection area obtained on the plane and the screen area of the terminal device Coincident area. In some embodiments, the projection area may be represented by the coordinates of the screen coordinate system of the terminal device, and the screen coordinate system uses one corner of the screen, for example, the lower left corner, as the origin.

In some embodiments, the projection area of the virtual content on the screen of the terminal device may be obtained according to the virtual content and the spatial position of the terminal device in the virtual space. The virtual space may include a virtual camera, which is used to simulate the human eye of the user, and the position of the virtual camera in the virtual space may be regarded as the position of the head-mounted display device in the virtual space.

In some embodiments, the spatial position of the virtual content and the terminal device in the virtual space may be the position of the virtual content and the terminal device in a preset coordinate system of the virtual space, and the preset coordinate system may include a world coordinate system, The spatial coordinate system with the virtual camera as the origin, etc. The terminal device can obtain the extension line of the connection between the virtual camera and each vertex in the virtual content according to the virtual camera, the virtual content, and the spatial position of the terminal device in the preset coordinate system, and calculate each extension line and the terminal in the virtual space separately The intersection point of the screen of the device determines the projection point of each vertex of the virtual content on the screen of the terminal device, and obtains the spatial coordinates of each projection point in the preset coordinate system. According to the conversion parameters of the preset coordinate system of the virtual space and the screen coordinate system of the terminal device in the real environment, the spatial coordinates of the projection point can be converted into the two-dimensional coordinates of the screen coordinate system of the terminal device to obtain the virtual content on the screen of the terminal device On the projection area.

As shown in FIG. 22, the terminal device 220 is a tablet computer, with the human eye 221 as the origin (also can be regarded as the origin of the head-mounted display device), the extension line of the human eye 221 and the virtual content 223, and the tablet screen The area formed by the intersection point on the screen is the projection area 225.

The specific method for obtaining the projection area may not be limited in this application. For example, the outline of the projection area of the virtual content on the screen of the terminal device may be obtained based only on the outline vertex of the virtual content and the spatial position of the terminal device in the virtual space. To get the projection area.

In some embodiments, the head-mounted display device may transmit the display data of the virtual content to the terminal device, the display data may include the spatial location of the virtual content in the virtual space, and the terminal device may use the above-mentioned methods according to the display data of the virtual content Get the projection area. In some embodiments, the terminal device may also obtain the projection area from the head-mounted display device. The head-mounted display device may obtain the projection area in the above manner, and then transmit the data of the projection area to the terminal device.

Step 2130: Acquire image content corresponding to the projection area in the screen content according to the screen content to be displayed on the screen.

As shown in FIG. 22, the image 227 is the image content corresponding to the projection area in the screen content. The terminal device can obtain the two-dimensional coordinates of the projection area on the screen of the terminal device, and can convert the two-dimensional coordinates to the coordinates of the image coordinate system of the screen content according to the conversion parameters of the screen coordinate system and the image coordinate system of the screen content to obtain the projection The area corresponds to the image area in the screen content, and the image content in the image area is acquired.

Step 2140: Perform specified processing on the image content and display the screen content containing the specified processed image content. The difference between the first hue of the specified processed image content and the second hue of the virtual content is greater than the first threshold .

The terminal device can specify the image content corresponding to the projection area in the screen content and display the processed screen content, which can reduce the interference with the virtual content displayed by the head-mounted display device.

In some embodiments, the difference between the first color tone of the designated processed image content and the second color tone of the virtual content is greater than a first threshold, which can be set reasonably according to the user's viewing comfort. There is a contrast between the first hue of the designated image content and the second hue of the virtual content superimposed. The virtual content overlay display can be readable and recognizable on the designated image content after processing. The image content can highlight the virtual content and reduce the interference of the content displayed by the terminal device with the virtual content.

Hue can refer to the lightness or darkness of an image. In some embodiments, the hue includes color and transparency, and the specified processing may be to set the color of the image content or to set the transparency of the image content. For example, adjust the color of the image content to a solid color (such as black, gray, etc.), adjust the transparency of the image content to 50% transparency, and overlay a solid color picture on the image content.

As shown in FIG. 23, the terminal device 230 establishes a wireless communication connection with the head-mounted display device, the head-mounted display device recognizes the marker 231 on the terminal device 230 and displays the virtual animal 233, and the user can see the virtual animal 233 through the worn head-mounted display device Superimposed and displayed on the terminal device 230 in the real space, and the color of the image content 235 (stone, grass, etc.) corresponding to the projection area of the virtual content in the screen content displayed on the terminal device 230 is adjusted to gray, which can highlight the virtual animal 233, Reduce the interference of image content 235 to virtual animals.

The image processing method of the above embodiment reduces the interference of the screen content displayed by the terminal device on the virtual content when the virtual content is displayed by the head-mounted display device, highlights the virtual content displayed by the head-mounted display device, and improves the realism of the virtual content in augmented reality And display effects; and only process the screen content corresponding to the projection area of the virtual content on the screen, while highlighting the virtual content, to ensure the normal display of other content on the screen.

Referring to FIG. 24, another embodiment of the present application provides an image processing method, which is applied to a terminal device. The terminal device and a head-mounted display device are communicatively connected, including the following steps.

Step 2410: Obtain the relative spatial position relationship between the terminal device and the head-mounted display device.

Step 2420: Obtain a projection area of the virtual content displayed on the head-mounted display device on the screen of the terminal device according to the relative spatial position relationship, and the projection area is a projection area of the virtual content observed on the screen by the human eye through the head-mounted display device.

In one embodiment, referring to FIG. 25, the above-mentioned obtaining the projection area of the virtual content displayed on the head-mounted display device on the screen of the terminal device according to the relative spatial position relationship may include steps 2422-2428.

Step 2422: Obtain a left-eye image and a right-eye image for displaying virtual content in the head-mounted display device.

The display data of the virtual content may include a left-eye image and a right-eye image for forming three-dimensional virtual content in the human eye through the reflection of the optical element. The left-eye image and the right-eye image have parallax. The left-eye image is used to project to the user's left eye through the optical lens during display, and the right-eye image is used to project to the user's right eye through the optical lens during display. The fusion of the image and the right-eye image through the user's brain can form a stereoscopic image, so that the user can see the display effect of the stereoscopic image.

The head-mounted display device can obtain rendering coordinates of the virtual content, and render the virtual content based on the rendering coordinates. The rendering coordinates may be the spatial coordinates of each point of the virtual content in the virtual space with the virtual camera as the origin. The virtual camera includes a left virtual camera and a right virtual camera. The left virtual camera is used to simulate the left eye of the human eye, and the right virtual camera is used to simulate the right eye of the human eye. The left rendering coordinates of the virtual content in the first spatial coordinate system with the left virtual camera as the origin and the right rendering coordinates in the second spatial coordinate system with the right virtual camera as the origin can be obtained. The virtual content can be rendered according to the left rendering coordinates to obtain the left-eye image of the virtual content, and the virtual content can be rendered according to the right rendering coordinates to obtain the right-eye image of the virtual content. The head-mounted display device can transmit the left-eye image and the right-eye image of the virtual content to the terminal device.

Step 2424: Acquire the first projection area of the left-eye image on the screen of the terminal device according to the relationship between the left-eye image and the relative spatial position.

The terminal device may acquire the first projection area of the left-eye image on the screen according to the relative spatial position relationship between the left-eye image and the head-mounted display device. The first projection area may be a plane on which the shape of the virtual content in the left-eye image is projected onto the screen of the terminal device, and the projection area on the plane coincides with the screen area of the terminal device. The projection area may partially overlap the screen area of the terminal device, or may be entirely contained in the screen area. As shown in FIG. 26, the terminal device 260 is a tablet computer, and the left-eye display image of the virtual content 261 can be incident on the left eye 262 of the user after being reflected by the optical lens of the head-mounted display device, and the left-eye display image corresponds to the screen of the tablet computer 'S first projection area 264.

In some embodiments, the terminal device may acquire the virtual content in the left virtual camera and the left-eye image according to the spatial position of the left virtual camera, the left-eye image, and the terminal device in the same spatial coordinate system (such as the world coordinate system) in the virtual space. The intersection of the extension line of each vertex and the screen of the terminal device, the intersection is the projection point of the vertex on the screen. The space coordinates of the projection point of the left-eye image on the screen of the virtual space can be converted into two-dimensional coordinates of the screen coordinate system of the terminal device according to the conversion parameters of the space coordinate system of the virtual space and the screen coordinate system of the terminal device The first projection area of the left-eye image on the screen.

Further, the first projection area of the left-eye image on the screen of the terminal device may be acquired based on the spatial position of the outline area of the virtual content in the left-eye image on the virtual space to simplify the calculation steps and optimize the processing process of the terminal device.

In some embodiments, please refer to FIG. 27, the acquiring the first projection area of the left-eye image on the screen of the terminal device according to the left-eye image and the relative spatial position relationship may include steps 2702-2708.

Step 2702: Acquire the first outline coordinate set of virtual content in the virtual space in the left-eye image.

In some embodiments, the terminal device may acquire the first outline coordinate set of the virtual content of the left-eye image in the virtual space to obtain the first projection area according to the first outline coordinate set. The first set of outline coordinates may be the coordinates of each vertex of the outline area of the virtual content in the left-eye image in the space coordinate system. The space coordinates of the outline area of the virtual content in the first space coordinate system can be directly used as the first outline coordinate set, or the conversion parameters of the first space coordinate system and the world coordinate system can be used to display the content of the virtual content in the left-eye display image. The spatial coordinates of the contour area in the first spatial coordinate system are converted into the spatial coordinates in the world coordinate system to obtain the first contour coordinate set.

Step 2704: Acquire the screen coordinate set of the screen in the virtual space according to the relative spatial position relationship.

The terminal device may acquire the spatial position of the screen of the terminal device in the real space according to the relative position relationship, and convert the spatial position into spatial coordinates in the virtual space to obtain the screen coordinate set of the screen in the virtual space.

Step 2706: Establish a first connection between the left virtual camera in the virtual space and each point in the first outline coordinate set, and obtain the coordinates of the point on each first connection in the screen coordinate set to obtain the first A set of coordinates.

The terminal device may establish a first connection between the left virtual camera in the virtual space and each point in the first outline coordinate set, and obtain the coordinates of the point on each first connection established in the screen coordinate set to obtain the first A set of intersection coordinates. Among them, the coordinates of the left virtual camera, the first outline coordinate set, the screen coordinate set, and the first intersection coordinate set are coordinates under the same spatial coordinate system.

The terminal device may determine whether the coordinates of the point on the first connection line exist in the screen coordinate set. If there is a point on the first line, it can be determined that the first line crosses the screen, otherwise, it is determined that the first line does not cross the screen. The first intersection coordinate set can be obtained according to the coordinates of the point on the first line in the screen coordinate set. The first intersection coordinate set includes coordinates of projection points of each vertex of the outline area of the virtual content in the left-eye image on the screen.

Step 2708: Acquire the first projection area of the left-eye image on the screen according to the first intersection coordinate set.

The terminal device may convert the first intersection coordinate set into two-dimensional coordinates of the terminal device's screen coordinate system according to the conversion parameters of the space coordinate system in the virtual space and the screen coordinate system of the terminal device to obtain the first projection area.

Step 2426: Acquire the second projection area of the right-eye display image on the screen according to the relationship between the right-eye image and the relative spatial position.

The terminal device may refer to the above corresponding step of acquiring the first projection area of the left-eye image on the screen of the terminal device according to the left-eye image and the relative spatial position relationship, and acquire the second projection area of the right-eye image on the screen. As shown in FIG. 26, the right-eye image of the virtual content 261 can be incident on the right eye 263 of the user after being reflected by the optical lens of the head-mounted display device, and the right-eye image corresponds to the second projection area 265 on the screen of the tablet computer.

In one embodiment, the terminal device may acquire the second outline coordinate set of the virtual content in the right-eye image in the virtual space, obtain the screen coordinate set of the screen in the virtual space according to the relative spatial position relationship, and respectively establish a right virtual camera in the virtual space The second connection line with each point in the second outline coordinate set, and obtain the coordinates of the points on each established second connection line in the screen coordinate set to obtain the second intersection coordinate set, and then according to the second coordinate set, Acquire the second projection area of the right-eye image on the screen.

Step 2428: Acquire the synthesis area of the first projection area and the second projection area, and use the synthesis area as the projection area of the virtual content on the screen.

The combined area of the first projection area and the second projection area refers to the area formed by the coordinate set obtained by combining the first intersection coordinate set corresponding to the first projection area and the second intersection coordinate set corresponding to the second projection area, That is, the union area of the first projection area and the second projection area. In some embodiments, the above process of obtaining the projection area of the virtual content on the screen may also be performed on the head-mounted display device, and then transmit the data of the projection area to the terminal device to reduce the calculation amount of the terminal device and optimize The processing of terminal equipment.

Step 2430: Acquire image content corresponding to the projection area in the screen content according to the screen content to be displayed on the screen.

Step 2440: Perform specified processing on the image content and display the screen content containing the specified processed image content. The difference between the first hue of the specified processed image content and the second hue of the virtual content is greater than the first threshold .

In some embodiments, the above-mentioned process of specifying the image content includes any one or more of the following: overlaying the content of the specified color on the image content; adjusting the color of the image content to the specified color; and transparency of the image content The value is adjusted to the specified transparency value. The color difference between the specified color and the color of the virtual content is greater than the second threshold, and the difference between the specified transparency value and the transparency value of the virtual content is greater than the third threshold.

Among them, the overlay content may be a picture of a specified color, or a newly-created layer of a specified color, which is not limited herein, and only needs to cover the image content, and the presentation color may be the specified color.

The specified color can be a solid color, such as black, gray, etc. The second threshold can be set reasonably according to the user's viewing comfort. For example, when the color of the virtual content is blue, the specified color can be gray.

The specified transparency value can be used to reduce the display effect of the image content. The smaller the transparency value, the higher the transparency. If the specified transparency value can be set to 5, which is 50% transparent, it can also be set to 0, which is 100% transparent. The third threshold can be set reasonably according to the user's observation comfort, which is not limited here. For example, the third threshold is set to 5, when the transparency value of the virtual content is 10, that is, 0% transparency, the specified transparency value may be 1, that is, 90% transparency.

As an implementation manner, the transparency value of the image content may also be set to be less than a preset threshold, for example, the preset threshold may be 1 or 2, and so on.

As an embodiment, the head-mounted display device may only include a display module, a communication module, and a camera for display, and control of the displayed virtual content may be performed by a processor, memory, etc. of the terminal device.

In some embodiments, the terminal device includes a manipulation area, and when the manipulation area detects a manipulation operation, a control instruction may be generated according to the manipulation operation, the virtual content displayed in the head-mounted display device may be adjusted according to the control instruction, and the adjusted virtual content The corresponding display data is sent to the head-mounted display device. The terminal device may reacquire the projection area based on the adjusted virtual content, and specify the image content corresponding to the newly acquired projection area in the screen content.

As shown in FIGS. 23 and 28, the user views the virtual animal 233 superimposed on the screen of the terminal device through the worn head-mounted display device, can move the virtual animal 233 to the right, and re-determines the projection area of the virtual animal 233 on the screen, The color of the image content 235 corresponding to the projection area in the screen content is adjusted to gray, and the virtual animal is highlighted, so that the user can clearly observe the virtual animal 233 all the time, reducing the interference of the screen content on the virtual animal.

Referring to FIG. 29, an embodiment of the present application provides an image processing method, which is applied to a head-mounted display device, and the head-mounted display device is in communication connection with a terminal device. The method includes the following steps.

Step 2910: Display the virtual content.

Step 2920: Obtain the relative spatial position relationship between the terminal device and the head-mounted display device.

Step 2930: Acquire the projection area of the virtual content on the screen of the terminal device according to the relative spatial position relationship. The projection area is the projection area of the virtual content on the screen observed by the human eye through the head-mounted display device.

Step 2940: Send the data of the projection area to the terminal device. The data of the projection area is used to instruct the terminal device to specify the image content corresponding to the projection area in the screen content to be displayed, and display the content containing the image content after the specified processing. The screen content specifies the difference in tone between the first hue of the processed image content and the second hue of the virtual content is greater than the first threshold.

The image processing method provided by the above embodiment reduces the interference of the screen content displayed by the terminal device on the virtual content when the virtual content is displayed on the head-mounted display device, thereby highlighting the virtual content and improving the realism and display effect of the virtual content in augmented reality . Moreover, only the screen content corresponding to the projection area of the virtual content on the screen is processed, while the virtual content is highlighted, the normal display of other screen content is ensured, which is convenient for the user to interact with the mobile terminal.

In one embodiment, a terminal device is provided, and the terminal device may be an electronic device capable of running an application program, such as a smartphone or a tablet computer. The terminal device may include one or more of the following components: a processor and a memory, where the memory stores one or more application programs, the one or more application programs may be configured to be executed by one or more processors, one or Multiple programs are configured to perform the method as described in the foregoing method embodiments.

The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire terminal device, executes each instruction of the terminal device by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory Kinds of functions and processing data. Alternatively, the processor may be implemented in at least one hardware form of digital signal processing, field programmable gate array, and programmable logic array. The processor may integrate one or a combination of a central processor, an image processor, and a modem. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used for rendering and rendering of the displayed content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 120, and may be implemented by a communication chip alone.

The memory may include random access memory or read-only memory. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory 130 may include a storage program area and a storage data area, where the storage program area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , Instructions for implementing the following method embodiments. The storage data area can also store data created by the terminal device in use, and the like.

As shown in FIG. 30, a head-mounted display device is provided, which includes a camera for collecting images of real objects and collecting scene images of a target scene. The camera may be an infrared camera or a visible light camera, and the specific type is not limited.

In one embodiment, the first terminal 100 may further include one or more components as follows: a display module, an optical module, a communication module, and a power supply. The display module may include a display control unit for receiving a display image of the virtual content rendered by the processor, displaying and projecting the display image onto the optical module, so that the user can view the virtual content through the optical module . Among them, the display module may be a display screen or a projection device, etc., for displaying images. The optical module may use an off-axis optical system or a waveguide optical system. After the display image displayed by the display module passes through the optical module, it can be projected to the user's eyes. The user can see the display image projected by the display module through the optical module. In some embodiments, the user can also observe the real environment through the optical module and feel the visual effect of the superimposed virtual content and the real environment. The communication module may be a module such as Bluetooth, WiFi, or ZigBee. The terminal device may communicate with the interactive device through the communication module to exchange information and instructions. The power supply can supply power to the entire terminal equipment to ensure the normal operation of various components of the terminal equipment.

In some embodiments, the head-mounted display device may further include a processor and a memory, where the memory stores one or more application programs, the one or more application programs may be configured to be executed by one or more processors, one Or more programs are configured to perform the method as described in the foregoing method embodiments.

In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores program code, and the program code may be called by a processor to execute the method described in the foregoing method embodiments.

The computer-readable storage medium may be an electronic memory such as flash memory, EEPROM, EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-volatile computer-readable medium. The computer-readable storage medium 800 has a storage space for the program code 810 that performs any of the method steps described above. These program codes can be read from or written into one or more computer program products. The program code 810 may be compressed in an appropriate form, for example.

Claims (22)

1. A virtual content interaction system is characterized by comprising a terminal device and a head-mounted display device, the terminal device establishing a communication connection with the head-mounted display device, wherein,

   The terminal device is configured to acquire first relative spatial position information of the head-mounted display device and the terminal device, determine the display position of the virtual content according to the first relative spatial position information, and render according to the display position The virtual content, and obtaining display data of the virtual content, and transmitting the display data to the head-mounted display device;

   The head-mounted display device is configured to receive the display data sent by the terminal device and display the virtual content according to the display data.
2. The system according to claim 1, wherein the terminal device is provided with a marker;

   The head-mounted display device is also used to collect an image containing the marker through a camera and send the image to the terminal device;

   The terminal device is further configured to receive the image sent by the head-mounted display device, and identify markers contained in the image, to obtain a first relative space between the head-mounted display device and the terminal device position.
3. The system according to claim 1, wherein the terminal device is further configured to receive a control operation, generate a control instruction according to the control operation, and adjust the virtual content according to the control instruction, and adjust the adjusted virtual The display data of the content is transmitted to the head-mounted display device;

   The head-mounted display device is further configured to receive the display data of the adjusted virtual content, and update the displayed virtual content according to the adjusted display data of the virtual content.
4. The system according to claim 3, wherein the control operation includes a manipulation operation detected by a manipulation area of a terminal device, a user gesture collected by a head-mounted display device or a terminal device, and a control connected to the terminal device The controller receives at least one of the control operations.
5. The system according to claim 1, wherein the terminal device is further used to obtain screen content displayed on the screen and extended content corresponding to the screen content, and obtain virtual content display according to the screen content and the extended content data;

   The head-mounted display device is also used to display virtual content including the screen content and the extended content according to the display data.
6. The system according to claim 5, wherein the terminal device is further used to obtain a second relative spatial position between the virtual content and the terminal device, and according to the first relative spatial position and The second relative spatial position determines the display position of the virtual content.
7. The system according to claim 6, wherein the terminal device is further configured to determine the screen content of the overlapping portion of the screen when the display position of the virtual content overlaps with the screen of the terminal device, And determine the corresponding display area of the screen content of the overlapping portion in the virtual content, and perform preset processing on the display area, after the virtual content is displayed on the head-mounted display device, after the preset processing The content of the display area is invisible.
8. The system according to claim 1, wherein the terminal device is further configured to obtain a projection area of the virtual content displayed by the head-mounted display device on the screen of the terminal device according to the first relative spatial position , The projection area is the projection area of the virtual content observed on the screen through the head-mounted display device, obtains the image content corresponding to the projection area in the screen content to be displayed, and The image content is subjected to designation processing, and the screen content containing the image content after the designation processing is displayed on the screen, and the difference between the first color tone of the image content after the designation processing and the second color tone of the virtual content is greater than The first threshold.
9. The system according to claim 8, wherein the terminal device is further configured to acquire a left-eye image and a right-eye image for displaying virtual content in the head-mounted display device, and according to the left-eye image and the Acquiring the first relative spatial position, acquiring a first projection area of the left-eye image on the screen of the terminal device, and acquiring the right-eye image at all positions according to the right-eye image and the first relative spatial position A second projection area on the screen to determine a projection area of the virtual content on the screen, the projection area being a composite area of the first projection area and the second projection area.
10. The system according to claim 8, wherein the designated processing includes at least one of the following processing:

    Overlay the overlay content of the specified color on the image content;

    Adjust the color of the image content to the specified color; and

    Adjusting the transparency value of the image content to a specified transparency value;

    Wherein, the color difference between the specified color and the color of the virtual content is greater than a second threshold, and the difference between the specified transparency value and the transparency value of the virtual content is greater than a third threshold.
11. A virtual content interaction method is characterized in that it is applied to a terminal device that is in communication connection with an external head-mounted display device. The method includes:

    Acquiring a first relative spatial position of the head-mounted display device and the terminal device;

    Determine the display position of the virtual content according to the first relative spatial position;

    Rendering the virtual content according to the display position, and obtaining display data of the virtual content; and

    Transmitting the display data to the head-mounted display device, where the display data is used to instruct the head-mounted display device to display the virtual content.
12. The method according to claim 11, wherein the determining the display position of the virtual content according to the first relative spatial position comprises:

    Acquiring the second relative spatial position of the virtual content and the terminal device; and

    The display position of the virtual content is determined according to the first relative spatial position and the second relative spatial position.
13. The method according to claim 11, wherein after the transmitting the virtual content to the head-mounted display device, the method further comprises:

    Receiving a control operation and generating a control instruction according to the control operation; and

    According to the control instruction, the virtual content is controlled.
14. The method according to claim 13, wherein the control operation includes a manipulation operation detected by a manipulation area of a terminal device, a user gesture collected by a head-mounted display device or a terminal device, and a control connected to the terminal device The controller receives at least one of the control operations.
15. The method according to claim 13, wherein the generating a control instruction according to the control operation includes:

    A control instruction is generated according to at least one of the detected number of fingers when the control operation is performed, the gesture action when the control operation is performed, and the finger sliding trajectory when the control operation is performed.
16. The method according to claim 11, wherein the terminal device includes a marker, and the acquiring the first relative spatial position between the head-mounted display device and the terminal device includes:

    Receiving an image containing a marker sent by the head-mounted display device, the image being acquired by the head-mounted display device through a camera; and

    Identify the marker contained in the image, and obtain the first relative spatial position between the head-mounted display device and the terminal device.
17. The method according to claim 11, wherein the method further comprises:

    Acquiring a projection area of the virtual content displayed by the head-mounted display device on the screen of the terminal device according to the first relative spatial position, the projection area being the virtual content observed by the head-mounted display device The projection area on the screen;

    Acquiring image content corresponding to the projection area in the screen content to be displayed, and performing specified processing on the image content; and

    The screen content containing the designated processed image content is displayed through the screen, and the difference in tone between the first tone of the specified processed image content and the second tone of the virtual content is greater than the first threshold.
18. A virtual content interaction method is characterized in that it is applied to a head-mounted display device, and the head-mounted display device establishes a communication connection with a terminal device, including:

    Collect an image containing a marker through the camera, the marker is provided on the terminal device;

    Transmitting the marker image to the terminal device, where the image is used to instruct the terminal device to recognize the marker contained in the image, and obtain the first of the head-mounted display device and the terminal device Relative spatial position; and

    Receiving display data sent by the terminal device, and displaying virtual content according to the display data, the display data being obtained by the terminal device rendering the virtual content according to the first relative spatial position.
19. The method according to claim 18, wherein after the image containing the marker is collected by the camera, the method further comprises:

    Identify the markers contained in the image, and obtain the first relative spatial position of the head-mounted display device and the terminal device;

    Receiving screen content displayed on the screen of the terminal device from the terminal device, and extended content corresponding to the screen content; and

    Generate virtual content according to the screen content and extended content, and render virtual content including the screen content and extended content according to the first relative spatial position.
20. The method of claim 19, further comprising:

    When the display position of the virtual content overlaps with the screen of the terminal device, determine the screen content of the overlapping portion of the screen;

    Determine the corresponding display area of the screen content of the overlapping portion in the virtual content; and

    Perform preset processing on the display area, and when the virtual content is displayed on the head-mounted display device, the content of the display area after the preset processing is in an invisible state.
21. A virtual content display method is characterized in that it is applied to a first terminal, and the first terminal is in communication connection with a second terminal. The method includes:

    Acquiring first relative spatial position information between the first terminal and the second terminal;

    Acquiring display content data from the second terminal, wherein the display content data includes at least data of display content currently displayed by the second terminal; and

    The virtual content is displayed according to the first relative spatial position information and the display content data, where the virtual content includes the display content displayed by the second terminal and the extended content corresponding to the display content.
22. An image processing method, characterized in that it is applied to a terminal device that is in communication connection with a head-mounted display device, the method includes:

    Acquiring the relative spatial position relationship between the terminal device and the head-mounted display device;

    Acquiring a projection area of the virtual content displayed on the head-mounted display device on the screen of the terminal device according to the relative spatial position relationship, the projection area being the virtual content observed through the head-mounted display device The projection area on the screen;

    Acquiring image content corresponding to the projection area in the screen content according to the screen content to be displayed on the screen; and

    Performing a designated process on the image content, and displaying a screen content containing the designated processed image content, the difference in tone between the first hue of the designated processed image content and the second hue of the virtual content being greater than The first threshold.

Images