WO2023016107A1

WO2023016107A1 - Remote interaction method, apparatus and system, and electronic device and storage medium

Info

Publication number: WO2023016107A1
Application number: PCT/CN2022/101543
Authority: WO
Inventors: 秦禄东
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-08-10
Filing date: 2022-06-27
Publication date: 2023-02-16
Also published as: CN113672087A

Abstract

The present application relates to the technical field of data processing. Disclosed are a remote interaction method, apparatus and system, and an electronic device and a storage medium. The method comprises: acquiring, as first user data, user data of a user corresponding to a first head-mounted display device; processing the first user data, so as to obtain first data to be displayed; sending the first data to be displayed to the first head-mounted display device, so as to instruct the first head-mounted display device to display the first data to be displayed; and sending the first data to be displayed to a second head-mounted display device, so as to instruct the second head-mounted display device to display the first data to be displayed. By means of the present application, data to be displayed is obtained by means of user data of a user corresponding to a first head-mounted display device, and the data to be displayed is then respectively sent to the first head-mounted display device and a second head-mounted display device for display, such that the visual experience during interaction can be improved.

Description

Remote interaction method, device, system, electronic device and storage medium

Cross References to Related Applications

This application claims the priority of the Chinese application with application number CN202110915592.X filed on August 10, 2021, which is hereby incorporated by reference in its entirety for all purposes.

technical field

The present application relates to the field of interaction technologies, and more specifically, to a remote interaction method, device, system, electronic equipment, and storage medium.

Background technique

Due to the development of communication technology, remote interaction scenarios such as teleconferencing, remote assistance, and distance education have increasingly penetrated into people's lives.

Contents of the invention

In view of the above problems, the present application proposes a remote interaction method, device, system, electronic equipment and storage medium to solve the above problems.

In the first aspect, the embodiment of the present application provides a remote interaction method, which is applied to an electronic device, and the electronic device is connected to at least two head-mounted display devices, wherein the at least two head-mounted display devices include the first A head-mounted display device and a second head-mounted display device, the method includes: acquiring user data of a user corresponding to the first head-mounted display device as first user data; performing processing to obtain first data to be displayed; sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed; sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

In the second aspect, the embodiment of the present application provides a remote interaction method, which is applied to a first head-mounted display device, the first head-mounted display device is connected to an electronic device, and the electronic device is also connected to a second head-mounted display device. The method includes: collecting the user data of the user corresponding to the first head-mounted display device as the first user data; sending the first user data to the electronic device to indicate the The electronic device processes the first user data to obtain the first data to be displayed, and sends the first data to be displayed to the first head-mounted display device for display and the second head-mounted display device respectively. displaying by the display device; receiving the first data to be displayed sent by the electronic device, and displaying the first data to be displayed.

In the third aspect, the embodiment of the present application provides a remote interaction device, which is applied to electronic equipment, and the electronic equipment is respectively connected to the first head-mounted display device and the second head-mounted display device, and the device includes: a user The data obtaining module is used to obtain the user data of the user corresponding to the first head-mounted display device as the first user data; the data to be displayed obtaining module is used to process the first user data to obtain the first user data. Data to be displayed: a first data to be displayed sending module, configured to send the first data to be displayed to the first head-mounted display device, so as to instruct the first head-mounted display device to display the first Data to be displayed; a second data to be displayed sending module, configured to send the first data to be displayed to the second head-mounted display device, so as to instruct the second head-mounted display device to display the first Data to be displayed.

In a fourth aspect, the embodiment of the present application provides a remote interaction device, which is applied to a first head-mounted display device, the first head-mounted display device is connected to an electronic device, and the electronic device is also connected to a second head-mounted display device. connected to a head-mounted display device, the device includes: a user data acquisition module, configured to collect user data of a user corresponding to the first head-mounted display device as first user data; a user data sending module, configured to transmit the Send the first user data to the electronic device to instruct the electronic device to process the first user data to obtain first data to be displayed, and send the first data to be displayed to the first head The wearable display device performs display and the second head-mounted display device performs display; a data display module is configured to receive the first data to be displayed sent by the electronic device, and display the first data to be displayed.

In the fifth aspect, the embodiment of the present application provides a remote interaction system, the remote interaction system includes an electronic device and at least two head-mounted display devices, and the electronic device is connected to the at least two head-mounted display devices , wherein the at least two head-mounted display devices include a first head-mounted display device and a second head-mounted display device, wherein: the electronic device is configured to acquire the corresponding The user data of the user is used as the first user data; the electronic device is used to process the first user data to obtain the first data to be displayed; the electronic device is used to process the first data to be displayed The data is sent to the first head-mounted display device; the first head-mounted display device is used to display the first data to be displayed; the electronic device is used to send the first data to be displayed To the second head-mounted display device; the second head-mounted display device is configured to display the first data to be displayed.

In a sixth aspect, the embodiment of the present application provides an electronic device, including a memory and a processor, the memory is coupled to the processor, and the memory stores instructions, which are executed when the instructions are executed by the processor. The processor executes the above method.

In a seventh aspect, the embodiment of the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, and the program code can be invoked by a processor to execute the above method.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 shows a schematic diagram of an application environment of a remote interaction method provided by an embodiment of the present application;

Fig. 2 shows a schematic diagram of another application environment of the remote interaction method provided by the embodiment of the present application;

FIG. 3 shows a schematic diagram of the remote interaction method that can be used in the embodiment of the present application.

FIG. 4 shows a schematic flowchart of a remote interaction method provided by an embodiment of the present application;

Fig. 5 shows a schematic flowchart of a remote interaction method provided by another embodiment of the present application;

FIG. 6 shows a schematic diagram of a scene that can be used in the remote interaction method provided by the embodiment of the present application;

FIG. 7 shows a schematic flowchart of step S201 of the remote interaction method shown in FIG. 5 of the present application;

Fig. 8 shows a schematic flowchart of a remote interaction method provided by another embodiment of the present application;

FIG. 9 shows a schematic diagram of a teleconference scenario that can be used in the embodiment of the present application;

Fig. 10 shows a schematic diagram of a remote teaching scenario that can be used in the embodiment of the present application;

Fig. 11 shows a schematic flowchart of a remote interaction method provided by another embodiment of the present application;

Fig. 12 shows a schematic flowchart of a remote interaction method provided by still another embodiment of the present application;

Fig. 13 shows another schematic diagram that can be used in the remote teaching scenario provided by the embodiment of the present application;

Fig. 14 shows a schematic flow chart of a remote interaction method provided by yet another embodiment of the present application;

Fig. 15 shows a schematic flowchart of a remote interaction method provided by still another embodiment of the present application;

Fig. 16 shows a module block diagram of a remote interaction device provided by an embodiment of the present application;

Fig. 17 shows a module block diagram of a remote interaction device provided by another embodiment of the present application;

FIG. 18 shows a block diagram of an electronic device for performing a remote interaction method according to an embodiment of the present application;

Fig. 19 shows a storage unit used to store or carry program codes for realizing the remote interaction method according to the embodiment of the present application according to the embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

At present, in the remote interaction, the audio and video information of one party in the remote interaction is usually sent to the other party in the remote interaction. This interaction method will cause the interaction to be boring, and the visual experience of the interaction is not good. In view of the above problems, the inventor found after long-term research, and proposed the remote interaction method, device, system, electronic device and storage medium provided by the embodiment of the present application, through the user data of the user corresponding to the first head-mounted display device. The data to be displayed, and sending the data to be displayed to the first head-mounted display device and the second head-mounted display device for display, can improve the visual experience during interaction. Wherein, the specific remote interaction method is described in detail in the subsequent embodiments.

The following will describe an application environment that can be used for the remote interaction method provided by the embodiment of the present application.

Please refer to FIG. 1, which shows a schematic diagram of an application environment of the remote interaction method provided by the embodiment of the present application. The application scenario includes: an electronic device 100, a first head-mounted display device 200, and a second head-mounted display device. A device 300, wherein the electronic device 100 can be connected to at least two head-mounted display devices, wherein the at least two head-mounted display devices include the first head-mounted display device 200 and the second head-mounted display device 300, In this embodiment, the electronic device 100 is respectively connected to the first head-mounted display device 200 and the second head-mounted display device 300 to realize data interaction between the electronic device 100 and the first head-mounted display device 200, and electronic The device 100 interacts with the data of the second head-mounted display device 300 . Wherein, the number of the first head-mounted display device 200 may be one, and the number of the second head-mounted display device 300 may be one or more. The device 200 is connected to a second head-mounted display device 300 , and the electronic device 100 may also be connected to the first head-mounted display device 200 and multiple second head-mounted display devices 300 .

Please refer to FIG. 2, which shows a schematic diagram of another application environment of the remote interaction method provided by the embodiment of the present application. The application scenario includes: electronic device 100, a first head-mounted display device 200, a second head-mounted display device The display device 300 and the data collection device 40, and the electronic device are respectively connected to the first head-mounted display device 200, the second head-mounted display device 300, and the data collection device 400, so as to realize the connection between the electronic device 100 and the first head-mounted display device. 200, data interaction between the electronic device 100 and the second head-mounted display device 300, and data interaction between the electronic device 100 and the data collection device 400. Wherein, the data collection device 400 may be composed of at least one smart home device, and the data collection device 400 is used to collect the user data of the user corresponding to the first head-mounted display device 100, and send the collected user data to the electronic device 100 .

In the embodiment of the present application, the electronic device 100 may be a smart phone, a tablet computer, a desktop computer, a server and other devices with data processing functions, which is not limited here.

In the embodiment of the present application, the first head-mounted display device 200 and the second head-mounted display device 300 may be head-mounted display devices, or mobile devices such as mobile phones and tablets. When the first head-mounted display device 200 and the second head-mounted display device 300 are head-mounted display devices, for example, when they are head-mounted augmented reality devices or head-mounted virtual reality devices, the head-mounted display devices may be integrated head-mounted display device, the first head-mounted display device 200 and the second head-mounted display device 300 can also be smart terminals such as mobile phones connected to the external head-mounted display device, that is, the first head-mounted display device 200 and the second head-mounted display device 300 The second head-mounted display device 300 can be used as a processing and storage device of the head-mounted display device, plugged into or connected to an external head-mounted display device, and can display data (virtual objects) to be displayed in the head-mounted display device.

Please refer to FIG. 3 . FIG. 3 shows a schematic diagram of the remote interaction method that can be used in the embodiment of the present application. As shown in FIG. 3 , the first head-mounted display device 200 can be worn by a first user, the second head-mounted display device 300 can be worn by a second user, and the first head-mounted display device 200 and the head-mounted display device 300 can be worn by no one. The display device 300 can realize remote interaction through the electronic device 100 .

Please refer to FIG. 4 , which shows a schematic flowchart of a remote interaction method provided by an embodiment of the present application. The method is used to obtain the data to be displayed by using the user data of the user corresponding to the first head-mounted display device, and send the data to be displayed to the first head-mounted display device and the second head-mounted display device for display respectively, It can improve the visual experience during interaction. In a specific embodiment, the remote interaction method is applied to the remote interaction device 500 shown in FIG. 16 and the electronic device 100 ( FIG. 18 ) configured with the remote interaction device 500 . The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device can be a smart phone, a tablet computer, a desktop computer, a server, etc., and is not limited here. The process shown in Figure 4 will be described in detail below. In this embodiment, the electronic device is respectively connected to the first head-mounted display device and the second head-mounted display device. The remote interaction method may specifically include The following steps:

Step S101: Obtain user data of a user corresponding to the first head-mounted display device as first user data.

In this embodiment, the electronic device may acquire user data of the user corresponding to the first head-mounted display device, and use the acquired user data of the user corresponding to the first head-mounted display device as the first user data.

In some implementations, the first head-mounted display device may include a data collection module. The first head-mounted display device can detect the user data of the corresponding user through its data acquisition module, and send the detected user data to the electronic device. Correspondingly, the electronic device can obtain the user data sent by the first head-mounted display device. data, as the first user data.

For example, the data acquisition module may include motion sensors and audio sensors. The first head-mounted display device can detect the gesture information of the corresponding user through its motion sensor, and detect the voice information of the corresponding user through its audio sensor, and determine the detected gesture information and voice information as the user of the corresponding user. The data is sent to the electronic device. Correspondingly, the electronic device can obtain the user data sent by the first head-mounted display device as the first user data.

As a method, the first head-mounted display device is connected to the electronic device wirelessly, such as through WiFi, the first head-mounted display device can send the detected user data to the electronic device based on wireless transmission. equipment.

In some implementation manners, a data collection device may be provided around the first head-mounted display device. The data collection device can detect the user data of the user corresponding to the first head-mounted display device, and send the detected user data to the electronic device. Correspondingly, the electronic device can obtain the user data sent by the data collection device as the first user data. data.

For example, the data collection device consists of at least one smart home device, where the smart home device may include a smart speaker, a smart computer, a smart TV, a camera, a door and window sensor, and the like. The data acquisition device can collect the voice information, gesture information, image information, etc. of the user corresponding to the first head-mounted display device, and then determine the detected voice information, gesture information, image information, etc. as the corresponding user's user data and send it to The electronic device, correspondingly, the electronic device may obtain the user data sent by the data collection device as the first user data.

As a way, the data collection device and the electronic device are connected wirelessly, such as through WiFi, and the data collection device can send the detected user data to the electronic device based on wireless transmission.

In some implementations, the electronic device can set user data requirements. Wherein, when the demand characterization of user data set by the electronic device does not require complete image information of the user, the user data of the user may be collected by the first head-mounted display device, and the user data collected by the first head-mounted display device may be obtained. data as the first user data. Among them, when the demand characterization of the user data set by the electronic device requires the user's complete image information (such as a full-body image), the user data of the user can be collected through the data collection setting, and the user data collected by the data collection device can be obtained as the first User data.

Step S102: Process the first user data to obtain first data to be displayed.

In some implementations, after obtaining the first user data, the electronic device may process the first user data to obtain the data to be displayed as the first data to be displayed. For example, after the electronic device obtains the first user data, it may analyze, render and optimize the first user data, and obtain the data to be displayed as the first data to be displayed.

As a method, when the first user data is collected by the first head-mounted display device, then the first data to be displayed obtained by processing the first user data may include the corresponding The partial image information of the user may not include the image information of the user corresponding to the first head-mounted display device. Wherein, when the first data to be displayed includes partial image information of the user corresponding to the first head-mounted display device, the partial image information may include an image of the user's hand within the viewing angle of the first head-mounted display device Information, at this time, the first data to be displayed may be represented as including audio and video information including the user's hands presented in a certain scene (which may be a real scene or a virtual scene). Wherein, when the first data to be displayed does not include the image information of the user corresponding to the first head-mounted display device, the first data to be displayed may be presented in a certain scene as audio and video information that does not include the user image.

As a manner, when the first user data is collected by the data collection device, the first data to be displayed obtained by processing the first user data may include image information of the user's whole body. At this time, the first data to be displayed may be represented as audio and video information including the user's whole body presented in a certain scene (which may be a real scene or a virtual scene).

Step S103: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

In this embodiment, after the electronic device obtains the first data to be displayed, it can send the first data to be displayed to the first head-mounted display device. Correspondingly, the first head-mounted display device can receive the data sent by the electronic device. The first data to be displayed is displayed, and the first data to be displayed is displayed, so that the user corresponding to the first head-mounted display device can view his own data and make reference adjustments to improve interactive performance. Wherein, the first data to be displayed may be a resource in the real world, or a resource superimposed between a virtual object and the real world, which is not limited here.

Step S104: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

In this embodiment, after the electronic device obtains the first data to be displayed, it can send the first data to be displayed to the second head-mounted display device. Correspondingly, the second head-mounted display device can receive the data sent by the electronic device. The first data to be displayed is displayed, and the first data to be displayed is displayed, so that the user corresponding to the second head-mounted display device can view the data of the remote counterpart, thereby improving the interactive visual experience.

In the remote interaction method provided by an embodiment of the present application, the user data of the user corresponding to the first head-mounted display device is obtained as the first user data, and the first user data is processed to obtain the first data to be displayed, and the first The data to be displayed is sent to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed, and the first data to be displayed is sent to the second head-mounted display device to instruct the second head-mounted display device The wearable display device displays the first data to be displayed, thereby obtaining the data to be displayed through the user data of the user corresponding to the first head-mounted display device, and sending the data to be displayed to the first head-mounted display device and the second Displaying on a head-mounted display device can improve visual experience during interaction.

Please refer to FIG. 5 , which shows a schematic flowchart of a remote interaction method provided in another embodiment of the present application. The method is applied to the above-mentioned electronic device, and the electronic device is respectively connected with the first head-mounted display device, the second head-mounted display device, and the data acquisition device. The process shown in FIG. 5 will be described in detail below. The remote interaction method may specifically include the following steps:

Step S201: Obtain user data of a user corresponding to the first head-mounted display device sent by the data collection device as the first user data.

In this embodiment, the user data of the user corresponding to the first head-mounted display device may be collected by the data collection device, and the collected user data corresponding to the first head-mounted display device may be sent to the electronic device. Correspondingly, When the electronic device acquires the user data sent by the data collection device, the user data may be used as the first user data. Wherein, the data collection device may include a smart home device, or may include a smart terminal, a tablet computer, etc., which is not limited here.

Please refer to FIG. 6. FIG. 6 shows a schematic diagram of a scene applicable to the remote interaction method provided by the embodiment of the present application. As shown in FIG. 6 , the user corresponding to the first head-mounted display device 200 is located in a room, and the room includes a camera A1 and a TV A2. The camera A1 can be used to collect image information and posture information of the user, and the TV A2 can be used to Collect voice information of the user.

Please refer to FIG. 7 , which shows a schematic flowchart of step S201 of the remote interaction method shown in FIG. 5 of the present application. In this embodiment, the data acquisition device is composed of at least one smart home device. The following will describe the process shown in FIG. 7 in detail, and the method may specifically include the following steps:

Step S2011: Obtain at least one smart home device located within a preset range of the first head-mounted display device.

In some embodiments, the data collection device consists of at least one smart home device.

As a manner, the electronic device may preset and store a preset range, wherein the preset range may be used to represent a range in which user data can be effectively collected. That is to say, the smart home device whose distance from the first head-mounted display device is within the preset range can effectively collect the user data of the user corresponding to the first head-mounted display device, which is different from the first head-mounted display device. Smart home devices whose distance between the wearable display devices is outside the preset range cannot effectively collect the user data of the user corresponding to the first head-mounted display device, for example, the first head-mounted display device cannot be collected The user data of the corresponding user, or the collected user data of the user corresponding to the first head-mounted display device is inaccurate.

In this embodiment, at least one smart home device located within the preset range of the first head-mounted display device can be obtained. It can be understood that at least one smart home device located within the preset range of the first head-mounted display device The smart home device can effectively collect user data of the user corresponding to the first head-mounted display device.

As a manner, a circle may be established with the position of the first head-mounted display device as an origin and a preset range as a radius, and then at least one smart home device within the circle may be obtained.

As yet another manner, the room where the first head-mounted display device is located may be obtained, and at least one smart home device in the room may be obtained. As an implementable manner, the room where the first head-mounted display device is located can be obtained, and the size of the room can be obtained. When the size of the room is not larger than the preset range, at least one For smart home devices, when the room is larger than the preset range, at least one smart home device within the preset range in the room may be acquired.

Step S2012: When the at least one smart home device satisfies the preset data collection condition, acquire the user data of the user corresponding to the first head-mounted display device sent by the at least one smart home device as the first User data.

In some implementations, the electronic device may preset and store preset data collection conditions, and the preset data collection conditions are used as a basis for judging the at least one smart home device. Therefore, in this embodiment, after obtaining at least one smart home device, the at least one smart home device may be compared with preset data collection conditions to determine whether the at least one smart home device satisfies the preset data collection conditions.

Wherein, when the judging result indicates that the at least one smart home device satisfies the preset data collection condition, it means that the at least one smart home device can effectively collect the required user data, and the first head can be performed through at least one smart home device. The user data of the user corresponding to the head-mounted display device is collected, and the user data corresponding to the first head-mounted display device sent by at least one smart home device is acquired as the first user data.

Wherein, when the judging result indicates that the at least one smart home device does not meet the preset data collection conditions, it indicates that the at least one smart home device cannot effectively collect the required user data, and a prompt message can be output to the first head-mounted device. The display device is used to prompt the user to change the location of the first head-mounted display device, or it can be switched to collect user data through the first head-mounted display device, which is not limited here.

In some implementations, the preset data collection conditions may include: collection of image information, collection of audio information, and collection of posture information. Therefore, in this embodiment, after at least one smart home device is obtained, it can be judged whether the at least one home device can meet the requirements for image information collection, audio information collection, and gesture information collection. Wherein, when at least one smart home device can meet the collection of image information, audio information and gesture information, it can be determined that the at least one smart home device meets the preset data collection conditions, for example, when at least one smart home device When the device includes a camera (collection of image information), a smart phone (collection of attitude information), a smart speaker (collection of voice information), etc., it can be determined that the at least one smart home device satisfies the preset condition. Wherein, when at least one smart home device does not satisfy at least one of image information collection, audio information collection and attitude information collection, it may be determined that the at least one smart home device does not meet the preset condition.

Step S202: Process the first user data to obtain first data to be displayed.

Step S203: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

Step S204: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of step S202-step S204, please refer to step S102-step S104, which will not be repeated here.

In the remote interaction method provided by yet another embodiment of the present application, compared with the remote interaction method shown in FIG. Send user data to electronic devices, thereby improving the authenticity of collected user data.

Please refer to FIG. 8 . FIG. 8 shows a schematic flowchart of a remote interaction method provided by another embodiment of the present application. The method is applied to the above-mentioned electronic device, which is respectively connected to the first head-mounted display device and the second head-mounted display device. The process shown in FIG. 8 will be described in detail below. The remote interaction method is specifically Can include the following steps:

Step S301: Obtain user data of a user corresponding to the first head-mounted display device as first user data.

Step S302: Process the first user data to obtain first data to be displayed.

Step S303: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

Step S304: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of step S301-step S304, please refer to step S101-step S104, which will not be repeated here.

Step S305: Receive a virtual scene selection instruction sent by the first head-mounted display device, wherein the virtual scene selection instruction is triggered by a user corresponding to the first head-mounted display device.

In some implementations, the user corresponding to the first head-mounted display device may also trigger the scene selection when the first head-mounted display device triggers it. Correspondingly, when the first head-mounted display device detects that the user triggers the scene selection , then a virtual scene selection instruction may be generated in response to the scene selection triggered by the user. Wherein, when the first head-mounted display device generates the virtual scene selection instruction, it can send the virtual scene selection instruction to the electronic device, and correspondingly, the electronic device can receive the virtual scene selection instruction sent by the first head-mounted display device .

As a method, when the user needs to select a scene, the first head-mounted display device can display a plurality of selectable virtual scenes for the user to choose, and the user can select the desired scene from the displayed plurality of virtual scenes. scene, the first head-mounted display device may generate a virtual scene selection instruction based on the selection. For example, the multiple virtual scenarios may include remote meeting scenarios, remote teaching scenarios, remote assistance scenarios, etc., which are not limited here.

As an implementable manner, when the user needs to select a scene, the first head-mounted display device can display a plurality of selectable application programs for the user to select, and each application program in the multiple application programs corresponds to a virtual Scenes. As another implementable manner, when the user needs to select a scene, the first head-mounted display device may start the target application, and display multiple selectable virtual scenes in the target application for the user to choose.

Step S306: In response to the virtual scene selection instruction, determine a target virtual scene from multiple virtual scenes.

In some implementations, after receiving a virtual scene selection instruction, the electronic device may determine a target virtual scene from multiple virtual scenes in response to the virtual scene selection instruction. For example, assuming that multiple virtual scenes include a remote meeting scene, a remote teaching scene, and a remote assistance scene, and when the virtual scene selection instruction indicates that a remote meeting scene is selected, the remote meeting scene can be determined from the remote meeting scene, the remote teaching scene, and the remote assistance scene. The scene is the target virtual scene.

Step S307: Send the target virtual scene to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual scene.

In this embodiment, after obtaining the target virtual scene, the electronic device may send the target virtual scene to the first head-mounted display device, and accordingly, the first head-mounted display device may receive the target virtual scene sent by the electronic device, And display the target virtual scene, so that the user corresponding to the first head-mounted display device can experience the corresponding scene personally.

For example, when the target virtual scene is a teleconference scene, the first head-mounted display device may display a scene in which the user corresponding to the first head-mounted display device is located in the virtual conference room. Please refer to FIG. 9 . FIG. 9 shows a schematic diagram of a teleconferencing scenario that can be used in the embodiment of the present application. As shown in FIG. 9 , when the target virtual scene is a teleconference scene, the content displayed by the first head-mounted display device may include a scene in which user B1 corresponding to the first head-mounted display device is located in a virtual conference room. Of course, the content displayed by the first head-mounted display device may also include a scene in which the user B2 corresponding to the second head-mounted display device is located in the virtual meeting room.

For another example, when the target virtual scene is a remote teaching scene, the first head-mounted display device may display a scene in which the user corresponding to the first head-mounted display device is located in the virtual classroom. Please refer to FIG. 10 . FIG. 10 shows a schematic diagram that can be used in the remote teaching scenario provided by the embodiment of the present application. As shown in FIG. 10 , when the target virtual scene is a remote teaching scene, the content displayed by the first head-mounted display device may include a scene in which the user B1 corresponding to the first head-mounted display device is located in the virtual classroom. Of course, the content displayed by the first head-mounted display device may also include a scene where the user B2 corresponding to the second head-mounted display device is located in the virtual classroom.

As one way, the electronic device can send the first data to be displayed and the target virtual scene to the first head-mounted display device respectively, and the first head-mounted display device can render the first data to be displayed and the target virtual scene to display. As yet another manner, the electronic device may render the first data to be displayed and the target virtual scene and send them to the first head-mounted display device for display.

Step S308: Receive a virtual item selection instruction sent by the first head-mounted display device, wherein the virtual item selection instruction is triggered by a user corresponding to the first head-mounted display device in the target virtual scene.

In some implementations, the user corresponding to the first head-mounted display device may also trigger the selection of a virtual item in the target virtual scene. Correspondingly, when the first head-mounted display device detects that the user triggers the selection of a virtual item, Then a virtual item selection instruction may be generated in response to the user-triggered virtual item selection. Wherein, when the first head-mounted display device generates a virtual item selection instruction, it can send the virtual item selection instruction to the electronic device, and correspondingly, the electronic device can receive the virtual item selection instruction sent by the first head-mounted display device. .

As a method, when the user needs to select a virtual prop, the first head-mounted display device can display a plurality of selectable virtual props for the user to choose, and the user can select the desired virtual prop from the displayed plurality of virtual props. The first head-mounted display device may generate a virtual item selection instruction based on the selection. For example, when the target virtual scene is a teleconference scene, the multiple virtual props may include a virtual laser pointer, a virtual projection screen, a virtual microphone, and the like. When the target virtual scene is a remote teaching scene, the multiple virtual props may include a virtual writing board, a virtual writing pen, and the like.

Step S309: In response to the virtual item selection instruction, determine a target virtual item from a plurality of virtual items.

In some implementations, after receiving the virtual item selection instruction, the electronic device may determine the target virtual item from multiple virtual items in response to the virtual item selection instruction. For example, assuming that the target virtual scene is a teleconference scene, and multiple virtual props include a virtual laser pointer, a virtual projection screen, and a virtual microphone, when the virtual prop selection instruction indicates to select a virtual projection screen, the virtual laser pointer, virtual projection screen, and The virtual projection screen is determined as the target virtual prop in the virtual microphone.

Step S310: Send the target virtual item to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual item.

In this embodiment, after the electronic device obtains the target virtual item, it can send the target virtual item to the first head-mounted display device, and accordingly, the first head-mounted display device can receive the target virtual item sent by the electronic device, And display the target virtual prop, so that the user corresponding to the first head-mounted display device can personally enter the corresponding scene.

For example, when the target virtual scene is a teleconference scene and the target virtual prop is a virtual projection screen, the first head-mounted display device may display that the user corresponding to the first head-mounted display device is located in the virtual meeting with the virtual projection screen. room scene. Please continue to refer to FIG. 9. As shown in FIG. 9, when the target virtual scene is a teleconference scene, the content displayed by the first head-mounted display device may include that the user B1 corresponding to the first head-mounted display device is located in the virtual A scene in a meeting room, and a virtual projection screen C1 is set in the virtual meeting room.

When the target virtual scene is a remote teaching scene and the target virtual prop is a virtual writing board, the first head-mounted display device may display the information of the user corresponding to the first head-mounted display device located in the virtual classroom equipped with a virtual writing board. Scenes. Please continue to refer to FIG. 10. As shown in FIG. 10, when the target virtual scene is a remote teaching scene, the content displayed by the first head-mounted display device may include that the user B1 corresponding to the first head-mounted display device is located in the virtual A scene in a classroom, and a virtual writing board C2 is set in the virtual classroom.

As a method, the electronic device can send the first data to be displayed, the target virtual scene, and the target virtual prop to the first head-mounted display device respectively, and the first head-mounted display device can send the first data to be displayed, the target virtual scene The scene and target virtual props are rendered and then displayed. As yet another manner, the electronic device may render the first data to be displayed, the target virtual scene, and the target virtual prop and send it to the first head-mounted display device for display.

As an implementation manner, after obtaining the target virtual item, the electronic device may render the target virtual item based on the gesture information of the user corresponding to the first head-mounted display device, and then send the target virtual item to the first head-mounted display device for display.

Wherein, the posture information represents a view matrix, that is, when the user views an object in the real world or the virtual world through the head-mounted display device, the corresponding views are different from different directions or angles. Corresponding to this embodiment, the gesture information represents the corresponding view when the user views the target virtual prop from the direction or angle corresponding to the gesture information when viewing the target virtual prop through the first head-mounted display device. Therefore, as an implementable manner, the electronic device may render the target virtual prop based on the posture information, so as to obtain the corresponding display parameters of the target virtual prop under the posture information. Specifically, when the electronic device renders the target virtual prop, it will extract the texture of a frame image of the target virtual prop, and then build a 2D model, which will combine the pose information and put the extracted image texture into the 2D model Then perform corresponding rendering processing to obtain corresponding display parameters, and then perform coordinate conversion from 2D to 3D and then send it to the first head-mounted display device for display.

Step S311: Receive input information sent by the first head-mounted display device, wherein the input information is input by a user corresponding to the first head-mounted display device on the target virtual prop.

In some implementations, the user corresponding to the first head-mounted display device can also input information on the target virtual prop. Correspondingly, when the first head-mounted display device detects that the user performs information input on the target virtual prop , then the input information can be obtained and sent to the electronic device, and correspondingly, the electronic device can receive the input information sent by the first head-mounted display device. It can be understood that, at this time, the input information may be correspondingly displayed on the virtual prop of the first head-mounted display device.

As a method, when the target virtual prop is a virtual writing board, the input information can be written by the user on the virtual writing board; when the target virtual prop is a virtual projection screen, the input information can be triggered by the user to deliver onto the virtual projection screen.

Step S312: Send the input information to the second head-mounted display device to instruct the second head-mounted display device to correspondingly display the input information and the target virtual prop.

In this embodiment, after the electronic device obtains the input information sent by the first head-mounted display device, it can send the input information to the second head-mounted display device, and accordingly, the second head-mounted display device can receive Input information sent by electronic devices and displayed on virtual props to achieve more instant interaction.

For example, if the target virtual scene is a remote teaching scene, the user corresponding to the first head-mounted display device is a teacher, and the user corresponding to the second head-mounted display device is a student as an example, then the teacher can write on the virtual board, Correspondingly, the content of the blackboard writing will be sent to the second head-mounted display device for students to learn, thereby realizing more immediate learning.

Step S313: Send the target virtual item to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual item.

In this embodiment, after the electronic device obtains the target virtual item, it can send the target virtual item to the second head-mounted display device, and accordingly, the second head-mounted display device can receive the target virtual item sent by the electronic device, And display the target virtual props, so that the user corresponding to the second head-mounted display device can experience the corresponding scene personally, and make the props of the first head-mounted display device and the second head-mounted display device share , to enhance the interactive experience.

For example, when the target virtual scene is a teleconference scene and the target virtual prop is a virtual projection screen, the second head-mounted display device may display that the user corresponding to the first head-mounted display device is located in the virtual meeting with the virtual projection screen. The scene in the room; when the target virtual scene is a remote teaching scene and the target virtual prop is a virtual writing board, the second head-mounted display device can display that the user corresponding to the first head-mounted display device is located The scene in the virtual classroom.

As a method, the electronic device can send the first data to be displayed, the target virtual scene, and the target virtual prop to the second head-mounted display device respectively, and the second head-mounted display device can control the first data to be displayed, the target virtual scene, and the second head-mounted display device. The scene and target virtual props are rendered and then displayed. As yet another manner, the electronic device may render the first data to be displayed, the target virtual scene, and the target virtual prop and send it to the second head-mounted display device for display.

As an implementation manner, after obtaining the target virtual item, the electronic device may render the target virtual item based on the gesture information of the user corresponding to the second head-mounted display device, and then send the target virtual item to the second head-mounted display device for display.

Step S314: Send the target virtual scene to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual scene.

In this embodiment, after obtaining the target virtual scene, the electronic device may send the target virtual scene to the second head-mounted display device, and accordingly, the second head-mounted display device may receive the target virtual scene sent by the electronic device, And display the target virtual scene, so that the user corresponding to the second head-mounted display device can experience the corresponding scene personally, and make the scene sharing between the first head-mounted display device and the second head-mounted display device , to enhance the interactive experience.

For example, when the target virtual scene is a teleconference scene, the second head-mounted display device may display a scene in which the user corresponding to the first head-mounted display device is located in a virtual conference room; , the second head-mounted display device may display a scene in which the user corresponding to the first head-mounted display device is located in the virtual classroom.

As a method, the electronic device can send the first data to be displayed and the target virtual scene to the second head-mounted display device respectively, and the second head-mounted display device can render the first data to be displayed and the target virtual scene to display. As yet another manner, the electronic device may render the first data to be displayed and the target virtual scene and send them to the second head-mounted display device for display.

In the remote interaction method provided by another embodiment of the present application, compared with the remote interaction method shown in FIG. 3 , this embodiment also receives a virtual scene selection instruction sent by the first head-mounted display device, and in response to the virtual scene selection instruction, Determine the target virtual scene from multiple virtual scenes, and send the target virtual scene to the first head-mounted display device and the second head-mounted display device for display, so as to select and interactively display scenes according to user needs, so as to improve The experience of scene interaction. Furthermore, this embodiment also receives a virtual item selection instruction sent by the first head-mounted display device, and in response to the virtual item selection instruction, determines the target virtual item from multiple virtual items, and sends the target virtual item to the first head-mounted display device respectively. The wearable display device and the second head-mounted display device perform display, so as to select and interactively display props according to user needs, so as to enhance the experience of prop interaction. In addition, in this embodiment, the input information of the first head-mounted display device is received and displayed, so as to realize real-time interaction of the input information and improve user interaction experience.

Please refer to FIG. 11 , which shows a schematic flowchart of a remote interaction method provided by another embodiment of the present application. This method is applied to the above-mentioned electronic device, which is respectively connected to the first head-mounted display device and the second head-mounted display device. The process shown in FIG. 11 will be described in detail below. The remote interaction method is specifically Can include the following steps:

Step S401: Obtain user data of a user corresponding to the first head-mounted display device as first user data.

Step S402: Process the first user data to obtain first data to be displayed.

Step S403: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

Step S404: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of steps S401-S404, please refer to step S101-step S104, which will not be repeated here.

Step S405: Receive a virtual scene selection instruction sent by the first head-mounted display device, wherein the virtual scene selection instruction is triggered by a user corresponding to the first head-mounted display device.

Step S406: In response to the virtual scene selection instruction, determine a target virtual scene from multiple virtual scenes.

Step S407: Send the target virtual scene to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual scene.

Wherein, for the specific description of step S405-step S407, please refer to step S305-step S307, which will not be repeated here.

Step S408: Send the target virtual scene to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual scene.

Wherein, for the specific description of step S408, please refer to step S314, which will not be repeated here.

Step S409: Receive feedback information sent by the second head-mounted display device, wherein the feedback information is triggered by a user corresponding to the second head-mounted display device in the target virtual scene.

In some implementations, the user corresponding to the second head-mounted display device can also trigger information feedback in the target virtual scene. When giving feedback, the feedback information can be acquired and sent to the electronic device, and accordingly, the electronic device can receive the feedback information sent by the second head-mounted display device.

As a manner, when the target virtual scene is a teleconference scene, the feedback information triggered by the user corresponding to the second head-mounted display device in the target virtual scene may include: question information, reply information, and the like. As another way, when the target virtual scene is a remote teaching scene, the feedback information triggered by the user corresponding to the second head-mounted display device in the target virtual scene may include: hand-raising information, question information, answer information, Completed job information, etc.

Step S410: Send the feedback information to the first head-mounted display device to instruct the first head-mounted display device to display the feedback information.

In this embodiment, after obtaining the feedback information sent by the second head-mounted display device, the electronic device can send the feedback information to the first head-mounted display device, and accordingly, the first head-mounted display device can receive Feedback information sent by electronic devices and displayed to achieve more instant interaction.

For example, if the target virtual scene is a remote teaching scene, the user corresponding to the first head-mounted display device is a teacher, and the user corresponding to the second head-mounted display device is a student as an example, then the students can upload their completed homework as feedback information Correspondingly, the feedback information will be instantly sent to the first head-mounted display device for review by the teacher, thereby realizing more immediate learning. Wherein, after the review is completed, the teacher can return the reviewed homework to the second head-mounted display device, which will not be repeated here.

In the remote interaction method provided by another embodiment of the present application, compared with the remote interaction method shown in FIG. 3 , this embodiment also receives a virtual scene selection instruction sent by the first head-mounted display device, and in response to the virtual scene selection instruction, Determine the target virtual scene from multiple virtual scenes, and send the target virtual scene to the first head-mounted display device and the second head-mounted display device for display, so as to select and interactively display scenes according to user needs, so as to improve The experience of scene interaction. In addition, this embodiment also receives feedback information from the second head-mounted display device, and sends the feedback information to the first head-mounted display device for display, so as to realize real-time interaction of feedback information and improve user interaction experience.

Please refer to FIG. 12 . FIG. 12 shows a schematic flowchart of a remote interaction method provided by still another embodiment of the present application. This method is applied to the above-mentioned electronic equipment, and the electronic equipment is respectively connected to the first head-mounted display device and the second head-mounted display device. The process shown in FIG. 12 will be described in detail below. The remote interaction method is specifically Can include the following steps:

Step S501: Obtain user data of a user corresponding to the first head-mounted display device as first user data.

Step S502: Process the first user data to obtain first data to be displayed.

Step S503: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

Step S504: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of step S501-step S504, please refer to step S101-step S104, which will not be repeated here.

Step S505: Receive a data adjustment instruction sent by the second head-mounted display device, wherein the data adjustment instruction is triggered by a user corresponding to the second head-mounted display device.

Wherein, when the second head-mounted display device is displaying the first data to be displayed, the user corresponding to the second head-mounted display device can select the part that he or she pays more attention to from the first data to be displayed, and display it on the second head-mounted display device. The mounted display device focuses on displaying the more interesting part. For example, the second head mounted display device may enlarge the more focused part, or improve the definition of the more focused part, etc., which is not limited herein. For example, the user corresponding to the first head-mounted display device may be selected from the first data to be displayed as a part of more attention, and the explanation scene of the user corresponding to the first head-mounted display device may be selected from the first data to be displayed as For the part of more attention, the content demonstrated by the user corresponding to the first head-mounted display device may also be selected from the first data to be displayed as the part of more attention, etc.

In some implementations, the user corresponding to the second head-mounted display device can also trigger the data adjustment when the second head-mounted display device triggers it. Correspondingly, when the second head-mounted display device detects that the user triggers the data adjustment , then a data adjustment instruction can be generated in response to the data adjustment triggered by the user. Wherein, when the second head-mounted display device generates the data adjustment instruction, it can send the data adjustment instruction to the electronic device, and correspondingly, the electronic device can receive the data adjustment instruction sent by the second head-mounted display device.

Step S506: Adjust the first data to be displayed in response to the data adjustment instruction to obtain second data to be displayed.

In this embodiment, after receiving the data adjustment instruction, the electronic device may adjust the first data to be displayed in response to the data adjustment instruction, and obtain the second data to be displayed.

In some implementations, after receiving the data adjustment instruction, the electronic device can analyze the data adjustment instruction to obtain the part that the user focuses on in the first data to be displayed, and then based on the data adjustment instruction, the first data to be displayed The part of the data that the user focuses on is highlighted to obtain the second data to be displayed. Among them, highlighting the part that the user focuses on in the first data to be displayed may include: enlarging the part that the user focuses on, deleting the part that the user does not focus on and retaining the part that the user focuses on, and narrowing the part that the user does not focus on. This is not limited.

Step S507: Send the second data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the second data to be displayed.

In this embodiment, after the electronic device obtains the second data to be displayed, it can send the second data to be displayed to the second head-mounted display device. Correspondingly, the second head-mounted display device can receive the data sent by the electronic device. The second data to be displayed, and display the second data to be displayed, so that the user corresponding to the second head-mounted display device can view the part that he focuses on, making the displayed content more targeted and personalized, and improving user use experience.

Please refer to FIG. 13 . FIG. 13 shows another schematic diagram that can be used in the remote teaching scenario provided by the embodiment of the present application. As shown in FIG. 13, when the target virtual scene is a remote teaching scene, the content displayed by the first head-mounted display device may include a scene in which the user B1 corresponding to the first head-mounted display device is located in the virtual classroom, and A virtual tablet C2 is set in the virtual classroom. It can be understood that, compared with the first data to be displayed shown in FIG. 10 , the third data to be displayed shown in FIG. 13 highlights the part of the teacher's lecture.

Yet another embodiment of the present application provides a remote interaction method. Compared with the remote interaction method shown in FIG. 3 , this embodiment also receives a data adjustment instruction sent by the second head-mounted display device, and responds to the data adjustment instruction to the first The first data to be displayed is adjusted to obtain the second data to be displayed, so that the user can select the data that the user focuses on in the data to be displayed according to his own needs, which has high selectivity and strong pertinence.

Please refer to FIG. 14 , which shows a schematic flowchart of a remote interaction method provided in yet another embodiment of the present application. This method is applied to the above-mentioned electronic device, which is respectively connected to the first head-mounted display device and the second head-mounted display device. The process shown in FIG. 14 will be described in detail below. The remote interaction method specifically Can include the following steps:

Step S601: Obtain user data of a user corresponding to the first head-mounted display device as first user data.

Step S602: Process the first user data to obtain first data to be displayed.

Step S603: Sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of step S601-step S603, please refer to step S101-step S103, which will not be repeated here.

Step S604: Receive a user viewing instruction sent by the first head-mounted display device, wherein the user viewing instruction is triggered by a user corresponding to the first head-mounted display device.

In some implementation manners, the user corresponding to the first head-mounted display device can also trigger the user to view the interactive party on the first head-mounted display device, and correspondingly, the first head-mounted display device detects that the user triggers the interaction When the user checks, a user check instruction may be generated in response to the user check triggered by the user. Wherein, when the first head-mounted display device generates the user viewing instruction, it may send the user viewing instruction to the electronic device, and correspondingly, the electronic device may receive the user viewing instruction sent by the first head-mounted display device.

Step S605: In response to the user viewing instruction, acquire user data of a user corresponding to the second head-mounted display device as second user data.

In some implementations, after receiving the user viewing instruction sent by the first head-mounted display device, the electronic device may respond to the user viewing instruction and obtain the user data of the user corresponding to the second head-mounted display device as the second User data. As a method, the user data of the user corresponding to the second head-mounted display device can be collected by the second head-mounted display device, or can be obtained by a data collection device (smart home device) located around the second head-mounted display device ) collected and obtained, and is not limited here.

Step S606: Process the second user data to obtain third data to be displayed.

In some implementations, after obtaining the second user data, the electronic device may process the second user data to obtain the data to be displayed as the third data to be displayed. For example, after obtaining the second user data, the electronic device may analyze, render and optimize the second user data, and obtain the data to be displayed as the third data to be displayed.

Step S607: Send the third data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the third data to be displayed.

In this embodiment, after the electronic device obtains the third data to be displayed, it can send the third data to be displayed to the first head-mounted display device, and accordingly, the first head-mounted display device can receive the data sent by the electronic device. The third data to be displayed is displayed, and the third data to be displayed is displayed, so that the user corresponding to the first head-mounted display device can view the data of the remote counterparty, thereby improving the interactive visual experience. As a method, after receiving the third data to be displayed, the first head-mounted display device can simultaneously display the first data to be displayed and the third data to be displayed, or can replace the first data to be displayed with the third data to be displayed. The data is displayed, which is not limited here.

For example, taking the target virtual scene as a remote teaching scene, the user corresponding to the first head-mounted display device is a teacher, and the user corresponding to the second head-mounted display device is a student. In the process of displaying the third data to be displayed (student data), the teacher can also observe the learning status of the students at any time, and can ask students questions, etc., so as to improve the interactivity of the classroom.

Step S608: Sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.

Wherein, for the specific description of step S608, please refer to step S104, which will not be repeated here.

In the remote interaction method provided by yet another embodiment of the present application, compared with the remote interaction method shown in FIG. 3 , this embodiment also receives the user viewing instruction sent by the first head-mounted display device, responds to the user viewing instruction, and obtains the first The user data of the user corresponding to the second head-mounted display device is used as the second user data, the second user data is processed to obtain the third data to be displayed, and the third data to be displayed is sent to the first head-mounted display device for display , so as to improve the interactivity of remote interaction.

Please refer to FIG. 15 , which shows a schematic flowchart of a remote interaction method provided by still another embodiment of the present application. The method is applied to a first head-mounted display device, the first head-mounted display device is connected to an electronic device, and the electronic device is also connected to a second head-mounted display device. The following will describe the process shown in FIG. 15 in detail. According to the elaboration, the remote interaction method may specifically include the following steps:

Step S701: Collect user data of a user corresponding to the first head-mounted display device as first user data.

Step S702: Send the first user data to the electronic device to instruct the electronic device to process the first user data to obtain first data to be displayed, and send the first data to be displayed respectively to the first head-mounted display device for display and the second head-mounted display device for display.

Step S703: Receive the first data to be displayed sent by the electronic device, and display the first data to be displayed.

Wherein, for the specific description of step S701-step S703, please refer to step S101-step S104, which will not be repeated here.

According to the remote interaction method provided by still another embodiment of the present application, the first head-mounted display device collects the user data of its corresponding user as the first user data, and sends the first user data to the electronic device to instruct the electronic device Process the first user data to obtain the first data to be displayed, and send the first data to be displayed to the first head-mounted display device for display and the second head-mounted display device for display, and receive the first data to be displayed sent by the electronic device data to be displayed, and display the band display data, so as to obtain the data to be displayed through the user data of the user corresponding to the first head-mounted display device, and send the data to be displayed to the first head-mounted display device and the second head-mounted display device respectively The display by the two head-mounted display devices can improve the visual experience during interaction.

Please refer to FIG. 16 , which shows a module block diagram of a remote interaction device provided by an embodiment of the present application. The remote interaction device 500 is applied to the above-mentioned electronic equipment, and the electronic device is respectively connected to the first head-mounted display device and the second head-mounted display device. The following describes the block diagram shown in FIG. 16. The remote interaction device 500 Including: a user data acquisition module 510, a data to be displayed acquisition module 520, a first data to be displayed sending module 530 and a second data to be displayed sending module 540, wherein:

The user data obtaining module 510 is configured to obtain user data of a user corresponding to the first head-mounted display device as first user data.

Further, the electronic device is also connected to a data collection device, and the user data acquisition module 510 includes: a user data acquisition submodule, wherein:

The user data acquisition sub-module is configured to acquire the user data of the user corresponding to the first head-mounted display device sent by the data acquisition device as the first user data.

Further, the data collection device is composed of at least one smart home device, and the user data acquisition submodule includes: a smart home device acquisition unit and a user data acquisition unit, wherein:

The smart home device obtaining unit is configured to obtain at least one smart home device within a preset range of the first head-mounted display device.

A user data acquiring unit, configured to acquire user data of a user corresponding to the first head-mounted display device sent by the at least one smart home device when the at least one smart home device satisfies preset data collection conditions, as The first user data.

The data to be displayed obtaining module 520 is configured to process the first user data to obtain the first data to be displayed.

The first data to be displayed sending module 530, configured to send the first data to be displayed to the first head-mounted display device, so as to instruct the first head-mounted display device to display the first data to be displayed .

The second data to be displayed sending module 540, configured to send the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed .

Further, the remote interaction device 500 also includes: a virtual scene selection instruction receiving module, a target virtual scene determination module, a first target virtual scene sending module and a second target virtual scene sending module, wherein:

The virtual scene selection instruction receiving module is configured to receive the virtual scene selection instruction sent by the first head-mounted display device, wherein the virtual scene selection instruction is triggered by a user corresponding to the first head-mounted display device.

The target virtual scene determination module is used to determine the target virtual scene from multiple virtual scenes in response to the virtual scene selection instruction.

The first target virtual scene sending module is configured to send the target virtual scene to the first head-mounted display device, so as to instruct the first head-mounted display device to display the target virtual scene.

The second target virtual scene sending module is configured to send the target virtual scene to the second head-mounted display device, so as to instruct the second head-mounted display device to display the target virtual scene.

Further, the remote interaction device 500 also includes: a virtual item selection instruction receiving module, a target virtual item determination module, a first target virtual item sending module and a second target virtual item sending module, wherein:

The virtual item selection instruction receiving module is configured to receive the virtual item selection instruction sent by the first head-mounted display device, wherein the virtual item selection instruction is issued by the user corresponding to the first head-mounted display device in the Triggered in the target virtual scene.

The target virtual item determination module is configured to determine the target virtual item from multiple virtual items in response to the virtual item selection instruction.

The first target virtual item sending module is configured to send the target virtual item to the first head-mounted display device, so as to instruct the first head-mounted display device to display the target virtual item.

The second target virtual item sending module is configured to send the target virtual item to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual item.

Further, the remote interaction device 500 also includes: an input information receiving module and an input information sending module, wherein:

The input information receiving module is configured to receive the input information sent by the first head-mounted display device, wherein the input information is input by the user corresponding to the first head-mounted display device on the target virtual prop.

The input information sending module is configured to send the input information to the second head-mounted display device to instruct the second head-mounted display device to display the input information and the target virtual prop correspondingly.

Further, the remote interaction device 500 also includes: a feedback information receiving module and a feedback information sending module, wherein:

The feedback information receiving module is configured to receive the feedback information sent by the second head-mounted display device, wherein the feedback information is triggered by the user corresponding to the second head-mounted display device in the target virtual scene.

A feedback information sending module, configured to send the feedback information to the first head-mounted display device to instruct the first head-mounted display device to display the feedback information.

Further, the remote interaction device 500 also includes: a data adjustment instruction receiving module, a module for obtaining data to be displayed, and a third module for sending data to be displayed, wherein:

The data adjustment instruction receiving module is configured to receive the data adjustment instruction sent by the second head-mounted display device, wherein the data adjustment instruction is triggered by the user corresponding to the second head-mounted display device.

The data to be displayed obtaining module is configured to adjust the first data to be displayed in response to the data adjustment instruction, and obtain second data to be displayed.

The third data to be displayed sending module is configured to send the second data to be displayed to the second head-mounted display device, so as to instruct the second head-mounted display device to display the second data to be displayed.

Further, the remote interaction device 500 also includes: a user viewing instruction receiving module, a user viewing instruction response module, a data to be displayed obtaining module, and a fourth data to be displayed sending module, wherein:

A user viewing instruction receiving module, configured to receive a user viewing instruction sent by the first head-mounted display device, wherein the user viewing instruction is triggered by a user corresponding to the first head-mounted display device;

The user viewing instruction response module is configured to respond to the user viewing instruction and obtain user data of the user corresponding to the second head-mounted display device as the second user data.

The data to be displayed obtaining module is configured to process the second user data to obtain third data to be displayed.

The fourth data to be displayed sending module is configured to send the third data to be displayed to the first head-mounted display device, so as to instruct the first head-mounted display device to display the third data to be displayed.

Please refer to FIG. 17 , which shows a module block diagram of a remote interaction device provided by another embodiment of the present application. The remote interaction device 600 can be applied to the above-mentioned first head-mounted display device, the first head-mounted display device is connected to the electronic device, and the electronic device is also connected to the second head-mounted display device, the following will refer to FIG. 17 For illustration, the remote interaction device 600 further includes: a user data collection module 610, a user data sending module 620, and a data display module 630, wherein:

The user data collection module 610 is configured to collect user data of a user corresponding to the first head-mounted display device as first user data.

The user data sending module 620 is configured to send the first user data to the electronic device, so as to instruct the electronic device to process the first user data to obtain first data to be displayed, and respectively send the first user data to the electronic device. The data to be displayed is sent to the first head-mounted display device for display and the second head-mounted display device for display.

The data display module 630 is configured to receive the first data to be displayed sent by the electronic device, and display the first data to be displayed.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the devices and modules described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.

In some embodiments, this embodiment also provides a remote interaction system, the remote interaction system includes an electronic device and at least two head-mounted display devices, the electronic device is connected to at least two head-mounted display devices, wherein, The at least two head-mounted display devices include a first head-mounted display device and a second head-mounted display device. Among them: the electronic device is used to obtain the user data of the user corresponding to the first head-mounted display device as the first user data; the electronic device is used to process the first user data to obtain the first data to be displayed; the electronic device , used to send the first data to be displayed to the first head-mounted display device; the first head-mounted display device is used to display the first data to be displayed; the electronic device is used to send the first data to be displayed to the first head-mounted display device Two head-mounted display devices; the second head-mounted display device is used to display the first data to be displayed.

Please refer to FIG. 18 , which shows a structural block diagram of an electronic device 100 provided by an embodiment of the present application. The electronic device 100 may be an electronic device capable of running application programs, such as a smart phone, a tablet computer, and an e-book. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein one or more application programs may be stored in the memory 120 and configured to be executed by a or a plurality of processors 110, and one or more programs are configured to execute the methods described in the foregoing method embodiments.

Wherein, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes or executes instructions, programs, code sets or instruction sets stored in the memory 120, and calls data stored in the memory 120 to execute Various functions of the electronic device 100 and processing data. Optionally, the processor 110 may adopt at least one of Digital Signal Processing (Digital Signal Processing, DSP), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable Logic Array, PLA). implemented in the form of hardware. The processor 110 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used to render and draw the content to be displayed; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may also not be integrated into the processor 110, but implemented by a communication chip alone.

The memory 120 may include random access memory (Random Access Memory, RAM), and may also include read-only memory (Read-Only Memory). The memory 120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. The storage data area can also store data created during use of the electronic device 100 (such as phonebook, audio and video data, chat record data) and the like.

Please refer to FIG. 19 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program codes are stored in the computer-readable medium 700, and the program codes can be invoked by a processor to execute the methods described in the foregoing method embodiments.

The computer readable storage medium 700 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 700 includes a non-transitory computer-readable storage medium (non-transitory computer-readable storage medium). The computer-readable storage medium 700 has a storage space for program code 710 for executing any method steps in the above methods. These program codes can be read from or written into one or more computer program products. Program code 710 may, for example, be compressed in a suitable form.

To sum up, the remote interaction method, device, system, electronic device, and storage medium provided by the embodiments of the present application acquire the user data of the user corresponding to the first head-mounted display device as the first user data, and the first user processing the data, obtaining the first data to be displayed, sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed, and sending the first data to be displayed To the second head-mounted display device, to instruct the second head-mounted display device to display the first data to be displayed, so as to obtain the data to be displayed through the user data of the user corresponding to the first head-mounted display device, and transfer the data to be displayed The data are respectively sent to the first head-mounted display device and the second head-mounted display device for display, which can improve visual experience during interaction.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions away from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

A remote interaction method, characterized in that it is applied to an electronic device, and the electronic device is connected to at least two head-mounted display devices, wherein the at least two head-mounted display devices include the first head-mounted display device and a second head-mounted display device, the method comprising:

Obtaining user data of a user corresponding to the first head-mounted display device as first user data;

Processing the first user data to obtain first data to be displayed;

sending the first data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the first data to be displayed;

sending the first data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the first data to be displayed.
The method according to claim 1, wherein the electronic device is also connected to a data collection device, and the obtaining the user data of the user corresponding to the first head-mounted display device as the first user data includes:

Acquiring the user data of the user corresponding to the first head-mounted display device sent by the data collection device as the first user data.
The method according to claim 2, wherein the data collection device is composed of at least one smart home device, and the acquisition of the user corresponding to the first head-mounted display device sent by the data collection device Data, as said first user data, includes:

Acquiring at least one smart home device located within a preset range of the first head-mounted display device;

When the at least one smart home device satisfies the preset data collection condition, acquire the user data of the user corresponding to the first head-mounted display device sent by the at least one smart home device as the first user data.
The method according to claim 3, wherein said obtaining at least one smart home device within a preset range of said first head-mounted display device comprises:

Acquiring the room where the first head-mounted display device is located, and acquiring the size of the room where the first head-mounted display device is located;

When the size of the room where the first head-mounted display device is located is greater than a preset range, at least one smart home device located within the preset range of the first head-mounted display device is acquired.
The method according to claim 3, further comprising:

When the at least one smart home device does not meet the preset data collection conditions, output prompt information to the first head-mounted display device to instruct to change the location of the first head-mounted display device; or

When the at least one smart home device does not meet the preset data collection condition, switch to the first head-mounted display device to collect user data.
The method according to any one of claims 1-3, wherein the method further comprises:

receiving a virtual scene selection instruction sent by the first head-mounted display device, wherein the virtual scene selection instruction is triggered by a user corresponding to the first head-mounted display device;

Responding to the virtual scene selection instruction, determine a target virtual scene from a plurality of virtual scenes;

sending the target virtual scene to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual scene;

sending the target virtual scene to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual scene.
The method according to claim 6, wherein after sending the target virtual scene to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual scene ,Also includes:

receiving a virtual item selection instruction sent by the first head-mounted display device, wherein the virtual item selection instruction is triggered by a user corresponding to the first head-mounted display device in the target virtual scene;

Responding to the virtual item selection instruction, determine a target virtual item from a plurality of virtual items;

sending the target virtual item to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual item;

sending the target virtual item to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual item.
The method according to claim 7, wherein after sending the target virtual item to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual item ,Also includes:

receiving input information sent by the first head-mounted display device, wherein the input information is input by a user corresponding to the first head-mounted display device on the target virtual item;

Sending the input information to the second head-mounted display device to instruct the second head-mounted display device to display the input information and the target virtual prop correspondingly.
The method according to claim 7, wherein the sending the target virtual item to the first head-mounted display device to instruct the first head-mounted display device to display the target virtual item ,include:

acquiring gesture information of a user corresponding to the first head-mounted display device;

Based on the posture information of the user corresponding to the first head-mounted display device, the target virtual prop is rendered and then sent to the first head-mounted display device to instruct the first head-mounted display device to display The rendered target virtual prop.
The method according to claim 7, wherein after sending the target virtual scene to the second head-mounted display device to instruct the second head-mounted display device to display the target virtual scene ,Also includes:

receiving feedback information sent by the second head-mounted display device, wherein the feedback information is triggered by a user corresponding to the second head-mounted display device in the target virtual scene;

sending the feedback information to the first head-mounted display device to instruct the first head-mounted display device to display the feedback information.
The method according to claim 7, wherein the virtual scene selection instruction is a plurality of selectable virtual scenes displayed by the user corresponding to the first head-mounted display device through the first head-mounted display device. Scene triggering, or, the virtual scene selection instruction is triggered by multiple selectable application programs displayed by the user corresponding to the first head-mounted display device through the first head-mounted display device, wherein the multiple Each application corresponds to a virtual scene.
The method according to any one of claims 1-3, wherein the first data to be displayed is sent to the second head-mounted display device to instruct the second head-mounted display device After displaying the first data to be displayed, it also includes:

receiving a data adjustment instruction sent by the second head-mounted display device, wherein the data adjustment instruction is triggered by a user corresponding to the second head-mounted display device;

adjusting the first data to be displayed in response to the data adjustment instruction to obtain second data to be displayed;

sending the second data to be displayed to the second head-mounted display device to instruct the second head-mounted display device to display the second data to be displayed.
The method according to any one of claims 1-3, wherein the first data to be displayed is sent to the first head-mounted display device to indicate that the first head-mounted display device After displaying the first data to be displayed, it also includes:

receiving a user viewing instruction sent by the first head-mounted display device, wherein the user viewing instruction is triggered by a user corresponding to the first head-mounted display device;

Responding to the user viewing instruction, acquiring user data of a user corresponding to the second head-mounted display device as second user data;

Processing the second user data to obtain third data to be displayed;

sending the third data to be displayed to the first head-mounted display device to instruct the first head-mounted display device to display the third data to be displayed.
A remote interaction method, characterized in that it is applied to a first head-mounted display device, the first head-mounted display device is connected to an electronic device, and the electronic device is also connected to a second head-mounted display device, so The methods described include:

collecting user data of a user corresponding to the first head-mounted display device as first user data;

sending the first user data to the electronic device to instruct the electronic device to process the first user data to obtain first data to be displayed, and sending the first data to be displayed to the displaying by the first head-mounted display device and displaying by the second head-mounted display device;

receiving the first data to be displayed sent by the electronic device, and displaying the first data to be displayed.
The method according to claim 14, wherein the first head-mounted display device includes a motion sensor and an audio sensor, and the user data of the user corresponding to the first head-mounted display device is collected as the second 1. User data, including:

Detecting gesture information of the user corresponding to the first head-mounted display device through the motion sensor;

Detecting the voice information of the user corresponding to the first head-mounted display device through the audio sensor;

Determining the gesture information of the user corresponding to the first head-mounted display device and the voice information of the user corresponding to the first head-mounted display device as the first user data.
A remote interaction device, characterized in that it is applied to electronic equipment, and the electronic equipment is connected to at least two head-mounted display devices, wherein the at least two head-mounted display devices include the first head-mounted display device and a second head-mounted display device, the apparatus comprising:

A user data acquisition module, configured to acquire user data of a user corresponding to the first head-mounted display device as first user data;

The data to be displayed obtaining module is configured to process the first user data to obtain the first data to be displayed;

The first data to be displayed sending module is configured to send the first data to be displayed to the first head-mounted display device, so as to instruct the first head-mounted display device to display the first data to be displayed;

The second data to be displayed sending module is configured to send the first data to be displayed to the second head-mounted display device, so as to instruct the second head-mounted display device to display the first data to be displayed.
A remote interaction device, characterized in that it is applied to a first head-mounted display device, the first head-mounted display device is connected to an electronic device, and the electronic device is also connected to a second head-mounted display device, so Said devices include:

A user data collection module, configured to collect user data of a user corresponding to the first head-mounted display device as first user data;

a user data sending module, configured to send the first user data to the electronic device, to instruct the electronic device to process the first user data to obtain first data to be displayed, and respectively send the first The data to be displayed is sent to the first head-mounted display device for display and the second head-mounted display device for display;

A data display module, configured to receive the first data to be displayed sent by the electronic device, and display the first data to be displayed.
A remote interaction system, characterized in that the remote interaction system includes an electronic device and at least two head-mounted display devices, the electronic device is connected to the at least two head-mounted display devices, wherein the at least The two head-mounted display devices include a first head-mounted display device and a second head-mounted display device, wherein:

The electronic device is configured to acquire user data of a user corresponding to the first head-mounted display device as first user data;

The electronic device is configured to process the first user data to obtain first data to be displayed;

The electronic device is configured to send the first data to be displayed to the first head-mounted display device;

The first head-mounted display device is configured to display the first data to be displayed;

The electronic device is configured to send the first data to be displayed to the second head-mounted display device;

The second head-mounted display device is configured to display the first data to be displayed.
An electronic device, characterized in that it includes a memory and a processor, the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs The method described in any one of claims 1-13.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores program codes, and the program codes can be invoked by a processor to execute the method according to any one of claims 1-13 .