WO2023130435A1

WO2023130435A1 - Interaction method, head-mounted display device, and system and storage medium

Info

Publication number: WO2023130435A1
Application number: PCT/CN2022/070966
Authority: WO
Inventors: 张鼎
Original assignee: 深圳市闪至科技有限公司
Priority date: 2022-01-10
Filing date: 2022-01-10
Publication date: 2023-07-13

Abstract

An interaction method, a head-mounted display device, and a system and a storage medium. The interaction method comprises: detecting whether a wearer of a head-mounted display device has an intention to operate an external device; and in response to detecting that the wearer has the intention to operate the external device, displaying, in a superimposed manner and in a display image of the head-mounted display device, an image captured by a camera of the head-mounted display device, wherein the image captured by the camera comprises an image area corresponding to the external device. In this way, the wearer of the head-mounted display device can operate the external device while "seeing" the external device, thereby improving the operation convenience of the external device; and the image area corresponding to the external device is displayed in a superimposed manner only when it is detected that the wearer has the intention to operate the external device, thereby preventing the display image of the head-mounted display device from being covered.

Description

Interaction method, head-mounted display device, system and storage medium

technical field

The present application relates to the field of head-mounted display devices, and in particular, relates to an interaction method, a head-mounted display device, a system, and a storage medium.

Background technique

With the development of technology and the improvement of living standards, head-mounted display devices are gradually becoming popular. The appearance of the head-mounted display device is usually in the form of a goggle or a helmet, which puts the display screen close to the user's eyes and projects a picture to the user's eyes at a close distance. Head-mounted display devices are divided into two types: translucent and non-transparent. For non-transparent head-mounted display devices, users lose their eyes' perception of the outside world after wearing the head-mounted display device. It creates a lot of inconvenience when trying to operate the equipment.

Contents of the invention

In view of this, one of the objectives of the present application is to provide an interaction method, a head-mounted display device, a system, and a storage medium.

First, in some scenarios, when the user wears the head-mounted display device, there is a need to operate some external devices, and in this case the user cannot see the external devices in the physical environment, and the user may operate the external devices The unfamiliar way makes it less convenient for users to operate external devices.

Therefore, an embodiment of the present application provides an interaction method for a head-mounted display device, the method including:

Detecting whether the wearer of the head-mounted display device intends to operate an external device;

In response to detecting that the wearer has an intention to operate an external device, superimposing and displaying an image captured by a camera of the head-mounted display device on the display image of the head-mounted display device, the image captured by the camera An image area corresponding to the external device is included.

In this embodiment, the image area corresponding to the external device can be superimposed and displayed in the display image of the head-mounted display device, so that the wearer of the head-mounted display device can operate the external device while "seeing" the external device, improving The operation convenience of the external device; further, the image area corresponding to the external device is superimposed and displayed only when the wearer is detected to have the intention to operate the external device, and there is no need to display the image area corresponding to the external device at all times, so as to avoid blocking the head-mounted display image of the display device.

Second, in some scenarios, when the user wears the head-mounted display device, there is a need to operate some virtual devices, such as pointing and pressing the virtual device with the finger to realize data input. The operating position of the device is not accurately grasped, which makes it less convenient for the user to operate the virtual device.

Detecting whether the wearer of the head-mounted display device intends to operate the virtual device;

In response to detecting that the wearer has an intention to operate the virtual device, superimposing and displaying the virtual device in the display image of the head-mounted display device, the virtual device is obtained based on the camera of the head-mounted display device The virtual device ID is generated.

In this embodiment, the virtual device is superimposed and displayed on the display image of the head-mounted display device by means of the virtual device identifier set in the physical environment. In order to realize the precise operation of the virtual device and improve the operation convenience of the virtual device; further, the virtual device is superimposed and displayed only when it is detected that the wearer has the intention to operate the virtual device, and there is no need to display the virtual device at all times. Avoid blocking the display image of the head-mounted display device.

In a third aspect, the embodiment of the present application provides a head-mounted display device, including:

one or more processors;

memory for storing said processor-executable instructions;

a display for displaying images;

The one or more processors individually or jointly execute the executable instructions to perform the method as described in the first aspect or the second aspect.

In a fourth aspect, an embodiment of the present application provides an interactive system, including the head-mounted display device and an external device as described in the third aspect.

In the fifth aspect, the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the first aspect or the second aspect is implemented. the method described.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a usage scene diagram of a head-mounted display device and external devices provided by an embodiment of the present application;

Fig. 2 is a schematic structural diagram of an interactive system provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an image area corresponding to a keyboard superimposed on a display image of a head-mounted display device according to an embodiment of the present application;

FIG. 4 is a schematic structural diagram of another interactive system provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a usage scenario of the interactive system shown in FIG. 4 provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of superimposing an image area corresponding to a remote control device in a display image of a head-mounted display device provided by an embodiment of the present application;

FIG. 7 is a schematic flowchart of an interaction method for a head-mounted display device provided in an embodiment of the present application;

Fig. 8 is a schematic flowchart of another interaction method for a head-mounted display device provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a head-mounted display device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In some scenarios of related technologies, for example, as shown in FIG. 1 , a head-mounted display device 100 and an external device 200 are shown. When the user wears the head-mounted display device 100, there is a need for the user to operate some external devices 200, but in this case the user cannot see the external devices in the physical environment, and the user may not be familiar with the operation mode of the external devices, so that The convenience for the user to operate the external device is poor.

Among them, the physical environment refers to the physical world that people can sense and/or interact with without the help of electronic devices. People are able to directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste and smell.

Based on the problems in the related technologies, the embodiment of the present application provides an interaction method for a head-mounted display device, which can shoot external devices through the camera of the head-mounted display device, and the captured image includes the image area corresponding to the external device , and then can superimpose and display the image area corresponding to the external device in the display image of the head-mounted display device, then the wearer of the head-mounted display device can operate the external device while "seeing" the external device, and improve the safety of the external device. Ease of operation. Furthermore, the inventor found that if the image area corresponding to the external device is superimposed and displayed on the display image of the head-mounted display device, it will cause continuous occlusion of the display image of the head-mounted display device, affecting the viewing experience of the user. Therefore, the embodiment of the present application further realizes that the image area corresponding to the external device is superimposed on the display image of the head-mounted display device only when it is detected that the wearer has the intention to operate the external device, and there is no need to display the corresponding image area of the external device at all times. The image area of the head-mounted display device is avoided from blocking the display image of the head-mounted display device, and the operation convenience of the external device is improved while reducing the impact on the normal display of the display image of the head-mounted display device.

In some embodiments, the head-mounted display device mentioned in the embodiment of the present application includes but is not limited to smart glasses, a helmet provided with a display, goggles or a head-up display and the like. The displayed image of the head-mounted display device may be a virtual image, an image taken from a physical environment, or an image obtained by superimposing a virtual image and an image taken from a physical environment.

In an exemplary embodiment, the head-mounted display device includes a glasses body, one end of the glasses body is provided with a wearing part for cooperating with a user, and the other end of the glasses body is provided with at least a display. After the user wears the head-mounted display device, the display can display images within the wearer's field of vision, and the images displayed on the display include but are not limited to virtual images and/or images captured from physical environments.

The head-mounted display device may also be provided with an inward-facing or outward-facing sensor/device, so as to provide a good interaction effect for the wearer. Exemplarily, the head-mounted display device further includes a camera installed on the main body of the glasses, and the camera can shoot external devices in the physical environment to obtain images including the external devices; wherein, the camera includes not limited to a color camera, Monochrome cameras, monochrome cameras, 3D depth-sensing cameras, stereo cameras or time-of-flight (ToF) cameras, etc. Exemplarily, the head-mounted display device further includes an attitude sensor provided on the glasses body, and the attitude sensor is used to detect the head posture of the wearer, for example, the content displayed on the display can be determined according to the head posture of the wearer. Exemplarily, the head-mounted display device further includes an eye tracking sensor, which is used to detect the wearer's eye movement, for example, the wearer's eye gaze range can be determined according to the wearer's eye movement, and then the display corresponding to the displayed content.

In an exemplary application scenario, the external device may be an input device of a head-mounted display device. In the related art, since the user cannot see the external device in the physical environment, in order to facilitate the user's operation, the head-mounted display device in the related art The input device of the display device may be some input device with a relatively simple operation mode. For example, the input device may only include two or three operation controls with specified functions, and the input function of the first input device is limited. However, in the embodiment of the present application, when the user can "see" the external device, the external device may not only include an input device with a relatively simple operation mode, but also include an input device with a variety of different input functions, so as to satisfy Diversified input requirements of users. Exemplarily, the input device of the head-mounted display device includes, but is not limited to, at least one of a keyboard, a mouse, a touch screen, a stylus, a rocker (such as a game joystick), and a handwriting pad. Various input devices can be Meet the user's personalized input needs.

In an example, please refer to FIG. 2 and FIG. 3 , FIG. 2 shows an interactive system, and the interactive system includes a head-mounted display device 100 and a keyboard 201 . FIG. 3 shows a schematic diagram of superimposing the image area corresponding to the keyboard 201 in the display image 20 of the head-mounted display device 100 . In the process of wearing the head-mounted display device 100, the user can not only see the display image 20 of the head-mounted display device, but also when there is a need to operate an external device, the head-mounted display device 100 responds to the detection that the wearer has The intention of operating the external device is to obtain the image of the keyboard 201 captured by the camera, the image includes the image area 10 corresponding to the keyboard 201, and then the image area 10 corresponding to the keyboard 201 can be superimposed on the display image 20 of the head-mounted display device 100, so that The wearer can "see" the keyboard 201 in the physical environment when he intends to operate the keyboard 201, so as to realize the convenient operation of the keyboard 201 and improve the use experience of the wearer.

In another exemplary embodiment, the external device may be a device related to a head-mounted display device, for example, the external device may be a remote control device of a movable platform. The movable platform is provided with a photographing device, and the remote control device can control the movement process of the movable platform and/or the photographing process of the photographing device, for example, the remote control device can directly or indirectly send a message to the movable platform under the user's manipulation. The control command is transmitted to control the movement direction or movement speed of the movable platform. The movable platform is also communicatively connected with the head-mounted display device, and the movable platform can transmit the image taken by the shooting device to the head-mounted display device, and then the image taken by the movable platform based on the shooting device can be displayed on the In the head-mounted display device. Wherein, the movable platform includes but not limited to vehicles, unmanned aerial vehicle (UAV), unmanned ship or mobile robot and so on.

Here we take drones as an example for illustration: please refer to FIG. 4 and FIG. 5, FIG. 4 shows an interactive system, which includes a head-mounted display device 100, a remote control device 202 and a drone 301 . FIG. 5 shows a usage scenario of the interactive system described in FIG. 4 . Wherein, the drone 301 is provided with a photographing device, and the head-mounted display device 100 is provided with a camera. The remote control device 202 can directly or indirectly transmit control instructions to the UAV 301 based on the operation of the wearer of the head-mounted display device 100, so as to control the flight of the UAV 301 and/or the shooting device in the UAV 301 Take an image. The UAV 301 can take pictures during the flight and transmit them to the head-mounted display device 100 . The head-mounted display device 100 can display images captured by the drone 301 . Among them, in order to enable the user to conveniently operate the remote control device 202 to achieve precise control of the UAV 301, and then see the desired display image in the head-mounted display device 100, the head-mounted display device 100 can respond to the detected The wearer intends to operate the remote control device 202 and acquires an image captured by the camera, and the image includes an image area corresponding to the remote control device 202 . Please refer to FIG. 6 , the head-mounted display device 100 can superimpose the image area 10 corresponding to the remote control device 202 on the display image 20 of the head-mounted display device, so that the wearer can " "See" the remote control device 202 in the physical environment, so as to realize the convenient operation of the remote control device 202 and improve the wearer's use experience.

In some embodiments, please refer to FIG. 7. FIG. 7 is a schematic flowchart of a method for interacting with a head-mounted display device according to an embodiment of the present application. The method can be executed by the head-mounted display device, and the method includes :

In step S101, it is detected whether the wearer of the head-mounted display device intends to operate an external device.

In step S102, in response to detecting that the wearer has an intention to operate an external device, superimposing and displaying an image captured by a camera of the head-mounted display device on the display image of the head-mounted display device, the The image captured by the camera includes the image area corresponding to the external device.

In some embodiments, when the user is wearing the head-mounted display device, the head-mounted display device can display images within the user's field of view. Wherein, the displayed image of the head-mounted display device may be a virtual image (for example, as shown in FIG. 3 ), or an image obtained from a physical environment, or a superimposed image of a virtual image and an image obtained from a physical environment. Image. Exemplarily, the virtual image may be generated based on a preset modeling rule and using the principle of imaging geometry; for example, the virtual image includes but is not limited to a virtual game screen, a virtual building screen, or a virtual character screen. Exemplarily, the head-mounted display device communicates with a mobile platform equipped with a shooting device, and the mobile platform includes but is not limited to drones, vehicles, ships, or mobile robots; The images include real-time images collected by the photographing device of the movable platform or images or video frames collected in a historical time period. Exemplarily, the image captured from the physical environment may be an image obtained from a third-party platform, such as an image or a video frame shared by other users on a social platform.

In some embodiments, when the user wears the head-mounted display device, the head-mounted display device can detect whether the wearer of the head-mounted display device intends to operate the external device, so as to determine whether to wear the head-mounted display device The external device is superimposed on the display image of the display device. Exemplarily, after detecting that the user wears the head-mounted display device, the head-mounted display device may detect whether the wearer intends to operate the external device in real time, or intermittently detect whether the wearer has the intention to operate the external device.

In some possible implementation manners, the head-mounted display device may detect whether the wearer intends to operate the external device according to the wearer's head posture and/or eye gaze range. Exemplarily, the head-mounted display device is equipped with an attitude sensor, and the wearer's head attitude can be determined through the data collected by the attitude sensor; the attitude sensor can include motion sensors such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass . Exemplarily, the head-mounted display device may be installed with an eye tracking sensor, and the gaze range of the wearer's eyes may be determined through data collected by the eye tracking sensor.

Exemplarily, in order to facilitate the user to operate the external device, the height of the external device is generally lower than that of the head-mounted display device. When the user wants to operate the external device, he may lower his head subconsciously to determine the location of the external device. Therefore, if the head-mounted display device detects that the head posture indicates the wearer's head-down posture, it can be determined that the wearer has an intention to operate the external device.

Exemplarily, when the head-mounted display device displays visual information, it determines the gaze range of the wearer's eyes according to the data collected by the eye tracking sensor and/or the data collected by the posture sensor, and then obtains the display corresponding to the gaze range of the eyes. The content is presented to the user. The user can change the gaze range of the eyes by turning the head or eyeballs, so as to watch different display contents. Therefore, a virtual component corresponding to the external device can be set, and the virtual component can be displayed on the head-mounted display device; when the wearer wants to operate the external device, he can change the gaze range of the eye by turning the head or eyeballs to find If the display content of the virtual component is displayed, if the head-mounted display device detects that the display content within the gaze range of the eyes includes the virtual component corresponding to the external device, it may determine that the wearer has the intention to operate the external device. Exemplarily, the external device is an input device, and the virtual component corresponding to the external device may be an input virtual component. Exemplarily, the external device is a remote control device, and the virtual component corresponding to the external device may be a virtual remote control.

Of course, in order to improve the detection accuracy, it is possible to detect that the head posture indicates the wearer's head-down posture and that the display content within the gaze range of the eyes includes a virtual component corresponding to the external device. In this case, it is determined that the wearer has an intention to operate the external device.

In other possible implementation manners, the head-mounted display device is provided with designated controls, and the wearer can communicate his needs to the head-mounted display device by operating the external device when the wearer needs to operate the external device. The head-mounted display device may determine that the wearer intends to operate the external device in response to the triggering of the specified control, and then control the camera installed in the head-mounted display device to capture images.

In some embodiments, the head-mounted display device may not display the image captured by the camera of the head-mounted display device in response to the wearer not intending to operate the external device. In this embodiment, when the wearer has no intention to operate the external device, the image area corresponding to the external device is not displayed, so as to avoid blocking the display image of the head-mounted display device. Exemplarily, the head-mounted display device may determine that the wearer has no intention to operate the external device when detecting any one or more of the following situations: the wearer's head posture indicates that the wearer is looking up The posture, the displayed content within the gaze range of the wearer's eyes does not include the virtual component corresponding to the external device, or the specified part of the wearer does not exist in the image captured by the camera.

In a possible implementation, for example, when the head-mounted display device superimposes and displays the image area corresponding to the external device in the displayed image, it may periodically detect whether the wearer of the head-mounted display device operates the external device. Intent, if it is detected that the wearer has no intention to operate the external device, stop displaying the image area corresponding to the external device, so that the user can see the complete content of the displayed image.

In some embodiments, the camera of the head-mounted display device may be fixedly arranged in the head-mounted display device. Exemplarily, the head-mounted display device is provided with a wearing part that cooperates with the user, and the camera is arranged on the outside of the head-mounted display device and is opposite to the wearing part of the head-mounted display device, so that The physical environment can be photographed to obtain images including external equipment. Exemplarily, considering that when the user wears the head-mounted display device, the height of the external device is generally lower than the height of the head-mounted display device based on the user's habit of operating the external device, then the camera can be set on the head-mounted display device. The outside of the display device, and the direction of the field of view of the camera can be tilted downward (for example, the direction of the field of view of the camera and the direction of gravity are at a preset angle, and the angle is less than 90 degrees), so as to capture images including external devices; of course , when the field of view of the camera is large, for example, if the camera is a fisheye camera, any setting of the camera on the outside of the head-mounted display device can capture images including external devices. There is also no restriction on the setting method on the outside of the display device.

In some other embodiments, considering the uncertainty of the location of the external device, the camera may be arranged on the head-mounted display device and be rotatably connected to the head-mounted display device. Exemplarily, the camera can be rotated artificially (for example, the camera is rotatably connected to the head-mounted display device through a knob), or the camera can also be rotated automatically under the action of a driving component.

Taking the automatic rotation of the camera as an example, the head-mounted display device can control the rotation of the camera so that the lens assembly of the camera faces the external device, so that the camera can capture images including the external device, so that In response to the wearer's intention to operate the external device, the head-mounted display device superimposes and displays the image captured by the camera on the display image of the head-mounted display device, where the image includes an image area corresponding to the external device.

Exemplarily, the head-mounted display device may pre-store the reference information of the external device, and during the process of turning the camera, the head-mounted display device may check whether there is an external device in the image captured by the camera according to the reference information of the external device , when it is determined that the image captured by the camera includes the external device, the camera is controlled to stop rotating.

In some embodiments, in order to obtain images including external devices with better display effects, the head-mounted display device includes one or more supplementary lights. When the camera of the head-mounted display device is capturing images, the head-mounted display device can detect the brightness of the environment where the external device is located, and when the brightness of the environment where the external device is located is less than a preset brightness threshold, Turn on the fill light. It can be understood that the preset brightness threshold can be specifically set according to the actual application scene, and this embodiment does not impose any limitation on this. For example, the preset brightness threshold can be determined according to the brightness requirement of the camera when capturing images. In this embodiment, by turning on the supplementary light at an appropriate time, the brightness of the environment where the external device is located can meet the brightness requirement of the camera, thereby obtaining an image with better display effect and improving the recognizability of the external device in the image.

Exemplarily, for the detection of the brightness of the environment where the external device is located, the ambient light brightness may be determined according to the number of photons received by the photosensitive element in the camera and/or the pixel value of the pixel in the image collected by the camera. Wherein, the number of photons is positively correlated with the brightness of the environment where the external device is located, that is, the more the number of photons is, the greater the brightness of the environment where the external device is located. The pixel value of the pixel is also positively correlated with the brightness of the environment where the external device is located. For example, the pixel value of the pixel is represented by a grayscale value, and a higher grayscale value indicates a greater brightness of the environment where the external device is located. It can be understood that the brightness of the environment where the external device is located can be determined according to any one of the number of photons and the pixel values of the pixels in the image, or can be jointly determined based on the two. The embodiment does not impose any limitation on this, and specific settings can be made according to actual application scenarios.

In some possible implementation manners, the brightness of the supplementary light can be adjusted according to the brightness of the environment where the external device is located, for example, the brightness of the supplementary light is negatively correlated with the brightness of the environment where the external device is located Relationship, that is, the darker the brightness of the environment where the external device is located, the brighter the brightness of the supplementary light, on the contrary, the brighter the brightness of the environment where the external device is located, the darker the brightness of the supplementary light, so that the supplementary light can be avoided or reduced The brightness of the lamp affects others.

In some possible implementations, when there are multiple supplementary lights, the head-mounted display device can determine the supplementary light that needs to be turned on according to the difference between the brightness of the environment where the external device is located and the preset brightness threshold. number of lights. For example, when the brightness of the environment where the external device is located is less than the preset brightness threshold, the number of fill lights that need to be turned on is positively correlated with the difference, that is, the greater the difference, the more the number of fill lights that need to be turned on. Realize that the auxiliary camera captures images that meet the brightness requirements. The smaller the difference, the fewer the number of fill lights that need to be turned on, which can avoid or reduce the impact of the brightness of the fill lights on others.

In some embodiments, considering that the images captured by the camera may contain other content that the user does not pay attention to, if the images captured by the camera are superimposed on the display image of the head-mounted display device, it may affect the head-mounted display device. The display image of the display device causes a large area of occlusion, which is not conducive to the user's viewing experience. Therefore, after the head-mounted display device acquires the original image captured by the camera, it can identify the original image to determine the image position corresponding to the external device, and then according to the image position corresponding to the external device in the original image, The original image is cropped to obtain a cropped image, the cropped image includes the image area corresponding to the external device, and then the cropped image can be superimposed and displayed on the display image. In this embodiment, only the image area corresponding to the external device is superimposed and displayed in the display image of the head-mounted display device, so as to avoid invalid content in the original image from blocking the display image of the head-mounted display device, and improve the viewing experience of the user. experience.

In some embodiments, the display position of the image area corresponding to the external device in the display image is determined at least according to the relative positional relationship between the wearer and the external device, so that the external The image area corresponding to the device is adapted to the relative positional relationship between the wearer and the relative positional relationship between the wearer and the external device in the physical environment, so that the image area corresponding to the external device is compatible with the physical environment The scales of the external devices are the same, and the wearer can find the external devices in the physical environment based on the display position of the "seeing" external devices, so as to realize the convenient operation of the external devices in the physical environment.

Exemplarily, in order to improve the accuracy of the displayed position, and in order to realize that the image area corresponding to the external device is consistent with the scale of the external device in the physical environment, comprehensively considering the user's eyes, the display, the camera and the factors of the external device Correlation, the display position of the image area corresponding to the external device in the display image can be based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, the camera The relative positional relationship between the display and the display, and the relative positional relationship between the camera and the external device are determined. In this way, the relative positional relationship between the image area corresponding to the external device and the wearer's eyes in the display space is the same as the relative positional relationship between the wearer's eyes and the external device in the physical environment, and the wearer can The display position of the "seen" external device corresponds to find the external device in the physical environment, so as to realize the convenient operation of the external device in the physical environment.

In some embodiments, in order to allow the user to see the complete content of the display image of the head-mounted display device during the operation of the external device, the image captured by the camera can be superimposed in a semi-transparent or semi-opaque form displayed in the display image, so that the user can also see the blocked part of the display image of the head-mounted display device through the image captured by the camera, so as to improve the viewing experience of the user.

In some other embodiments, considering that the color shades of the displayed images are different, if the images captured by the camera are superimposed and displayed with a fixed transparency, it may not be applicable to different display situations of the displayed images; for example, in the display When the overall tone of the image is a dark color, the fixed translucency may prevent the wearer from seeing the image captured by the camera (that is, the external device), and the operation of the external device may be affected; When the overall tone is light, the fixed translucency may make it impossible for the wearer to see the occluded content in the displayed image, affecting the user's viewing experience. Therefore, in order to achieve a balance between viewing the complete content of the displayed image and viewing the image captured by the camera (that is, the external device), this embodiment can set the adaptive change of the transparency of the image captured by the camera. The transparency of the image captured by the camera can be determined according to the color system of the displayed image in the superimposed area; wherein, the superimposed area refers to that when the displayed image and the image captured by the camera are superimposed and displayed, the displayed image is captured by the camera The resulting image occluded regions.

Exemplarily, the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the dark color system is smaller than the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the light color system. In other words, the darker the color system of the superimposed area, the smaller the transparency of the image captured by the camera is set, making the image captured by the camera clearer, so that the wearer can clearly see the image area corresponding to the external device, realizing convenient The operation process is permanent; the lighter the color system of the superimposed area, the greater the transparency of the image captured by the camera is set, so that the superimposed area (that is, the blocked area in the display area) is clearer, so that the wearer can see the display clearly. The full content of the image.

In some embodiments, this embodiment does not impose any restrictions on the operation mode of the external device. For example, the external device can be operated by the user's hand, or the external device can be operated by the user's foot. Therefore , when the wearer uses a designated part (such as a hand or a foot) to operate the external device, the image captured by the camera may further include an image area corresponding to the wearer's designated part. In an example, referring to FIG. 3 , when the wearer operates the keyboard with hands, the image area corresponding to the wearer's hands is superimposed on the displayed image. In another example, please refer to FIG. 6 , when the wearer operates the remote control device by hand, the image area corresponding to the wearer's hand is superimposed on the displayed image. Then, in some possible implementation manners, the head-mounted display device may recognize the operation position of the designated part of the wearer on the external device from the image captured by the camera of the head-mounted display device, and then The operation data generated by the wearer operating the external device is acquired according to the content indicated by the operation position.

In some other possible implementation manners, the external device is communicatively connected to the head-mounted display device; then the head-mounted display device can directly obtain the operation generated by the wearer operating the external device through the communication connection. data. In some other possible implementation manners, the external device is communicatively connected to the target device corresponding to the head-mounted display device; operation data. Exemplarily, the target device includes a terminal device or a server. In an example, the terminal device includes, but is not limited to, a mobile phone, a computer, or a tablet, and the like. In another example, for example, in the scenario where a head-mounted display device is used to participate in an online meeting, the operation data generated by the wearer operating the external device can be sent to the server, so that the server can send the operation data to the participant Other equipment for the meeting.

Exemplarily, the external device is an input device, and the operation data may be input text or image. Exemplarily, the external device is a remote control device of the movable platform, and the operation data may be control content corresponding to the movable platform, such as controlling the acceleration of the movable platform.

In some embodiments, in order to let the user understand the operation mode of the external device, the display image of the head-mounted display device can be superimposed and displayed with the operation prompt information corresponding to the external device, so as to assist the wearer to accurately operate the external device . Exemplarily, the external device is a keyboard, and the operation prompt information corresponding to the external device may be prompt information related to an input method. Exemplarily, the external device is a remote control device including a joystick and/or buttons, and the operation prompt information corresponding to the external device may be the control logic of the movable platform corresponding to different operation modes of the joystick and/or buttons.

In some embodiments, in order to improve operation accuracy, after obtaining the operation data generated by the wearer operating the external device, the wearer's operation can also be superimposed and displayed in the display image of the head-mounted display device. The operation data generated by the external device, so that the user can confirm the accuracy of the operation data.

In some embodiments, the head-mounted display device communicates with other devices, and the display image of the head-mounted display device and the image area corresponding to the external device can also be sent to the remote device, so that other devices can The display image superimposed with the image area corresponding to the external device is displayed in . And the head-mounted display device or the target device corresponding to the head-mounted display device can also send the operation data generated by the wearer's operation of the external device to other devices, so that other devices display on the head-mounted display device The operation data is superimposed on the image.

In an exemplary scenario, such as an online conference scenario, a user may wear a head-mounted display device to participate in an online conference, and the head-mounted display device may directly or indirectly communicate with other devices belonging to the same conference channel, such as The head-mounted display device can communicate with other devices belonging to the same conference channel through the server, and all or part of the content displayed on the head-mounted display device can also be correspondingly displayed on other devices. In another exemplary scenario, such as a live broadcast scenario, the anchor user can wear a head-mounted display device for live broadcast, the head-mounted display device can communicate with the live broadcast server, and the head-mounted display device can display images, superimpose The image area or operation data corresponding to the external device of the host is sent to the live broadcast server, and the live broadcast server can send the content displayed on the head-mounted display device to the audience client who enters the live broadcast channel where the host is located, and the head-mounted display device All or part of the displayed content can also be correspondingly displayed in the viewer client.

In an exemplary embodiment, the external device is an input device (such as a keyboard) as an example for illustration: the head-mounted display device may include one or more cameras, which are placed on the outside of the head-mounted display device and Opposite to the wearing part of the head-mounted display device, it is used to photograph the physical environment in front of or below the wearer. The head-mounted display device includes a posture sensor and an eye-tracking sensor. The head-mounted display device can determine the wearer's head posture based on the data collected by the posture sensor, and determine the wearer's eye gaze range based on the data collected by the eye-tracking sensor. . When the head-mounted display device determines that the wearer is lowering his head based on the wearer's head posture and eye gaze range, the display content within the eye gaze range is a text input component, and the current display content has a text input component, It is determined that the wearer has the intent to enter text (in other words, to operate the keyboard in the physical environment). Furthermore, the head-mounted display device can recognize the original image captured by the camera to obtain the image position corresponding to the external device, and then extract the keyboard image area from the original image according to the image position corresponding to the external device, and display it on the head-mounted display The keyboard image area is superimposed on the display image of the device; the keyboard image area seen by the wearer is consistent with the scale and shape of the keyboard in the physical environment, so that the user can operate the keyboard image in the physical environment based on the seen keyboard image. keyboard. The head-mounted display device can recognize information such as the wearer's finger movements and/or operated keys based on the original image captured by the camera, and obtain the text content entered by the wearer's keyboard in the physical environment.

Various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features, so any combination of various technical features in the above embodiments also falls within the scope of the disclosure of this specification.

In some scenarios, when the user wears the head-mounted display device, there is a need to operate some virtual devices, such as pointing and pressing the virtual device with the finger to realize data input. Since the position of the virtual device in the physical environment is uncertain, the user is The operating position of the virtual device is inaccurate, which makes it less convenient for the user to operate the virtual device.

Based on this, please refer to FIG. 8. The embodiment of the present application also provides another interaction method for a head-mounted display device, which includes:

In step S201, it is detected whether the wearer of the head-mounted display device intends to operate the virtual device.

In step S202, in response to detecting that the wearer intends to operate a virtual device, superimpose and display a virtual device in the display image of the head-mounted display device, the virtual device is based on the head-mounted display device It is generated by the virtual device identification obtained by the camera shooting.

In some embodiments, the virtual device logo can be a pattern, which can be set on different types of carriers, such as printed matter or the screen of an electronic device. When the camera captures the pattern, it can be expanded on the reference surface corresponding to the pattern. A virtual device, so that the user can "see" the virtual device and operate it.

In some embodiments, the virtual device is a virtual input device, and the virtual input device includes at least one of a virtual keyboard, a virtual mouse, a virtual touch screen, and a virtual tablet; or, the virtual device is a virtual remote control device , the head-mounted display device communicates with the movable platform, the head-mounted display device can generate control instructions based on user operations on the virtual device, and directly or indirectly transmit the control instructions to the movable platform, the Images captured by the movable platform can be displayed on the head-mounted display device.

In some embodiments, the detecting whether the wearer of the head-mounted display device intends to operate the virtual device includes: whether the operator has the intention to operate the virtual device. Exemplarily, if the head posture indicates the wearer's head-down posture and/or the display content within the gaze range of the eyes includes a virtual component corresponding to the virtual device, it is determined that the wearer has an operation The intent of the virtual device. Exemplarily, the head posture is determined through the data collected by the posture sensor arranged in the head-mounted display device; and/or, the gaze range of the eyes is determined through the The data collected by the eye tracking sensor is determined.

In some embodiments, the display position of the virtual device in the display image is determined at least according to the relative positional relationship between the wearer and the virtual device identifier, so that the virtual device and the wearer in the display space The relative positional relationship between the physical environment and the relative positional relationship between the wearer and the virtual device logo in the physical environment is adapted to provide a reference surface for the virtual device in the physical environment, ensuring that the user can accurately locate the device by means of the reference surface Virtual equipment, and realize the precise operation of virtual equipment, and improve the accuracy of operation.

Exemplarily, the display position of the virtual device in the display image is based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, and the relationship between the camera and the display. The relative positional relationship between the cameras and the relative positional relationship between the camera and the virtual device is determined.

In some embodiments, the camera is arranged on the outside of the head-mounted display device, facing away from the wearing part of the head-mounted display device.

In some embodiments, the camera is arranged on the head-mounted display device and is rotatably connected to the head-mounted display device. Exemplarily, the method further includes: controlling the rotation of the camera so that a lens assembly of the camera faces the virtual device identification.

In some embodiments, the virtual device is superimposed and displayed in the display image in a semi-transparent or semi-opaque form.

In some embodiments, the transparency of the virtual device is determined according to the color system of the display image in the overlay area. Exemplarily, the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the dark color system is smaller than the transparency corresponding to the color system of the displayed image in the superimposed area belonging to the light color system.

In some embodiments, the head-mounted display device includes a supplementary light; the method further includes: if the brightness of the environment where the virtual device is identified is lower than a preset brightness threshold, turning on the supplementary light.

In some embodiments, the brightness of the supplementary light is adjusted according to the brightness of the environment where the virtual device identifier is located, and the brightness of the supplementary light is negatively correlated with the brightness of the environment where the virtual device identifier is located.

In some embodiments, the image captured by the camera also includes the image area corresponding to the designated part of the wearer; the method further includes: identifying from the image captured by the camera of the head-mounted display device The operating position of the designated part of the wearer on the virtual device; obtaining the operation data generated by the wearer operating the virtual device according to the content indicated by the operating position.

In some embodiments, the method further includes: superimposing and displaying the operation data generated by the wearer operating the virtual device in the display image of the head-mounted display device.

In some embodiments, the method further includes: superimposing and displaying operation prompt information corresponding to the virtual device in the display image of the head-mounted display device.

In some embodiments, further comprising: in response to the wearer having no intention to operate the virtual device, not displaying the virtual device. Exemplarily, if any one or more of the following conditions are met, it is determined that the wearer has no intention of operating the virtual device: the wearer's head posture indicates the wearer's head-up posture, the wearer's eyes The displayed content within the gazing range does not include the virtual component corresponding to the virtual device, or the specified part of the wearer does not exist in the image captured by the camera.

In some embodiments, the displayed image includes a virtual image and/or an image captured from a physical environment.

In some embodiments, the head-mounted display device is connected in communication with a movable platform equipped with a photographing device; the images captured from the physical environment include images collected by the movable platform in real time.

Correspondingly, referring to FIG. 9, the embodiment of the present application also provides a head-mounted display device 100, including:

one or more processors 101;

A memory 102 for storing instructions executable by the processor 101;

A display 103 for displaying images;

The one or more processors 101 individually or jointly execute the executable instructions, so as to perform any one of the methods described above.

The processor 101 executes the executable instructions included in the memory 102, and the processor 101 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory 102 stores executable instructions of the interaction method of the head-mounted display device, and the memory 102 may include at least one type of storage medium, and the storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic storage, magnetic disks, optical disks, etc. Also, the device may cooperate with a network storage device that performs a storage function of the memory through a network connection. The memory 102 may be an internal storage unit of the head-mounted display device 100 , such as a hard disk or memory of the head-mounted display device 100 . The memory 102 can also be an external storage device of the head-mounted display device 100, such as a plug-in hard disk equipped on the head-mounted display device 100, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Further, the memory 102 may also include both an internal storage unit of the head-mounted display device 100 and an external storage device. The memory 102 is used to store computer programs and data. The memory 102 can also be used to temporarily store data that has been output or will be output.

Displays include but are not limited to LCD (Liquid Crystal Display) displays, LED (Light Emitting Diode) displays, or OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) displays.

In an exemplary embodiment, the processor is configured to: detect whether the wearer of the head-mounted display device intends to operate an external device; in response to detecting that the wearer intends to operate an external device, The image captured by the camera of the head-mounted display device is superimposed on the displayed image of the display, and the image captured by the camera includes the image area corresponding to the external device.

Optionally, the external device is an input device, and the input device includes at least one of a keyboard, a mouse, a touch screen, and a tablet; or, the external device is a remote control device, and the remote control device can directly or indirectly The control instruction is transmitted to the movable platform, and the image captured by the movable platform can be displayed on the head-mounted display device.

Optionally, the processor is further configured to detect whether the wearer intends to operate the external device according to the wearer's head posture and/or eye gaze range. Exemplarily, if the head posture indicates the wearer's head-down posture and/or the display content within the gaze range of the eyes includes a virtual component corresponding to the external device, it is determined that the wearer has an operation The intent of the external device.

Optionally, the head-mounted display device includes an attitude sensor and/or an eye tracking sensor. The head posture is determined by data collected by a posture sensor arranged in the head-mounted display device; and/or, the gaze range of the eyes is collected by an eye tracking sensor arranged in the head-mounted display device The data is confirmed.

Optionally, the processor is further configured to obtain an original image captured by the camera of the head-mounted display device; and according to the image position corresponding to the external device in the original image, crop the original image to obtain A cropped image, where the cropped image includes the image area corresponding to the external device; and the cropped image is superimposed and displayed on the displayed image in the display.

Optionally, the display position of the image area corresponding to the external device in the display image is determined at least according to a relative positional relationship between the wearer and the external device. Exemplarily, the display position of the image area corresponding to the external device in the display image is based on the relative positional relationship between the wearer's eyes and the display of the head-mounted display device, the camera and the The relative positional relationship between the displays and the relative positional relationship between the camera and the external device are determined.

Optionally, the camera is arranged outside the head-mounted display device, facing away from the wearing part of the head-mounted display device.

Optionally, the camera is arranged on the head-mounted display device and is rotatably connected to the head-mounted display device. Optionally, the processor is further configured to: control the rotation of the camera so that the lens assembly of the camera faces the external device.

Optionally, the image captured by the camera is superimposed and displayed in the display image in a semi-transparent or semi-opaque form.

Optionally, the transparency of the image captured by the camera is determined according to the color system of the displayed image in the superimposed area. For example, the color system of the display image in the overlay area belongs to the transparency corresponding to the dark color system, which is less than the transparency corresponding to the color system of the display image in the overlay area belongs to the light color system.

Optionally, the head-mounted display device includes a supplementary light; the processor is further configured to turn on the supplementary light if the brightness of the environment where the external device is located is less than a preset brightness threshold. For example, the brightness of the supplementary light is adjusted according to the brightness of the environment where the external device is located, and the brightness of the supplementary light is negatively correlated with the brightness of the environment where the external device is located.

Optionally, the image captured by the camera further includes an image area corresponding to the designated part of the wearer; the processor is further configured to identify the The operating position of the designated part of the wearer on the external device; and the operation data generated by the wearer operating the external device is acquired according to the content indicated by the operating position.

Optionally, the external device is communicatively connected to the head-mounted display device; the processor is further configured to acquire operation data generated by the wearer operating the external device through the communication connection.

Optionally, the external device is communicatively connected to a target device corresponding to the head-mounted display device; the target device is configured to acquire operation data generated by the wearer operating the external device through the communication connection.

Optionally, the target device includes a terminal device or a server.

Optionally, the processor is further configured to superimpose and display the operation data generated by the wearer operating the external device in the display image of the head-mounted display device.

Optionally, the processor is further configured to superimpose and display operation prompt information corresponding to the external device in the display image of the head-mounted display device.

Optionally, the processor is further configured to not display the image captured by the camera of the head-mounted display device in response to the wearer not intending to operate the external device.

Optionally, the processor is further configured to determine that the wearer has no intention to operate the external device if any one or more of the following conditions are met: the wearer's head posture indicates the wearer's head-up posture , the displayed content within the gaze range of the wearer's eyes does not include the virtual component corresponding to the external device, or there is no designated part of the wearer in the image captured by the camera.

Optionally, the displayed image includes a virtual image and/or an image captured from a physical environment.

Optionally, the head-mounted display device is communicatively connected to a movable platform equipped with a photographing device; the images captured from the physical environment include images collected by the movable platform in real time.

Various implementations described herein can be implemented using a computer readable medium such as computer software, hardware, or any combination thereof. For hardware implementation, the embodiments described herein can be implemented by using Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays ( FPGA), processors, controllers, microcontrollers, microprocessors, electronic units designed to perform the functions described herein. For software implementation, an embodiment such as a procedure or a function may be implemented with a separate software module that allows at least one function or operation to be performed. The software codes can be implemented by a software application (or program) written in any suitable programming language, stored in memory and executed by a controller.

For the implementation process of the functions and functions of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method for details, and details will not be repeated here.

Correspondingly, referring to FIG. 1 , the embodiment of the present application also provides an interactive system, including a head-mounted display device 100 and an external device 200 .

For example, please refer to Figure 2, the external device is an input device, and the input device includes at least one of a keyboard, mouse, touch screen, and tablet; or, please refer to Figure 4, the external device is a remote control device, the remote control device can directly or indirectly transmit control commands to the movable platform, and the images captured by the movable platform can be displayed on the head-mounted display device.

Exemplarily, please refer to FIG. 4 , the system further includes a movable platform, and the movable platform is communicatively connected with the head-mounted display device.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, which are executable by a processor of an apparatus to perform the above method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the terminal, the terminal can execute the above method. It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. The term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The methods and devices provided by the embodiments of the present application have been described in detail above. The principles and implementation methods of the present application have been explained by using specific examples in this paper. The descriptions of the above embodiments are only used to help understand the methods and methods of the present application. core idea; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application .

Claims

An interaction method for a head-mounted display device, characterized in that the method includes:

Detecting whether the wearer of the head-mounted display device intends to operate an external device;

In response to detecting that the wearer has an intention to operate an external device, superimposing and displaying an image captured by a camera of the head-mounted display device on the display image of the head-mounted display device, the image captured by the camera An image area corresponding to the external device is included.
The method according to claim 1, wherein the external device is an input device, and the input device includes at least one of a keyboard, a mouse, a touch screen and a tablet; or,

The external device is a remote control device, and the remote control device can directly or indirectly transmit control commands to the movable platform, and images captured by the movable platform can be displayed on the head-mounted display device.
The method according to claim 1, wherein the detecting whether the wearer of the head-mounted display device intends to operate an external device comprises:

Whether the wearer intends to operate the external device is detected according to the wearer's head posture and/or eye gaze range.
The method according to claim 3, wherein if the head posture indicates the wearer's head-down posture and/or the display content within the gaze range of the eyes includes virtual images corresponding to the external device A component that determines that the wearer has an intent to operate the external device.
The method according to claim 3, wherein the head posture is determined by data collected by a posture sensor arranged in the head-mounted display device;

And/or, the gaze range of the eyes is determined by data collected by an eye tracking sensor arranged in the head-mounted display device.
The method according to claim 1, wherein the superimposing the image captured by the camera of the head-mounted display device on the display image of the head-mounted display device includes:

Obtain the original image captured by the camera of the head-mounted display device;

cropping the original image according to the image position corresponding to the external device in the original image to obtain a cropped image, where the cropped image includes an image area corresponding to the external device;

The cropped image is superimposed and displayed in the displayed image.
The method according to claim 1 or 6, wherein the display position of the image area corresponding to the external device in the displayed image is determined at least according to the relative positional relationship between the wearer and the external device .
The method according to claim 7, wherein the display position of the image area corresponding to the external device in the display image is based on the distance between the wearer's eyes and the display of the head-mounted display device. The relative positional relationship between the camera and the display, and the relative positional relationship between the camera and the external device are determined.
The method according to claim 1, wherein the camera is arranged outside the head-mounted display device, and is opposite to the wearing part of the head-mounted display device.
The method according to claim 1, wherein the camera is arranged on the head-mounted display device and is rotatably connected to the head-mounted display device.
The method according to claim 10, characterized in that the method further comprises:

The camera is controlled to rotate so that the lens assembly of the camera faces the external device.
The method according to claim 1, wherein the image captured by the camera is superimposed and displayed in the display image in a semi-transparent or semi-opaque form.
The method according to claim 1, wherein the transparency of the image captured by the camera is determined according to the color system of the displayed image in the overlapping area.
The method according to claim 13, wherein the color system of the displayed image in the superimposed area belongs to the transparency corresponding to the dark color system, which is smaller than the transparency corresponding to the color system of the displayed image in the superimposed area belongs to the light color system.
The method according to claim 1, wherein the head-mounted display device comprises a fill light;

The method also includes:

If the brightness of the environment where the external device is located is less than a preset brightness threshold, the supplementary light is turned on.
The method according to claim 15, wherein the brightness of the supplementary light is adjusted according to the brightness of the environment where the external device is located, and the brightness of the supplementary light and the brightness of the environment where the external device is located into a negative correlation.
The method according to any one of claims 1 to 16, wherein the image captured by the camera also includes an image area corresponding to the designated part of the wearer;

The method also includes:

From the image captured by the camera of the head-mounted display device, identify the operating position of the designated part of the wearer on the external device;

The operation data generated by the wearer operating the external device is acquired according to the content indicated by the operation position.
The method according to any one of claims 1 to 16, wherein the external device is communicatively connected to the head-mounted display device;

The method also includes:

The operation data generated by the wearer operating the external device is acquired through the communication connection.
The method according to any one of claims 1 to 16, wherein the external device is communicatively connected to the target device corresponding to the head-mounted display device;

The target device is configured to acquire the operation data generated by the wearer operating the external device through the communication connection.
The method according to claim 19, wherein the target device comprises a terminal device or a server.
The method according to any one of claims 1, 17 to 19, further comprising:

The operation data generated by the wearer operating the external device is superimposed on the display image of the head-mounted display device.
The method according to claim 1, further comprising:

The operation prompt information corresponding to the external device is superimposed and displayed in the display image of the head-mounted display device.
The method according to claim 1, further comprising:

In response to the wearer not intending to operate the external device, the image captured by the camera of the head-mounted display device is not displayed.
The method according to claim 23, wherein, if any one or more of the following conditions are met, it is determined that the wearer has no intention to operate the external device: the wearer's head posture indicates the wearer's The head-up posture, the display content within the gaze range of the wearer's eyes does not include the virtual component corresponding to the external device, or the specified part of the wearer does not exist in the image captured by the camera.
The method according to any one of claims 1 to 24, wherein the displayed image includes a virtual image and/or an image captured from a physical environment.
The method according to claim 25, characterized in that, the head-mounted display device is communicatively connected to a movable platform equipped with a camera;

The images captured from the physical environment include images collected by the movable platform in real time.
An interaction method for a head-mounted display device, characterized in that the method includes:

Detecting whether the wearer of the head-mounted display device intends to operate the virtual device;

In response to detecting that the wearer has an intention to operate the virtual device, superimposing and displaying the virtual device in the display image of the head-mounted display device, the virtual device is obtained based on the camera of the head-mounted display device The virtual device ID is generated.
A head-mounted display device, characterized in that it includes:

one or more processors;

memory for storing said processor-executable instructions;

a display for displaying images;

The one or more processors individually or jointly execute the executable instructions to perform the method as claimed in any one of claims 1-27.
An interactive system, characterized by comprising the head-mounted display device as claimed in claim 28 and external devices.
The interactive system according to claim 29, wherein the external device is an input device, and the input device includes at least one of a keyboard, a mouse, a touch screen, and a tablet; or,

The external device is a remote control device, and the remote control device can directly or indirectly transmit control commands to the movable platform, and images captured by the movable platform can be displayed on the head-mounted display device.
The interactive system of claim 29, further comprising a movable platform.
A computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the method according to any one of claims 1 to 27 is implemented.