CN115131547A

CN115131547A - Method, device and system for image interception by VR/AR equipment

Info

Publication number: CN115131547A
Application number: CN202110322569.XA
Authority: CN
Inventors: 姜永涛; 许哲; 卢曰万; 郜文美
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-03-25
Filing date: 2021-03-25
Publication date: 2022-09-30
Also published as: WO2022199597A1

Abstract

The application provides a method, a device and a system for intercepting images by VR/AR equipment. In the method, the VR/AR device displays an image, and in the process of presenting a 3D scene for a user, the user can input an operation after seeing the image presented by the VR/AR device, and the VR/AR device can intercept the image in response to the operation. By implementing the method, the personalized requirements of the user can be met, and the user experience is improved.

Description

Method, device and system for intercepting images by VR/AR equipment

Technical Field

The application relates to the field of terminals, in particular to a method, a device and a system for intercepting an image by VR/AR equipment.

Background

Virtual Reality (VR) technology uses computer simulation to create a three-dimensional (3D) virtual reality scene and provides a simulated experience in the visual, auditory, tactile, or other senses that the user feels as if he or she is experiencing it. Augmented Reality (AR) technology may display a virtual image for a user by superimposing virtual reality while the user views the real world, and the user may interact with the virtual image to achieve an effect of augmented reality.

With the popularization of VR and AR devices, more and more users begin to experience VR and AR technologies, and how to meet the increasing personalized requirements of users is a major problem in current and future research.

Disclosure of Invention

The application provides a method, a device and a system for intercepting an image by VR/AR equipment, which can meet the personalized requirements of a user, intercept the image in a region of interest (ROI) of the user, and improve the user experience.

In a first aspect, the present application provides a method for capturing an image by a VR/AR device, where the method is applied to the VR/AR device, the VR/AR device includes an optical component for providing a 3-dimensional 3D scene and is worn on a head of a user, and the method for capturing the image includes: the VR/AR device displays a first user interface; and the VR/AR equipment responds to the first operation, determines an ROI in a display screen of the VR/AR equipment, and intercepts a first image, wherein the first image is an image in the ROI.

By implementing the method provided by the first aspect, the VR/AR device can intercept the image in the ROI according to the operation of the user, thereby satisfying the user personalization and improving the user experience.

With reference to the first aspect, in one embodiment, the first operation includes: the operation of moving the hand of the user, or the operation of moving the gaze point of the eyes of the user, or the operation of moving an input device connected with the VR/AR device.

Therefore, the user can realize the enforceability of intercepting the image through various different operations, and the operability of the user is improved.

With reference to the first aspect, in an embodiment, before the VR/AR device determines the ROI, the method for the VR/AR device to intercept the image further includes: the VR/AR device responds to the first operation, displays a moving track of a cursor on a display screen, wherein the moving track of the cursor corresponds to a moving track of a hand of a user, or a moving track of a fixation point of eyeballs of the user, or a moving track of an input device; the ROI is a region containing a movement trajectory of the cursor.

Therefore, the user can watch the moving track of the cursor in the display screen of the VR/AR equipment, the interaction process of the user and the VR/AR equipment is further displayed, and the user experience is improved.

With reference to the first aspect, in an embodiment, the ROI is a region including a movement trajectory of the cursor, for example, the ROI may be a minimum rule region including the movement trajectory of the cursor; alternatively, the ROI is a minimum irregular region including the movement trajectory of the cursor, and the minimum irregular region is a region surrounded by the movement trajectory of the cursor.

Therefore, the VR/AR equipment can determine multiple possible ROIs according to the operation of the user, and the user experience is improved.

With reference to the first aspect, in an embodiment, after the VR/AR device intercepts the first image in response to the first operation, the method further includes: the VR/AR device displays a second user interface that displays the identification of the one or more electronic devices; and the VR/AR equipment responds to a second operation of selecting the first identification, sends the first image to the first equipment, the first equipment corresponds to the first identification, and the identification of one or more electronic equipment in the second user interface comprises the first identification.

Therefore, the user can send the intercepted first image to any one or more electronic devices around, user requirements are met, and user experience is improved.

With reference to the first aspect, in one embodiment, before the VR/AR device displays the second user interface, the method further includes: the VR/AR device detects a third operation; in response to the third operation, the VR/AR device displays a second user interface.

Therefore, the VR/AR device can display the second user interface after receiving the operation of triggering the display of the identification of the surrounding electronic device, and the personalized requirements of the user are met.

With reference to the first aspect, in one implementation, the location of the identification of the one or more electronic devices in the second user interface is used to indicate the location of the one or more electronic devices relative to the VR/AR device.

With reference to the first aspect, in an embodiment, the identification of the one or more electronic devices includes an image of the one or more electronic devices captured by the VR/AR device, and before the VR/AR device displays the second user interface, the method further includes: the VR/AR device captures images of one or more electronic devices; the VR/AR device determines a location of the one or more electronic devices relative to the VR/AR device based on the image of the one or more electronic devices.

In this way, a user may perceive the location of one or more electronic devices in the surroundings relative to themselves by viewing the identity of the one or more electronic devices, providing a more immersive experience.

With reference to the first aspect, in one implementation, the identification of the one or more electronic devices includes one or more of: an icon, type, or model of one or more electronic devices; after the VR/AR device captures images of one or more electronic devices, the method further comprises: the VR/AR device obtains one or more of an icon, a type, or a model of the one or more electronic devices from the image of the one or more electronic devices.

Therefore, the VR/AR equipment can display images, virtual icons, types or models and the like of one or more electronic equipment according to the requirements of the user, provide rich content for the user and meet the personalized requirements of the user.

With reference to the first aspect, in one embodiment, before the VR/AR device displays the second user interface, the method further comprises: after the VR/AR equipment sends the first request message, the VR/AR equipment receives a first response message sent by one or more electronic equipment, and the first response message carries the communication address of the one or more electronic equipment; the VR/AR equipment acquires the positions of one or more electronic equipment relative to the VR/AR equipment according to the receiving condition of the first response message; the VR/AR device sends the first image to the first device in response to the second operation of selecting the first identifier, and specifically includes: the VR/AR device determines the position of the first device according to the corresponding relation between the images of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device; the VR/AR device determines a communication address of the first device according to the corresponding relation between the communication addresses of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device; and the VR/AR equipment sends the first image to the first equipment according to the communication address of the first equipment.

Therefore, the VR/AR equipment can share the first image with the first equipment according to the operation of selecting the first identifier of the first equipment by the user and the communication address corresponding to the first identifier, and the personalized requirements of the user are met.

With reference to the first aspect, in one embodiment, the first operation includes one or more of: gestures, voice commands, eye states of users, and operations of pressing keys; the first operation is detected by the VR/AR device or, alternatively, by the input device.

Therefore, the realization mode of the first operation can be various, the implementability of the screenshot method provided by the embodiment of the application is improved, and the user experience is improved.

With reference to the first aspect, in an embodiment, after the VR/AR device intercepts the first image in response to the first operation, the first image may be further saved.

Therefore, the user can further operate the first graph conveniently, for example, the first graph is shared in other electronic equipment, and user experience is improved.

In a second aspect, embodiments of the present application provide a VR/AR device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories being configured to store computer program code comprising computer instructions that, when executed by the one or more processors, cause the VR/AR device to perform the method as described in an embodiment of the first aspect.

In a third aspect, the present application provides a computer program product containing instructions, which is characterized in that when the computer program product is run on an electronic device, the electronic device is caused to execute the method described in the embodiment of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, which includes instructions, and is characterized in that when the instructions are executed on an electronic device, the electronic device is caused to execute the method described in the implementation manner of the first aspect.

After the method for intercepting the image by the VR/AR equipment is implemented, the VR/AR equipment can intercept the image in the ROI, and the image is shared to any one or more electronic equipment around, so that the user experience is improved.

Drawings

Fig. 1 is a schematic diagram of a communication system provided in an embodiment of the present application;

fig. 2A is a schematic structural diagram of a VR/AR device according to an embodiment of the present application;

FIG. 2B is a schematic diagram illustrating an imaging principle of a VR/AR device provided by an embodiment of the application;

FIG. 3 is a flowchart of a method for capturing an image according to an embodiment of the present disclosure;

4A-4B are a set of user interfaces for intercepting images provided by embodiments of the present application;

5A-5E are another set of user interfaces for intercepting images provided by embodiments of the present application;

fig. 6 is a user interface for sharing images according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described in detail and clearly with reference to the accompanying drawings. Wherein in the description of the embodiments of the present application, "/" indicates an inclusive meaning, for example, a/B may indicate a or B; "and/or" in the text is only an association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: three cases of a alone, a and B both, and B alone exist, and in addition, "a plurality" means two or more than two in the description of the embodiments of the present application.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of embodiments of the application, unless stated otherwise, "plurality" means two or more.

The term "User Interface (UI)" in the following embodiments of the present application is a media interface for interaction and information exchange between an application program or an operating system and a user, and implements conversion between an internal form of information and a form acceptable to the user. The user interface is source code written by java, extensible markup language (XML) and other specific computer languages, and the interface source code is analyzed and rendered on the electronic equipment and finally presented as content which can be identified by a user. A commonly used presentation form of the user interface is a Graphical User Interface (GUI), which refers to a user interface related to computer operations and displayed in a graphical manner. It may be a visual interface element such as text, an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc. displayed in the display of the electronic device. The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments herein only and is not intended to be limiting of the application.

A user may want to intercept an image in the process of experiencing VR/AR, and how VR/AR equipment intercepts the image is a problem to be solved urgently.

In the method, the VR/AR device displays an image, and can input a first operation after seeing the image displayed by the VR/AR device in the process of presenting a 3D scene for a user, and the VR/AR device can intercept the first image in response to the first operation. In the embodiment of the application, when the VR/AR device presents a 3D scene to a user, an image displayed on a display screen of the VR/AR device and an image seen by the user may be the same or different.

The first image may be any one of:

1. all or part of the image displayed on the display screen of the VR/AR device.

In some embodiments, the VR/AR device may intercept all images displayed on the display screen as the first image in response to the first operation.

In some embodiments, the VR/AR device may determine a region of interest (ROI) of a user in the display screen in response to the first operation, and then truncate an image in the ROI on the display screen as the first image. In the embodiment of the present application, the ROI refers to a region in the display screen determined by the VR/AR device according to the user operation, where the ROI determined by the user operation may be a moving track of a cursor displayed on the display screen correspondingly according to a moving track of a hand of the user, a moving track of a gaze point of an eyeball, or a moving track of the input device 300, and then the VR/AR device determines the ROI according to the moving track of the cursor, which is specifically described in detail in the method embodiments below. The ROI may be a regular region and the ROI may be an irregular region.

2. And when the VR/AR device presents the VR/AR scene for the user, all images or part of images seen by the user.

In some embodiments, based on the imaging principle of the VR/AR device, the image displayed on the display screen of the VR/AR device and the image seen by the user may be different, so that when the VR/AR device intercepts all or part of the image displayed on the display screen in response to the first operation, the VR/AR device may further need to fuse the intercepted all or part of the image into the image seen by the user according to the imaging principle of the VR/AR device, so as to serve as the first image. Reference is made, inter alia, to the following description of the VR/AR device with regard to its imaging principles.

In some embodiments, the VR/AR device may also share the first image onto surrounding electronic devices.

In particular, the VR/AR device, in response to the detected user action, may display a user interface including an identification of surrounding electronic devices, where a location of the identification of the electronic device in the user interface indicates a location of the electronic device relative to the user. The identification of the electronic device includes, but is not limited to: an image, a virtual icon, a type, a model, etc. of the electronic device. After the user sees the user interface, the position of the surrounding electronic devices relative to themselves can be identified. And finally, the VR/AR equipment responds to the detected operation of the identification of the selected electronic equipment, establishes communication connection with the electronic equipment corresponding to the identification, and shares the first image through the communication connection.

After the method for intercepting the image by the VR/AR device is implemented, the VR/AR device can intercept the first image at any time, the size of the intercepted first image can be defined by the user, in addition, the user can share the intercepted first image to surrounding electronic devices, other users can see the first image, and personalized requirements of the user are met.

Referring to fig. 1, fig. 1 illustrates a communication system 10 provided by an embodiment of the present application.

As shown in fig. 1, the communication system 10 includes a VR/AR device 100, one or more electronic devices such as an electronic device 201, an electronic device 202, an electronic device 203, and an electronic device 204 surrounding the VR/AR device 100. In some embodiments, communication system 10 may also include an input device 300.

The VR/AR device 100 may be a helmet, glasses, or other electronic device that can be worn on the head of a user, and the VR/AR device 100 may be used with other electronic devices (e.g., mobile phones) to receive and display data or content (e.g., rendered images) processed by the GPU of the other electronic devices (e.g., mobile phones). In this case, the VR/AR device 100 may be a terminal device such as VR glasses with limited computing power. The VR/AR device 100 is used to display images to present a 3D scene to a user, giving the user a VR/AR/MR experience. The 3D scene may include 3D images, 3D video, and so on.

In an embodiment of the application, the VR/AR device 100 may intercept the first image in response to the detected first operation. Reference may be made to the above detailed description regarding the definition of the first image. The specific step of capturing the first image may refer to the following description of the method embodiment.

In some embodiments, the VR/AR device, in response to the detected user operation, may display a user interface containing an identification (e.g., image, virtual icon, type, model, etc.) of the surrounding electronic devices. Then, the VR/AR device responds to the detected operation of selecting the identifier of the electronic device, for example, a gesture pointing to the identifier of the electronic device in the user interface, establishes a communication connection with the electronic device corresponding to the identifier, and shares the first image through the communication connection.

The communication connection may be a wired or wireless connection. The wired connection may include a wired connection that communicates through an interface such as a USB interface, an HDMI interface, or the like. The wireless connection may include a wireless connection that communicates through one or more of a BT communication module, a Wireless Local Area Network (WLAN) communication module, and a UWB communication module.

The

electronic devices

201, 202, 203 and 204 are electronic devices around the VR/AR device 100, such as smart televisions, computers, mobile phones, VR/AR devices, tablet computers, and non-portable terminal devices such as Laptop computers (Laptop computers) with touch-sensitive surfaces or touch panels, desktop computers with touch-sensitive surfaces or touch panels, and the like.

In the embodiment of the present application, the electronic device 201, the electronic device 202, the electronic device 203, and the electronic device 204 have one or more of a BT communication module, a WLAN communication module, and a UWB communication module. Taking the electronic device 201 as an example, the electronic device 201 may listen to a signal, such as a probe request, a scan signal, etc., transmitted by the VR/AR device 100 through one or more of the BT communication module, the WLAN communication module, and the UWB communication module, and may transmit a response signal, such as a probe response, a scan response, etc., so that the VR/AR device 100 may discover the electronic device 201, determine the location and the network address of the electronic device 201, then establish a communication connection with the electronic device 201, and receive the first image shared by the VR/AR device 100 based on the communication connection.

The input device 300 is a device for controlling the VR/AR device 100, such as a handle, a mouse, a keyboard, a stylus, a bracelet, and so forth.

In the embodiment of the present application, the input device 300 may be configured with various sensors, such as an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, and the like. The pressure sensor may be disposed under the confirm button/cancel button of the input device 300.

In the embodiment of the present application, the input device 300 may collect movement data of the input device 300 and data indicating whether a confirm button/cancel button, etc. of the input device 300 is pressed. The movement data includes a sensor of the input device 300, such as an acceleration sensor, for acquiring an acceleration of the input device 300, a gyroscope sensor for acquiring a movement speed of the input device 300, and the like. The data indicating whether the confirm button/cancel button or the like of the input device 300 is pressed includes a pressure value collected by a pressure sensor provided under the confirm button/cancel button, a level generated by the input device 300, and the like.

In this embodiment, the input device 300 may establish a communication connection with the VR/AR device 100, and transmit the collected movement data of the input device 300 and data indicating whether a confirmation key/cancel key or the like of the input device 300 is pressed to the VR/AR device 100 through the communication connection. The communication connection may be a wired or wireless connection. The wired connection may include a wired connection that communicates through a USB interface, an HDMI interface, a custom interface, or the like. The wireless connection may include one or more of wireless connection communicating through a Near Field Communication (NFC), ZigBee, or other close range transmission technology.

In the embodiment of the present application, the movement data of the input device 300, and the data indicating whether the confirm key/cancel key, etc. of the input device 300 is pressed may be used by the VR/AR device 100 to determine the movement condition and/or state of the input device 300. The movement of the input device 300 may include, but is not limited to: whether to move, direction of movement, speed of movement, distance moved, trajectory moved, etc. The state of input device 300 may include: a confirmation key of the input device 300 is pressed.

The VR/AR device 100 may initiate the corresponding function based on the movement and/or state of the input device 300. That is, the user may trigger the VR/AR device 100 to perform a corresponding function by inputting a user operation on the input device 300. For example, the user may hold the input device 300 and move 3cm to the left to cause a cursor displayed on a display screen of the VR/AR device 100 to move 6cm to the left. This may allow a user to move a cursor to any location on a display screen in the VR/AR device 100 through manipulation of the input device 300. For another example, after a cursor is moved over a control displayed in the VR/AR device 100, the user may press a confirmation button of the input device 300 to cause the VR/AR device 100 to enable the function corresponding to the control by the VR/AR device 100.

In this embodiment of the application, the input device 300 may be configured to receive a user operation for triggering the VR/AR device to intercept the first image, and specific implementation of the user operation may refer to the following description related to the method embodiment.

Referring to fig. 2A, fig. 2A shows a schematic structural diagram of a VR/AR device 100 provided in an embodiment of the present application.

As shown in FIG. 2A, the VR/AR device 100 may include: processor 201, memory 202, communication module 203, sensor module 204, camera 205, display device 206, audio device 207. The above components may be coupled and in communication with each other, for example, the above components may be connected by a bus.

It is to be understood that the configuration shown in FIG. 2A does not constitute a specific limitation on the VR/AR device 100. In other embodiments of the present application, the VR/AR device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. For example, the VR/AR device 100 may also include physical keys such as on/off keys, volume keys, various interfaces such as a USB interface for supporting the connection of the VR/AR device 100 and the input device 300, and so on. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 201 may include one or more processing units, such as: the processor 110 may include a memory, an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals, and complete the control of instruction fetching and instruction execution, so that each component can execute corresponding functions, such as man-machine interaction, motion tracking/prediction, rendering display, audio processing and the like.

In the embodiment of the present application, the memory in the processor 201 may recognize the operation input by the user and perform the corresponding function. For example, the processor 201 may recognize a gesture input by the user such as a fist, a palm, and the like, a voice such as a screen shot, a share, and the like, and an operation acting on the input device 300 such as shaking up and down, shaking left and right, moving, and the like, and then determine to perform a corresponding function according to the operations.

The memory 202 stores executable program code for performing the display method provided by the embodiments of the present application, the executable program code including instructions. The memory 202 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data) created during use of the VR/AR device 100, and the like. Further, the memory 202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like. The processor 201 executes various functional applications of the VR/AR device 100 and data processing by executing instructions stored in the memory 202 and/or instructions stored in a memory disposed in the processor.

In an embodiment of the application, the memory 202 may store an image for display by the display device 206 of the VR/AR device 100 and send the image to the display device 206 in the form of an image stream.

In this embodiment, the memory 202 may also store the first image intercepted by the VR/AR device 100.

In some embodiments, the memory 202 may store a database including image models of the electronic device 201, the electronic device 202, the electronic device 203, the electronic device 204, and other more electronic devices, and types, virtual icons, models, and the like of the electronic devices corresponding to the image models of the respective electronic devices. For the contents specifically contained in the database, see the detailed description of table 1 in the method examples hereinafter. In some embodiments, the memory 202 may also store images of surrounding electronic devices to which the camera 205 captured the device, locations and communication addresses of the surrounding electronic devices, and correspondence therebetween. For details, see the detailed description in table 2, which will not be repeated herein.

The communication module 203 may include a wireless communication module and a wired communication module. The wireless communication module may provide solutions for wireless communication of BT, WLAN, UWB, GNSS, FM, IR, etc. applied on the VR/AR device 100. The wireless communication module may be one or more devices integrating at least one communication processing module. The wired communication module may provide wired connections including communication through a USB interface, an HDMI interface, or the like. The communication module 203 may support communication between the VR/AR device 100 and surrounding electronic devices (electronic device 201, electronic device 202, electronic device 203, electronic device 204, etc.).

In this embodiment, the VR/AR device 100 may establish a communication connection with the input device 300 through a wireless communication module or a wired communication module, and acquire the movement data of the input device 300 collected by the input device 300 and data indicating whether a confirmation key/cancel key, etc. of the input device 300 is pressed based on the communication connection. Thereafter, the processor 201 may identify an operation input by the user according to the data transmitted by the input device 300, and determine a function to be performed corresponding to the operation.

In some embodiments, the VR/AR device 100 may establish a communication connection with surrounding electronic devices via one or more of BT, WLAN, UWB, etc. communication modules, and share the first image based on the communication connection.

The sensor module 204 may include an accelerometer, compass, gyroscope, magnetometer, or other sensor for detecting motion, etc. The sensor module 204 is used for acquiring corresponding data, for example, an acceleration sensor acquires acceleration of the VR/AR device 100, a gyroscope sensor acquires a movement speed of the VR/AR device 100, and the like. The data collected by the sensor module 204 may reflect the movement of the head of the user wearing the VR/AR device 100. In some embodiments, the sensor module 204 may be an Inertial Measurement Unit (IMU) disposed within the VR/AR device 100. In some embodiments, the VR/AR device 100 may send data acquired by the sensor module to the processor 201 for analysis. The processor 201 may determine the movement of the head of the user according to the data collected by the sensors, and execute corresponding functions according to the movement of the head of the user.

The sensor module 204 may also include optical sensors for tracking the user's eye position and capturing eye movement data in conjunction with the camera 205. This eye movement data may be used, for example, to determine the inter-eye distance of the user, the 3D position of each eye relative to the VR/AR device 100, the magnitude and gaze direction of the twist and rotation (i.e., roll, pitch, and roll) of each eye, and so on. In one example, infrared light is emitted within the VR/AR device 100 and reflected from each eye, the reflected light is detected by the camera 205 or optical sensor, and the detected data is transmitted to the processor 201 such that the processor 201 analyzes the position, pupil diameter, motion state, etc. of the user's eyes from changes in the infrared light reflected by each eye.

In the embodiment of the present application, the optical sensor may be combined with the camera 205 to collect an image of an eyeball of the user, and track a movement track of a fixation point of the eyeball of the user. And transmitting the data of the movement track of the gaze point to the processor 201, the processor 201 analyzing the data, displaying the movement track of the cursor corresponding to the movement track of the eye gaze point on the display screen, and then determining the ROI on the display screen according to the movement track of the cursor.

The camera 205 may be used to capture still images or video. The still image or video may be an externally facing image or video of the user's surroundings, or may be an internally facing image or video. The camera 205 may track the movement of one or both eyes of the user. The camera 205 includes, but is not limited to, a conventional color camera (RGB camera), a depth camera (RGB depth camera), a Dynamic Vision Sensor (DVS) camera, and the like. The depth camera can acquire depth information of a subject to be photographed.

In the embodiment of the present application, the camera 205 includes at least a pair of cameras, and the pair of cameras can acquire images facing the inside of the VR/AR device 100, that is, images of eyes of the user, and can also acquire images facing the outside of the VR/AR device 100, that is, images of hands of the user. The pair of cameras may send the collected images of the eyes of the user or the images of the hands of the user to the processor 201 for analysis, and the processor 201 may identify the state of the eyes of the user, such as whether the eyes of the user blink, the number of blinks, and the like, according to the eye images, and execute corresponding functions according to the state of the eyes of the user. Alternatively, the processor 201 may also recognize the user's gesture such as fist making, palm, "OK", etc. according to the hand image, and correspondingly perform different functions according to different gestures.

In some embodiments, the camera 205 may also include two pairs of cameras, one pair for capturing images externally facing the outside of the VR/AR device 100 and the other pair for capturing images internally facing the VR/AR device 100

In embodiments of the present application, the VR/AR device 100 may render or display images via the GPU, the display device 206, and the application processor 100.

The GPU is a microprocessor for image processing, and is connected to the display device 206 and an application processor. Processor 201 may include one or more GPUs that execute program instructions to generate or alter display information.

The display device 206 may include: one or more display screens, one or more optical components. The one or more display screens may include, for example, display screen 101 and display screen 103. The one or more optical components include, for example, optical component 102 and optical component 104. The display screens, such as the display screen 101 and the display screen 103, may include a display panel, which may be used to display images to present a stereoscopic virtual scene to a user. The display panel can adopt a liquid crystal display device LCD, OLED, AMOLED, FLED, Miniled, MicroLED, Micro-oLED, QLED and the like. Optical components, such as the optical component 102, the optical component 104, are used to direct light from the display screen to the exit pupil for perception by the user. In some implementations, one or more optical elements (e.g., lenses) in an optical assembly can have one or more coatings, such as an anti-reflective coating. The magnification of the image light by the optical assembly allows the display screen to be physically smaller, lighter, and consume less power. In addition, the magnification of the image light may increase the field of view of the content displayed by the display screen. For example, the optical assembly may cause the field of view of the content displayed by the display screen to be the full field of view of the user. The optical assembly may also be used to correct one or more optical errors. Examples of optical errors include: barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma, field curvature, astigmatism, and the like. In some embodiments, the content provided to the display screen is pre-distorted, and the distortion is corrected by the optical assembly when the content-based image light is received from the display screen.

Referring to FIG. 2B, FIG. 2B is a schematic diagram of the imaging principle of the VR/AR device 100.

As shown in FIG. 2B, the display device 206 of the VR/AR device 100 may include: display screen 101, optical assembly 102, display screen 103, optical assembly 104. A first line at which the center of the display screen 101 and the center of the optical assembly 102 are located, and a third line at which the center of the optical assembly 102 and the center of the optical assembly 104 are located, are perpendicular. The display screen 101 and the optical assembly 102 correspond to the left eye of the user. The image a1 may be displayed on the display screen 101 when the user is wearing the VR/AR device 100. The light emitted by the display screen 101 when displaying image a1 will form a virtual image a1 'of image a1 in front of the user's left eye after transmission through the optical assembly 102.

A second line at the center of the display screen 103 and the center of the optical component 104 is perpendicular to a third line at the center of the optical component 102 and the center of the optical component 104. The display screen 103 and the optical assembly 104 correspond to the right eye of the user. The display screen 103 may display an image a2 when the user is wearing the VR/AR device 100. The light emitted by the display screen 103 when displaying image a2 will form a virtual image a2 'of image a2 in front of the user's right eye after transmission through the optical assembly 104.

The image a1 and the image a2 are two images having parallax for the same object, for example, the object a. Parallax is the difference in the position of an object in the field of view when the same object is viewed from two points at a distance. The virtual image a1 'and the virtual image a 2' are located on the same plane, which may be referred to as an virtual image plane.

While wearing the VR/AR device 100, the user's left eye will focus on virtual image a 1', and the user's right eye will focus on virtual image a 2'. Then, the virtual image a1 ' and the virtual image a2 ' are superimposed into a complete and stereoscopic image in the user's brain, which is called vergence. During convergence, the convergence point of the two eye lines is considered by the user to be the position where the object described by the images a1 and a2 is actually located. Due to the vergence process, the user can feel the 3D scene provided by the VR/AR device 100.

Not limited to the imaging principles of the VR/AR device 100 illustrated in FIG. 2B described above, the VR/AR device may also use other ways to provide a 3D scene to a user.

It is understood that the display device 206 may be used to receive and display data or content (e.g., rendered images) processed by a GPU of another electronic device (e.g., a cell phone). In this case, the VR/AR device 100 may be a terminal device such as VR glasses with limited computing power, and may be used with other electronic devices (e.g., mobile phones) to present a 3D scene to the user, thereby providing the VR/AR/MR experience to the user.

The audio device 207 is used for acquiring and outputting audio. Audio device 207 may include, but is not limited to: microphones, speakers, headphones, and the like.

In an embodiment of the present application, a microphone may be used to receive voice input by a user and convert the voice into an electrical signal. The processor 201 may be configured to receive an electrical signal corresponding to a voice from the microphone, and upon receiving the voice signal, the processor 201 may recognize an operation corresponding to the voice and perform the corresponding operation. For example, when the microphone receives a "sharing" voice, the processor 201 may recognize the voice instruction to share the first image with surrounding electronic devices (e.g., the electronic device 201).

In this embodiment of the application, the camera 205 may continuously capture a plurality of palm images of the user, and transmit the palm images to the processor 201, the processor 201 analyzes the plurality of palm images by using a binocular positioning technology, identifies a moving track of the palm of the user, displays a moving track of a cursor corresponding to the moving track of the palm on the display screen, and then determines the ROI according to the moving track. The processor 201 may then clip the image in the ROI off the display screen as a first image. In some embodiments, the processor 201 may retrieve the image stream sent by the internal memory to the display device 206 to obtain the first image. In other embodiments, the processor 201 may directly capture the image displayed by the display device 206 and cut the image to obtain the first image. In some embodiments, the camera 205 may capture an image of a surrounding electronic device and send the image of the surrounding electronic device to the processor 201, the processor 201 analyzes the image of the surrounding electronic device using a binocular positioning technique to identify a location, e.g., three-dimensional coordinates, of the surrounding electronic device relative to the VR/AR device 100, and the processor 201 may also match the image of the surrounding electronic device with an image model of each electronic device in the database to identify a type, virtual icon, model, etc. of the surrounding electronic device. The processor 201 may then schedule the display 206 to display the identification of the surrounding electronic devices, including but not limited to images, types, virtual icons, models, etc. of the electronic devices. The processor 201 may then schedule one or more of the BT, WLAN, UWB, etc. communication modules to obtain a communication address, such as a MAC address or IP address, etc., of the surrounding electronic devices. Meanwhile, the processor 201 may further acquire the respective corresponding positions, such as the angle and the distance, of the communication addresses of the surrounding electronic devices by combining the BT positioning technology/the WiFi positioning technology/the UWB positioning technology. Thereafter, the processor 201 may match the positions of the surrounding electronic devices obtained by the binocular positioning technique with the positions of the surrounding electronic devices obtained by the BT positioning technique/WiFi positioning technique/UWB positioning technique using the coincidence positioning algorithm. And then the corresponding relation is established between the identification of the surrounding electronic equipment and the communication address. Finally, the processor 201 receives an operation of selecting an identifier of the electronic device by a user, responds to the operation, establishes a communication connection with the corresponding electronic device based on a communication address corresponding to the identifier, and shares the first image.

The binocular positioning technology is as follows: the VR/AR apparatus 100 uses one pair of cameras among the cameras 205 to capture images of surrounding electronic apparatuses (for example, the electronic apparatus 201), and obtains pixel coordinates (two-dimensional coordinates) of the electronic apparatus 201 in the images captured by the two cameras from the two captured images of the electronic apparatus 201. The position, e.g. three-dimensional coordinates, of the electronic device 201 is then obtained from a geometric relation algorithm, in combination with the relative positions of the two cameras.

The BT positioning technology/WiFi positioning technology/UWB positioning technology refers to: the VR/AR device 100 sends a probe request to a surrounding electronic device, receives a probe response from the surrounding electronic device, measures an angle of the surrounding electronic device (e.g., the electronic device 201) with respect to the VR/AR device 100 according to an angle of arrival (AoA) of a BT/WIFI/UWB signal of the probe response, and calculates a distance of the electronic device 201 by using a triangulation method, or a Received Signal Strength Indicator (RSSI) value, or a time of flight (TOF) of the BT/WIFI/UWB signal, thereby obtaining a position (angle and distance) of the electronic device 201 with respect to the VR/AR device 100.

In the embodiment of the present application, the probe request is also referred to as a first request message, the probe response is also referred to as a first response message, and the reception condition of the first response message is the BT/WIFI/UWB signal AoA and TOF of the probe response.

Based on the foregoing detailed description of the communication system 10 shown in fig. 1, and the structure of the VR/AR device 100 shown in fig. 2A. The method provided by the embodiment of the present application is described in detail below with reference to a flowchart of the method and a series of user interfaces.

Referring to fig. 3, fig. 3 schematically shows a flow of a method for intercepting a first image by a VR/AR device according to an embodiment of the present application.

S101, the VR/AR device 100 intercepts a first image in response to the detected first operation.

In some embodiments, this first operation is used to intercept all images displayed on the display screen of the VR/AR device 100.

Referring to fig. 4A-4B, fig. 4A-4B illustrate a user interface 410 for the VR/AR device 100 to intercept all images displayed by the display screen. In the present embodiment, the user interface 410 may also be referred to as a first user interface.

FIG. 4A illustrates one possible implementation, such as a gesture operation, in which the VR/AR device 100 detects the first operation.

As shown in fig. 4A, the user interface 410 displays an image 411, a cursor 412A. The image 411 is the entire image displayed on the display screen of the VR/AR device 100. The representation of cursor 412A may be an "arrow" icon.

The cursor may be an icon, an image, or the like, and when the cursor is located at different positions of a page displayed on the display screen of the VR/AR device, the icon corresponding to the cursor may be different, for example, when the cursor is located at a position of a control in the page, the icon corresponding to the cursor may be an arrow or an icon of a hand, and when the cursor is located at a position of a text input box in the page, the icon corresponding to the cursor may be a vertical line or a horizontal line.

The VR/AR device 100 may detect a first operation in the user interface 410 shown in fig. 4A, such as detecting a user input "punch" gesture 412B, which user input "punch" gesture 412B may be captured by the camera 205 of the VR/AR device 100, i.e., the gesture operation of the user input "punch" gesture 412B should be within the capture range of the camera 205. In response to the first operation, the VR/AR device 100 intercepts the first image and displays the user interface 410 as shown in fig. 4B.

Fig. 4B illustrates a user interface where the VR/AR device 100 intercepts the first image in response to the first operation.

As shown in fig. 4B, the user interface 410 displays an image 411, a cursor 412A, a control 413B, and a rectangular wire frame 414.

Image 411 is the first image that the user intercepts. In some embodiments, the processor 201 may retrieve the image stream sent by the internal memory to the display device 206 to obtain the first image. In other embodiments, the processor 201 may directly capture the entire image displayed by the display device 206 and cut the image to obtain the first image.

The cursor 412A may appear in the form of a change to a "fist" image for prompting the user that the VR/AR device 100 has received a first operation of a user input.

The control 413A is used to share the first image captured, and the control 413B is used to cancel sharing the first image captured.

A rectangular wire frame 414, i.e., a border of the image 411, is used to prompt the user that the VR/AR device 100 has intercepted all of the images in the user interface 410.

It is understood that fig. 4A-4B are merely exemplary of one implementation in which the VR/AR device 100 intercepts the first image. The first operation may be an operation of the user shaking the input device 300 up and down in addition to the operation of the user inputting the "fist making" gesture 412B, specifically, the input device 300 may detect that the input device 300 is shaken, transmit data related to the shaking operation to the VR/AR device 100, and determine that the input device 300 receives the first operation according to the data by the processor 201 in the VR/AR device 100. Alternatively, the first operation may also be the user inputting a voice instruction of "intercept full screen", or the like.

In some embodiments, after the VR/AR device 100 detects the first operation, the first image may be automatically captured within a certain time, e.g., 1 second. In other embodiments, the operation of determining to intercept the first image, which is input by the user, may be received again after the VR/AR device 100 detects the first operation, for example, the user presses a button on the input device 300, and the VR/AR device 100 intercepts the first image in response to detecting the operation acting on the control confirming to intercept the first image.

In some embodiments, after the VR/AR device 100 detects the first operation, the user input of an operation of adjusting the size or position of the first image may be further received, for example, a gesture of pinching or expanding the index finger and the middle finger together to control the reduction or expansion of the first image, and a gesture of moving the position of the first image, for example, a movement of the index finger.

In other embodiments, the first operation is to intercept an image in the ROI.

The first operation may include: for triggering the operation of the VR/AR device 100 to select the ROI, and for the operation of the VR/AR device 100 to determine the ROI, and for the operation of the VR/AR device 100 to intercept the image in the ROI.

Referring to fig. 5A-5E, fig. 5A-5E illustrate a user interface 510 for the VR/AR device 100 to intercept an image in the ROI. In the present embodiment, the user interface 510 may also be referred to as a first user interface.

FIG. 5A illustrates one possible implementation of the VR/AR device 100 detecting a first operation, such as a gesture operation, for triggering the VR/AR device 100 to select the ROI.

As shown in fig. 5A, user interface 510 displays image 511, cursor 512A. The user interface shown in fig. 5A is the same as the user interface shown in fig. 4A, with specific reference to the description of fig. 4A above.

The VR/AR device 100 may detect an operation in the user interface 510 shown in fig. 5A to trigger the VR/AR device 100 to select the ROI, such as detecting a user input "palm" gesture 512B, in response to which the VR/AR device 100 displays the user interface 510 as shown in fig. 5B. Specifically, the camera 205 captures an image of a "palm" gesture 512B input by the user, sends the image to the processor 201, the processor 201 analyzes the image, determines an operation corresponding to the "palm" gesture 512B in the image, and in response to the operation, the VR/AR device 100 displays a user interface 510 as shown in fig. 5B.

As shown in fig. 5B, user interface 510 displays image 511, cursor 512A. The representation of the cursor 512A becomes a "palm" image for prompting the user that the VR/AR device 100 has received a gesture operation of the user input that triggers the electronic device to select the ROI.

It is understood that fig. 5A-5B merely illustrate one possible implementation of the first operation for triggering the VR/AR device 100 to select the ROI, and in some embodiments, the operation for triggering the VR/AR device 100 to select the ROI may also be a user inputting a voice instruction such as "screenshot", specifically, the voice is received by a microphone of the VR/AR device 100, transmitted to the processor 201, and analyzed by the processor 201 to perform a corresponding function according to the voice. In other embodiments, the operation for triggering the VR/AR device 100 to select the ROI may also be an operation of shaking the input device 300 left and right by the user, specifically, the input device 300 may detect the operation of shaking itself, transmit data related to the shaking operation to the VR/AR device 100, and determine, by the processor 201 in the VR/AR device 100, that the input device 300 receives the operation for triggering the VR/AR device 100 to select the ROI in the first operation according to the data, and execute a function corresponding to the operation.

The VR/AR device 100 may detect an operation for the VR/AR device 100 to determine the ROI in the user interface 510 shown in fig. 5B, such as an operation of detecting a movement of the user's "palm" 512B, and the movement trace 512C of the "palm" 512B encloses a regular-like figure, such as a rectangle, a diamond, a triangle, a circle, etc. In response to this operation, the VR/AR device 100 displays the user interface 510 as shown in fig. 5C.

As shown in fig. 5C, the user interface 510 displays an image 511, a cursor 512A, and a movement trajectory 512D of the cursor 512A, a rectangular wire frame 513, a first image 514, a control 515A, and a control 515B.

The moving track 512D of the cursor 512A is: the movement locus 512C of the "palm" 512B corresponds to the movement locus of the cursor. Specifically, the VR/AR device 100 invokes the camera 205 to continuously capture images of the user's hand over a period of time (e.g., 3 seconds). Then, the camera 205 transmits the acquired images of the plurality of hands to the processor 201, the processor 201 analyzes the images of the plurality of palms, identifies the coordinates of the palms of the hands in each image, the coordinates of the palms in the plurality of images form a moving track 512C, and then displays a moving track 512D of a cursor corresponding to the moving track 512C of the palms on the display screen. For example, if the user's hand is moved 20cm to the left, the cursor displayed on the display of the electronic device 100 is moved 10cm to the left.

Wherein the rectangular wire frame 513 is: a rectangular box surrounding the ROI to prompt the user that the VR/AR device 100 has determined the ROI. Specifically, the processor in the VR/AR device 100 may determine the ROI corresponding to the movement track 512D by using an algorithm for determining the ROI according to the movement track 512D. The ROI may be a minimum regular region containing the movement trajectory 512D, or a minimum irregular region. When the ROI is a minimum regular region, for example, a minimum rectangular region, containing the movement trajectory 512D, the VR/AR device 100 may take the minimum value and the maximum value of the abscissa, and the minimum value and the maximum value of the ordinate, of the pixel coordinates of the image displayed on the current display screen of the movement trajectory 512D, and the four maximum values may be combined to obtain four two-dimensional coordinates, where a rectangular wire frame surrounded by the four two-dimensional coordinates is the rectangular wire frame 513 surrounding the ROI. It is understood that the ROI corresponding to the movement track 512D may not be the smallest rectangular region containing the movement track 512D, but may be a circular region or other regions, which are specifically determined by the algorithm stored by the VR/AR device 100 for determining the ROI. When the ROI is the minimum irregular region including the movement trajectory 512D, the region surrounded by the movement trajectory 512D itself is the ROI region.

Therein, the image 514 is the image in the ROI, i.e. the first image. In some embodiments, the processor 201 may retrieve the image stream sent by the internal memory to the display device 206 and truncate the image in the ROI in the image stream to obtain the first image. In other embodiments, the processor 201 may directly capture the image displayed by the display device 206 and truncate the image in the ROI on the display screen to obtain the first image.

Control 515A is for determining to intercept image 514. Control 515B is for canceling the intercepted image 514.

It is to be understood that fig. 5C merely illustrates one possible implementation for triggering the VR/AR device 100 to select an ROI in the first operation. In some embodiments, the operation for triggering the VR/AR device 100 to select the ROI may be an operation in which the user's gaze point moves to control the movement of a cursor, and the movement trajectory of the cursor may constitute a similar regular graph. Specifically, the camera 205 continuously collects eyeball images of the user, sends the collected eyeball images to the processor 201, the processor 201 analyzes the images, determines the movement track of the gaze point of the user, displays the corresponding movement track of the cursor on the display screen according to the movement track of the gaze point, and then the processor 201 determines the ROI according to the movement track of the cursor. In other embodiments, the operation for the VR/AR device 100 to determine the ROI may be an operation in which the user controls the movement of a cursor by moving the handheld input device 300, and the movement trace of the cursor may form a regular graph. Specifically, the input device 300 may detect an operation of moving itself, transmit data related to the movement operation to the VR/AR device 100, determine, by the processor 201 in the VR/AR device 100, that the input device 300 receives an operation for determining the ROI by the VR/AR device 100 in the first operation according to the data, and perform a function corresponding to the operation, that is, display a movement trajectory of a cursor corresponding to the movement trajectory of the input device 300 on the display screen, and then the processor 201 determines the ROI according to the movement trajectory of the cursor.

It should be noted that, as the hand-held input device 300 is moved to control the cursor movement by the movement of the user's palm and the movement of the gaze point, the cursor movement track may or may not be configured to be similar to a regular pattern. When the cursor movement trajectory does not constitute a regular pattern like, the VR/AR device 100 processor may automatically determine a regular pattern ROI, such as a rectangle, circle, diamond, triangle, etc. The ROI contains the movement locus of the cursor which does not constitute a regular-like figure as described above. When the cursor movement track does not form a regular-like graph or a non-regular-like graph, such as a point, a short line, or the like, the VR/AR device 100 may prompt the user that the operation is an invalid operation and ask the user to re-input the operation.

In some embodiments, after the VR/AR device 100 detects the operation of the first operation for the VR/AR device 100 to determine the ROI, the operation of adjusting the size or position of the first image may be received, for example, a gesture of pinching or expanding the index finger and the middle finger together to control the zoom-out or zoom-in of the first image, and a gesture of moving the index finger to control the position of the first image.

FIG. 5D illustrates one possible implementation of the VR/AR device 100 detecting a first operation to intercept an image in the ROI, such as a gesture operation.

The user's action on the control 515A, which may be detected by the VR/AR device 100 in the user interface 510 shown in FIG. 5D, may be the user moving the hand control cursor 512A to the control 515A and staying, for example, for 1 second. In response to this operation, the VR/AR device 100 intercepts the image in the ROI and displays the user interface 510 shown in fig. 5E.

As shown in fig. 5E, user interface 510 includes cursor 512A, first image 514, control 516A, control 516B.

Wherein:

the first image 514 is the image in the ROI. Control 516A is used to share the first image 514 and control 516B is used to cancel sharing the first image 514.

It will be appreciated that fig. 5D-5E merely illustrate one possible implementation for intercepting the ROI in the first operation. In some embodiments, the operation for intercepting the ROI may also be a gesture of user input such as "ok", a voice input such as "ok", an operation such as "blink twice in succession", or the like.

It is to be understood that fig. 5A-5E are merely exemplary of user interfaces for the VR/AR device 100 to intercept an image in the ROI in response to the detected first operation, and should not be construed as limiting embodiments of the present application.

In some embodiments, the operation for triggering the VR/AR device 100 to select the ROI and the operation for the VR/AR device 100 to determine the ROI in the first operation described above may be merged into one operation. For example, the VR/AR device 100 may detect an operation of a user input such as a movement of the "palm" 512B in the user interface shown in fig. 5A, and the movement trace 512C of the "palm" 512B encloses a similar regular graph, such as a similar rectangle, diamond, triangle, circle, etc. In response to this operation, the VR/AR device 100 displays the user interface 510 as shown in fig. 5C.

In other embodiments, of the first operations described above, the operation for the VR/AR device 100 to determine the ROI, and the operation for the VR/AR device 100 to intercept the image in the ROI may be merged into one operation. For example, the VR/AR device 100 detects a user input such as a movement of the "palm" 512B in the user interface shown in fig. 5B, and the movement trace 512C of the "palm" 512B encloses a regular-like graph, such as a rectangle, a diamond, a triangle, a circle, and so on. In response to this operation, the VR/AR device 100 does not display the user interface 510 as shown in fig. 5C, but directly intercepts the first image and displays the user interface as shown in fig. 5D.

In other embodiments, the operation for triggering the VR/AR device 100 to select the ROI, the operation for the VR/AR device 100 to determine the ROI, and the operation for the VR/AR device 100 to intercept the image in the ROI may be integrated into one operation in the first operation. For example, the VR/AR device 100 may detect a user input such as a movement of the "palm" 512B in the user interface shown in fig. 5A, and the movement trace 512C of the "palm" 512B may form a regular-like graph, such as a rectangle, a diamond, a triangle, a circle, etc. In response to this operation, the VR/AR device 100 does not display the user interface 510 as shown in fig. 5C, but directly intercepts the first image and displays the user interface as shown in fig. 5D.

In some embodiments, the method flow shown in fig. 3 further comprises: and saving the first image.

S102, the VR/AR device 100 saves the captured first image.

After the first image is intercepted by the VR/AR device 100, the first image may be automatically saved in the gallery of the VR/AR device 100 after a certain time. Or, after the VR/AR device 100 intercepts the first image, it may further receive a user operation, and in response to the user operation, store the first image in the gallery of the VR/AR device 100.

In some embodiments, after the VR/AR device 100 intercepts the first image, the VR/AR device 100 may fuse the first image into a first image with a 3D effect viewed by the user according to an imaging principle, and save the first image with the 3D effect into the gallery. In this case, if the user shares the first image with 3D effect to the surrounding electronic devices, such as a mobile phone, a computer, a tablet, a smart screen, etc., the user still has the same 3D effect when viewing the first image with 3D effect while wearing the VR/AR device 100 while viewing the first image with 3D effect.

In some embodiments, the method flow shown in fig. 3 further comprises: and sharing the first image.

Steps S103-S104 are a method flow for enabling the VR/AR device 100 to share the first image with surrounding electronic devices, such as the electronic device 201.

S103, the VR/AR device 100 displays the identification of the surrounding electronic devices in response to the detected user operation.

The user operation is an operation that triggers the VR/AR device 100 to display the identities of the surrounding electronic devices. In this embodiment, the third operation is a user operation for triggering the VR/AR device 100 to display the identifiers of the surrounding electronic devices.

For example, the VR/AR device 100 may detect in the user interface 410 shown in fig. 4B or the user interface 510 shown in fig. 5E that a user acting on the control 412A or the control 516A operates the user operation shown in fig. 5E, or that the user inputs a gesture of "ok," shares "speech, blinks twice in succession, and so on.

In some embodiments, the VR/AR device 100 may also detect the above-described operation in a user interface (e.g., in a user interface provided by a gallery) storing the first image that triggers the VR/AR device 100 to display the identities of the surrounding electronic devices, in response to which the user interface 610 shown in fig. 6 is displayed. In the present embodiment, the user interface 610 may also be referred to as a second user interface.

As shown in fig. 6, user interface 610 displays an identification of surrounding electronic devices, such as an image of the surrounding electronic devices, including: image 1, image 2, image 3 and image 4. In some embodiments, the identification of surrounding electronic devices displayed by the user interface 610 may also include: virtual icons, types, models of surrounding electronic devices.

The specific steps of displaying the identification of the surrounding electronic device by the VR/AR device 100 are as follows:

firstly, the VR/AR device 100 acquires the position and image of the surrounding electronic device relative to the VR/AR device 100, for example, the position 1 and image 1 of the electronic device 201, according to the binocular positioning technology, and then matches the image 1 with an image model of the electronic device in a pre-stored database, so as to determine the virtual icon 1, the type 1, the model 1, and the like corresponding to the image 1. Among them, the definition of the database stored by the VR/AR device 100 can be described in detail with reference to table 1.

Referring to table 1, table 1 exemplarily shows image models of an electronic device and virtual icons, types, models, and the like corresponding to the respective image models.

Image model of electronic device	Virtual icon	Types of	Type number
				Image model 1	Virtual icon 1	Type 1	Type 1
Image model 2	Virtual icon 2	Type 2	Type 2
				Image model 3	Virtual icon 3	Type 3	Type 3
Image model 4	Virtual icon 4	Type 4	Type 4

TABLE 1

The database shown in table 1 may further include image models of more electronic devices, and virtual icons, types, models, and the like corresponding to the respective image models.

Then, the VR/AR device 100 acquires the locations of the surrounding electronic devices with respect to the VR/AR device 100 and the corresponding communication addresses according to the BT positioning technology/WiFi positioning technology/UWB positioning technology. Such as location 1 and corresponding communication address 1 of the electronic device 201. The image of the surrounding electronic devices is then displayed at the corresponding location in the user interface 610. Then, the VR/AR device 100 matches the positions of the peripheral electronic devices obtained by the binocular positioning technology relative to the VR/AR device 100 with the positions of the peripheral electronic devices obtained by the BT positioning technology/WiFi positioning technology/UWB positioning technology relative to the VR/AR device 100, and further determines the correspondence between the identifiers, positions, and communication addresses of the peripheral electronic devices. The correspondence between the identification, the location and the communication address of the surrounding electronic device can refer to the detailed description in table 2.

Electronic device image	Virtual icon	Type (B)	Type number	Position of	Communication address
						Image 1	Virtual icon 1	Type 1	Type 1	Position 1	Communication address 1
Image 2	Virtual icon 2	Type 2	Type 2	Position 2	Communication address 2
						Image 3	Virtual icon 3	Type 3	Type 3	Position 3	Communication address 3
Image 4	Virtual icon 4	Type 4	Type 4	Position 4	Communication address 4

TABLE 2

Finally, the VR/AR device 100 displays the identifications of the surrounding electronic devices, such as image 1, image 2, image 3, and image 4, in the user interface 610 according to the image device identifications of the surrounding electronic devices, such as image, virtual icon, type, model, and location in table 2.

Where the positions of image 1, image 2, image 3, and image 4 in the user interface 610 indicate: the location of surrounding electronic devices relative to the VR/AR device 100. For example, if a surrounding electronic device, such as the electronic device 201, is located in front of the VR/AR device 100 on the left, the image of the electronic device 201 is displayed in the user interface 610 at a position to the left of the user interface 610, and the size of the image of the electronic device 201 is inversely proportional to the distance between the electronic device 201 and the VR/AR device 100, i.e., the closer the electronic device 201 is to the VR/AR device, the larger the image of the electronic device 201, and the farther the electronic device 201 is from the VR/AR device, the smaller the image of the electronic device 201. Thus, after seeing the user interface 610, the user can perceive that the electronic device 201 is located in front of the left of the user, and can perceive the distance of the electronic device 201 from the user.

In some embodiments, one manifestation of the identity of the surrounding electronic devices displayed by the VR/AR device 100 may be: the VR/AR device 100 displays a spatial coordinate system in the user interface 610 with the identifications of the surrounding electronic devices displayed therein, the locations of the identifications of the surrounding electronic devices in the spatial coordinate system indicating the spatial locations of the surrounding electronic devices relative to the VR/AR device 100.

In some embodiments, the VR/AR device 100 may obtain the content shown in table 2 in response to the operation after detecting the operation that triggers the VR/AR device 100 to display the identifiers of the surrounding electronic devices, and display the identifiers of the electronic devices shown in table 6 according to the content in table 2.

In other embodiments, the VR/AR device 100 may pre-fetch the contents shown in table 2 before detecting the operation that triggers the VR/AR device 100 to display the identities of the surrounding electronic devices, for example, when the VR/AR device 100 is enabled or when the VR/AR device 100 intercepts the first image. Then, after detecting an operation that triggers the VR/AR device 100 to display the identities of the surrounding electronic devices, in response to the operation, the identities of the electronic devices shown in fig. 6 are displayed according to the contents in table 2 acquired in advance. In this case, the content in table 2 may be updated periodically, which not only saves the time delay for the VR/AR device 100 to display the identifications of the surrounding electronic devices, but also ensures the accuracy of the locations of the surrounding electronic devices relative to the VR/AR device 100.

In some embodiments, the VR/AR device 100 may directly display the identities of the surrounding electronic devices without receiving an operation to trigger the VR/AR device 100 to display the identities of the surrounding electronic devices. For example, the VR/AR device 100 may directly display the identifier of the surrounding electronic device shown in fig. 6 after capturing the first image in the user interface shown in fig. 5E.

S104, the VR/AR device 100 responds to the detection of the second operation and shares the first image with the surrounding electronic devices.

The second operation is an operation for selecting the identification of the surrounding electronic devices.

For example, the VR/AR device 100 may detect in the user interface shown in fig. 6 that the user inputs the "finger" gesture 611 and points to the identification of the surrounding electronic device, such as the electronic device 201, for example, image 1, or the identification of the input device, such as the voice of "type 1" or "model 1", or may also control the cursor 512A to move to the position of the identification of the electronic device 201 by moving the input device 300 and clicking the operation of pressing a key, and so on. In response to the operation for selecting the identifier of the electronic device 201, the VR/AR device 100 establishes a communication connection with the electronic device 201 based on the communication address 1 according to the communication address 1 corresponding to the identifier of the electronic device 201 in table 2, and shares the first image based on the communication connection.

In the embodiment of the present application, the electronic device 201 may be referred to as a first device, and the identifier of the electronic device 201, for example, the image 1, may also be referred to as a first identifier.

In some embodiments, the VR/AR device 100 may detect, in the user interface shown in fig. 6, an operation of continuously selecting the identities of the plurality of electronic devices within a certain time, for example, within 2 seconds, and in response to the operation, the VR/AR device 100 establishes a communication connection with the plurality of electronic devices based on the communication addresses according to the communication addresses corresponding to the identities of the plurality of electronic devices in table 2, and shares the first image based on the communication connection.

In some embodiments, the method of sharing the first image by the VR/AR device 100 is not limited to the method shown in steps S103-S104. For example, it may also be: after intercepting the first image, the VR/AR device 100 receives a user operation, the VR/AR device 100 acquires a communication address of a peripheral electronic device through the BT communication module/WiFi communication module/UWB module, and displays a list of the peripheral electronic device, and then the VR/AR device 100 receives an operation of selecting any one or more electronic devices in the list of the peripheral electronic device by the user, establishes a communication connection with the electronic device according to the communication address of the electronic device, and shares the first image.

It is understood that the operations of intercepting the first image by the VR/AR device 100, establishing a communication connection with surrounding electronic devices, and sharing the first image based on the communication connection are not limited to the sequence shown in fig. 3.

In some embodiments, the VR/AR device 100 may first establish a communication connection with the surrounding electronic devices, then intercept the first image, and then share the first image with the surrounding electronic devices based on the pre-established communication connection. For example, after the VR/AR device 100 is started, the identities of the surrounding electronic devices may be displayed, and the VR/AR device 100 detects an operation of selecting the identities of the surrounding electronic devices to establish a communication connection with the surrounding electronic devices. After the VR/AR device 100 intercepts the first image, it detects that an operation for sharing the first image is performed, such as acting on the control 516A in fig. 5E, and in response to the operation, the VR/AR device 100 shares the first image with the surrounding electronic devices based on the pre-established communication connection.

In this case, the VR/AR device 100 may establish a connection with surrounding electronic devices in advance, thereby improving the efficiency of the VR/AR device 100 in sharing the first image.

It can be seen that, after the method provided by the embodiment of the application is adopted, the VR/AR device can intercept the first image at any time. Further, the size of the first image may be defined by the user himself, and the user may share the first image with any one or more electronic devices around the user. Specifically, the user may intercept the first image, and then establish a connection with the surrounding electronic device and share the first image. Therefore, when the user intercepts different first images, the different first images can be shared in different electronic devices respectively, and the personalized requirements of the user are met. Or the user may establish a connection with the surrounding electronic device first and then share the first image. Therefore, the first image can be shared only in the fixed electronic equipment, and the efficiency of sharing the first image can be improved.

The embodiments of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in accordance with the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk), among others.

One of ordinary skill in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the above method embodiments. And the aforementioned storage medium includes: various media capable of storing program codes, such as ROM or RAM, magnetic or optical disks, etc.

In short, the above description is only an example of the technical solution of the present invention, and is not intended to limit the scope of the present invention. Any modifications, equivalents, improvements and the like made in accordance with the disclosure of the present invention are intended to be included within the scope of the present invention.

Claims

1. A method of intercepting an image by a Virtual Reality (VR)/Augmented Reality (AR) device, the method being applied to a VR/AR device, wherein the VR/AR device includes optical components for providing a 3-dimensional (3D) scene and is worn on a user's head, the method comprising:

the VR/AR device displaying a first user interface;

the VR/AR device responds to a first operation, determines a user region of interest (ROI) in a display screen of the VR/AR device, and intercepts the first image, wherein the first image is an image in the ROI.

2. The method of claim 1, wherein the first operation comprises: the method comprises the following steps that the hand of a user moves, or the eyeball of the user moves, or an input device connected with the VR/AR device moves;

before the VR/AR device determines the ROI, the method further comprising:

the VR/AR device responds to the first operation, and displays a moving track of a cursor on the display screen, wherein the moving track of the cursor corresponds to a moving track of a hand of the user, or a moving track of a fixation point of eyeballs of the user, or a moving track of the input device;

the ROI is a region including a movement trajectory of the cursor.

3. The method according to claim 2, wherein the ROI is a region containing a movement trajectory of the cursor, and specifically comprises:

the ROI is a minimum rule area containing a moving track of the cursor;

or, the ROI is a minimum irregular region including the movement trajectory of the cursor, and the irregular region is a region surrounded by the movement trajectory of the cursor.

4. The method of claims 1-3, wherein the VR/AR device, in response to the first operation, after intercepting the first image, further comprises:

the VR/AR device displays a second user interface that displays an identification of one or more electronic devices;

the VR/AR device responds to a second operation of selecting the first identification, the first image is sent to a first device, the first device corresponds to the first identification, and the identification of one or more electronic devices in the second user interface comprises the first identification.

5. The method of claim 4, wherein before the VR/AR device displays the second user interface, the method further comprises:

the VR/AR device detecting a third operation;

the VR/AR device displays a second user interface, specifically including: in response to the third operation, the VR/AR device displays a second user interface.

6. The method of any of claims 4 or 5, wherein the location of the identity of the one or more electronic devices in the second user interface is used to indicate the location of the one or more electronic devices relative to the VR/AR device.

7. The method of claim 6, wherein the identification of the one or more electronic devices comprises an image of the one or more electronic devices taken by the VR/AR device, and wherein before the VR/AR device displays the second user interface, the method further comprises:

the VR/AR device captures images of the one or more electronic devices;

the VR/AR device determines a location of the one or more electronic devices relative to the VR/AR device from the image of the one or more electronic devices.

8. The method of claim 7, wherein the identification of the one or more electronic devices comprises one or more of: an icon, type, or model of the one or more electronic devices; after the VR/AR device captures an image of the one or more electronic devices, the method further includes:

and the VR/AR equipment acquires one or more of icons, types or models of the one or more electronic equipment according to the images of the one or more electronic equipment.

9. The method of any of claims 7 or 8, prior to the VR/AR device displaying a second user interface, the method further comprising:

the VR/AR equipment sends a first request message;

the VR/AR equipment receives a first response message sent by the one or more electronic equipment, wherein the first response message carries the communication address of the one or more electronic equipment;

the VR/AR equipment acquires the positions of the one or more electronic equipment relative to the VR/AR equipment according to the receiving condition of the first response message;

the VR/AR device sends the first image to the first device in response to a second operation of selecting the first identifier, which specifically includes:

the VR/AR device determines the position of the first device according to the corresponding relation between the images of the one or more electronic devices and the position of the one or more electronic devices relative to the VR/AR device;

the VR/AR device determines a communication address of the first device according to correspondence between the communication addresses of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device;

and the VR/AR equipment sends the first image to the first equipment according to the communication address of the first equipment.

10. The method according to any one of claims 1-9, wherein the first operation comprises one or more of: gestures, voice commands, eye states of users, and operations of pressing keys;

the first operation is detected by the VR/AR device or detected by the input device.

11. The method of any of claims 1-10, wherein the VR/AR device, in response to the first operation, after intercepting the first image, the method further comprises:

the VR/AR device saves the first image.

12. A VR/AR device, wherein the VR/AR comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors for storing computer program code comprising computer instructions that, when executed by the one or more processors, cause the VR/AR device to perform the method of any of claims 1-11.

13. A computer program product comprising instructions for causing an electronic device to perform the method according to any of claims 1-11 when the computer program product is run on the electronic device.

14. A computer-readable storage medium comprising instructions that, when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-11.