WO2022199597A1 - Method, apparatus and system for cropping image by vr/ar device - Google Patents

Abstract

The present application provides a method, apparatus and system for cropping an image by a VR/AR device. In the method, a VR/AR device displays an image; during the process of presenting a 3D scene for a user, the user can input an operation after viewing the image presented by the VR/AR device; and the VR/AR device can crop the image in response to the operation. By implementing the method, personalized requirements of the user can be met, and the user experience can be improved.

Description

Method, device and system for capturing images from VR/AR equipment

This application claims the priority of the Chinese patent application filed on March 25, 2021 with the application number 202110322569.X and the application title "Method, Apparatus and System for Intercepting Images by VR/AR Devices", the entire contents of which are Incorporated herein by reference.

technical field

The present application relates to the field of terminals, and in particular, to a method, an apparatus, and a system for capturing an image by a VR/AR device.

Background technique

Virtual reality (VR) technology uses computer simulation to generate a three-dimensional (3D) virtual reality scene, and provides a visual, auditory, tactile or other sensory simulation experience, allowing users to feel as if they are in the real world . Augmented reality (AR) technology can superimpose virtual reality to display virtual images for users while users watch the real world, and users can also interact with virtual images to achieve the effect of augmented reality.

With the popularization of VR equipment and AR equipment, more and more users are beginning to experience VR and AR technology. How to meet the growing personalized needs of users is the main problem of current and future research.

SUMMARY OF THE INVENTION

The present application provides a method, device and system for capturing images by a VR/AR device, which can meet the personalized needs of users, capture images in a region of interest (ROI) of users, and improve user experience.

In a first aspect, the present application provides a method for capturing an image from a VR/AR device, the method is applied to a VR/AR device, the VR/AR device includes an optical component for providing a 3D 3D scene, and is worn on a user's head part, the method for capturing an image includes: the VR/AR device displays a first user interface; the VR/AR device determines the ROI in the display screen of the VR/AR device in response to the first operation, and captures the first image, the first image is the image in the ROI.

By implementing the method provided in the first aspect, the VR/AR device can intercept the image in the ROI according to the user's operation, so as to satisfy the user's personalization and improve the user experience.

With reference to the first aspect, in one embodiment, the above-mentioned first operation includes: an operation of moving a user's hand, or an operation of moving a user's eye gaze point, or an operation of moving an input device connected to the VR/AR device.

In this way, the user can achieve the practicability of capturing images through various operations, and improve the operability of the user.

With reference to the first aspect, in an embodiment, before the VR/AR device determines the ROI, the method for capturing an image by the VR/AR device further includes: the VR/AR device displays the movement track of the cursor on the display screen in response to the first operation , the movement trajectory of the cursor corresponds to the movement trajectory of the user's hand, or the movement trajectory of the user's eye gaze point, or the movement trajectory of the input device; the ROI is an area containing the movement trajectory of the cursor.

In this way, the user can watch the movement track of the cursor on the display screen of the VR/AR device, further displaying the interaction process between the user and the VR/AR device, and improving the user experience.

With reference to the first aspect, in one embodiment, the ROI is an area containing the movement track of the cursor, for example, the ROI may be the smallest regular area containing the movement track of the cursor; Regular area, the minimum irregular area is the area enclosed by the movement track of the cursor.

In this way, the VR/AR device can determine a variety of possible ROIs according to the user's operation, improving the user experience.

With reference to the first aspect, in an embodiment, after the VR/AR device captures the first image in response to the first operation, the method further includes: the VR/AR device displays a second user interface, and the second user interface displays one or more The identifiers of multiple electronic devices; the VR/AR device sends the first image to the first device in response to the second operation of selecting the first identifier, the first device corresponds to the first identifier, and one or more electronic devices in the second user interface The identification of the device includes the first identification.

In this way, the user can send the captured first image to any one or more surrounding electronic devices, so as to meet user requirements and improve user experience.

With reference to the first aspect, in one embodiment, before the VR/AR device displays the second user interface, the method further includes: the VR/AR device detects a third operation; in response to the third operation, the VR/AR device displays the first 2. User interface.

In this way, the VR/AR device can display the second user interface only after receiving the operation of triggering the display of the identifiers of the surrounding electronic devices, so as to meet the personalized needs of the user.

In conjunction with the first aspect, in one embodiment, the location of the identifier of the one or more electronic devices in the second user interface is used to indicate the location of the one or more electronic devices relative to the VR/AR device.

With reference to the first aspect, in one embodiment, the identifiers of one or more electronic devices include images of one or more electronic devices captured by the VR/AR device. Before the VR/AR device displays the second user interface, the method It also includes: the VR/AR device captures images of one or more electronic devices; the VR/AR device determines the position of one or more electronic devices relative to the VR/AR device according to the images of the one or more electronic devices.

In this way, the user can perceive the position of one or more surrounding electronic devices relative to himself by viewing the identifiers of one or more electronic devices, thereby providing a more immersive experience.

In conjunction with the first aspect, in one embodiment, the identification of one or more electronic devices includes one or more of the following: icons, types or models of one or more electronic devices; VR/AR devices photographing one or more electronic devices After the image of the device, the method further includes: the VR/AR device acquires one or more of icons, types or models of the one or more electronic devices according to the images of the one or more electronic devices.

In this way, the VR/AR device can display images, virtual icons, types or models, etc. of one or more electronic devices according to user needs, so as to provide users with rich content to meet the individual needs of users.

With reference to the first aspect, in an embodiment, before the VR/AR device displays the second user interface, the method further includes: after the VR/AR device sends the first request message, the VR/AR device receives one or more The first response message sent by the electronic device, the first response message carries the communication addresses of one or more electronic devices; the VR/AR device obtains the relative information of one or more electronic devices relative to the VR/AR device according to the reception of the first response message. position; the VR/AR device sends the first image to the first device in response to the second operation of selecting the first identifier, which specifically includes: the VR/AR device interacts with one or more electronic devices according to the images of one or more electronic devices The position of the first device is determined relative to the correspondence between the positions of the VR/AR devices; the VR/AR device is based on the correspondence between the communication addresses of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device The communication address of the first device is determined; the VR/AR device sends the first image to the first device according to the communication address of the first device.

In this way, the VR/AR device can share the first image with the first device according to the user's operation of selecting the first identifier of the first device and the communication address corresponding to the first identifier to meet the user's personalized needs.

With reference to the first aspect, in one embodiment, the first operation includes one or more of the following: gestures, voice commands, the user's eye state, and an operation of pressing a button; the first operation is detected by the VR/AR device, Alternatively, detected by and input devices.

In this way, the first operation can be implemented in multiple ways, which improves the practicability of the screenshot method provided by the embodiment of the present application and improves the user experience.

With reference to the first aspect, in an implementation manner, the VR/AR device may further save the first image after capturing the first image in response to the first operation.

In this way, it is convenient for the user to perform further operations on the first graphic, such as sharing it with other electronic devices, so as to improve the user experience.

In a second aspect, embodiments of the present application provide a VR/AR device, where the VR/AR includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, The one or more memories are used to store computer program code, the computer program code comprising computer instructions which, when executed by the one or more processors, cause the VR/AR device to perform the method described in the embodiment of the first aspect.

In a third aspect, embodiments of the present application provide a computer program product containing instructions, characterized in that, when the computer program product runs on an electronic device, the electronic device is caused to execute the method described in the implementation manner of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, including instructions, characterized in that, when the instructions are executed on an electronic device, the electronic device is caused to execute the method described in the implementation manner of the first aspect.

After implementing the method for capturing an image by a VR/AR device provided in this application, the VR/AR device can capture an image in the ROI, and share the image to any one or more surrounding electronic devices, thereby improving user experience.

Description of drawings

1 is a schematic diagram of a communication system provided by an embodiment of the present application;

2A is a schematic structural diagram of a VR/AR device provided by an embodiment of the present application;

FIG. 2B is a schematic diagram of an imaging principle of a VR/AR device provided by an embodiment of the present application;

3 is a flow chart of a method for capturing an image provided by an embodiment of the present application;

4A-4B provide a set of user interfaces for capturing images according to an embodiment of the present application;

5A-5E provide another set of user interfaces for capturing images according to an embodiment of the present application;

FIG. 6 is a user interface for sharing an image provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the accompanying drawings. Wherein, in the description of the embodiments of the present application, unless otherwise specified, “/” means or, for example, A/B can mean A or B; “and/or” in the text is only a description of an associated object The association relationship indicates that there can be three kinds of relationships, for example, A and/or B can indicate that A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application , "plurality" means two or more.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as implying or implying relative importance or implying the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

The term "user interface (UI)" in the following embodiments of this application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the internal form of information and the form acceptable to the user. conversion between. The user interface is the source code written in a specific computer language such as java and extensible markup language (XML). A commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed graphically. It can be text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements displayed in the display screen of the electronic device. The terms used in the implementation part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

When users experience VR/AR, they may want to capture images. How to capture images in VR/AR devices is an urgent problem to be solved.

An embodiment of the present application provides a method for capturing an image by a VR/AR device. In this method, the VR/AR device displays an image and presents a 3D scene to a user. After the user sees the image presented by the VR/AR device, A first operation may be input, and the VR/AR device may capture a first image in response to the first operation. In this embodiment of the present application, when the VR/AR device presents a 3D scene to the user, the image displayed on the display screen of the VR/AR device may be the same or different from the image seen by the user.

The above-mentioned first image can be any of the following:

1. All or part of the image displayed on the display screen of the VR/AR device.

In some embodiments, the VR/AR device may, in response to the first operation, capture all images displayed on the display screen as the first image.

In some embodiments, the VR/AR device may determine a user's region of interest (ROI) in the display screen in response to the first operation, and then capture the image in the ROI on the display screen as the first operation. an image. In this embodiment of the present application, the ROI refers to an area in the display screen determined by the VR/AR device according to the user's operation. Here, the ROI determined by the user's operation may be based on the movement trajectory of the user's hand and the gaze point of the eyeball. The movement trajectory or the movement trajectory of the input device 300 corresponds to displaying the movement trajectory of the cursor on the display screen, and then the VR/AR device determines the ROI according to the movement trajectory of the cursor. The ROI can be a regular area, and the ROI can be an irregular area.

2. When the VR/AR device presents the VR/AR scene to the user, all or part of the image that the user sees.

In some embodiments, based on the imaging principle of the VR/AR device, it can be known that the image displayed on the display screen of the VR/AR device may be different from the image seen by the user. Therefore, the VR/AR device can respond to the first operation by intercepting and displaying the image. When the entire image or part of the image displayed on the screen is displayed, it is also necessary to fuse the captured image or part of the image into the image seen by the user as the first image according to the imaging principle of the VR/AR device. Among them, for the imaging principle of VR/AR devices, please refer to the introduction of VR/AR devices later.

In some embodiments, the VR/AR device may also share the first image to surrounding electronic devices.

Specifically, in response to the detected user operation, the VR/AR device may display a user interface including the identifiers of surrounding electronic devices, and the location of the identifiers of the electronic devices in the user interface indicates the location of the electronic device relative to the user. The identification of the electronic device includes, but is not limited to: an image, a virtual icon, a type, a model, and the like of the electronic device. After the user sees the user interface, he can recognize the position of the surrounding electronic devices relative to himself. Finally, the VR/AR device establishes a communication connection with the electronic device corresponding to the identification in response to the detected operation of the identification of the selected electronic device, and shares the above-mentioned first image through the communication connection.

After implementing the method for capturing an image by a VR/AR device provided in this embodiment of the present application, the VR/AR device can capture the first image at any time, and the size of the captured first image can be defined by the user. The first image is shared on the surrounding electronic devices, so that other users can also see the first image, which satisfies the user's personalized needs.

Referring to FIG. 1 , FIG. 1 exemplarily shows a communication system 10 provided by an embodiment of the present application.

As shown in FIG. 1 , the communication system 10 includes a VR/AR device 100, and one or more electronic devices such as electronic device 201, electronic device 202, electronic device 203, and electronic device 204 around the VR/AR device 100. In some embodiments, the communication system 10 may also include an input device 300 .

The VR/AR device 100 can be an electronic device such as a helmet, glasses, etc. that can be worn on the user's head. The VR/AR device 100 can be used in conjunction with other electronic devices (such as mobile phones) to receive GPUs of other electronic devices (such as mobile phones). Processed data or content (such as a rendered image) and display it. In this case, the VR/AR device 100 may be a terminal device such as VR glasses with limited computing power. The VR/AR device 100 is used for displaying images, so as to present a 3D scene to a user and bring a VR/AR/MR experience to the user. The 3D scene may include 3D images, 3D videos, and the like.

In this embodiment of the present application, the VR/AR device 100 may capture the first image in response to the detected first operation. For the definition of the first image, reference may be made to the above detailed description. For the specific steps of intercepting the first image, reference may be made to the related descriptions of the following method embodiments.

In some embodiments, the VR/AR device may display a user interface containing identification (eg, images, virtual icons, types, models, etc.) of surrounding electronic devices in response to detected user actions. Then, in response to the detected operation of selecting the identification of the electronic device, such as a gesture pointing to the identification of the electronic device in the user interface, the VR/AR device establishes a communication connection with the electronic device corresponding to the identification, and shares the above-mentioned first section through the communication connection. an image.

The above-mentioned communication connection may be a wired or wireless connection. The wired connection may include a wired connection that communicates through an interface such as a USB interface, an HDMI interface, or the like. The wireless connection may include a wireless connection that communicates through one or more of a BT communication module, a wireless local area networks (WLAN) communication module, and a UWB communication module.

The electronic device 201 , the electronic device 202 , the electronic device 203 and the electronic device 204 are the electronic devices around the VR/AR device 100 , such as a smart TV, a computer, a mobile phone, a VR/AR device, a tablet computer, or a touch-sensitive device. Surface or touch panel laptop computer (Laptop), non-portable terminal equipment such as desktop computer with touch sensitive surface or touch panel, etc.

In this embodiment of the present application, the electronic device 201 , the electronic device 202 , the electronic device 203 , and the electronic device 204 have one or more of a BT communication module, a WLAN communication module, and a UWB communication module. Taking the electronic device 201 as an example, the electronic device 201 can monitor the signals transmitted by the VR/AR device 100, such as detection requests, scanning signals, etc., through one or more of the BT communication module, the WLAN communication module, and the UWB communication module. And can send response signals, such as detection response, scan response, etc., so that the VR/AR device 100 can discover the electronic device 201, and determine the location and network address of the electronic device 201, and then establish a communication connection with the electronic device 201, and based on the communication Connect to receive the first image shared by the VR/AR device 100 .

The input device 300 is a device for controlling the VR/AR device 100, for example, a handle, a mouse, a keyboard, a stylus, a wristband, and the like.

In this embodiment of the present application, the input device 300 may be configured with various sensors, such as an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, and the like. The pressure sensor may be disposed under the confirm key/cancel key of the input device 300 .

In this embodiment of the present application, the input device 300 may collect movement data of the input device 300, and data indicating whether the confirmation button/cancel button of the input device 300 is pressed. The movement data includes sensors of the input device 300 , for example, an acceleration sensor to collect the acceleration of the input device 300 , a gyro sensor to collect the movement speed of the input device 300 , and the like. Wherein, the data indicating whether the confirm button/cancel button of the input device 300 is pressed includes the pressure value collected by the pressure sensor disposed under the confirm button/cancel button or the level generated by the input device 300, etc.

In this embodiment of the present application, the input device 300 can establish a communication connection with the VR/AR device 100, and through the communication connection, the collected movement data of the input device 300, and the confirmation button/cancel indicating the input device 300 The data of whether a key or the like is pressed is sent to the VR/AR device 100 . The communication connection may be a wired or wireless connection. The wired connection may include a wired connection that communicates through a USB interface, an HDMI interface, a custom interface, or the like. The wireless connection may include one or more of wireless connections that communicate via Bluetooth, near field communication (NFC), ZigBee, and other short-range transmission technologies.

In this embodiment of the present application, the movement data of the input device 300 and the data indicating whether the confirmation button/cancel button of the input device 300 is pressed can be used by the VR/AR device 100 to determine the movement of the input device 300 and/or state. The movement of the input device 300 may include, but is not limited to, whether to move, the direction of movement, the speed of movement, the distance of movement, the trajectory of movement, and the like. The state of the input device 300 may include whether the confirmation key of the input device 300 is pressed.

The VR/AR device 100 may activate a corresponding function according to the movement situation and/or state of the input device 300 . That is, the user can trigger the VR/AR device 100 to perform a corresponding function by inputting a user operation on the input device 300 . For example, when the user holds the input device 300 and moves it to the left by 3 cm, the cursor displayed on the display screen of the VR/AR device 100 can be moved to the left by 6 cm. In this way, the user can move the cursor to any position on the display screen of the VR/AR device 100 by manipulating the input device 300 . For another example, after the cursor is moved to a certain control displayed in the VR/AR device 100, the user can press the confirmation button of the input device 300, so that the VR/AR device 100 enables the VR/AR device 100 to activate the control corresponding to the control Function.

In this embodiment of the present application, the input device 300 may be configured to receive a user operation for triggering the VR/AR device to capture the first image. For the specific implementation of the user operation, reference may be made to the related description of the method embodiments later.

Referring to FIG. 2A , FIG. 2A shows a schematic structural diagram of a VR/AR device 100 provided by an embodiment of the present application.

As shown in FIG. 2A , the VR/AR device 100 may include: a processor 201 , a memory 202 , a communication module 203 , a sensor module 204 , a camera 205 , a display device 206 , and an audio device 207 . The above components can be coupled and connected and communicate with each other, for example, the above components can be connected through a bus.

It is understandable that the structure shown in FIG. 2A does not constitute a specific limitation to the VR/AR device 100 . In other embodiments of the present application, the VR/AR device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. For example, the VR/AR device 100 may further include physical keys such as on-off keys, volume keys, various interfaces such as a USB interface for supporting the connection between the VR/AR device 100 and the input device 300, and the like. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 201 may include one or more processing units, for example, the processor 110 may include a memory, an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal Image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. The controller can generate operation control signals according to the instruction opcode and timing signal, and complete the control of fetching and executing instructions, so that each component can perform corresponding functions, such as human-computer interaction, motion tracking/prediction, rendering display, audio processing, etc.

In this embodiment of the present application, the memory in the processor 201 can identify the operation input by the user, and execute the corresponding function. For example, the processor 201 can recognize user input gestures such as fist, palm, etc., voice such as screenshot, share, etc., and operations on the input device 300 such as shaking up and down, shaking left and right, moving, etc., and then determine to execute the corresponding operation according to these operations. Function.

The memory 202 stores executable program codes for executing the display methods provided by the embodiments of the present application, where the executable program codes include instructions. The memory 202 may include a stored program area and a stored data area. The storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, etc.) created during the use of the VR/AR device 100 and the like. In addition, the memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like. The processor 201 executes various functional applications and data processing of the VR/AR device 100 by executing the instructions stored in the memory 202 and/or the instructions stored in the memory provided in the processor.

In this embodiment of the present application, the memory 202 may store an image for display by the display device 206 of the VR/AR device 100, and send the image to the display device 206 in the form of an image stream.

In this embodiment of the present application, the memory 202 may further store the first image captured by the VR/AR device 100 .

In some embodiments, the memory 202 may store a database, the database including the image models of the electronic device 201, the electronic device 202, the electronic device 203, the electronic device 204 and other more electronic devices, and the electronic devices corresponding to the image models of each electronic device The type, virtual icon and model of the device, etc. For the content specifically included in the database, please refer to the detailed description of Table 1 in the following method embodiments for details. In some embodiments, the memory 202 may also store images of the surrounding electronic devices captured by the camera 205 , the locations and communication addresses of the surrounding electronic devices, and the corresponding relationship between them. For details, please refer to the detailed description in Table 2 below, which will not be repeated here.

The communication module 203 may include a wireless communication module and a wired communication module. The wireless communication module can provide solutions for wireless communication such as BT, WLAN, UWB, GNSS, FM, IR, etc. applied to the VR/AR device 100 . The wireless communication module may be one or more devices integrating at least one communication processing module. The wired communication module may provide a wired connection including communication through an interface such as a USB interface, an HDMI interface, and the like. The communication module 203 can support the communication between the VR/AR device 100 and surrounding electronic devices (electronic device 201 , electronic device 202 , electronic device 203 , electronic device 204 , etc.).

In this embodiment of the present application, the VR/AR device 100 may establish a communication connection with the input device 300 through a wireless communication module or a wired communication module, and based on the communication connection, acquire the movement data of the input device 300 collected by the input device 300 , and represent Data of whether the confirmation button/cancel button of the input device 300 is pressed. Afterwards, the processor 201 can identify the operation input by the user according to the data sent by the input device 300, and determine the function to be executed corresponding to the operation.

In some embodiments, the VR/AR device 100 may establish a communication connection with surrounding electronic devices through one or more of communication modules such as BT, WLAN, and UWB, and share the first image based on the communication connection.

The sensor module 204 may include an accelerometer, a compass, a gyroscope, a magnetometer, or other sensors for detecting motion, or the like. The sensor module 204 is used to collect corresponding data, for example, the acceleration sensor collects the acceleration of the VR/AR device 100, and the gyroscope sensor collects the motion speed of the VR/AR device 100, and the like. The data collected by the sensor module 204 can reflect the movement of the head of the user wearing the VR/AR device 100 . In some embodiments, the sensor module 204 may be an inertial measurement unit (IMU) disposed within the VR/AR device 100 . In some embodiments, the VR/AR device 100 may send the data acquired by the sensor module to the processor 201 for analysis. The processor 201 can determine the movement of the user's head according to the data collected by each sensor, and execute corresponding functions according to the movement of the user's head.

The sensor module 204 may also include optical sensors for tracking the user's eye position and capturing eye movement data in conjunction with the camera 205. The eye movement data may be used, for example, to determine the user's eye separation, the 3D position of each eye relative to the VR/AR device 100, the magnitude of each eye's twist and rotation (ie, roll, pitch, and pan) and gaze direction, etc. . In one example, infrared light is emitted within the VR/AR device 100 and reflected from each eye, the reflected light is detected by a camera 205 or an optical sensor, and the detected data is transmitted to the processor 201 to enable The processor 201 analyzes the position, pupil diameter, motion state, etc. of the user's eyes from the changes of the infrared light reflected by each eye.

In this embodiment of the present application, the above-mentioned optical sensor may be combined with the camera 205 to collect the image of the user's eyeball, and track the movement trajectory of the user's eyeball gaze point. and transmit the data of the movement trajectory of the gaze point to the processor 201, the processor 201 analyzes the data, and displays the movement trajectory of the cursor corresponding to the movement trajectory of the eye gaze point in the display screen, and then determines according to the movement trajectory of the cursor ROI in the display.

Camera 205 may be used to capture still images or video. The still image or video can be an externally facing image or video around the user, or an internal facing image or video. The camera 205 can track the movement of one or both eyes of the user. The camera 205 includes, but is not limited to, a traditional color camera (RGB camera), a depth camera (RGB depth camera), a dynamic vision sensor (DVS) camera, and the like. The depth camera can obtain the depth information of the photographed object.

In this embodiment of the present application, the camera 205 includes at least a pair of cameras, and the pair of cameras can capture images facing the inside of the VR/AR device 100, that is, the user's eye image, and can also capture images facing the outside of the VR/AR device 100. That is, the image of the user's hand. The pair of cameras can send the collected images of the user's eyes or the user's hand to the processor 201 for analysis, and the processor 201 can identify the state of the user's eyes, such as whether to blink, the number of blinks, etc. The state the eye is in performs the corresponding function. Alternatively, the processor 201 may also recognize the user's gestures, such as making a fist, palm, "OK", etc. according to the hand image, and perform different functions correspondingly according to different gestures.

In some embodiments, the camera 205 may also include two pairs of cameras, one of which is used to capture images facing the outside of the VR/AR device 100 , and the other pair is used to capture images facing the interior of the VR/AR device 100

In this embodiment of the present application, the VR/AR device 100 may present or display images through the GPU, the display device 206, and the application processor.

The GPU is a microprocessor for image processing, and is connected to the display device 206 and the application processor. Processor 201 may include one or more GPUs that execute program instructions to generate or alter display information.

Display device 206 may include: one or more display screens, one or more optical components. The one or more display screens may include, for example, display screen 101 and display screen 103 . The one or more optical assemblies include, for example, optical assembly 102 and optical assembly 104 . The display screen, such as the display screen 101 and the display screen 103, may include a display panel, and the display panel may be used to display images, thereby presenting a three-dimensional virtual scene to the user. The display panel may adopt a liquid crystal display device LCD, OLED, AMOLED, FLED, Miniled, MicroLed, Micro-oLed, QLED, and the like. Optical components, such as the optical component 102, the optical component 104, are used to direct light from the display screen to the exit pupil for user perception. In some embodiments, one or more optical elements (eg, lenses) in an optical assembly may have one or more coatings, such as anti-reflection coatings. The amplification of the image light by the optical components allows the display to be physically smaller, lighter, and consume less power. In addition, the magnification of the image light can increase the field of view of the content displayed on the display screen. For example, the optical assembly can make the field of view of the content displayed on the display screen the full field of view of the user. Optical assemblies may also be used to correct for one or more optical errors. Examples of optical errors include: barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and the like. In some embodiments, the content provided to the display screen for display is pre-distorted, and the distortion is corrected by the optical assembly upon receiving the content-based image light from the display screen.

Referring to FIG. 2B , FIG. 2B is a schematic diagram of an imaging principle of the VR/AR device 100 .

As shown in FIG. 2B , the display device 206 of the VR/AR device 100 may include: a display screen 101 , an optical component 102 , a display screen 103 , and an optical component 104 . The first straight line where the center of the display screen 101 and the center of the optical component 102 are located is perpendicular to the third straight line where the center of the optical component 102 and the center of the optical component 104 are located. Display screen 101 and optical assembly 102 correspond to the user's left eye. When the user wears the VR/AR device 100 , an image a1 may be displayed on the display screen 101 . When the display screen 101 displays the image a1, the light emitted by the optical component 102 will form a virtual image a1' of the image a1 in front of the user's left eye.

The second straight line where the center of the display screen 103 and the center of the optical component 104 are located is perpendicular to the third straight line where the center of the optical component 102 and the center of the optical component 104 are located. Display screen 103 and optical assembly 104 correspond to the user's right eye. When the user wears the VR/AR device 100, the display screen 103 may display the image a2. When the display screen 103 displays the image a2, the light emitted by the optical component 104 will form a virtual image a2' of the image a2 in front of the user's right eye.

Image a1 and image a2 are two images with parallax for the same object, eg, object a. Parallax refers to the difference in the position of the object in the field of view when the same object is viewed from two points that are a certain distance away. The virtual image a1' and the virtual image a2' are located on the same plane, which can be called a virtual image plane.

When wearing the VR/AR device 100, the user's left eye will focus on the virtual image a1', and the user's right eye will focus on the virtual image a2'. Then, the virtual image a1' and the virtual image a2' are superimposed in the user's brain to form a complete and three-dimensional image, a process called vergence. During the vergence process, the meeting point of the eyes of both eyes will be considered by the user as the actual location of the object described by the image a1 and the image a2. Due to the vergence process, the user can feel the 3D scene provided by the VR/AR device 100 .

Not limited to the imaging principle of the VR/AR device 100 shown in FIG. 2B above, the VR/AR device may also use other methods to provide the user with a 3D scene.

It can be understood that the display device 206 can be used to receive data or content (eg, rendered images) processed by the GPU of other electronic devices (eg, mobile phones), and display them. In this case, the VR/AR device 100 can be a terminal device such as VR glasses with limited computing power, and needs to be used in conjunction with other electronic devices (such as mobile phones) to present a 3D scene to the user and provide VR/AR/MR to the user experience.

The audio device 207 is used to realize the collection and output of audio. Audio devices 207 may include, but are not limited to, microphones, speakers, headphones, and the like.

In this embodiment of the present application, the microphone may be used to receive the voice input by the user, and convert the voice into an electrical signal. The processor 201 may be configured to receive an electrical signal corresponding to the voice from the microphone, and after receiving the voice signal, the processor 201 may recognize the operation corresponding to the voice and perform the corresponding operation. For example, when the microphone receives the voice of "share", the processor 201 may recognize the voice instruction and share the first image with the surrounding electronic devices (eg, the electronic device 201).

In this embodiment of the present application, the camera 205 can continuously capture multiple palm images of the user, and transmit the palm images to the processor 201. The processor 201 analyzes the multiple palm images by using binocular positioning technology, and identifies the user The movement trajectory of the palm is displayed, and the movement trajectory of the cursor corresponding to the movement trajectory of the palm is displayed on the display screen, and then the ROI is determined according to the trajectory. Afterwards, the processor 201 may capture the image in the ROI on the display screen as the first image. In some embodiments, the processor 201 may obtain the image stream sent from the internal memory to the display device 206 to obtain the first image. In other embodiments, the processor 201 may directly acquire the image displayed by the display device 206, and capture the image to obtain the first image. In some embodiments, the camera 205 can capture an image of the surrounding electronic device, and send the image of the surrounding electronic device to the processor 201, and the processor 201 analyzes the image of the surrounding electronic device using a binocular positioning technology to identify the surrounding electronic device. The position of the electronic device relative to the VR/AR device 100 is, for example, three-dimensional coordinates. The processor 201 can also match the image of the surrounding electronic device with the image model of each electronic device in the database, so as to identify the type, virtual icon, model, etc. Afterwards, the processor 201 may schedule the display device 206 to display the identifiers of the surrounding electronic devices, where the identifiers of the electronic devices include, but are not limited to, images, types, virtual icons, models, and the like of the electronic devices. Afterwards, the processor 201 may also schedule one or more of communication modules such as BT, WLAN, and UWB to obtain the communication addresses of surrounding electronic devices, such as MAC addresses or IP addresses. At the same time, the processor 201 may also obtain positions such as angles and distances respectively corresponding to the communication addresses of the surrounding electronic devices in combination with the BT positioning technology/WiFi positioning technology/UWB positioning technology. After that, the processor 201 can use the coincidence positioning algorithm to match the positions of the surrounding electronic devices obtained by the binocular positioning technology with the positions of the surrounding electronic devices obtained by the BT positioning technology/WiFi positioning technology/UWB positioning technology. Then, a corresponding relationship is established between the identifiers of the surrounding electronic devices and the communication addresses. Finally, the processor 201 receives the operation of the user selecting the identification of the electronic device, and in response to the operation, establishes a communication connection with the corresponding electronic device based on the communication address corresponding to the identification, and shares the above-mentioned first image.

The above binocular positioning technology refers to: the VR/AR device 100 uses a pair of cameras in the camera 205 to separately collect images of the surrounding electronic devices (for example, the electronic device 201 ), and respectively obtains two images according to the collected two images of the electronic device 201 . Pixel coordinates (two-dimensional coordinates) of the electronic device 201 in the images captured by the two cameras. Then, in combination with the relative positions of the two cameras, the position of the electronic device 201, such as three-dimensional coordinates, is obtained according to a geometric relationship algorithm.

The above BT positioning technology/WiFi positioning technology/UWB positioning technology refers to: the VR/AR device 100 sends a detection request to the surrounding electronic devices, and then receives the detection response from the surrounding electronic devices, and then arrives according to the BT/WIFI/UWB signal of the detection response. The angle of arrival (AoA) measures the angle of the surrounding electronic device (eg electronic device 201) relative to the VR/AR device 100, and then uses triangulation, or the received signal strength indicator (RSSI) value, or BT The time of flight (TOF) of the /WIFI/UWB signal calculates the distance of the electronic device 201, and then obtains the position (angle and distance) of the electronic device 201 relative to the VR/AR device 100.

In this embodiment of the present application, the probe request is also referred to as the first request message, the probe response is also referred to as the first response message, and the reception of the first response message is the BT/WIFI/UWB signal AoA and TOF of the probe response.

Based on the foregoing detailed introduction to the structure of the communication system 10 shown in FIG. 1 and the VR/AR device 100 shown in FIG. 2A . Next, the method provided by the embodiments of the present application will be described in detail with reference to the method flowchart and a series of user interfaces.

Referring to FIG. 3 , FIG. 3 exemplarily shows a flow of a method for capturing a first image by a VR/AR device according to an embodiment of the present application.

S101, the VR/AR device 100 captures a first image in response to the detected first operation.

In some embodiments, the first operation is used to capture all images displayed on the display screen of the VR/AR device 100 .

Referring to FIG. 4A-FIG. 4B, FIG. 4A-FIG. 4B exemplarily show a user interface 410 in which the VR/AR device 100 captures all images displayed on the display screen. In this embodiment of the present application, the user interface 410 may also be referred to as a first user interface.

FIG. 4A exemplarily shows a possible implementation manner in which the VR/AR device 100 detects a first operation, such as a gesture operation.

As shown in Figure 4A, user interface 410 displays image 411, cursor 412A. The image 411 is all the images displayed on the display screen of the VR/AR device 100 VR/AR device 100 . Cursor 412A may take the form of an "arrow" icon.

The above-mentioned cursor can be an icon, an image, etc. When the cursor is at different positions on the page displayed on the VR/AR device display screen, the icon corresponding to the cursor can be different. For example, when the cursor is at the position of the control on the above-mentioned page, the corresponding icon The icon can be an arrow or a hand icon. When the cursor is at the position of the text input box in the above page, the icon corresponding to the cursor can be a vertical line or a horizontal line.

The VR/AR device 100 may detect a first operation in the user interface 410 shown in FIG. 4A , for example, detecting a user input of a "make a fist" gesture 412B, which may be captured by the camera of the VR/AR device 100 The gesture operation captured by the device 205 , that is, the user inputting the “make a fist” gesture 412B should be within the shooting range of the camera device 205 . In response to the first operation, the VR/AR device 100 captures the first image and displays the user interface 410 as shown in FIG. 4B .

FIG. 4B exemplarily shows a user interface in which the VR/AR device 100 captures a first image in response to the first operation.

As shown in FIG. 4B , the user interface 410 displays an image 411 , a cursor 412A, a control 413A, a control 413B and a rectangular wireframe 414 .

The image 411 is the first image captured by the user. In some embodiments, the processor 201 may obtain the image stream sent from the internal memory to the display device 206 to obtain the first image. In other embodiments, the processor 201 may directly acquire all the images displayed by the display device 206, and capture the images to obtain the first image.

The expression form of the cursor 412A may be changed to a "fist" image, which is used to prompt the user that the VR/AR device 100 has received the first operation input by the user.

The control 413A is used to share the captured first image, and the control 413B is used to cancel the sharing of the captured first image.

The rectangular wire frame 414 is the frame of the image 411 , which is used to prompt the user that the VR/AR device 100 has captured all the images in the user interface 410 .

It can be understood that FIG. 4A-FIG. 4B only exemplarily show an implementation manner in which the VR/AR device 100 captures the first image. In addition to the above-mentioned user inputting the "make a fist" gesture 412B operation, the above-mentioned first operation may also be an operation in which the user shakes the input device 300 up and down. The relevant data is transmitted to the VR/AR device 100, and the processor 201 in the VR/AR device 100 determines that the input device 300 has received the first operation according to the data. Alternatively, the first operation may also be the user inputting a voice instruction of "capture full screen", or the like.

In some embodiments, after the VR/AR device 100 detects the first operation, it can automatically capture the first image within a certain period of time, for example, within 1 second. In other embodiments, after the VR/AR device 100 detects the first operation, it may also receive a user input again to determine the operation to capture the first image. For example, the user presses a button on the input device 300 and the VR/AR device 100 The first image is captured in response to detecting an operation of the control for confirming the capture of the first image.

In some embodiments, after detecting the first operation, the VR/AR device 100 may also receive an operation input by the user to adjust the size or position of the first image, for example, the first finger and the middle finger are pinched or expanded to control the first image. For the reduction or expansion of the image, for example, the position movement of the first image is controlled by moving the index finger.

In other embodiments, the first operation is used to capture the image in the ROI.

The first operation may include: an operation for triggering the VR/AR device 100 to select an ROI, an operation for the VR/AR device 100 to determine the ROI, and an operation for the VR/AR device 100 to capture an image in the ROI.

Referring to FIGS. 5A-5E , FIGS. 5A-5E exemplarily illustrate a user interface 510 for the VR/AR device 100 to capture an image in an ROI. In this embodiment of the present application, the user interface 510 may also be referred to as a first user interface.

FIG. 5A exemplarily shows a possible implementation manner in which the VR/AR device 100 detects a first operation for triggering the VR/AR device 100 to select an ROI, such as a gesture operation.

As shown in Figure 5A, user interface 510 displays image 511, cursor 512A. The user interface shown in FIG. 5A is the same as the user interface shown in FIG. 4A , please refer to the description of FIG. 4A above for details.

The VR/AR device 100 may detect an operation for triggering the selection of an ROI by the VR/AR device 100 in the user interface 510 shown in FIG. 5A , for example, detecting that the user inputs a “palm” gesture 512B, and in response to the operation, the VR/AR Device 100 displays user interface 510 as shown in FIG. 5B. Specifically, the camera 205 captures an image of the "palm" gesture 512B input by the user, sends the image to the processor 201, and the processor 201 analyzes the image to determine the operation corresponding to the "palm" gesture 512B in the image, In response to this operation, the VR/AR device 100 displays a user interface 510 as shown in FIG. 5B .

As shown in Figure 5B, user interface 510 displays image 511, cursor 512A. The representation form of the cursor 512A becomes a "palm" image, which is used to prompt the user that the VR/AR device 100 has received the user-input gesture operation that triggers the electronic device to select the ROI.

It can be understood that FIGS. 5A-5B only exemplarily show a possible implementation manner for triggering the VR/AR device 100 to select an ROI in the first operation. In some embodiments, this is used to trigger the VR/AR device 100. 100 The operation of selecting the ROI can also be a user inputting a voice command such as "screenshot". Specifically, the microphone of the VR/AR device 100 receives the voice, and transmits the voice to the processor 201, and the processor 201 analyzes the voice, The corresponding function is executed according to the voice. In other embodiments, the operation for triggering the selection of the ROI by the VR/AR device 100 may also be an operation of the user shaking the input device 300 left and right. Specifically, the input device 300 may detect the operation of being shaken, and use the The relevant data of the shaking operation is transmitted to the VR/AR device 100, and the processor 201 in the VR/AR device 100 determines, according to the data, that the input device 300 receives the first operation for triggering the VR/AR device 100 to select the ROI. operation, and execute the corresponding function of the operation.

The VR/AR device 100 may detect an operation for the VR/AR device 100 to determine the ROI in the user interface 510 shown in FIG. 5B , for example, an operation to detect the movement of the user's "palm" 512B, and the "palm" 512B moves the trajectory 512C Form a similar regular figure, such as a rectangle, a diamond, a triangle, a circle, etc. In response to this operation, the VR/AR device 100 displays a user interface 510 as shown in FIG. 5C .

As shown in FIG. 5C , the user interface 510 displays an image 511, a cursor 512A, and a movement track 512D of the cursor 512A, a rectangular wire frame 513, a first image 514, a control 515A, and a control 515B.

The movement track 512D of the cursor 512A is the movement track of the cursor corresponding to the movement track 512C of the "palm" 512B. Specifically, the VR/AR device 100 calls the camera 205 to continuously collect images of the user's hand within a certain period of time (for example, 3 seconds). The camera 205 then transmits the collected images of the hands to the processor 201, and the processor 201 analyzes the images of the multiple palms to identify the coordinates of the hands and palms in each image, and the coordinates of the palms in the multiple images form A moving track 512C, and then a moving track 512D of the cursor corresponding to the moving track 512C of the palm is displayed on the display screen. For example, when the user's hand moves to the left by 20 cm, the cursor displayed on the display screen of the electronic device 100 moves to the left by 10 cm.

The rectangular wire frame 513 is a rectangular wire frame surrounding the ROI, which is used to prompt the user that the VR/AR device 100 has determined the ROI. Specifically, the processor in the VR/AR device 100 may determine the ROI corresponding to the movement track 512D by using an algorithm for determining the ROI according to the movement track 512D. The ROI may be the smallest regular area containing the movement trajectory 512D, or the smallest irregular area. When the ROI is the smallest regular area including the movement track 512D, such as the smallest rectangular area, the VR/AR device 100 can take the minimum and maximum abscissa and the minimum ordinate among the pixel coordinates of the image displayed by the movement track 512D on the current display screen. value and the maximum value, the four maximum values can be combined to obtain four two-dimensional coordinates, and the rectangular wire frame enclosed by the four two-dimensional coordinates is the rectangular wire frame 513 surrounding the ROI. It can be understood that the ROI corresponding to the movement track 512D may not be the smallest rectangular area including the movement track 512D, but may be a circular area or other, which is specifically determined by the algorithm for determining the ROI stored in the VR/AR device 100 . When the ROI is the smallest irregular area including the movement track 512D, the area enclosed by the movement track 512D itself is the ROI area.

The image 514 is the image in the ROI, that is, the first image. In some embodiments, the processor 201 may acquire the image stream sent from the internal memory to the display device 206, and capture the image in the ROI in the image stream to obtain the first image. In other embodiments, the processor 201 may directly acquire the image displayed by the display device 206, and capture the image in the ROI in the display screen to obtain the first image.

Control 515A is for determining the captured image 514 . Control 515B is used to cancel the captured image 514 .

It can be understood that FIG. 5C only exemplarily shows a possible implementation manner for triggering the VR/AR device 100 to select an ROI in the first operation. In some embodiments, the operation for triggering the VR/AR device 100 to select an ROI may be an operation of controlling the movement of the cursor by moving the user's gaze point, and the movement track of the cursor may form a similar regular graph. Specifically, the camera 205 continuously collects images of the user's eyeballs, and sends the collected multiple eyeball images to the processor 201. The processor 201 analyzes the images, determines the movement trajectory of the user's gaze point, and displays the user's gaze point according to the movement trajectory of the gaze point. The corresponding movement track of the cursor is displayed on the screen, and then the processor 201 determines the ROI according to the movement track of the cursor. In other embodiments, the operation for the VR/AR device 100 to determine the ROI may also be an operation for the user to control the movement of the cursor by moving the handheld input device 300, and the movement track of the cursor may form a similar regular graph. Specifically, the input device 300 can detect the operation of its own movement, and transmit the relevant data of the movement operation to the VR/AR device 100, and the processor 201 in the VR/AR device 100 determines the input device 300 according to the data. The operation for determining the ROI of the VR/AR device 100 in the first operation is received, and the function corresponding to the operation is performed, that is, the movement trajectory of the cursor corresponding to the movement trajectory of the input device 300 is displayed on the display screen, and then the processor 201 according to the cursor's movement trajectory. Move the trajectory to determine the ROI.

It is worth noting that the above-mentioned movement of the cursor is controlled by the user's palm movement, gaze point movement, and handheld input device 300 movement, and the cursor movement trajectory may or may not constitute a similar regular figure. When the movement track of the cursor does not form a similar regular figure, the processor of the VR/AR device 100 can automatically determine an ROI of a regular figure, such as a rectangle, a circle, a diamond, a triangle, and the like. The ROI includes the movement track of the above-mentioned cursor that does not constitute a similar regular pattern. When the movement track of the cursor does not form a similar regular graphic or a non-similar regular graphic, such as a point or a short line, the VR/AR device 100 may prompt the user that the operation is an invalid operation, and ask the user to re-enter the operation.

In some embodiments, after detecting the operation for the VR/AR device 100 to determine the ROI in the first operation, the VR/AR device 100 may also receive an operation input by the user to adjust the size or position of the first image, for example, by using an index finger A gesture of pinching or expanding with the middle finger is used to control the reduction or expansion of the first image, and for example, the position movement of the first image is controlled by moving the index finger.

FIG. 5D exemplarily shows a possible implementation manner in which the VR/AR device 100 detects the first operation for capturing the image in the ROI, such as a gesture operation.

The VR/AR device 100 may detect an operation performed by the user on the control 515A in the user interface 510 shown in FIG. 5D . The operation may be that the user controls the cursor 512A to move to the control 515A by moving the hand, and stays there for, for example, 1 second. operate. In response to this operation, the VR/AR device 100 captures the image in the ROI, and displays the user interface 510 shown in FIG. 5E .

As shown in FIG. 5E, the user interface 510 includes a cursor 512A, a first image 514, a control 516A, and a control 516B. in:

The first image 514 is the image in the ROI. The control 516A is used to share the first image 514 , and the control 516B is used to cancel the sharing of the first image 514 .

It can be understood that FIG. 5D-FIG. 5E only exemplarily show a possible implementation manner for capturing the ROI in the first operation. In some embodiments, the operation for capturing the ROI may also be a user inputting a gesture such as "ok", a voice such as inputting "OK", an operation such as "blinking eyes twice in a row", and the like.

It can be understood that FIGS. 5A-5E are only illustrative, and the VR/AR device 100 captures the user interface of the image in the ROI in response to the detected first operation, which should not constitute a limitation to the embodiments of the present application.

In some embodiments, in the above-mentioned first operation, the operation for triggering the VR/AR device 100 to select the ROI and the operation for the VR/AR device 100 to determine the ROI may be merged into one operation. For example, the VR/AR device 100 can detect the user input operation such as the movement of the "palm" 512B in the user interface shown in FIG. 5A, and the movement track 512C of the "palm" 512B encloses a similar regular figure, such as a rectangle, a diamond , triangle, circle, etc. In response to this operation, the VR/AR device 100 displays a user interface 510 as shown in FIG. 5C .

In other embodiments, in the above-mentioned first operation, the operation for the VR/AR device 100 to determine the ROI and the operation for the VR/AR device 100 to capture the image in the ROI may be merged into one operation. For example, in the user interface shown in FIG. 5B, the VR/AR device 100 detects that the user inputs an operation such as the movement of the "palm" 512B, and the movement track 512C of the "palm" 512B encloses a similar regular figure, such as a rectangle, a diamond, Triangles, circles, etc. In response to this operation, the VR/AR device 100 does not display the user interface 510 as shown in FIG. 5C , but directly captures the first image and displays the user interface as shown in FIG. 5D .

In other embodiments, among the above-mentioned first operations, an operation for triggering the VR/AR device 100 to select an ROI, an operation for the VR/AR device 100 to determine an ROI, and an operation for the VR/AR device 100 to capture Operations on the images in the ROI can be fused into one operation. For example, the VR/AR device 100 can detect the user input operation such as the movement of the "palm" 512B in the user interface shown in FIG. 5A, and the movement track 512C of the "palm" 512B encloses a similar regular figure, such as a rectangle, a diamond , triangle, circle, etc. In response to this operation, the VR/AR device 100 does not display the user interface 510 as shown in FIG. 5C , but directly captures the first image and displays the user interface as shown in FIG. 5D .

In some embodiments, the method flow shown in FIG. 3 further includes: saving the above-mentioned first image.

S102, the VR/AR device 100 saves the captured first image.

After the VR/AR device 100 captures the first image, the first image may be automatically saved to the gallery of the VR/AR device 100 after a certain period of time. Alternatively, after the VR/AR device 100 captures the first image, it may also receive a user operation, and in response to the operation, save the first image to the gallery of the VR/AR device 100 .

In some embodiments, after the VR/AR device 100 captures the first image, the VR/AR device 100 may fuse the first image into a first image with a 3D effect viewed by the user according to an imaging principle, and combine the first image with a 3D effect. to save the first image of . In this case, if the user shares the first image with 3D effect to surrounding electronic devices such as mobile phones, computers, tablets, smart screens, etc., the user watches the first image with 3D effect on the surrounding electronic devices , it still has the same 3D effect as when wearing the VR/AR device 100 to watch the first image.

In some embodiments, the method flow shown in FIG. 3 further includes: sharing the above-mentioned first image.

Steps S103-S104 are the method flow for implementing the VR/AR device 100 to share the first image to surrounding electronic devices such as the electronic device 201 .

S103, the VR/AR device 100 displays the identifiers of surrounding electronic devices in response to the detected user operation.

The user operation is an operation that triggers the VR/AR device 100 to display the identifiers of surrounding electronic devices. In this embodiment of the present application, the third operation is a user operation for triggering the VR/AR device 100 to display the identifiers of surrounding electronic devices.

For example, the VR/AR device 100 may detect in the user interface 410 shown in FIG. 4B or the user interface 510 in FIG. 5E, the user operation on the control 412A or the control 516A shown in FIG. 5E, or the user input The gesture of "ok", the voice of "share", and the operation of "blink twice in a row", etc.

In some embodiments, the VR/AR device 100 may also detect the above-mentioned operation of triggering the VR/AR device 100 to display the identifiers of the surrounding electronic devices in the user interface storing the first image (for example, in the user interface provided by the gallery), and responds The user interface 610 shown in FIG. 6 is displayed for this operation. In this embodiment of the present application, the user interface 610 may also be referred to as a second user interface.

As shown in FIG. 6 , the user interface 610 displays identifications of surrounding electronic devices, such as images of the surrounding electronic devices, including: image 1 , image 2 , image 3 and image 4 . In some embodiments, the identifiers of the surrounding electronic devices displayed on the user interface 610 may further include: virtual icons, types, and models of the surrounding electronic devices.

The specific steps for the VR/AR device 100 to display the identifiers of surrounding electronic devices are as follows:

First, the VR/AR device 100 obtains the position and image of the surrounding electronic devices relative to the VR/AR device 100 according to the binocular positioning technology, such as the position 1 and the image 1 of the electronic device 201, and then compares the image 1 with the pre-stored database. The image model of the electronic device is matched, and then the virtual icon 1, type 1 and model 1 corresponding to the image 1 are determined. The definition of the database stored in the VR/AR device 100 may be introduced in detail with reference to Table 1.

Referring to Table 1, Table 1 exemplarily shows image models of electronic devices and virtual icons, types, models, etc. corresponding to each image model.

Image model of electronic equipment	virtual icon	type	model
Image Model 1	virtual icon 1	Type 1	Model 1
Image Model 2	virtual icon 2	Type 2	Model 2
Image Model 3	virtual icon 3	Type 3	Model 3
Image Model 4	virtual icon 4	Type 4	Model 4

Table 1

The database shown in Table 1 may also include more image models of electronic devices, as well as virtual icons, types, and models corresponding to each image model.

Then, the VR/AR device 100 obtains the position of the surrounding electronic devices relative to the VR/AR device 100 and the corresponding communication address according to the BT positioning technology/WiFi positioning technology/UWB positioning technology. For example, the position 1 of the electronic device 201 and the corresponding communication address 1. Images of surrounding electronic devices are then displayed at corresponding locations in the user interface 610 . Afterwards, the VR/AR device 100 uses the binocular positioning technology to obtain the position of the surrounding electronic devices relative to the VR/AR device 100 according to the coincidence positioning algorithm, and uses the BT positioning technology/WiFi positioning technology/UWB positioning technology to obtain the relative position of the surrounding electronic devices relative to the VR/AR device 100 . The location of the VR/AR device 100 is matched, and then the correspondence between the identifiers, locations and communication addresses of the surrounding electronic devices is determined. For the corresponding relationship between the identifiers, locations and communication addresses of the surrounding electronic devices, reference may be made to the detailed introduction in Table 2.

electronic equipment images	virtual icon	type	model	Location	contact address
image 1	virtual icon 1	Type 1	Model 1	position 1	Mailing address 1
image 2	virtual icon 2	Type 2	Model 2	position 2	Mailing address 2
image 3	virtual icon 3	Type 3	Model 3	position 3	Mailing address 3
image 4	virtual icon 4	Type 4	Model 4	position 4	Mailing address 4

Table 2

Finally, the VR/AR device 100 displays the identifications of the surrounding electronic devices in the user interface 610 such as image 1, image 2, Image 3 and Image 4.

The positions of Image 1 , Image 2 , Image 3 and Image 4 in the user interface 610 indicate the positions of the surrounding electronic devices relative to the VR/AR device 100 . For example, if the surrounding electronic devices such as electronic device 201 are located in the front left of the VR/AR device 100, the image of the electronic device 201 is displayed in the left position of the user interface 610 in the user interface 610, and the size of the image of the electronic device 201 is the same as that of the electronic device 610. The distance between 201 and the VR/AR device 100 is said to be inversely proportional, that is, the closer the electronic device 201 is to the VR/AR device, the larger the image of the electronic device 201, and the farther the electronic device 201 is from the VR/AR device, the higher the distance between the electronic device 201 and the VR/AR device. the smaller the image. In this way, after seeing the user interface 610, the user can sense that the electronic device 201 is located in front of the user, and can also sense the distance between the electronic device 201 and the user.

In some embodiments, a representation form of the identification of the surrounding electronic devices displayed by the VR/AR device 100 may be: The identifier of the device, the location of the identifier of the surrounding electronic device in the spatial coordinate system indicates the spatial position of the surrounding electronic device relative to the VR/AR device 100 .

In some embodiments, the VR/AR device 100 may, after detecting the above-mentioned operation that triggers the VR/AR device 100 to display the identifiers of the surrounding electronic devices, in response to the operation, obtain the content shown in the above Table 2, according to Table 2 content to display the identity of the electronic device shown in FIG. 6 .

In other embodiments, the VR/AR device 100 may detect the above-mentioned operation that triggers the VR/AR device 100 to display the identifiers of surrounding electronic devices, for example, when the VR/AR device 100 is enabled or the VR/AR device 100 intercepts the first When one image is used, the contents shown in Table 2 above are obtained in advance. Then, after detecting an operation that triggers the VR/AR device 100 to display the identifiers of surrounding electronic devices, in response to the operation, the identifiers of the electronic devices shown in FIG. 6 are displayed according to the content in Table 2 acquired in advance. In this case, the content in Table 2 can be updated regularly, which not only saves the time delay for the VR/AR device 100 to display the identifiers of the surrounding electronic devices, but also ensures the accuracy of the position of the surrounding electronic devices relative to the VR/AR device 100 sex.

In some embodiments, the VR/AR device 100 may directly display the identifiers of the surrounding electronic devices without receiving an operation for triggering the VR/AR device 100 to display the identifiers of the surrounding electronic devices. For example, in the user interface shown in FIG. 5E , the VR/AR device 100 directly displays the identifiers of the surrounding electronic devices shown in FIG. 6 after capturing the first image.

S104, in response to detecting the second operation, the VR/AR device 100 shares the first image to surrounding electronic devices.

The second operation is an operation for selecting the identifiers of the surrounding electronic devices.

For example, the VR/AR device 100 may detect in the user interface shown in FIG. 6 that the user inputs a "finger" gesture 611 and points to a surrounding electronic device such as an identification of the electronic device 201, such as image 1, or an input device identification such as The voice of "Type 1", "Type 1", or by moving the input device 300, the cursor 512A is controlled to move to the position of the logo of the electronic device 201, and the operation of pressing the button is clicked and so on. In response to the operation for selecting the identification of the electronic device 201, the VR/AR device 100 establishes a communication connection with the electronic device 201 based on the communication address 1 corresponding to the identification of the electronic device 201 in Table 2, and The first image is shared based on the communication connection.

In this embodiment of the present application, the electronic device 201 may be referred to as the first device, and an identifier of the electronic device 201, such as image 1, may also be the first identifier.

In some embodiments, the VR/AR device 100 may, in the user interface shown in FIG. 6 , detect an operation of continuously selecting the identifiers of multiple electronic devices within a certain period of time, such as 2 seconds, and in response to the operation, VR The AR device 100 establishes a communication connection with the plurality of electronic devices based on the communication addresses corresponding to the identifiers of the plurality of electronic devices in Table 2, and shares the first image based on the communication connection.

In some embodiments, the method for the VR/AR device 100 to share the first image is not limited to the methods shown in steps S103-S104. For example, after the VR/AR device 100 intercepts the first image, the user operation is received, and the VR/AR device 100 obtains the communication address of the surrounding electronic devices through the BT communication module/WiFi communication module/UWB module, and displays the surrounding electronic devices. A list of devices, and then the VR/AR device 100 receives the user's operation of selecting any one or more electronic devices in the list of surrounding electronic devices, establishes a communication connection with the electronic device according to the communication address of the electronic device, and shares the first image.

It can be understood that, the VR/AR device 100 captures the first image, establishes a communication connection with surrounding electronic devices, and shares the first image based on the communication connection and is not limited to the sequence shown in FIG. 3 .

In some embodiments, the VR/AR device 100 may first establish a communication connection with the surrounding electronic devices, then capture the first image, and then share the first image to the surrounding electronic devices based on the pre-established communication connection. For example, after the VR/AR device 100 is started, the identifiers of the surrounding electronic devices can be displayed. The VR/AR device 100 detects the operation of selecting the identifiers of the surrounding electronic devices, and establishes a communication connection with the surrounding electronic devices. After the VR/AR device 100 captures the first image, an operation for sharing the first image is detected, such as acting on the control 516A in FIG. 5E . The first image is shared to surrounding electronic devices.

In this case, the VR/AR device 100 can establish a connection with the surrounding electronic devices in advance, thereby improving the efficiency of the VR/AR device 100 sharing the first image.

It can be seen that after using the method provided by the embodiment of the present application, the VR/AR device can capture the first image at any time. Further, the size of the first image can be defined by the user, and the user can also share the first image to any one or more surrounding electronic devices. Specifically, the user can first capture the first image, and then establish a connection with the surrounding electronic devices and share the first image. In this way, when the user captures different first images, the different first images can be shared to different electronic devices respectively, so as to meet the personalized needs of the user. Or the user may first establish a connection with the surrounding electronic devices, and then share the first image. In this way, the first image can only be shared to the fixed electronic device, but the efficiency of sharing the first image can be improved.

The various embodiments of the present application can be arbitrarily combined to achieve different technical effects.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer program instructions, when loaded and executed on a computer, produce, in whole or in part, the processes or functions described herein. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk), and the like.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented. The process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed , which may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store program codes.

In a word, the above descriptions are merely examples of the technical solutions of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included within the protection scope of the present invention.

Claims (14)

1. A method for capturing an image from a virtual reality VR/augmented reality AR device, the method is applied to the VR/AR device, wherein the VR/AR device includes an optical component for providing a 3D 3D scene, and is worn on the VR/AR device. User head, the method includes:

   the VR/AR device displays a first user interface;

   In response to the first operation, the VR/AR device determines the ROI of the region of interest of the user in the display screen of the VR/AR device, and captures the first image, where the first image is the image in the ROI .
2. The method according to claim 1, wherein the first operation comprises: an operation of moving a user's hand, or an operation of moving a user's eye gaze point, or an input device connected to the VR/AR device moving operations;

   Before the VR/AR device determines the ROI, the method further includes:

   In response to the first operation, the VR/AR device displays a movement trajectory of the cursor on the display screen, where the movement trajectory of the cursor corresponds to the movement trajectory of the user's hand, or the user's eye gaze the movement trajectory of the point, or the movement trajectory of the input device;

   The ROI is an area including the movement track of the cursor.
3. The method according to claim 2, wherein the ROI is an area containing the movement track of the cursor, and specifically includes:

   The ROI is the smallest regular area including the movement track of the cursor;

   Alternatively, the ROI is a minimum irregular area including the movement trajectory of the cursor, and the irregular area is an area surrounded by the movement trajectory of the cursor.
4. The method according to claims 1-3, wherein after the VR/AR device captures the first image in response to the first operation, the method further comprises:

   the VR/AR device displays a second user interface, and the second user interface displays the identifiers of one or more electronic devices;

   In response to the second operation of selecting the first identifier, the VR/AR device sends the first image to the first device, the first device corresponds to the first identifier, and one of the second user interface or The identifications of the plurality of electronic devices include the first identification.
5. The method according to claim 4, wherein before the VR/AR device displays the second user interface, the method further comprises:

   the VR/AR device detects a third operation;

   The VR/AR device displaying the second user interface specifically includes: in response to the third operation, the VR/AR device displaying the second user interface.
6. The method according to any one of claims 4 or 5, wherein the position of the identifier of the one or more electronic devices in the second user interface is used to indicate that the one or more electronic devices are relatively the location of the VR/AR device.
7. The method according to claim 6, wherein the identification of the one or more electronic devices comprises images of the one or more electronic devices captured by the VR/AR device, and the VR/AR device Before displaying the second user interface, the method further includes:

   the VR/AR device captures images of the one or more electronic devices;

   The VR/AR device determines the position of the one or more electronic devices relative to the VR/AR device according to the images of the one or more electronic devices.
8. The method according to claim 7, wherein the identification of the one or more electronic devices comprises one or more of the following: icons, types or models of the one or more electronic devices; the VR/AR After the device captures the image of the one or more electronic devices, the method further includes:

   The VR/AR device acquires one or more of icons, types or models of the one or more electronic devices according to the images of the one or more electronic devices.
9. The method according to any one of claims 7 or 8, before the VR/AR device displays the second user interface, the method further comprises:

   The VR/AR device sends a first request message;

   The VR/AR device receives a first response message sent by the one or more electronic devices, where the first response message carries the communication address of the one or more electronic devices;

   The VR/AR device acquires the position of the one or more electronic devices relative to the VR/AR device according to the reception of the first response message;

   The VR/AR device sends the first image to the first device in response to the second operation of selecting the first identifier, specifically including:

   The VR/AR device determines the position of the first device according to the correspondence between the images of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device;

   The VR/AR device determines the communication address of the first device according to the correspondence between the communication addresses of the one or more electronic devices and the positions of the one or more electronic devices relative to the VR/AR device ;

   The VR/AR device sends the first image to the first device according to the communication address of the first device.
10. The method according to any one of claims 1-9, wherein the first operation includes one or more of the following: gesture, voice command, user's eye state, and operation of pressing a button;

    The first operation is detected by the VR/AR device, or is detected by the input device.
11. The method according to any one of claims 1-10, wherein after the VR/AR device captures the first image in response to the first operation, the method further comprises:

    The VR/AR device saves the first image.
12. A VR/AR device, characterized in that the VR/AR includes one or more processors and one or more memories; wherein the one or more memories are coupled with the one or more processors, the one or more memories for storing computer program code comprising computer instructions which, when executed by the one or more processors, cause the VR/AR device to perform as claimed The method of any one of 1-11.
13. A computer program product comprising instructions, wherein, when the computer program product is run on an electronic device, the electronic device is caused to perform the method of any one of claims 1-11.
14. A computer-readable storage medium comprising instructions, characterized in that, when the instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1-11.

Images