CN115793841A

CN115793841A - Display method and electronic equipment

Info

Publication number: CN115793841A
Application number: CN202111056782.7A
Authority: CN
Inventors: 李昱霄; 毛春静; 沈钢
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-09-09
Filing date: 2021-09-09
Publication date: 2023-03-14
Also published as: WO2023035911A1

Abstract

A display method and an electronic device are provided. The method comprises the following steps: displaying the first image to a user through a display screen; at least one of the display positions or the shapes of a first object on the first image and a first object on a second image are different, and the display positions and the shapes of the second object on the first image and the second object on the second image are the same; the second image is an image acquired by the camera; the first object is located in the area where the user's point of regard is located, and the second object is located in an area outside the area where the user's point of regard is located. Through this kind of mode, help solving because of camera and display screen position difference and the camera that leads to shoots the problem that the visual angle is different with the human eye observation visual angle.

Description

Display method and electronic equipment

Technical Field

The present application relates to the field of electronic technologies, and in particular, to a display method and an electronic device.

Background

Virtual Reality (VR) technology is a man-machine interaction means created by computer and sensor technologies. The VR technology integrates a plurality of scientific technologies such as computer graphics technology, computer simulation technology, sensor technology, display technology, etc., and can create a virtual world. The user can be immersed in the virtual world by wearing VR-worn devices (e.g., VR glasses, VR helmets, etc.).

The objects in the virtual world can be all the fictional objects or can comprise three-dimensional models of real objects, so that the experience is more real when the user sees the virtual world including both the fictional objects and the real objects. For example, a camera may be disposed on the VR-worn device to capture an image of the real object, construct a three-dimensional model of the real object based on the image, and display the three-dimensional model in the virtual world. Taking VR glasses as an example, fig. 1 is a schematic diagram of VR glasses. VR glasses are last to include camera and display screen. Generally, for the purpose of light, thin and not heavy appearance of VR glasses, the camera is not disposed at the position of the display screen, and is usually disposed at the position below the display screen, as shown in fig. 1.

This arrangement may cause the viewing angle direction of the human eye to be different from the viewing angle direction of the camera (or referred to as shooting direction). For example, in fig. 1, the shooting direction of the camera is downward, and the viewing direction of the human eye is forward. If direct through the display screen with the image of camera collection to people's eye show, can give other people uncomfortable and feel, so can appear dizzy feeling for a long time, experience is relatively poor.

Disclosure of Invention

The application aims to provide a display method and electronic equipment for improving VR experience.

In a first aspect, a display method is provided, which is applied to a wearable device, where the wearable device includes at least one display screen and at least one camera; the method comprises the following steps: displaying a first image to a user through the display screen; at least one of the display positions or the shapes of a first object on the first image and a first object on a second image are different, and the display positions and the shapes of the second object on the first image and the second object on the second image are the same; the second image is an image acquired by the camera; the first object is located in the area where the user's point of regard is located, and the second object is located in an area outside the area where the user's point of regard is located.

In the embodiment of the application, the wearable device can perform visual angle reconstruction on the area where the user's gaze point is located on the second image acquired by the camera, and does not perform visual angle reconstruction on the area outside the area where the user's gaze point is located. Can alleviate dizzy sense (the position of display screen and camera leads to the camera to shoot the visual angle and the dizzy sense that the visual angle was observed to people's eye and was brought) through visual angle reconstruction to user's point of fixation place region, promote VR experience, moreover, it is few only to make visual angle reconstruction work load to point of fixation place region to can reduce the probability or the degree that take place the picture distortion.

In one possible design, an offset of the displacement between the first display position of the first object on the first image and the second display position of the first object on the second image is related to a distance between the camera and the display screen. For example, the greater the distance between the camera and the display screen, the greater the offset between the first object and the second object.

In one possible design, the displacement offset between the first display position and the second display position increases with increasing distance between the camera and the display screen and decreases with decreasing distance between the camera and the display screen.

Illustratively, when the distance between the camera and the display screen is a first distance, the displacement offset between the first display position and the second display position is a first displacement offset. And when the distance between the camera and the display screen is a second distance, the displacement offset between the first display position and the second display position is a second displacement offset. When the first distance is greater than or equal to the second distance, the first displacement offset is greater than or equal to the second displacement offset. When the first distance is smaller than the second distance, the first displacement offset is smaller than the second displacement offset.

In one possible design, a direction of an offset between a first display position of the first object on the first image and a second display position of the first object on the second image is related to a positional relationship between the camera and the display screen. For example, the camera is located on the left side of the display screen, and the first object on the second image is shifted to the left to the position of the first object on the first image.

In one possible design, the offset direction between the first display position and the second display position varies as the direction between the camera head and the display screen varies.

For example, when the camera is located in a first direction of the display screen, the offset direction between the second display position and the first display position is the first direction. When the camera is located in a second direction of the display screen, the offset direction between the second display position and the first display position is the second direction.

In one possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is positioned in the area where the user gaze point is positioned and is closer to the edge of the area where the gaze point is positioned than the first object; the second offset is less than the first offset. That is, the first object at the central position in the area where the user's gaze point is located has a larger offset than the third object at the edge position, so that smooth transition of the edge of the area where the user's gaze point is located and other areas can be realized.

In one possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is positioned in an area outside the area where the user fixation point is positioned, and the area surrounds the edge of the area where the user fixation point is positioned; the second offset amount is smaller than the first offset amount. That is, the offset of the first object in the area where the user gaze point is located is larger than the offset of the third object in the peripheral area (the area outside the area where the user gaze point is located and surrounding the edge of the area where the user gaze point is located), so that smooth transition between the area where the user gaze point is located and the edge of the other area can be realized.

In one possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than the degree of morphological change of a third object on the first image relative to the third object on the second image; the third object is located in the area where the user's gaze point is located, and the third object is closer to the edge of the area where the gaze point is located than the first object. That is, the degree of morphological change of the object from the center position to the edge position in the region where the user's gaze point is located is smaller. Therefore, smooth transition of the edge of the area where the user gazes and other areas can be realized.

In one possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than the degree of morphological change of a third object on the first image relative to the third object on the second image; the third object is located in a region outside the region where the user's gaze point is located, and the region surrounds an edge of the region where the gaze point is located. That is, the smaller the degree of morphological change of the object from the area where the user's gaze point is located outward to the peripheral area (the area outside the area where the user's gaze point is located and surrounding the edge of the area where the gaze point is located). Therefore, smooth transition of the edge of the area where the user gazes and other areas can be realized.

In one possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is in the area of the user's gaze point, and the third object is in a first directional range of the first object, the first directional range including a direction of positional offset of the first object on the first image relative to the first object on the second image; the second offset is greater than the first offset. Taking the example that the offset direction is offset to the lower left, in the area where the user's gaze point is located, the amount of object offset in the lower left range is large, and the amount of object offset in the upper right range is small. In this way, when the region where the user's gaze point is located is shifted to the lower left, the image in the upper right region can smoothly transition with other regions.

In one possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is positioned in an area outside the area where the user gaze point is positioned and the area surrounds the edge of the area where the user gaze point is positioned; the third object is within a first directional range of the first object, the first directional range including a direction of positional offset of the first object on the first image relative to the first object on the second image; the second offset is greater than the first offset. Taking the example that the offset direction is offset to the lower left, the offset amount of the object in the lower left range in the region where the user's gaze point is located is smaller than the offset amount of the object in the lower left range in the peripheral region surrounding the region where the user's gaze point is located. That is, the amount of shift is larger as the object is farther from the lower left of the region where the user's gaze point is located, and the image in the upper right region can smoothly transition with other regions.

In one possible design, the first image includes a first pixel point, a second pixel point and a third pixel point, the first pixel point and the second pixel point are located in the area where the user gaze point is located, and the first pixel point is closer to the edge of the area where the user gaze point is located than the second pixel point; the third image information is an area outside the area where the user's gaze point is located; the image information of the first pixel point is located between the image information of the second pixel point and the image information of the third pixel point.

That is to say, the image information of the pixel point (i.e., the first pixel point) in the edge region in the first region is the intermediate value between the image information of the pixel point (i.e., the second pixel point) in the center region and the image information of the pixel point (i.e., the third pixel point) in the region outside the first region, so that the first region and the other regions can be in smooth transition. For example, the color, brightness, resolution, and the like of the pixel point gradually change from the region outside the first region to the first region.

In one possible design, the image information includes: resolution, color, brightness, and color temperature. It should be noted that the image information may also include more information, and the embodiment of the present application is not limited.

In one possible design, the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; the first display screen is configured to display an image acquired by the first camera; the second display screen is configured to display an image of the second camera; under the condition that the positions of the first display screen and the first camera are different, at least one of the display positions or the forms of a first object on the image displayed by the first display screen and the first object on the image acquired by the first camera is different, and the display positions and the forms of a second object on the image displayed by the first display screen and the second object on the second image acquired by the first camera are the same; under the condition that the positions of the second display screen and the second camera are different, at least one of the display positions or the forms of a first object on the image displayed by the second display screen and the first object on the image acquired by the second camera are different, and the display positions and the forms of a second object on the image displayed by the second display screen and the second object on the image acquired by the second camera are the same. That is to say, among the technical scheme that this application embodiment provided, can be applicable to the wearing equipment that includes two display screens and two cameras. Such as VR glasses.

In one possible design, the morphology of the first object on the first image is different from the morphology of the first object on the second image, including: the edge contour of the first object on the second image is flatter than the edge contour of the first object on the first image. Since the first object is reconstructed from the viewing angle on the first image, the edge of the first object may not be flat on the first image, and the edge of the second object is flat on the second image because the first object is not reconstructed from the viewing angle on the second image. Because first object passes through visual angle reconsitution, so the user sees first object and does not have dizzy sense (the position of display screen and camera leads to the camera to shoot visual angle and the dizzy sense that the visual angle was observed to people's eye and bring) when wearing equipment, promotes VR and experiences.

In a second aspect, a display method is further provided, which is applied to a wearable device, where the wearable device includes at least one display screen, at least one camera, and a processor; the camera is configured to transmit images it captures to the processor, the images being displayed on the display screen via the processor, including: displaying a first image to a user through the display screen; at least one of the display positions or the shapes of a first object on the first image and a first object on a second image are different, and the display positions and the shapes of the second object on the first image and the second object on the second image are the same; the second image is an image acquired at the camera; the first object is located in the area where the user's point of regard is located, and the second object is located in an area outside the area where the user's point of regard is located.

In the embodiment of the application, the wearable device can perform visual angle reconstruction on the area where the user's gaze point is located on the second image acquired by the camera, and does not perform visual angle reconstruction on the area outside the area where the user's gaze point is located. Can alleviate dizzy sense (the position of display screen and camera leads to the camera to shoot visual angle and the dizzy sense that the visual angle was observed to people's eye and was brought) through visual angle reconstruction to user's point of fixation place region, promote VR to experience.

In one possible design, an additional camera is provided at the position of the camera, and the image acquired by the additional camera is the same as the image acquired by the camera. That is, the image observed (by a person or taken with another camera) at the location of the camera is the same as the image captured by the camera.

In one possible design, an offset of the displacement between the first display position of the first object on the first image and the second display position of the first object on the second image is related to a distance between the camera and the display screen.

In one possible design, a direction of an offset between a first display position of the first object on the first image and a second display position of the first object on the second image is related to a positional relationship between the camera and the display screen.

For example, when the camera is located in a first direction of the display screen, the offset direction between the first display position and the second display position is the first direction. When the camera is located in a second direction of the display screen, the offset direction between the first display position and the second display position is the second direction.

In one possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is positioned in the area where the user gaze point is positioned and is closer to the edge of the area where the gaze point is positioned than the first object; the second offset is less than the first offset.

In one possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is located in a first area, the first area is outside the area where the user gaze point is located, and surrounds the edge of the area where the user gaze point is located; the second offset is less than the first offset.

In one possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than the degree of morphological change of a third object on the first image relative to the third object on the second image; the third object is located in the area where the user's gaze point is located, and the third object is closer to the edge of the area where the gaze point is located than the first object.

In one possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than the degree of morphological change of a third object on the first image relative to the third object on the second image; the third object is in a first area, the first area is outside the area where the user's gaze point is located and surrounds the edge of the area where the gaze point is located.

In one possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is in the area of the user's gaze point, and the third object is in a first directional range of the first object, the first directional range including a direction of positional offset of the first object on the first image relative to the first object on the second image; the second offset is greater than the first offset.

In one possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; the position offset of a third object on the first image relative to the third object on the second image is a second offset; the third object is located in a first area, and the first area is located outside the area where the user gaze point is located and surrounds the edge of the area where the user gaze point is located; the third object is within a first directional range of the first object, the first directional range including a direction of positional offset of the first object on the first image relative to the first object on the second image; the second offset is greater than the first offset.

In one possible design, the first image includes a first pixel point, a second pixel point and a third pixel point, the first pixel point and the second pixel point are located in the area where the user gaze point is located, and the first pixel point is closer to the edge of the area where the user gaze point is located than the second pixel point; the third image information is an area outside the area where the user's gaze point is located; and the image information of the first pixel point is positioned between the image information of the second pixel point and the image information of the third pixel point.

That is to say, the image information of the pixel point (i.e., the first pixel point) in the edge region in the first region is the intermediate value between the image information of the pixel point (i.e., the second pixel point) in the center region and the image information of the pixel point (i.e., the third pixel point) in the region outside the first region, so that the edge region of the first region can be in smooth transition.

In one possible design, the image information includes: resolution, color, brightness, and color temperature.

It should be noted that the image information may also include more information, and the embodiment of the present application is not limited.

In one possible design, the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; the first display screen is configured to display an image acquired by the first camera; the second display screen is configured to display an image of the second camera; under the condition that the positions of the first display screen and the first camera are different, at least one of the display positions or the forms of a first object on the image displayed by the first display screen and the first object on the image acquired by the first camera is different, and the display positions and the forms of a second object on the image displayed by the first display screen and the second object on the second image acquired by the first camera are the same; under the condition that the positions of the second display screen and the second camera are different, at least one of the display positions or the forms of a first object on the image displayed by the second display screen and the first object on the image acquired by the second camera are different, and the display positions and the forms of a second object on the image displayed by the second display screen and the second object on the image acquired by the second camera are the same.

That is to say, among the technical scheme that this application embodiment provided, can be applicable to the wearing equipment that includes two display screens and two cameras.

In one possible design, the morphology of the first object on the first image is different from the morphology of the first object on the second image, including: the edge contour of the first object on the second image is flatter than the edge contour of the first object on the first image.

In a third aspect, an electronic device is further provided, including:

a processor, a memory, and one or more programs;

wherein the one or more programs are stored in the memory, the one or more programs comprising instructions which, when executed by the processor, cause the electronic device to perform the method steps of the first or second aspect.

In a fourth aspect, there is also provided a computer readable storage medium for storing a computer program which, when run on a computer, causes the computer to perform the method of the first or second aspect.

In a fifth aspect, there is also provided a computer program product comprising a computer program which, when run on a computer, causes the computer to perform the method of the first or second aspect.

In a sixth aspect, there is also provided a graphical user interface on an electronic device, the electronic device having a display screen, a memory, and a processor configured to execute one or more computer programs stored in the memory, the graphical user interface comprising a graphical user interface displayed when the electronic device performs the method of the first or second aspect.

In a seventh aspect, an embodiment of the present application further provides a chip, where the chip is coupled to a memory in an electronic device, and is used to call a computer program stored in the memory and execute the technical solutions of the first aspect to the second aspect of the embodiment of the present application, and "coupled" in the embodiment of the present application means that two components are directly or indirectly combined with each other.

For the beneficial effects of the second to seventh aspects, reference is made to the beneficial effects of the first aspect, and repeated description is omitted.

Drawings

Fig. 1 is a schematic diagram of VR glasses provided in an embodiment of the present application;

fig. 2A is a schematic diagram of a VR system provided by an embodiment of the present application;

fig. 2B is a schematic diagram of a VR wearable device provided in an embodiment of the present application;

FIG. 2C is a schematic view of eye tracking according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a human eye according to an embodiment of the present application;

FIG. 4A is a schematic view of a human eye viewing an object with naked eyes according to an embodiment of the present application;

fig. 4B is a schematic diagram of an object being observed by wearing VR glasses on human eyes according to an embodiment of the present application;

fig. 4C is a schematic view of a human eye wearing VR glasses to observe an object according to an embodiment of the present application;

fig. 5A to fig. 5B are schematic diagrams of an application scenario provided in an embodiment of the present application;

fig. 6A-6B are schematic diagrams of a visual reconstruction process provided in an embodiment of the present application;

FIGS. 7-8 are schematic diagrams of visual reconstructions provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a first coordinate system and a second coordinate system provided by an embodiment of the present application;

fig. 10 to fig. 11 are schematic diagrams illustrating a view angle reconstruction performed on a first region according to an embodiment of the present application;

fig. 12 is a flowchart illustrating a display method according to an embodiment of the present application;

fig. 13 is a schematic diagram of an application scenario provided in an embodiment of the present application;

FIG. 14 is a schematic diagram of a planar two-dimensional image provided by an embodiment of the present application;

fig. 15 is a schematic view of the convergence angle according to an embodiment of the present application;

FIG. 16 is a schematic diagram illustrating a planar two-dimensional image converted into a three-dimensional point cloud according to an embodiment of the present application;

fig. 17 is a schematic view of a virtual camera provided in an embodiment of the present application;

FIG. 18 is a schematic illustration of an image before reconstruction and an image after reconstruction provided by an embodiment of the present application;

fig. 19 is a schematic view of an electronic device according to an embodiment of the present application.

Detailed Description

Hereinafter, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

(1) The embodiments of the present application relate to at least one, including one or more; wherein a plurality means greater than or equal to two. In addition, it should be understood that the terms "first," "second," and the like in the description of the present application are used for descriptive purposes only and are not intended to indicate or imply relative importance, nor order to be construed as indicating or implying any order. For example, the first region and the second region do not represent the importance of the two, or represent the order of the two, but merely to distinguish the regions. In the embodiment of the present application, "and/or" is only one kind of association relationship describing an association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

(2) Virtual Reality (VR) technology is a man-machine interaction means created by computer and sensor technologies. VR technology combines multiple scientific technologies such as computer graphics, computer simulation, sensor, and display technologies to create a virtual environment. The virtual environment comprises a three-dimensional vivid image which is generated by a computer and dynamically played in real time, so that visual perception is brought to a user; in addition to visual perception generated by computer graphics technology, there are also perceptions such as auditory sensation, tactile sensation, force sensation, and movement, and even olfactory sensation and taste sensation, which are also called multi-perception; in addition, head rotation, eyes, gestures or other human body behavior actions of the user can be detected, data adaptive to the actions of the user are processed by the computer, the real-time response is carried out on the actions of the user, and the data are respectively fed back to the five sense organs of the user, so that a virtual environment is formed. Exemplarily, a user wears the VR wearable device to see the VR game interface, and can interact with the VR game interface through operations such as gestures and a handle as if in a game.

(3) Augmented Reality (AR) technology refers to superimposing a computer-generated virtual object on a scene of the real world, thereby implementing an enhancement to the real world. That is, in the AR technology, a real-world scene needs to be collected, and then a virtual environment is added to the real world.

Therefore, VR technology differs from AR technology in that VR technology creates a complete virtual environment, and all users see is a virtual object; in the AR technology, a virtual object is superimposed on the real world, that is, both an object in the real world and the virtual object are included. For example, a user wears transparent glasses through which the real environment around the user can be seen, and virtual objects can be displayed on the glasses, so that the user can see both the real objects and the virtual objects.

(4) The Mixed Reality technology (MR) is a bridge for building interactive feedback information among a virtual environment, a real world and a user by introducing real scene information (or called real scene information) into the virtual environment, so as to enhance the sense of Reality of user experience. In particular, a real object is virtualized (e.g., a camera is used to scan the real object for three-dimensional reconstruction to generate a virtual object), and the virtualized real object is introduced into a virtual environment such that the user can view the real object in the virtual environment.

It should be noted that the technical solution provided by the embodiment of the present application may be applied to a VR scene, an AR scene, or an MR scene; or, the method can be applied to other scenes besides VR, AR and MR, and in any case, any scene that needs to be displayed to the user by an image with a shooting visual angle different from the human eye observation visual angle.

For ease of understanding, the following description will mainly take VR scenarios as an example.

For example, please refer to fig. 2A, which is a schematic diagram of a VR system according to an embodiment of the present application. The VR system comprises a VR wearable device 100 and an image processing device 200.

The image processing apparatus 200 may include a host (e.g., VR host) or a server (e.g., VR server), among others. VR wearable device 100 is connected (wired or wirelessly) to a VR host or VR server. The VR host or VR server may be a device with greater computing power. For example, the VR host may be a device such as a mobile phone, a tablet computer, and a notebook computer, and the VR server may be a cloud server.

Therein, the VR-worn device 100 may be a Head Mounted Device (HMD), such as glasses, a helmet, and the like. Be provided with at least one camera on the VR wearing equipment 100 to and at least one display screen. In fig. 2A, two display screens, i.e., a display screen 110 and a display screen 112, are provided on the VR-worn device 100. Wherein the display screen 110 is used to present images to the right eye of the user. The display screen 112 is used to present images to the left eye of the user. It should be noted that the display screen 110 and the display screen 112 are wrapped inside the VR glasses, so the arrows indicating the display screen 110 and the display screen 112 in fig. 2A are represented by dashed lines. The display screen 110 and the display screen 112 may be two separate display screens or may be two different display areas on the same display screen, which is not limited in this application. Moreover, in fig. 2A, two cameras, that is, the camera 120 and the camera 122, are provided on the VR wearable device 100 as an example. The

cameras

120 and 122 are each used to capture real world images. Wherein, the image collected by the camera 120 can be displayed through the display screen 110. The images captured by the camera 122 may be displayed via the display screen 112. Generally, when the user wears the VR wearable device 100, human eyes are located close to the display screen, for example, a right eye is located close to the display screen 110 to view an image on the display screen 110, and a left eye is located close to the display screen 112 to view an image on the display screen 112. The camera is located at a different position than the human eye because the camera is located at a different position than the display screen (e.g., the camera 120 is located at the lower right of the display screen 110 and the camera 122 is located at the lower left of the display screen 112). In this case, the shooting angle of view of the camera is different from the observation angle of view of human eyes. For example, with continued reference to fig. 2A, the camera 120 has a different shooting angle of view than the right eye, and the camera 122 has a different shooting angle of view than the left eye. Therefore, discomfort can be caused to the user, the vertigo can be caused for a long time, and the experience is poor.

In this embodiment of the application, the VR wearable device 100 may send the image collected by the camera to the image processing device 200 for processing. For example, the image processing device 200 performs perspective reconstruction on the image by using the perspective reconstruction scheme provided in the present application (the specific implementation process will be described later), and sends the image subjected to perspective reconstruction to the VR wearable device 100 for display. For example, the VR wearable device 100 sends the image 1 collected by the camera 120 to the image processing device 200 for view angle reconstruction to obtain an image 2, and then the display screen 110 displays the image 2. The VR wearable device 100 sends the image 3 collected by the camera 122 to the image processing device 200 for view angle reconstruction to obtain an image 4, and then the display screen 112 displays the image 4. In this way, the user glasses view the image with the reconstructed viewing angle, which can alleviate the discomfort (described later). In some embodiments, the image processing device 200 may not be included in the VR system of fig. 2A. For example, the VR-worn device 100 has image processing capabilities (e.g., the ability to reconstruct the perspective of an image) locally, without processing by the image processing device 200 (VR host or VR server). For convenience of understanding, the VR-equipped apparatus 100 performs the angle-of-view reconstruction locally, and the VR-equipped apparatus 100 is mainly used as VR glasses.

For example, please refer to fig. 2B, which shows a schematic structural diagram of a VR wearable device 100 provided in an embodiment of the present application. As shown in fig. 2B, the VR-worn device 100 may include a processor 111, a memory 101, a sensor module 130 (which may be used to obtain gestures of a user), a microphone 140, keys 150, an input-output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 1100, an eye tracking module 1200, and the like.

It is to be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation to the VR-worn device 100. In other embodiments of the present application, the VR-worn device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 111, which is typically used to control the overall operation of the VR wearable device 100, may include one or more processing units, such as: the processor 111 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a Video Processing Unit (VPU) controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

A memory may also be provided in processor 111 for storing instructions and data. In some embodiments, the memory in the processor 111 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 111. If the processor 111 needs to use the instruction or data again, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 111, thereby increasing the efficiency of the system.

In some embodiments of the present application, the processor 111 may be used to control the optical power of the VR wearable device 100. Illustratively, the processor 111 may be configured to control the optical power of the optical display module 1100, and implement the function of adjusting the optical power of the wearing apparatus 100. For example, the processor 111 may adjust the focal power of the optical display module 1100 by adjusting the relative position between each optical device (e.g., lens, etc.) in the optical display module 1100, so that the position of the corresponding virtual image plane may be adjusted when the optical display module 1100 images human eyes. Thereby achieving the effect of controlling the focal power of the wearable device 100.

In some embodiments, processor 111 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, a Serial Peripheral Interface (SPI) interface, and/or the like.

In some embodiments, the processor 111 may blur objects at different depths to make the sharpness of objects at different depths different.

The I2C interface is a bidirectional synchronous serial bus including a serial data line (SDA) and a Serial Clock Line (SCL). In some embodiments, processor 111 may include multiple sets of I2C buses.

The UART interface is a universal serial data bus used for asynchronous communications. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 111 with the communication module 170. For example: the processor 111 communicates with the bluetooth module in the communication module 170 through the UART interface to implement the bluetooth function.

The MIPI interface may be used to connect the processor 111 with peripheral devices such as a display screen and a camera 180 in the optical display module 1100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal and may also be configured as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 111 with the camera 180, a display screen in the optical display module 1100, the communication module 170, the sensor module 130, the microphone 140, and the like. The GPIO interface may also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, and the like. In some embodiments, the camera 180 may capture an image including a real object, and the processor 111 may fuse the image captured by the camera with a virtual object, and the image obtained by the real fusion is displayed by the optical display module 1100. In some embodiments, camera 180 may also capture images including the human eye. The processor 111 performs eye tracking from the image.

The USB interface is an interface which accords with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface and the like. The USB interface can be used to connect the charger to charge the VR wearable device 100, and also can be used to transmit data between the VR wearable device 100 and the peripheral device. And the earphone can also be used for connecting an earphone and playing audio through the earphone. The interface can also be used for connecting other electronic equipment, such as mobile phones and the like. The USB interface may be USB3.0, and is configured to be compatible with Display Port (DP) signaling, and may transmit video and audio high-speed data.

It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present application is only an illustration, and does not constitute a structural limitation for the wearing apparatus 100. In other embodiments of the present application, the wearable device 100 may also adopt different interface connection manners or a combination of multiple interface connection manners in the above embodiments.

In addition, the VR-worn device 100 may include wireless communication functionality, for example, the VR-worn device 100 may receive images from other electronic devices (e.g., a VR host) for display. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function may be implemented by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), a baseband processor (not shown), and the like. The antenna is used for transmitting and receiving electromagnetic wave signals. Multiple antennas may be included in VR-worn device 100, each antenna operable to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module may provide a solution for wireless communication including a second generation (2th generation, 2g) network/a third generation (3th generation, 3g) network/a fourth generation (4th generation, 4g) network/a fifth generation (5th generation, 5g) network applied to the VR-worn device 100. The mobile communication module may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module can receive electromagnetic waves by the antenna, filter and amplify the received electromagnetic waves, and transmit the electromagnetic waves to the modulation and demodulation processor for demodulation. The mobile communication module can also amplify the signal modulated by the modulation and demodulation processor and convert the signal into electromagnetic wave to radiate the electromagnetic wave through the antenna. In some embodiments, at least part of the functional modules of the mobile communication module may be provided in the processor 111. In some embodiments, at least part of the functional modules of the mobile communication module may be provided in the same device as at least part of the modules of the processor 111.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then passes the demodulated low frequency baseband signal to a baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker, etc.) or displays images or videos through a display screen in the optical display module 1100. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be separate from the processor 111, and may be disposed in the same device as the mobile communication module or other functional modules.

The wireless communication module may provide a solution for wireless communication applied to the VR-equipped device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), global Navigation Satellite System (GNSS), frequency Modulation (FM), near Field Communication (NFC), infrared (IR), and the like. The wireless communication module may be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via the antenna, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 111. The wireless communication module may also receive a signal to be transmitted from the processor 111, frequency-modulate and amplify the signal, and convert the signal into electromagnetic waves via the antenna to radiate the electromagnetic waves.

In some embodiments, the antenna and the mobile communication module of the VR-worn device 100 are coupled such that the VR-worn device 100 can communicate with networks and other devices through wireless communication techniques. The wireless communication technology may include global system for mobile communications (GSM), general Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), wideband Code Division Multiple Access (WCDMA), time-division code division multiple access (time-division code division multiple access, TD-SCDMA), long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technologies, etc. GNSS may include Global Positioning System (GPS), global navigation satellite system (GLONASS), beidou satellite navigation system (BDS), quasi-zenith satellite system (QZSS), and/or Satellite Based Augmentation System (SBAS).

The VR-equipped device 100 implements a display function through a GPU, an optical display module 1100, and an application processor. The GPU is a microprocessor for image processing, and is connected to the optical display module 1100 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 111 may include one or more GPUs that execute program instructions to generate or alter display information.

Memory 101 may be used to store computer executable program code, which includes instructions. Processor 111 executes various functional applications and data processing of VR-worn device 100 by executing instructions stored in memory 101, memory 101. Memory 101 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The data storage area may store data (such as audio data, a phone book, etc.) created during use of the wearable device 100, and the like. Further, the memory 101 may include a high speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

VR wearable device 100 may implement audio functions via an audio module, a speaker, a microphone 140, a headset interface, and an application processor, among others. Such as music playing, recording, etc. The audio module is used for converting digital audio information into analog audio signals to be output and converting the analog audio input into digital audio signals. The audio module may also be used to encode and decode audio signals. In some embodiments, the audio module may be disposed in the processor 111, or some functional modules of the audio module may be disposed in the processor 111. Loudspeakers, also known as "horns," are used to convert electrical audio signals into sound signals. The wearable device 100 can listen to music through a speaker or listen to a hands-free call.

The microphone 140, also known as a "microphone", is used to convert sound signals into electrical signals. The VR-worn device 100 may be provided with at least one microphone 140. In other embodiments, the VR-worn device 100 may be provided with two microphones 140 to achieve a noise reduction function in addition to collecting sound signals. In other embodiments, three, four or more microphones 140 may be further disposed on the VR wearable device 100 to collect sound signals, reduce noise, identify sound sources, perform directional sound recording, and so on.

The earphone interface is used for connecting a wired earphone. The headset interface may be a USB interface, or may be a 3.5 millimeter (mm) open mobile platform (OMTP) standard interface, or a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

In some embodiments, the VR-worn device 100 may include one or more keys 150 that may control the VR-worn device to provide a user with access to functionality on the VR-worn device 100. Keys 150 may be in the form of buttons, switches, dials, and touch or near touch sensitive devices (e.g., touch sensors). Specifically, for example, the user may open the optical display module 1100 of the VR wearable device 100 by pressing a button. The keys 150 include a power-on key, a volume key, and the like. The keys 150 may be mechanical keys. Or may be touch keys. The wearable device 100 may receive key inputs, generating key signal inputs related to user settings and function control of the wearable device 100.

In some embodiments, the VR-worn device 100 may include an input-output interface 160, and the input-output interface 160 may connect other devices to the VR-worn device 100 through suitable components. The components may include, for example, audio/video jacks, data connectors, and the like.

The optical display module 1100 is used for presenting images to a user under the control of the processor 111. The optical display module 1100 may convert real pixel image display into virtual image display of near-eye projection through one or more optical devices of a mirror, a transmission mirror, an optical waveguide, or the like, so as to realize virtual interactive experience, or realize interactive experience combining virtual and reality. For example, the optical display module 1100 receives the image data information sent by the processor 111 and presents the corresponding image to the user.

In some embodiments, the VR wearable device 100 may further include an eye tracking module 1200, where the eye tracking module 1200 is configured to track the movement of the human eye and determine the gaze point of the human eye. For example, the pupil position can be located by image processing technology, the pupil center coordinates can be obtained, and the fixation point of the person can be calculated. In some embodiments, the eye tracking system may determine the position of the user's gaze point (or determine the direction of the user's line of sight) by video eye mapping or photodiode response or pupil-corneal reflection, etc., to achieve eye tracking of the user.

In some embodiments, the user's gaze direction is determined by using pupillary-corneal reflex, for example. As shown in fig. 2C, the eye tracking system may include one or more near-infrared Light Emitting Diodes (LEDs) and one or more near-infrared cameras. The near-infrared LED and the near-infrared camera are not shown in fig. 2B. In various examples, the near-infrared LEDs may be disposed around an eyepiece to provide full illumination of the human eye. In some embodiments, the center wavelength of the near infrared LED may be 850nm or 940nm. The eye tracking system can acquire the sight direction of the user by the following method: the human eyes are illuminated by the near-infrared LEDs, the near-infrared cameras shoot images of the eyeballs, and then the optical axis direction of the eyeballs is determined according to the positions of light reflecting points of the near-infrared LEDs on the cornea in the images of the eyeballs (namely the images of LED light spots on the near-infrared cameras in fig. 2C) and the centers of pupils (namely the images of the centers of the pupils on the near-infrared cameras in fig. 2C), so that the sight line direction of a user is obtained.

It should be noted that, in some embodiments of the present application, respective eye-tracking systems may be provided for the two eyes of the user, so as to perform eye-tracking on the two eyes synchronously or asynchronously. In other embodiments of the present application, an eye tracking system may be only disposed near one human eye, the eye tracking system may acquire the gaze direction corresponding to the human eye, and the gaze direction or the gaze point position of another human eye may be determined according to the relationship between the gaze points of the two eyes (for example, when a user observes an object through the two eyes, the gaze point positions of the two eyes are generally close to or the same) and by combining the distance between the two eyes of the user.

It is to be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation to the VR-worn device 100. In other embodiments of the present application, the VR-worn device 100 may include more or fewer components than those shown in fig. 2A, or combine certain components, or split certain components, or a different arrangement of components, and the embodiments of the present application are not limited thereto.

In order to clearly explain the technical solution of the present application, first, a mechanism for generating the visual sensation of the human eye is briefly described below.

Fig. 3 is a schematic diagram of the composition of a human eye. As shown in fig. 3, the human eye may include a lens and ciliary muscles therein, and a retina located at the fundus of the eye. The lens can play a role of a zoom lens, and can converge light rays entering human eyes so as to converge the incident light rays on retinas of eyegrounds of the human eyes, so that scenes in an actual scene can form clear images on the retinas. The ciliary muscle can be used for adjusting the shape of the crystalline lens, for example, the ciliary muscle can adjust the diopter of the crystalline lens by contracting or relaxing, so as to achieve the effect of adjusting the focal length of the crystalline lens. So that objects at different distances in the actual scene can be clearly imaged on the retina through the crystalline lens.

In the real world, when a user (without wearing VR glasses) views an object, the viewing angles of the left and right eyes are different. The user's brain can determine the depth of the same object according to the parallax of the object in the left and right eyes, so that the world seen by the human eyes is stereoscopic. Generally, the larger the parallax, the smaller the depth, and the smaller the parallax, the larger the depth. For example, referring to fig. 4A, the real world includes an observed object 400 (taking a triangle as an example). When the human eye observes the observed object 400, the left eye captures an image 401 in which a triangle is located at a position of (A1, B1) in the image 401. And an image 402 captured by the right eye, wherein the triangle in the image 402 is located at the position of (A2, B2). The brain can determine the position of the same object (e.g., a triangle) in the real world by the pixel difference (alternatively referred to as disparity) of the object on the image 401 and the image 402. For example, the brain determines the position of the triangle in the real world as (A3, B3, L1) based on the position of the triangle in the image 401 (A1, B1) and the position of the triangle in the image 402 (A2, B2), where L1 is the depth of the triangle, i.e., the distance between the triangle and the eyes of the user. That is, the distance between the triangle and the user's eye seen by the user without wearing VR glasses is L1, and the distance between the triangle and the user's eye in the real world and the distance between the triangle and the user's eye perceived by the brain are equal.

At this time, the user position is kept still and wears VR glasses through which the same observed object 400 (i.e., a triangle) is viewed. However, the position of the observed object 400 seen by the user after wearing VR glasses is different from the position of the observed object 400 seen when not wearing VR glasses.

For example, referring to fig. 4B and fig. 2A, in some embodiments, the camera 120 on the VR glasses is located at the lower right of the display screen 110, and the camera 122 is located at the lower left of the display screen 112, so that the distance B between the two cameras is greater than the distance B between the two eyes of the human eye. Briefly, the camera 122 is further to the left than the left eye of the person, and the camera 120 is further to the right than the right eye of the person. As shown in fig. 4B, the triangle on the image 422 captured by the camera 122 is located at the position of (A1, B1'). Since the camera 122 is further to the left than the left eye of the person, the position of the triangle on the image 422 captured by the camera 122 is further to the right than the position of the triangle on the image 401 (see fig. 4A) captured by the left eye when the VR glasses are not worn, that is, (A1, B1,) is further to the right than (A1, B1). Continuing with FIG. 4B, the triangle on image 420 captured by camera 120 is at the position of (A2, B2,). Since the camera 120 is further to the right than the right eye of the person, the position of the triangle on the image 420 captured by the camera 120 is further to the left than the position of the triangle on the image 402 (see fig. 4A) captured by the right eye when the VR glasses are not worn, i.e., (A2, B2,) is further to the left than (A2, B2). With continued reference to fig. 4B, assuming that the image 422 is displayed via the display 112, the image 420 is displayed via the display 110, and the position where the brain sees the triangle based on the image 422 and the image 420 is (A3, B3, L2), that is, the user sees the triangle after wearing VR glasses and the distance between the user's eyes is L2. Since (A1, B1,) is closer to the right than (A1, B1) or (A1, B1,) is closer to the center of the image than (A1, B1,), (A2, B2,) is closer to the left than (A2, B2,) or (A2, B2,) is closer to the center of the image than (A2, B2,) is, the pixel difference between (A2, B2,) and (A1, B1,) in fig. 4B is larger than the pixel difference between (A2, B2,) and (A1, B1,) in fig. 4A. Therefore, the depth L2 of the triangle that the user sees based on the pixel difference between (A1, B1,) and (A2, B2,) is smaller than the depth L1 of the triangle that is seen based on the pixel difference between (A1, B1,) and (A2, B2).

That is, when the user is at the same position, the object seen when wearing VR glasses is closer to the user than the object seen when not wearing VR glasses. For example, in the real world, human eyes (without wearing VR glasses) see objects 1 meter away from a user, and when the user wears VR glasses, the objects are 0.7 meter away from the user and are closer to the user, which is inconsistent with the real situation.

The above embodiment takes one observed object as an example (i.e. triangle), and the lower embodiment takes two observed objects as an example, such as an observed object 400 (triangle) and an observed object 401 (square) in fig. 4C. For ease of understanding, the observed object 402 is located at an infinite distance, such as the sun, for example. Without VR glasses, the left eye would see image 460 and the right eye would see image 470. Since the square is at infinity and close to the left and right eyes, the square is centered on both image 460 and image 470. In this way, the brain can see a real environment based on the

images

460 and 470. When the user wears VR glasses, the left eye sees the image 480 captured by the camera 122 and the right eye sees the image 490 captured by the camera 120. Because the camera 122 is to the left of the human eye, the distance between the triangle and square on the image 480 viewed at the position of the camera 122 is greater than the distance between the triangle and square on the image 460 viewed at the position of the left eye. Similarly, because the camera 120 is positioned to the right of the right eye, the distance between the triangle and the square on the image 490 viewed at the position of the camera 120 is greater than the distance between the triangle and the square on the image 490 viewed at the position of the right eye. Therefore, in the case of wearing VR glasses, the triangle seen by the brain based on the

images

480 and 490 is closer to the user, which is not in accordance with the real situation.

In the above embodiment, the example where the camera 120 on the VR glasses is located at the lower right of the display screen 110 and the camera 122 is located at the lower left of the display screen 112 is taken as an example for description. It is understood that in other embodiments, the camera 120 and the camera 122 may be located in other positions, for example, the camera 120 is located above the display screen 110, and the camera 122 is located above the display screen 112; or the distance between the camera 120 and the camera 122 is smaller than the distance between the two display screens, and the like, as long as the camera position is different from the display screen position, the distance between the object and the eyes of the user when wearing the VR glasses is different from the distance between the object and the eyes of the user when not wearing the VR glasses.

For convenience of understanding, an application scenario in which a user plays a game at home wearing VR glasses is illustrated as fig. 5A. VR glasses can show real scene to the user in this application scene, so the user can see the environment in family when wearing VR glasses, for example sofa, desk in family etc.. In other embodiments, the VR glasses may present a real scene and a virtual object to the user, so that the user may see an environment and a virtual object (e.g., a game character, a game interface, etc., and the virtual object is not an object in the real scene) at home when wearing the VR glasses, and in this way, the user may play a virtual game in a familiar environment, and experience is better.

As shown in fig. 5B, what the user sees without wearing VR glasses would be the real world 501 shown in (a) of fig. 5B. What the human eye sees when the user wears VR glasses is a virtual world 502 shown in (B) in fig. 5B. As can be seen, each object in the virtual world 502 is closer to the user, and especially, an object originally closer to the user in the real world, such as a table, is.

To solve this problem, the embodiments of the present application provide a solution, namely view angle reconstruction. View reconstruction may be simply understood as view adjustment/reconstruction, etc. As mentioned above, since the shooting angle of view of the camera is different from the observation angle of view of the human eyes, the user sees a different scene when wearing VR glasses than when not wearing VR glasses. Therefore, in brief, the visual angle reconstruction refers to adjusting the shooting visual angle of the camera to the observation visual angle of human eyes. However, the difficulty of adjusting the shooting angle of view of the camera is high, for example, the camera is fixed at a certain position on VR glasses, and corresponding hardware/mechanical structure is needed to adjust the shooting angle of view, so that the cost is high, and the device is not light and thin. Therefore, in order to avoid increasing hardware cost, the effect of adjusting the shooting visual angle of the camera to the observation visual angle of human eyes can be realized through an image post-processing mode, namely, the image collected by the camera is processed, and when the processed image is displayed on VR glasses, the difference between the scene seen by a user and the scene seen when the user does not wear the VR glasses is reduced. The image processing is called image view angle reconstruction, and in brief, the image view angle reconstruction is to adjust the display position of a pixel point on an image acquired by a camera, so that an object seen by human eyes based on the adjusted image conforms to a real situation. Taking fig. 4B and 4A as an example, the image view angle reconstruction may include adjusting the positions (A1 ', B1') of the triangles in the image 422 in fig. 4B to (A1, B1); the position (A2 ', B2') of the triangle in the image 420 in fig. 4B is adjusted to (A2, B2). That is, the images before the perspective reconstruction are the image 422 and the image 420 in fig. 4B, and the images after the perspective reconstruction are the image 401 and the image 402 in fig. 4A. In this way, the images reconstructed from the viewing angles (i.e., the display images 401 and 402) can be displayed on the display screen of the VR glasses, so that the human brain can accurately determine the real position of the object (i.e., the triangle) based on the

images

401 and 402.

In one implementation, when performing view angle reconstruction on an image, view angle reconstruction may be performed on the entire image (which may be referred to as global view angle reconstruction).

The following takes the scene of fig. 5A as an example, and takes global perspective reconstruction of an image acquired by one camera on VR glasses as an example. Assume that the image acquired by the camera is the image in fig. 6A (a). As in fig. 6A (b), in some embodiments, the image is divided into four regions, region 601, region 602, region 603, and region 604. It is assumed that the display positions of the

regions

602 and 604 are shifted down after the viewing angle reconstruction, and the display positions of the

regions

601 and 603 are shifted up after the viewing angle reconstruction. As shown in fig. 6A (c), it can be seen that objects such as a wall, a sofa, and a table are deformed (or distorted, dislocated, and the like).

It should be noted that fig. 6A is an example of dividing an image into four regions for view angle reconstruction, and actually, the granularity of dividing the region on the image is finer during global view angle reconstruction, for example, the region is divided into 9 or 16 regions or a greater number of regions; or even to reconstruct each pixel point. It can be understood that, when the visual angle reconstruction is performed on the region with the finer granularity or the visual angle reconstruction is performed on each pixel point, the deformation of the object on the image is more serious. For example, as shown in fig. 6B, the wall surface of the image reconstructed from the global view is distorted (e.g., wavy distortion), and the edge of the table is also distorted (e.g., wavy distortion). Therefore, the global view reconstruction scheme not only has huge workload, but also has serious picture distortion after view reconstruction, and has great influence on user experience.

In other implementations, no view angle reconstruction of the entire image is required. For example, only the first region on the image (the image acquired by the camera) needs to be subjected to view angle reconstruction, and the second region (the other region except the first region) on the image does not need to be subjected to view angle reconstruction. Wherein the first area may be an area where a user's gaze point is located, a user-interested area, a default area, a user-specified area, and the like. For convenience of understanding, the view angle reconstruction of the first region on the image may be referred to as region view angle reconstruction. Since only the first region is reconstructed from the view angle and the second region is not reconstructed from the view angle, the workload is reduced, and as described above, the view angle reconstruction may generate picture distortion, and since the second region does not need to be reconstructed, the image in the second region may not be distorted. That is, the probability or degree of picture distortion occurring in the local view reconstruction is much smaller than that occurring in the global view reconstruction, which is helpful for improving the picture distortion phenomenon occurring in the global view reconstruction.

Illustratively, continuing with the scenario of fig. 5A as an example, assume that the image captured by the VR glasses is an image as shown in fig. 7 (a). Assuming that the area where the user gazing point is located is the area surrounded by the dotted line, only the area surrounded by the dotted line is subjected to view angle reconstruction, and no other area is subjected to view angle reconstruction, so that the display position and/or the form of the object in the dotted line area on the image subjected to view angle reconstruction are changed, and the display position and/or the form of the object in the other area are not changed, as shown in fig. 7 (b). Therefore, the distortion degree of the object on the image after the perspective reconstruction is obviously lower than that of the object on the image after the global perspective reconstruction. For example, please compare fig. 6B with fig. 7 (B), fig. 6B is the image reconstructed from the global viewing angle, and fig. 7 is the image reconstructed by using the technical solution of the present application, it can be seen that other regions (regions outside the region where the gazing point is located) on the image reconstructed by using the technical solution of the present application are stable, for example, the sofa, the wall surface, etc. are not distorted, and the degree and probability of image distortion are significantly reduced.

It should be noted that, in fig. 7, taking the area where the user's gaze point is located as an area surrounded by a dotted line as an example, the area surrounded by the dotted line may be the minimum bounding rectangle of the table, or may be greater than or equal to the minimum bounding rectangle of the table. It will be appreciated that a square shape, a circle shape, etc. may be used to define the minimum circumscribed shape of the table, and the shape is not limited. In other embodiments, the area where the user's gaze point is located may also be a partial area on the table.

Fig. 7 exemplifies the regional view angle reconstruction of the image acquired by one camera, and it can be understood that when two cameras are included on the VR glasses, the regional view angle reconstruction of the image acquired by each camera can be performed respectively.

Illustratively, as shown in fig. 8, the camera 122 on the vr glasses captures an image 622, and the camera 120 captures an image 620. The VR glasses may perform a local view reconstruction on a dashed area on the image 622, to obtain an image 624. The objects in the dashed area in image 624 and the objects in the dashed area in image 622 are displayed in different positions and/or morphologies. For example, the display position of the table in image 624 is to the left of the display position of the table in image 622, and/or the table is deformed to some extent. The other region (region other than the dotted line region) in the image 622 is not subjected to view angle reconstruction, so that the object (for example, a table) in the other region on the image 624 is displayed in the same position and shape as the object in the other region on the image 622. The VR glasses may also perform a local view reconstruction on the dashed area on image 620 to obtain image 626. The objects within the dashed area in image 626 are displayed in a different position and/or morphology than the objects within the dashed area in image 620. For example, the display position of the table in image 626 is to the right of the display position of the table in image 620, and/or the table is deformed to some extent. The other region (region other than the dotted line region, for example, the region where the couch is located) in the image 620 is not reconstructed from the angle of view, and therefore the display position and the form of the object in the other region in the image 626 are the same as those of the object in the other region in the image 620.

Display screen 112 of VR glasses displays image 624 and display screen 120 displays image 626. Thus, after wearing VR glasses, a user can see the image 624 with the left eye and the image 626 with the right eye, and determine that the depth information of the table is accurate based on the parallax of the table on the

images

624 and 626, because the display positions of the table on the

images

624 and 626 are adjusted, the parallax of the table on the two images becomes smaller after adjustment, and the determined depth information is larger based on the smaller parallax, so the user does not feel that the table approaches the user, and the view of the user is consistent with the real situation. In addition, since the area where the dotted line is located is reconstructed as the viewing angle, the table viewed by the user wearing VR glasses is deformed to some extent, but since the other area is not reconstructed as the viewing angle, the object in the other area viewed by the user is not distorted, and the distortion/shape degree is reduced compared with the global viewing angle reconstruction. Moreover, since the other areas are not reconstructed in view angle, the display positions of the objects in the other areas, which are seen by the user wearing the VR glasses, are inaccurate and are different from the real world, but because the other areas are not the user gazing areas, the user attention is low, so that the influence on the user experience is not great when the display positions of the objects in the other areas are inaccurate, the workload is saved, and the efficiency is improved.

The following describes the implementation principle of the above-mentioned view angle reconstruction with reference to the drawings.

First, two coordinate systems, a first coordinate system (X1-O1-Y1) and a second coordinate system (X2-O2-Y2), will be described. Wherein the first coordinate system (X1-O1-Y1) is a coordinate system established based on the display screen. For example, as shown in fig. 9, the first coordinate system uses the center of the display screen 112 as the origin of coordinates, and the display direction is the Y-axis direction. It will be appreciated that the first coordinate system may also be a coordinate system established on the basis of the human eye, such as the left eye in fig. 9. The difficulty of establishing the coordinate system based on the human eyes is high, the difficulty of establishing the coordinate system based on the display screen is low, and the position of the display screen is close to the position of the human eyes, so that the coordinate system established based on the display screen can be considered to be the same as the coordinate system established based on the human eyes to a certain extent. A second coordinate system (X2-O2-Y2) is established based on camera 122. For example, as shown in FIG. 9, the second coordinate system (X2-O2-Y2) is created based on the camera 122, i.e., the camera 122 images in the second coordinate system (X2-O2-Y2) when shooting the object. Since the image collected by the camera 122 and the image displayed by the display screen 112 are not in the same coordinate system, the shooting angle of the camera is different from the observation angle of the human eye. Therefore, the perspective reconstruction of the image acquired by the camera 122 is understood to be a coordinate transformation of the image acquired by the camera 122 from the second coordinate system into the first coordinate system.

The offset is used to transfer from the second coordinate system to the first coordinate system, and the offset is an offset between the second coordinate system and the first coordinate system, and can also be understood as a distance between the center of the camera 122 and the center of the display screen 112. The visual angle reconstruction of the image collected by the camera 122 includes: and shifting the pixel point on the image to a target position according to the offset. For example, taking the foregoing fig. 4B as an example, the position of the triangle on the image 422 acquired by the camera 122 is (A1 ', B1'), then (A1 ', B1') + the offset = (A1, B1), that is, the position (A1, B1) of the triangle in fig. 4A is obtained, and the view angle reconstruction of the triangle is completed.

In some embodiments, the offset amount includes an offset direction and/or an offset distance (the offset distance may also be referred to as a displacement offset amount).

The offset distance may be the distance between the origin of the second coordinate system to the origin of the first coordinate system. That is, the offset distance is related to the distance between the display screen 112 and the camera 122. For example, as the distance between the display screen 112 and the camera 122 is greater, the distance between the first coordinate system and the second coordinate system is greater, i.e., the offset distance is greater. In some embodiments, the offset distance increases as the distance between camera 122 and display screen 112 increases and decreases as the distance between camera 122 and display screen 112 decreases. Illustratively, when the distance between the camera 122 and the display screen 112 is a first distance, the offset distance is a first displacement offset. When the distance between the camera 122 and the display screen 112 is a second distance, the displacement distance is a second displacement offset. If the first distance is greater than or equal to the second distance, the first displacement offset is greater than or equal to the second displacement offset. If the first distance is less than the second distance, the first displacement offset is less than the second displacement offset. For example, taking the previous fig. 4B as an example, if the distance between the camera 122 and the display screen 112 increases, the displacement offset between the position (A1 ', B1') of the triangle on the image 422 acquired by the camera 122 and the position (A1, B1) of the triangle in fig. 4A increases.

The offset direction may be a direction from the origin of the second coordinate system to the origin of the first coordinate system. That is, the offset direction is related to the positional relationship between the display screen and the camera. In some embodiments, the offset direction varies as the direction between the camera and the display screen varies. Illustratively, when the camera is positioned in a first orientation on the display screen, the offset direction is the first orientation. When the camera is located in a second direction of the display screen, the offset direction is the second direction. For example, when the camera 122 is located on the left side of the display screen 112, i.e., the second coordinate system is on the left side of the first coordinate system, then the direction of the shift is to the left. For example, taking the previous fig. 4B as an example, the position (A1 ', B1') of the triangle on the image 422 acquired by the camera 122 is shifted to the left to the position (A1, B1) of the triangle in fig. 4A. Similarly, the camera 120 is located at the right side of the display screen 110, and the offset direction is offset to the right. For example, in the previous example of fig. 4B, the position (A2 ', B2') of the triangle on the image 420 acquired by the camera 120 is shifted to the right to the position (A2, B2) of the triangle in fig. 4A.

The first coordinate system, the second coordinate system, the offset and the like may be stored in the VR glasses in advance.

In other embodiments, the offset may vary, and in some embodiments the relative position between the display screen and the camera may vary, for example, the display screen may be movable on the VR glasses, and/or the camera may be movable on the VR glasses. For example, with the position adjustment of the display screen on the VR glasses and/or the position adjustment or shooting angle change of the camera, the offset between the first coordinate system corresponding to the display screen and the second coordinate system corresponding to the camera changes correspondingly; or, the offset amount changes correspondingly with the adjustment of the distance between the two display screens on the VR glasses and/or the adjustment of the distance between the two cameras. For example, the distance between the two display screens and/or the distance between the two cameras may be adjusted with the distance between the left eye pupil and the right eye pupil of the user. This solution may be applicable to VR glasses with adjustable display and/or camera positions. The VR glasses can be suitable for various crowds, for example, when the VR glasses are used by users with wide eye distance, the relative distance between the display screen and the camera can be adjusted to be a little larger; when the device is used by a user with a narrow eye distance, the relative distance between the display screen and the camera can be adjusted to be smaller, and the like. Therefore, a piece of VR glasses can be suitable for multiple users, for example, one piece of VR glasses can be used for the whole family. No matter how the position of the display screen and/or the camera is adjusted, the offset is adjusted correspondingly, and the VR glasses can realize visual angle reconstruction based on the adjusted offset.

In some embodiments, the VR glasses may shift all pixel points on the image collected by the camera to the target position according to the offset (i.e., global view reconstruction).

In other embodiments, the VR glasses may first determine a first region on the image, and shift a pixel point in the first region to a target position according to the offset. That is, only the pixel points in the first region are shifted, and the pixel points in other regions may be fixed.

Illustratively, the first region may be a region on the image where the user's gaze point is located. In some embodiments, an eye tracking module is included in the VR glasses, by which the user's gaze point can be located. One way to achieve this is that the VR glasses determine that the user's gaze point is at a point on an object (such as a table in fig. 7), then determine that the minimum bounding rectangle of the object (such as a table) is the first area. It is to be understood that the minimum bounding rectangle may also be a minimum bounding square, a minimum bounding circle, and the like. In other embodiments, when the VR glasses determine that the user's gaze point is located at a point on an object (such as a table), a rectangle with a preset length as a side length and with the point as a center may be determined as the first area, or a circle with the point as a center and with a preset radius as a radius may be determined as the first area, and so on. Wherein the preset length, the preset radius, etc. may be set by default. In this case, the area where the user's gaze point is located may be a partial area of the object. In still other embodiments, the first region may also be the entire region with a depth at which the user's point of regard is located.

Alternatively, the first region may also be a user region of interest on the image. The user interest area may be an area of the image where the user interested object is located. For example, an object (e.g., a person, an animal, etc.) of interest of the user may be stored in the VR glasses, and when the object of interest of the user is recognized to exist on the image captured by the camera, the area where the object is located is determined to be the first area. The object of interest to the user can be an object which is stored in the VR glasses manually by the user, or in the virtual world, the user can interact with the object in the virtual world, and the object of interest to the user can also be an object which is recorded by the VR glasses and has a user interaction execution time greater than a preset time and/or an interaction execution time greater than a preset time, and the like.

Alternatively, the first region may also be a default region, such as a central region on the image. Considering that the user generally focuses first on the central region on the image, the first region defaults to the central region.

Alternatively, the first area may also be a user-specified area. For example, the user may set the first region on VR glasses or set the first region on an electronic device (e.g., a cell phone) connected to the VR glasses, and so on.

In other embodiments, the first area may also be determined according to different scenarios. Taking the VR game scene as an example, if the user a participates in the game with the role of the game player, the area where the game role corresponding to the user a is located is the first area, or if the user a battles the game of the user B, the area where the game role corresponding to the user B (i.e., the battled player) is located is the first area. For another example, taking a VR driving scene as an example, the user a wears VR glasses to see that the virtual vehicle being driven runs on the road, and the first area may be an area where the vehicle being driven by the user a is located, or an area where a steering wheel, a windshield and the like are located on the vehicle being driven by the user a, or an area where a vehicle located in front of the vehicle being driven by the user a is located on the running road is the first area.

In summary, the first area is an area on an image acquired by the camera, and the specific manner of determining the first area includes, but is not limited to, the above manner, which is not listed in this application.

In some embodiments, the offsets of all the pixel points in the first region may be the same. For example, the offset distances of all the pixels are the distances between the origin of the first coordinate system and the origin of the second coordinate system, and the cheap directions of all the pixels are the directions from the origin of the second coordinate system to the origin of the first coordinate system.

In other embodiments, the offsets of different pixels in the first region may be different. For example, as shown in fig. 10 (a), the first region 1000 includes a central region 1010 and an edge region 1020 (regions in which oblique lines are drawn). The area of the edge region 1020 may be a default, such as a region formed from the edge of the first region to a predetermined width within the first region. The offset of the pixel points in the center region 1010 is greater than the offset of the pixel points in the edge region 1020. For example, taking the distance between the origin of the first coordinate system and the origin of the second coordinate system as L as an example, the offset distance of the pixel points in the central region 1010 is equal to L, and the offset distance of the pixel points in the edge region 1020 is less than L, such as L/2, L/3, and so on. In this way, the displacement amplitude of the pixel point at the central position in the first area is larger, the displacement amplitude of the pixel point at the edge position is smaller, because the edge position is connected with other areas, if the displacement amplitude of the pixel point at the edge position is small, the connection part between the edge position and other areas can be smoother, and the phenomenon that obvious dislocation occurs at the edge of the area where the gazing point is located is avoided.

It can be understood that when the offset of the central area is large and the offset of the edge area is small, the deformation (i.e. morphological change) of the object in the central area is large, and the sound deformation of the object in the edge area is small. That is, the degree of deformation of the object in the first region gradually decreases from the center to the edge.

In other cases, the offset of the pixel points in the first region is greater than the offset of the pixel points in the second region, which may be a region outside the first region and surrounding the outer edge of the first region. The area of the second region is not limited, and may be, for example, a region formed with a predetermined width outward from the outer edge of the first region. Correspondingly, the displacement in the first area is large, and the displacement in the second area is small, so that the deformation degree of the object in the first area is large, and the deformation degree of the object in the second area is small. That is, the degree of deformation of the object gradually decreases outwardly of the first region to the second region.

In other cases, the offsets of different pixels in the edge area 1020 may be different. For example, as shown in fig. 10 (b), the edge region 1020 includes a first edge region 1022 (diagonal line portion) and a second edge region 1024 (black line portion), and assuming that the shift direction is the direction indicated by the arrow in the drawing, that is, the first region 1000 is shifted to the lower left, the first edge region 1022 is in the shift direction (i.e., the lower left of the first region 1000), and the second edge region is in the opposite direction to the shift direction (i.e., the upper right of the first region 1000). The offset of the pixel points in the two edge areas is different. Continuing with fig. 10 (b), assuming that the offset direction is the direction indicated by the arrow, the offset of the pixel in the first edge region 1022 (black region) < the offset of the pixel in the center region 1010 < the offset of the pixel in the second edge region 1024 (diagonal region). That is, objects in the first region that are within a range of the offset direction (i.e., objects within the first edge region 1022) are offset by a large amount, and objects in a range of directions opposite to the offset direction (i.e., the second edge region 1024) are offset by a small amount. Thus, when the first region is shifted in the shift direction, the edge of the first region on the opposite side to the shift direction and the other regions can smoothly transition.

In some embodiments, the first image information of the first pixel point in the edge region in the first region on the image after view angle reconstruction may be a median value, such as an average value, of the second image information and the third image information. The second image information is the image information of the second pixel point in the central area of the first area, and the third image information is the image information of the third pixel point in other areas. For example, as shown in fig. 11, the pixel a is located in the edge area 1020 of the first area 1000, the pixel B is located in other areas, and the pixel C is located in the center area 1010 of the first area 1000. The image information of the pixel point a may be an average value of the image information of the pixel point B and the pixel point C, and the image information includes one or more of resolution, color temperature, brightness, and the like. The pixel point C and the pixel point B may be pixel points close to the pixel point a. Since the edge region 1020 of the first region is a transition region between the first region and other regions, a smooth transition between the first region and other regions is achieved when the resolution, color temperature, brightness, etc. of the pixel points in the edge region 1020 are intermediate values.

It can be understood that, the above is described by taking the image collected by the camera 122 in fig. 8 as the view angle reconstruction as an example, it can be understood that the image collected by the camera 120 can also be taken as the view angle reconstruction, and the implementation principle is the same, and is not repeated.

In other embodiments, the present application provides a display method. The method is applicable to electronic devices, such as VR glasses, that include at least one camera and at least one display screen, where the camera location and the display screen location are different. For example, as shown in fig. 4c, the position of the camera on the VR glasses is different from the position of the display screen, so that when the user wears the VR glasses, the observation angle of the human eye is different from the shooting angle of the camera. Exemplarily, fig. 12 is a schematic flowchart of a display method provided in an embodiment of the present application. The method comprises the following steps:

s1, a camera collects a second image.

The camera may be any one of the cameras on the VR glasses shown in fig. 4C, such as the left camera 122 or the right camera 120. Taking left camera 122 as an example, as in fig. 4C, the triangle is located to the front left of the square (since the square is at infinity, such as the sun) in the viewing perspective of where the left camera is located. Since the image taken by the left camera 122 is a planar two-dimensional image, a triangle and a square are included on the imaging plane of the left camera 122, and the triangle is on the left side of the square, as shown in fig. 13, which is a schematic diagram of the planar two-dimensional image taken by the camera 122. The triangle is to the left of the square on the image. It will be appreciated that an additional camera is provided at the location of the camera 122, which captures the same image as the camera 122. That is, the image observed (observed by a person or taken using another camera) at the position of the camera 122 is the same as the image captured by the camera 122.

And S2, determining a first area on the second image.

The first area has a plurality of determination methods, please refer to the foregoing, and the description thereof is not repeated. Illustratively, as shown in fig. 13, the first area is a dashed area on a planar two-dimensional image acquired by the camera.

And S3, performing visual angle reconstruction on the first area.

As mentioned above, the reconstructing of the viewing angle of the first area includes performing coordinate transformation on the image in the first area, that is, transforming from the coordinate system corresponding to the camera to the coordinate system corresponding to the display screen or the human eye. Because the image collected by the camera is a planar two-dimensional image, one implementation of the coordinate transformation may be to convert the planar two-dimensional image collected by the camera into a three-dimensional point cloud, where the three-dimensional point cloud may reflect the position (including the depth) of each object in the real environment, then create a virtual camera by simulating the human eye, and capture the three-dimensional point cloud by the virtual camera to obtain an image seen at the observation angle of the human eye, so as to realize reconstruction from the observation angle of the camera to the observation angle of the human eye. Specifically, in step S3, the method for reconstructing the view angle of the first region includes the following steps:

firstly, determining depth information of pixel points in a first region. The mode for determining the depth information of the pixel points in the first region includes at least one of mode 1 and mode 2.

In the mode 1, the depth information of the pixel point is determined according to the pixel difference of the same pixel point on two images acquired by two cameras on the VR glasses. Illustratively, the depth information of the pixel point satisfies the following formula:

wherein f is the focal length of the cameras, B is the distance between the two cameras, and disparity is the pixel difference between the same pixel points on the two images; d is the depth information of the pixel.

In the mode 2, the depth information of the pixel point is determined according to the convergence angle between the left eye and the right eye of the user and the corresponding relationship between the convergence angle and the depth information.

Referring to fig. 14 (a), in a real environment, the angle formed by the left eye line and the right eye line when the two eyes observe an object is called a convergence angle θ. It is understood that the closer the object to be observed is to the human eyes, the greater the convergence angle θ and the smaller the convergence depth. Correspondingly, the farther the observed object is away from the eyes of the person, the smaller the convergence angle theta is, and the larger the convergence depth is. As shown in fig. 14 (b), when the user wears the VR glasses, the scenes seen by the user are displayed by the display screen of the VR glasses in the virtual environment presented by the VR glasses. The light rays emitted from the screen have no depth difference, so that the focal point of the eyes is fixed on the screen through the focal length adjustment, namely, the convergence angle theta is changed into the angle of the sight lines of the eyes pointing to the display screen. However, the depth of the object in the virtual environment actually seen by the user is different from the distance from the display screen to the user, so in this embodiment of the application, after the user wears the VR glasses, the convergence angle θ of the eyes of the user is determined. The VR glasses may store a database in which a correspondence relationship between the convergence angle θ and the depth information is stored, and when the VR glasses determine the convergence angle θ, the corresponding depth information is determined based on the correspondence relationship. Wherein, the database can be obtained according to experience and stored in VR glasses in advance; alternatively, the determination may be based on a deep learning manner.

And secondly, determining three-dimensional point cloud data corresponding to the first area according to the depth information of the pixels in the first area.

For example, taking the planar binary image captured by the camera 122 in fig. 13 as an example, after determining the depth information of the pixel points in the first area (the dashed area) on the planar two-dimensional image, the three-dimensional point cloud of the pixel points in the first area may be obtained, as shown in fig. 15. The three-dimensional point cloud corresponding to the first area can map the position of each pixel point in the first area in the real world. For example, in fig. 15, the point cloud corresponding to the triangle in the three-dimensional point cloud is in front of the point cloud corresponding to the square because the scene viewed at the camera 122 location is the triangle in front of the square.

And thirdly, creating a virtual camera.

It is understood that the principle of image capture by human eyes is similar to the principle of image capture by a camera, and in order to simulate the image capture process by human eyes, a virtual camera is created, which simulates human eyes, for example, the position of the virtual camera is the same as the position of human eyes, and/or the angle of view of the virtual camera is the same as the angle of view of human eyes.

For example, generally, the angle of view of human eyes is 110 degrees up and down and 110 degrees left and right, and the angle of view of the virtual camera is 110 degrees up and down and 110 degrees left and right. For another example, the VR glasses may determine the eye position, and the virtual camera is disposed at the eye position. There are various ways to determine the position of the human eye. For example, in the method 1, the position of the display screen is determined, and then the position of the human eye can be estimated by adding the distance a to the position of the display screen. The position of the human eye determined in this way is relatively accurate. The distance a is the distance between the display screen and the human eyes, and may be stored in advance. Mode 2, the position of the human eye is equal to the position of the display screen. The mode is simple, and the virtual camera is arranged at the position of the display screen, so that the discomfort caused by the fact that the shooting visual angle is different from the visual angle of human eyes can be relieved. Illustratively, the virtual camera is at a human eye position, as shown in fig. 16.

And fourthly, shooting the three-dimensional point cloud data corresponding to the first area by using the virtual camera to obtain an image, wherein the image is obtained by performing visual angle reconstruction on the first area.

For example, as shown in fig. 17, the virtual camera corresponding to the left eye captures a three-dimensional point cloud (a three-dimensional point cloud into which a two-dimensional image collected by the left camera 122 is converted), and since the virtual camera corresponding to the left eye is closer to the right than the left camera 122, the distance between the triangle and the square on the image captured by the virtual camera corresponding to the left eye is smaller. For comparison, as shown in fig. 18, an image 1701 is a planar two-dimensional image captured by the camera 122, and an image 1702 is an image captured by a virtual camera corresponding to the left eye (i.e., an image obtained by reconstructing the angle of view of the first region on the planar two-dimensional image captured by the camera 122). The distance between two objects on image 1702 is less than the distance between two objects on image 1701. The image 1702 corresponds to an image captured by the left eye of a human.

The above is described by taking the camera 122 as an example, and the same principle is applied to the camera 120, that is, a planar two-dimensional image acquired by the camera 120 is converted into a three-dimensional point cloud, and then a virtual camera corresponding to the right eye is created, and the virtual camera is used to shoot the three-dimensional point cloud. Illustratively, as shown in fig. 17, the image 1703 is a planar two-dimensional image captured by the camera 120, and the image 1704 is an image captured by a virtual camera corresponding to the right eye (i.e., an image after perspective reconstruction of the first region). The distance between two objects on image 1704 is less than the distance between two objects on image 1703. This is because the virtual camera for the right eye is located to the left of the camera 120. Therefore, the image 1704 corresponds to an image captured by the right eye of the person.

In the application, only the first area is subjected to three-dimensional point cloud mapping, and the second area is not subjected to three-dimensional point cloud mapping, so that the image shot by the virtual camera only comprises the first area and does not comprise the second area, and the workload is small.

And S4, synthesizing the image blocks in the second area on the second image and the image blocks of the first area after the view angle is reconstructed into the first image. Wherein the second region is a region on the second image other than the first region.

The second area is not subjected to view angle reconstruction, and the image shot by the virtual camera is an image subjected to view angle reconstruction of the first area, so that the first image performs view angle reconstruction of the first area and does not perform view angle reconstruction of the second area relative to the second image.

And S5, displaying the first image.

In some embodiments, the above S2 to S4 may be executed by a processor in the VR glasses, that is, after the second image (i.e., S1) is acquired by the camera, the second image is sent to the processor, the processor executes S2 to S4 to obtain the first image, and the processor displays the first image through the display screen.

Based on the same concept, fig. 19 illustrates an electronic device 1900 provided in the present application. The electronic device 1900 may be a VR-worn device (e.g., VR glasses) as in the foregoing. As shown in fig. 19, electronic device 1900 may include: one or more processors 1901; one or more memories 1902; a communications interface 1903, and one or more computer programs 1904, which may be connected via one or more communications buses 1905. Wherein the one or more computer programs 1904 are stored in the memory 1902 and configured to be executed by the one or more processors 1901, the one or more computer programs 1904 comprising instructions that can be used to perform steps associated with VR-worn devices as in the respective embodiments above. The communication interface 1903 is used to enable communication with other devices, e.g., the communication interface may be a transceiver.

In the embodiments provided in the present application, the method provided in the embodiments of the present application is described from the perspective of an electronic device (e.g., a VR-worn device) as an execution subject. In order to implement the functions in the method provided by the embodiments of the present application, the electronic device may include a hardware structure and/or a software module, and the functions are implemented in the form of a hardware structure, a software module, or a hardware structure and a software module. Whether any of the above-described functions is implemented as a hardware structure, a software module, or a hardware structure plus a software module depends upon the particular application and design constraints imposed on the technical solution.

As used in the above embodiments, the terms "when …" or "after …" may be interpreted to mean "if …" or "after …" or "in response to determining …" or "in response to detecting …", depending on the context. Similarly, the phrase "in determining …" or "if (a stated condition or event) is detected" may be interpreted to mean "if … is determined" or "in response to … is determined" or "in response to (a stated condition or event) is detected", depending on the context. In addition, in the above-described embodiments, relational terms such as first and second are used to distinguish one entity from another entity without limiting any actual relationship or order between the entities.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the present embodiments are all or partially performed when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others. The aspects of the above embodiments may all be used in combination without conflict.

It is noted that a portion of this patent application contains material which is subject to copyright protection. The copyright owner reserves the copyright rights whatsoever, except for making copies of the patent files or recorded patent document contents of the patent office.

Claims

1. The display method is characterized by being applied to wearable equipment, wherein the wearable equipment comprises at least one display screen and at least one camera; the method comprises the following steps:

displaying a first image to a user through the display screen;

at least one of the display positions or the shapes of a first object on the first image and a first object on a second image are different, and the display positions and the shapes of the second object on the first image and the second object on the second image are the same; the second image is an image acquired by the camera;

the first object is located in the area where the user's point of regard is located, and the second object is located in an area outside the area where the user's point of regard is located.

2. The method of claim 1,

the displacement offset between the first display position of the first object on the first image and the second display position of the first object on the second image is related to the distance between the camera and the display screen.

3. The method according to claim 1 or 2,

the offset direction between the first display position of the first object on the first image and the second display position of the first object on the second image is related to the positional relationship between the camera and the display screen.

4. The method according to any one of claims 1 to 3,

the position offset of the first object on the first image relative to the second object on the second image is a first offset;

the position offset of a third object on the first image relative to the third object on the second image is a second offset;

the third object is positioned in the area where the user gaze point is positioned and is closer to the edge of the area where the gaze point is positioned than the first object;

the second offset is less than the first offset.

5. The method according to any one of claims 1 to 4,

the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than the degree of morphological change of a third object on the first image relative to the third object on the second image;

the third object is located in the area where the user's gaze point is located, and the third object is closer to the edge of the area where the gaze point is located than the first object.

6. The method according to any one of claims 1 to 5,

the position offset of the first object on the first image relative to the first object on the second image is a first offset;

the third object is in the area of the user's gaze point, and the third object is in a first directional range of the first object, the first directional range including a direction of positional offset of the first object on the first image relative to the first object on the second image;

the second offset is greater than the first offset.

7. The method according to any one of claims 1 to 6,

the first image comprises a first pixel point, a second pixel point and a third pixel point, the first pixel point and the second pixel point are located in the area where the user fixation point is located, and the first pixel point is closer to the edge of the area where the user fixation point is located than the second pixel point; the third image information is an area outside the area where the user's gaze point is located;

and the image information of the first pixel point is positioned between the image information of the second pixel point and the image information of the third pixel point.

8. The method of claim 7, wherein the image information comprises: resolution, color, brightness, and color temperature.

9. The method of any of claims 1-8, wherein the at least one camera comprises a first camera and a second camera, and the at least one display screen comprises a first display screen and a second display screen; the first display screen is configured to display an image acquired by the first camera; the second display screen is configured to display an image of the second camera;

under the condition that the positions of the first display screen and the first camera are different, at least one of the display positions or the forms of a first object on the image displayed by the first display screen and the first object on the image acquired by the first camera is different, and the display positions and the forms of a second object on the image displayed by the first display screen and the second object on the second image acquired by the first camera are the same;

under the condition that the positions of the second display screen and the second camera are different, at least one of the display positions or the forms of a first object on the image displayed by the second display screen and the first object on the image acquired by the second camera are different, and the display positions and the forms of a second object on the image displayed by the second display screen and the second object on the image acquired by the second camera are the same.

10. The method of any one of claims 1-9, wherein the morphology of the first object on the first image is different from the morphology of the first object on the second image, comprising:

the edge contour of the first object on the second image is flatter than the edge contour of the first object on the first image.

11. The display method is characterized by being applied to wearable equipment, wherein the wearable equipment comprises at least one display screen, at least one camera and a processor; the camera is configured to transmit images it captures to the processor, the images being displayed on the display screen via the processor, including:

displaying a first image to a user through the display screen;

at least one of the display positions or the shapes of a first object on the first image and a first object on a second image are different, and the display positions and the shapes of the second object on the first image and the second object on the second image are the same; the second image is an image acquired at the camera;

12. The method of claim 11,

the second offset is less than the first offset.

13. The method according to any one of claims 11 to 12,

14. The method according to any one of claims 11 to 13,

the second offset is greater than the first offset.

15. The method according to any one of claims 11 to 14,

16. The method of claim 15, wherein the image information comprises: resolution, color, brightness, and color temperature.

17. The method of any of claims 11-16, wherein the at least one camera comprises a first camera and a second camera, and the at least one display screen comprises a first display screen and a second display screen; the first display screen is configured to display an image acquired by the first camera; the second display screen is configured to display an image of the second camera;

18. An electronic device, comprising:

a processor, a memory, and one or more programs;

wherein the one or more programs are stored in the memory, the one or more programs comprising instructions which, when executed by the processor, cause the electronic device to perform the method steps of any of claims 1-17.

19. A computer-readable storage medium, for storing a computer program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 17.

20. A computer program product, comprising a computer program which, when run on a computer, causes the computer to perform the method according to any one of claims 1-17.