WO2023035911A1

WO2023035911A1 - Display method and electronic device

Info

Publication number: WO2023035911A1
Application number: PCT/CN2022/113692
Authority: WO
Inventors: 李昱霄; 毛春静; 沈钢
Original assignee: 华为技术有限公司
Priority date: 2021-09-09
Filing date: 2022-08-19
Publication date: 2023-03-16
Also published as: CN115793841A

Abstract

A display method and an electronic device. The method comprises: displaying a first image to a user by means of a display screen, at least one of the display position or state of a first object on the first image being different from that of the first object on a second image, and the display position and state of a second object on the first image being the same as those of the second object on the second image; the second image being an image acquired by a camera, wherein the first object is located in a region where the gaze point of the user is located, and the second object is located in a region other than the region where the gaze point of the user is located. In this way, the problem that the photography angle of the camera is different from the viewing angle of human eyes caused by different positions of the camera and the display screen can be solved.

Description

A display method and electronic device

Cross References to Related Applications

This application claims the priority of a Chinese patent application with the application number 202111056782.7 and the application title "A Display Method and Electronic Device" submitted to the China Patent Office on September 9, 2021, the entire contents of which are incorporated in this application by reference .

technical field

The present application relates to the field of electronic technology, in particular to a display method and electronic equipment.

Background technique

Virtual Reality (VR) technology is a means of human-computer interaction created with the help of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other science and technology to create a virtual world. Users can immerse themselves in the virtual world by wearing VR wearable devices (eg, VR glasses, VR helmets, etc.).

The objects in the virtual world can be all fictitious objects, and can also include three-dimensional models of real objects, so that the virtual world seen by the user includes both fictitious objects and real objects, and the experience is more realistic. For example, a camera can be set on a VR wearable device to capture an image of a real object, and based on the image, a three-dimensional model of the real object can be constructed and displayed in the virtual world. Taking VR glasses as an example, FIG. 1 is a schematic diagram of VR glasses. The VR glasses include a camera and a display screen. Generally, in order to make the appearance of the VR glasses thin and not heavy, the camera will not be set at the position of the display screen, but is usually set at the position below the display screen, as shown in Figure 1.

This setting method will cause the angle of view direction of the human eye to be inconsistent with the angle of view direction (or called the shooting direction) of the camera. For example, in Figure 1, the shooting direction of the camera is facing downward, and the viewing angle of the human eye is ahead. If the images captured by the camera are directly displayed to the human eyes through the display screen, it will give people a sense of discomfort, and if this is done for a long time, they will feel dizzy and the experience is poor.

Contents of the invention

The purpose of the present application is to provide a display method and an electronic device for improving VR experience.

In a first aspect, a display method is provided, which is applied to a wearable device, and the wearable device includes at least one display screen and at least one camera; including: presenting a first image to the user through the display screen; displaying a first image on the first image The first object is different from at least one of the display position or form of the first object on the second image, and the display of the second object on the first image is different from the display of the second object on the second image The position and shape are the same; the second image is an image collected by the camera; wherein, the first object is in the area where the user's gaze point is located, and the second object is in an area other than the area where the user's gaze point is located.

In the embodiment of the present application, the wearable device may reconstruct the viewing angle of the area where the user's gaze point is located on the second image captured by the camera, and not perform viewing angle reconstruction for areas other than the area where the user's gaze point is located. By reconstructing the viewing angle of the area where the user's gaze point is located, it can alleviate the feeling of vertigo (the position of the display screen and the camera causes the sense of vertigo caused by the shooting angle of the camera and the viewing angle of the human eye), and improve the VR experience. The workload of viewing angle reconstruction in the area is less, and the probability or degree of picture distortion can be reduced.

In a possible design, the displacement offset between the first display position of the first object on the first image and the second display position of the first object on the second image is the same as that of the camera It is related to the distance between the display screens. For example, the greater the distance between the camera and the display screen, the greater the offset between the first object and the second object.

In a possible design, the displacement offset between the first display position and the second display position increases as the distance between the camera and the display screen increases. Decreases as the distance between the camera and the display screen decreases.

Exemplarily, when the distance between the camera and the display screen is the first distance, the displacement offset between the first display position and the second display position is the first displacement offset . When the distance between the camera and the display screen is the second distance, the displacement offset between the first display position and the second display position is the second displacement offset. When the first distance is greater than or equal to the second distance, the first displacement offset is greater than or equal to the second displacement offset. When the first distance is smaller than the second distance, the first displacement offset is smaller than the second displacement offset.

In a possible design, the offset direction between the first display position of the first object on the first image and the second display position of the first object on the second image is related to the camera and The positional relationship between the display screens is related. For example, the camera is located on the left side of the display screen, and the first object on the second image shifts to the left to the position of the first object on the first image.

In a possible design, the offset direction between the first display position and the second display position changes as the direction between the camera and the display screen changes.

Exemplarily, when the camera is located in the first direction of the display screen, the offset direction between the second display position and the first display position is the first direction. When the camera is located in the second direction of the display screen, the offset direction between the second display position and the first display position is the second direction.

In a possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is in the area where the user's gaze point is located, and is larger than the first object. Close to the edge of the area where the gaze point is located; the second offset is smaller than the first offset. That is to say, the offset of the first object at the center of the area where the user's gaze point is located is larger than that of the third object at the edge position, so that a smooth edge transition between the area where the user's gaze point is located and other areas can be achieved.

In a possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is located in an area other than the area where the user's gaze point is located, and the area is surrounded by The edge of the area where the user's gaze point is located; the second offset is smaller than the first offset. That is to say, the first object in the area where the user's gaze point is located is offset from the third object in the peripheral area (the area outside the area where the user's gaze point is located, and this area surrounds the edge of the area where the user's gaze point is located) The amount is large, so that the edge of the area where the user's gaze point is located and other areas can be smoothly transitioned.

In a possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the The morphological change degree of the third object on the second image is large; the third object is in the area where the user's gaze point is located, and the third object is closer to the area where the gaze point is located than the first object. edge. That is to say, from the center position to the edge position in the area where the user's gaze point is located, the degree of shape change of the object is smaller. In this way, a smooth edge transition between the area where the user's gaze point is located and other areas can be achieved.

In a possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the The shape of the third object on the second image changes greatly; the third object is located in an area other than the area where the user's gaze point is located, and this area surrounds an edge of the area where the gaze point is located. That is to say, from the area where the user's gazing point is located outward to the peripheral area (the area outside the area where the user's gazing point is located, and the area surrounds the edge of the area where the gazing point is located), the smaller the degree of shape change of the object is. In this way, a smooth edge transition between the area where the user's gaze point is located and other areas can be achieved.

In a possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is in the area where the user's gaze point is located, and the third object is in the Within the first direction range of the first object, the first direction range includes a position offset direction of the first object on the first image relative to the first object on the second image; The second offset is greater than the first offset. Taking the offset direction as an example of shifting to the lower left, in the area where the user gazes, the offset of objects in the lower left range is large, and the offset of objects in the upper right range is small. In this way, when the area where the user's gaze point is shifted to the lower left, the image in the upper right area can smoothly transition with other areas.

In a possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is located in an area other than the area where the user's gaze point is located and the area surrounds the The edge of the area where the user's gaze point is located; the third object is within the first direction range of the first object, and the first direction range includes the first object on the first image relative to the second The position of the first object on the image is offset in a direction; the second offset is greater than the first offset. Taking the offset direction as an example of shifting to the lower left, the offset of objects in the lower left range in the area where the user's gaze point is located is smaller than the offset amount of objects in the lower left range in the peripheral area surrounding the area where the user's gaze point is located. That is to say, from the lower left direction of the area where the user's gaze point is located, the farther the object is, the larger the offset is, and the image in the upper right area can smoothly transition with other areas.

In a possible design, the first image includes a first pixel point, a second pixel point and a third pixel point, and the first pixel point and the second pixel point are located at the user's gaze point In the area, and the first pixel point is closer to the edge of the area where the user's gaze point is located than the second pixel point; the third image information is related to the area outside the area where the user's gaze point is located; the first The image information of the pixel is located between the image information of the second pixel and the image information of the third pixel.

That is to say, the image information of the pixels (i.e. the first pixels) in the edge area in the first area is the image information of the pixels (i.e. the second pixels) in the center area and the image information of the pixels (i.e. the second pixels) in the area outside the first area ( That is, the intermediate value of the image information of the third pixel point), so that the transition between the first area and other areas can be smooth. For example, from an area outside the first area to within the first area, the color, brightness, resolution, etc. of the pixels gradually change.

In a possible design, the image information includes: at least one of resolution, color, brightness, and color temperature. It should be noted that the image information may also include more information, which is not limited in this embodiment of the present application.

In a possible design, the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; the first display screen is configured to display the the image collected by the first camera; the second display screen is configured to display the image of the second camera; if the position of the first display screen is different from that of the first camera, the first The first object on the image displayed on the display screen is different from at least one of the display position or form of the first object on the image captured by the first camera, and the second object on the image displayed on the first display screen is The display position and form of the second object on the second image captured by the object and the first camera are the same; if the positions of the second display screen and the second camera are different, the second The first object on the image displayed on the display screen is different from at least one of the display position or form of the first object on the image captured by the second camera, and the second object on the image displayed on the second display screen is The display position and shape of the object and the second object on the image captured by the second camera are the same. That is to say, the technical solutions provided in the embodiments of the present application can be applied to wearable devices including two display screens and two cameras. For example, VR glasses.

In a possible design, the shape of the first object on the first image is different from that of the first object on the second image, including: the edge profile of the first object on the second image is larger than the The edge contour of the first object on the first image is smoothed. Since the first object on the first image has undergone perspective reconstruction, the edge of the first object on the first image may be uneven, while the first object on the second image has not undergone perspective reconstruction, so the second Objects have smooth edges. Since the first object has been reconstructed from the angle of view, the user will not feel dizzy when seeing the first object when wearing the wearable device (the position of the display screen and the camera causes the dizziness caused by the shooting angle of the camera and the viewing angle of the human eye), improving VR experience.

In the second aspect, a display method is also provided, which is applied to a wearable device, and the wearable device includes at least one display screen, at least one camera, and a processor; the camera is configured to transmit the image it collects to the processing displaying the image on the display screen via the processor, including: displaying the first image to the user through the display screen; the first object on the first image and the first object on the second image At least one of the display position or form of the object is different, and the display position and form of the second object on the first image and the second object on the second image are the same; the second image is The image collected by the camera; wherein, the first object is located in the area where the user's gaze point is located, and the second object is located in an area other than the area where the user's gaze point is located.

In the embodiment of the present application, the wearable device may reconstruct the viewing angle of the area where the user's gaze point is located on the second image captured by the camera, and not perform viewing angle reconstruction for areas other than the area where the user's gaze point is located. By reconstructing the viewing angle of the area where the user's gaze is located, it can alleviate the feeling of vertigo (the position of the display screen and the camera causes the feeling of vertigo caused by the shooting angle of the camera and the viewing angle of the human eye), and improve the VR experience.

In a possible design, another camera is set at the location of the camera, and the image collected by the other camera is the same as the image collected by the camera. That is to say, the image observed at the location of the camera (observed by a person or taken by other cameras) is the same as the image collected by the camera.

In a possible design, the displacement offset between the first display position of the first object on the first image and the second display position of the first object on the second image is the same as that of the camera It is related to the distance between the display screens.

In a possible design, the offset direction between the first display position of the first object on the first image and the second display position of the first object on the second image is related to the camera and The positional relationship between the display screens is related.

Exemplarily, when the camera is located in the first direction of the display screen, the offset direction between the first display position and the second display position is the first direction. When the camera is located in the second direction of the display screen, the offset direction between the first display position and the second display position is the second direction.

In a possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is in the area where the user's gaze point is located, and is larger than the first object. Close to the edge of the area where the gaze point is located; the second offset is smaller than the first offset.

In a possible design, the position offset of the first object on the first image relative to the second object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is a second offset; the third object is in a first area, and the first area is where the user's gaze point is located Outside the area, and around the edge of the area where the user's gaze point is located; the second offset is smaller than the first offset.

In a possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the The morphological change degree of the third object on the second image is large; the third object is in the area where the user's gaze point is located, and the third object is closer to the area where the gaze point is located than the first object. edge.

In a possible design, the degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the The shape of the third object on the second image has a large degree of change; the third object is in a first area, and the first area is outside the area where the user's gaze point is located, and surrounds the area where the gaze point is located. edge.

In a possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is the second offset; the third object is in the area where the user's gaze point is located, and the third object is in the Within the first direction range of the first object, the first direction range includes a position offset direction of the first object on the first image relative to the first object on the second image; The second offset is greater than the first offset.

In a possible design, the position offset of the first object on the first image relative to the first object on the second image is a first offset; The position offset of the third object relative to the third object on the second image is a second offset; the third object is in a first area, and the first area is where the user's gaze point is located Outside the area and around the edge of the area where the user's gaze point is located; the third object is within the first direction range of the first object, and the first direction range includes the first object on the first image A direction is offset relative to the position of the first object on the second image; the second offset is greater than the first offset.

That is to say, the image information of the pixels in the edge area (i.e. the first pixel) in the first area is the image information of the pixels in the central area (i.e. the second pixel) and the image information of the pixels in the area outside the first area. The intermediate value of the image information of the pixel point (that is, the third pixel point), so that the edge area of the first area can transition smoothly.

In a possible design, the image information includes: at least one of resolution, color, brightness, and color temperature.

It should be noted that the image information may also include more information, which is not limited in this embodiment of the present application.

In a possible design, the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; the first display screen is configured to display the the image collected by the first camera; the second display screen is configured to display the image of the second camera; if the position of the first display screen is different from that of the first camera, the first The first object on the image displayed on the display screen is different from at least one of the display position or form of the first object on the image captured by the first camera, and the second object on the image displayed on the first display screen is The display position and form of the second object on the second image captured by the object and the first camera are the same; if the positions of the second display screen and the second camera are different, the second The first object on the image displayed on the display screen is different from at least one of the display position or form of the first object on the image captured by the second camera, and the second object on the image displayed on the second display screen is The display position and shape of the object and the second object on the image captured by the second camera are the same.

That is to say, the technical solutions provided in the embodiments of the present application can be applied to wearable devices including two display screens and two cameras.

In a possible design, the shape of the first object on the first image is different from that of the first object on the second image, including: the edge profile of the first object on the second image is larger than the The edge contour of the first object on the first image is smoothed.

In a third aspect, an electronic device is also provided, including:

processor, memory, and, one or more programs;

Wherein, the one or more programs are stored in the memory, the one or more programs include instructions, and when the instructions are executed by the processor, the electronic device performs the above-mentioned first aspect Or the method steps described in the second aspect.

In the fourth aspect, there is also provided a computer-readable storage medium, the computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer executes the above-mentioned first or second aspect. The method described in the two aspects.

In a fifth aspect, there is also provided a computer program product, including a computer program, which, when the computer program is run on a computer, causes the computer to execute the method as described in the first aspect or the second aspect above.

In a sixth aspect, there is also provided a graphical user interface on an electronic device, the electronic device has a display screen, a memory, and a processor, and the processor is configured to execute one or more computer programs stored in the memory, The graphical user interface includes a graphical user interface displayed when the electronic device executes the method described in the first aspect or the second aspect.

In the seventh aspect, the embodiment of the present application further provides a chip, the chip is coupled with the memory in the electronic device, and is used to call the computer program stored in the memory and execute the technical solutions of the first aspect to the second aspect of the embodiment of the present application , "Coupling" in the embodiments of the present application means that two components are directly or indirectly combined with each other.

For the above beneficial effects of the second aspect to the seventh aspect, refer to the beneficial effects of the first aspect, which will not be repeated.

Description of drawings

FIG. 1 is a schematic diagram of VR glasses provided by an embodiment of the present application;

FIG. 2A is a schematic diagram of a VR system provided by an embodiment of the present application;

FIG. 2B is a schematic diagram of a VR wearable device provided by an embodiment of the present application;

FIG. 2C is a schematic diagram of eye tracking provided by an embodiment of the present application;

Fig. 3 is a schematic structural diagram of a human eye provided by an embodiment of the present application;

FIG. 4A is a schematic diagram of naked-eye observation of an object provided by an embodiment of the present application;

FIG. 4B is a schematic diagram of human eyes wearing VR glasses to observe objects provided by an embodiment of the present application;

FIG. 4C is a schematic diagram of human eyes wearing VR glasses to observe objects provided by an embodiment of the present application;

5A to 5B are schematic diagrams of an application scenario provided by an embodiment of the present application;

6A to 6B are schematic diagrams of a visual reconstruction process provided by an embodiment of the present application;

7 to 8 are schematic diagrams of visual reconstruction provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a first coordinate system and a second coordinate system provided by an embodiment of the present application;

FIG. 10 to FIG. 11 are schematic diagrams of viewing angle reconstruction in the first area provided by an embodiment of the present application;

FIG. 12 is a schematic flowchart of a display method provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of an application scenario provided by an embodiment of the present application;

Fig. 14 is a schematic diagram of a planar two-dimensional image provided by an embodiment of the present application;

FIG. 15 is a schematic diagram of a convergence angle provided by an embodiment of the present application;

FIG. 16 is a schematic diagram of converting a plane two-dimensional image into a three-dimensional point cloud according to an embodiment of the present application;

FIG. 17 is a schematic diagram of a virtual camera provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of an image before reconstruction and an image after reconstruction provided by an embodiment of the present application;

Fig. 19 is a schematic diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In the following, some terms used in the embodiments of the present application are explained, so as to facilitate the understanding of those skilled in the art.

(1) At least one of the embodiments of the present application involves one or more; wherein, a plurality means greater than or equal to two. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as express or implied relative importance, nor can they be understood as express or imply order. For example, the first area and the second area do not represent the importance of the two, or represent the order of the two, but are only for distinguishing the areas. In the embodiment of this application, "and/or" is just a kind of relationship describing the relationship between related objects, which means that there may be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, and A and B exist at the same time. B, there are three situations of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

(2) Virtual Reality (VR) technology is a means of human-computer interaction created with the help of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other science and technology to create a virtual environment. The virtual environment includes three-dimensional realistic images generated by computers and dynamically played in real time to bring visual perception to users; moreover, in addition to the visual perception generated by computer graphics technology, there are also perceptions such as hearing, touch, force, and movement. It even includes the sense of smell and taste, also known as multi-sensing; in addition, it can also detect the user's head rotation, eyes, gestures, or other human behaviors, and the computer will process the data that is suitable for the user's actions, and The user's actions respond in real time and are fed back to the user's facial features respectively, thereby forming a virtual environment. Exemplarily, the user can see the VR game interface by wearing the VR wearable device, and can interact with the VR game interface through gestures, handles, and other operations, as if in a game.

(3) Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects on real-world scenes to enhance the real world. In other words, AR technology needs to collect real-world scenes, and then add a virtual environment to the real world.

Therefore, the difference between VR technology and AR technology is that VR technology creates a complete virtual environment, and all users see are virtual objects; while AR technology superimposes virtual objects on the real world, that is, it includes objects in the real world. Also includes dummy objects. For example, the user wears transparent glasses, through which the real environment around can be seen, and virtual objects can also be displayed on the glasses, so that the user can see both real objects and virtual objects.

(4) Mixed reality technology (Mixed Reality, MR) is to build a bridge of interactive feedback information between the virtual environment, the real world and users by introducing real scene information (or called real scene information) into the virtual environment. , thereby enhancing the realism of the user experience. Specifically, the real object is virtualized (for example, using a camera to scan the real object for 3D reconstruction to generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see in the virtual environment real object.

It should be noted that the technical solutions provided by the embodiments of this application can be applied to VR scenes, AR scenes, or MR scenes; or, they can also be applied to other scenes except VR, AR, and MR. A scene that needs to show the user an image with a shooting angle different from that observed by the human eye.

For ease of understanding, the following mainly uses VR scenarios as an example.

For example, please refer to FIG. 2A , which is a schematic diagram of a VR system according to an embodiment of the present application. The VR system includes a VR wearable device 100 and an image processing device 200 .

Wherein, the image processing device 200 may include a host (such as a VR host) or a server (such as a VR server). The VR wearable device 100 is connected (wired connection or wireless connection) with a VR host or a VR server. The VR host or VR server may be a device with relatively large computing capabilities. For example, the VR host can be a device such as a mobile phone, a tablet computer, or a notebook computer, and the VR server can be a cloud server, etc.

Wherein, the VR wearable device 100 may be a head mounted device (Head Mounted Display, HMD), such as glasses, a helmet, and the like. The VR wearable device 100 is provided with at least one camera and at least one display screen. In FIG. 2A , two display screens, ie, a display screen 110 and a display screen 112 , are set on the VR wearable device 100 as an example. Wherein, the display screen 110 is used to display images to the user's right eye. The display screen 112 is used to present images to the user's left eye. It should be noted that the display screen 110 and the display screen 112 are wrapped inside the VR glasses, so the arrows indicating the display screen 110 and the display screen 112 in FIG. 2A are represented by dotted lines. Wherein, the display screen 110 and the display screen 112 may be two independent display screens or may be two different display areas on the same display screen, which is not limited in this application. Moreover, in FIG. 2A , two cameras, that is, a camera 120 and a camera 122 are set on the VR wearable device 100 as an example. The camera 120 and the camera 122 are respectively used to collect images of the real world. Wherein, the image collected by the camera 120 can be displayed through the display screen 110 . Images collected by the camera 122 can be displayed on the display screen 112 . Generally, when the user wears the VR wearable device 100, the human eyes are located close to the display screen, for example, the right eye is close to the display screen 110 to view the images on the display screen 110, and the left eye is close to the display screen 112 to view the images on the display screen 110. The image on the display screen 112. Since the positions of the camera and the display screen are different (for example, the camera 120 is located at the lower right of the display screen 110 , and the camera 122 is located at the lower left of the display screen 112 ), the positions of the camera and human eyes are different. In this case, the viewing angle of the camera is different from that of the human eye. For example, continue to refer to FIG. 2A , the shooting angle of view of the camera 120 is different from that of the right eye, and the shooting angle of view of the camera 122 is different from that of the left eye. Like this, can cause discomfort to the user, can appear dizziness like this for a long time, and experience is poorer.

In the embodiment of the present application, the VR wearable device 100 may send the image collected by the camera to the image processing device 200 for processing. For example, the image processing device 200 uses the perspective reconstruction scheme provided in this application to reconstruct the perspective of the image (the specific implementation process will be introduced later), and sends the reconstructed image to the VR wearable device 100 for display. For example, the VR wearable device 100 sends the image 1 captured by the camera 120 to the image processing device 200 to perform perspective reconstruction to obtain an image 2, and then the display screen 110 displays the image 2. The VR wearable device 100 sends the image 3 collected by the camera 122 to the image processing device 200 to perform perspective reconstruction to obtain an image 4 , and then the display screen 112 displays the image 4 . In this case, what the user's glasses see is the image after the angle of view has been reconstructed, which can relieve the discomfort (will be introduced later). In some embodiments, the VR system in FIG. 2A may not include the image processing device 200 . For example, the VR wearable device 100 locally has image processing capabilities (for example, the ability to reconstruct the viewing angle of images), and does not need to be processed by the image processing device 200 (VR host or VR server). For the convenience of understanding, the following takes the VR wearable device 100 to locally perform perspective reconstruction as an example for illustration, and the following mainly takes the VR wearable device 100 as VR glasses as an example.

For example, please refer to FIG. 2B , which shows a schematic structural diagram of a VR wearable device 100 provided by an embodiment of the present application. As shown in FIG. 2B, the VR wearable device 100 may include a processor 111, a memory 101, a sensor module 130 (which may be used to obtain the user's posture), a microphone 140, a button 150, an input and output interface 160, a communication module 170, a camera 180, battery 190 , optical display module 1100 , eye tracking module 1200 and so on.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR wearable device 100 . In other embodiments of the present application, the VR wearable device 100 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 111 is generally used to control the overall operation of the VR wearable device 100, and may include one or more processing units, for example: the processor 111 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), image signal processor (image signal processor, ISP), video processing unit (video processing unit, VPU) controller, memory, video codec, digital signal processor (digital signal processor, DSP ), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 111 for storing instructions and data. In some embodiments, the memory in processor 111 is a cache memory. The memory may hold instructions or data that the processor 111 has just used or recycled. If the processor 111 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 111 is reduced, thus improving the efficiency of the system.

In some embodiments of the present application, the processor 111 may be used to control the optical power of the VR wearable device 100 . Exemplarily, the processor 111 may be used to control the optical power of the optical display module 1100 to realize the function of adjusting the optical power of the wearable device 100 . For example, the processor 111 can adjust the relative positions of the optical devices (such as lenses, etc.) When the human eye is imaging, the position of the corresponding virtual image plane can be adjusted. In this way, the effect of controlling the optical power of the wearable device 100 is achieved.

In some embodiments, processor 111 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, a universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general input and output (general -purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or universal serial bus (universal serial bus, USB) interface, serial peripheral interface (serial peripheral interface, SPI) interface etc.

In some embodiments, the processor 111 may perform blurring processing to different degrees on objects at different depths of field, so that objects at different depths of field have different sharpness.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 111 may include multiple sets of I2C buses.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 111 and the communication module 170 . For example: the processor 111 communicates with the Bluetooth module in the communication module 170 through the UART interface to realize the Bluetooth function.

The MIPI interface can be used to connect the processor 111 with the display screen in the optical display module 1100 , the camera 180 and other peripheral devices.

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 111 with the camera 180 , the display screen in the optical display module 1100 , the communication module 170 , the sensor module 130 , the microphone 140 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc. In some embodiments, the camera 180 can capture images including real objects, and the processor 111 can fuse the images captured by the camera with the virtual objects, and display the fused images through the optical display module 1100 . In some embodiments, the camera 180 can also capture images including human eyes. The processor 111 performs eye tracking through the images.

The USB interface is an interface that conforms to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the VR wearable device 100, and can also be used to transmit data between the VR wearable device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as mobile phones. The USB interface may be USB3.0, which is compatible with high-speed display port (DP) signal transmission, and can transmit video and audio high-speed data.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the wearable device 100 . In other embodiments of the present application, the wearable device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

In addition, the VR wearable device 100 may include a wireless communication function, for example, the VR wearable device 100 may receive images from other electronic devices (such as a VR host) for display. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function can be realized by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). Antennas are used to transmit and receive electromagnetic wave signals. Multiple antennas may be included in the VR wearable device 100, and each antenna may be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module can provide applications on the VR wearable device 100 including second generation (2th generation, 2G) network/third generation (3th generation, 3G) network/fourth generation (4th generation, 4G) network/fifth generation (5th generation, 5G) network and other wireless communication solutions. The mobile communication module may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module can receive electromagnetic waves through the antenna, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave and radiate it through the antenna. In some embodiments, at least part of the functional modules of the mobile communication module may be set in the processor 111 . In some embodiments, at least part of the functional modules of the mobile communication module and at least part of the modules of the processor 111 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speakers, etc.), or displays images or videos through the display screen in the optical display module 1100 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 111, and be set in the same device as the mobile communication module or other functional modules.

The wireless communication module can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the VR wearable device 100. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module may be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves through the antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 111 . The wireless communication module can also receive the signal to be sent from the processor 111 , frequency-modulate it, amplify it, and convert it into electromagnetic wave and radiate it through the antenna.

In some embodiments, the antenna of the VR wearable device 100 is coupled to the mobile communication module, so that the VR wearable device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. GNSS can include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou satellite navigation system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi-zenith) satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The VR wearable device 100 realizes the display function through the GPU, the optical display module 1100 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the optical display module 1100 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 111 may include one or more GPUs that execute program instructions to generate or change display information.

The memory 101 may be used to store computer-executable program code including instructions. The processor 111 executes various functional applications and data processing of the VR wearable device 100 by executing instructions stored in the memory 101 . The memory 101 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The data storage area can store data created during use of the wearable device 100 (such as audio data, phonebook, etc.) and the like. In addition, the memory 101 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash memory (universal flash storage, UFS) and the like.

The VR wearable device 100 can implement audio functions through an audio module, a speaker, a microphone 140, an earphone interface, and an application processor. Such as music playback, recording, etc. The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module may be set in the processor 111 , or some functional modules of the audio module may be set in the processor 111 . Loudspeakers, also called "horns", are used to convert audio electrical signals into sound signals. The wearable device 100 can listen to music through the speaker, or listen to hands-free calls.

The microphone 140, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The VR wearable device 100 may be provided with at least one microphone 140 . In other embodiments, the VR wearable device 100 can be provided with two microphones 140, which can also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the VR wearable device 100 can also be provided with three, four or more microphones 140 to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.

The headphone jack is used to connect wired headphones. The headphone interface can be a USB interface, or a 3.5mm (mm) open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface .

In some embodiments, the VR wearable device 100 may include one or more buttons 150 , and these buttons may control the VR wearable device and provide users with access to functions on the VR wearable device 100 . Keys 150 may be in the form of buttons, switches, dials, and touch or near-touch sensing devices such as touch sensors. Specifically, for example, the user can turn on the optical display module 1100 of the VR wearable device 100 by pressing a button. The keys 150 include a power key, a volume key and the like. The key 150 may be a mechanical key. It can also be a touch button. The wearable device 100 can receive key input and generate key signal input related to user settings and function control of the wearable device 100 .

In some embodiments, the VR wearable device 100 may include an input-output interface 160, and the input-output interface 160 may connect other devices to the VR wearable device 100 through suitable components. Components may include, for example, audio/video jacks, data connectors, and the like.

The optical display module 1100 is used for presenting images to the user under the control of the processor 111 . The optical display module 1100 can convert the real pixel image display into a near-eye projection virtual image display through one or several optical devices such as mirrors, transmission mirrors, or optical waveguides, so as to realize virtual interactive experience, or realize virtual and Interactive experience combined with reality. For example, the optical display module 1100 receives image data information sent by the processor 111 and presents corresponding images to the user.

In some embodiments, the VR wearable device 100 may further include an eye tracking module 1200, which is used to track the movement of human eyes, and then determine the point of gaze of the human eyes. For example, the position of the pupil can be located by image processing technology, the coordinates of the center of the pupil can be obtained, and then the gaze point of the person can be calculated. In some embodiments, the eye tracking system can determine the position of the user's fixation point (or determine the direction of the user's line of sight) through methods such as video eye diagram method, photodiode response method, pupil cornea reflection method, etc., so as to realize the user's eye tracking. motion tracking.

In some embodiments, it is taken as an example to determine a user's line of sight direction by using a pupil-cornea reflection method. As shown in FIG. 2C , the eye tracking system may include one or more near-infrared light-emitting diodes (Light-Emitting Diode, LED) and one or more near-infrared cameras. The NIR LED and NIR camera are not shown in Figure 2B. In a different example, the near-infrared LEDs can be positioned around the eyepiece so as to fully illuminate the human eye. In some embodiments, the near-infrared LED may have a center wavelength of 850 nm or 940 nm. The eye tracking system can obtain the user's line of sight direction through the following method: the human eye is illuminated by a near-infrared LED, and the near-infrared camera captures the image of the eyeball, and then according to the position of the reflective point of the near-infrared LED on the cornea in the eyeball image (i.e. The image of the LED spot on the near-infrared camera in Figure 2C) and the center of the pupil (that is, the image of the center of the pupil on the near-infrared camera in Figure 2C) determine the direction of the optical axis of the eyeball, thereby obtaining the direction of the user's line of sight.

It should be noted that, in some embodiments of the present application, eye-tracking systems corresponding to the two eyes of the user may be set respectively, so as to perform eye-tracking on the two eyes synchronously or asynchronously. In some other embodiments of the present application, an eye-tracking system can also be set only near one human eye, and the line-of-sight direction of the corresponding human eye can be obtained through the eye-tracking system, and according to the relationship between the gaze points of the two eyes (for example, when the user passes through When observing an object with both eyes, the fixation point positions of the two eyes are generally similar or the same), combined with the user's binocular distance, the line of sight direction or fixation point position of the other eye can be determined.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR wearable device 100 . In other embodiments of the present application, the VR wearable device 100 may include more or fewer components than those shown in FIG. 2A , or combine certain components, or split certain components, or arrange different components. limited.

In order to clearly illustrate the technical solution of the present application, a brief description of the human vision generation mechanism will be given below.

Figure 3 is a schematic diagram of the composition of the human eye. As shown in Figure 3, the human eye can include a lens, a ciliary muscle, and a retina located in the fundus. Among them, the lens can function as a zoom lens to converge the light rays entering the human eye, so that the incident light rays can be converged on the retina of the human eye fundus, so that the scene in the actual scene can form a clear image on the retina. The ciliary muscle can be used to adjust the shape of the lens. For example, the ciliary muscle can adjust the diopter of the lens by contracting or relaxing, so as to achieve the effect of adjusting the focal length of the lens. Therefore, objects at different distances in the actual scene can be clearly imaged on the retina through the lens.

In the real world, when a user (not wearing VR glasses) views an object, the perspectives of the left eye and the right eye are different. The user's brain can determine the depth of the object based on the parallax of the same object in the left and right eyes, so the world seen by the human eye is three-dimensional. Generally, the larger the parallax, the smaller the depth, and the smaller the parallax, the larger the depth. Exemplarily, please refer to FIG. 4A , the real world includes an observed object 400 (take a triangle as an example). When human eyes observe the observed object 400 , the left eye captures an image 401 , in which the triangle is located at the position (A1, B1). In the image 402 captured by the right eye, the triangle in the image 402 is located at (A2, B2). The brain can determine the position of the object in the real world through the pixel difference (or parallax) of the same object (such as a triangle) on the image 401 and the image 402 . For example, according to the position (A1, B1) of the triangle in the image 401 and the position (A2, B2) of the triangle in the image 402, the brain determines that the position of the triangle in the real world is (A3, B3, L1), where L1 is the Depth, which is the distance between the triangle and the user's eyes. That is to say, the distance between the triangle seen by the user without VR glasses and the user's eyes is L1. At this time, the distance between the triangle and the user's eyes in the real world and the distance between the triangle perceived by the brain and the user's eyes The distance is equal.

At this time, the user keeps his position still and puts on the VR glasses, and watches the same observed object 400 (ie, a triangle) through the VR glasses. However, the position of the observed object 400 seen by the user wearing the VR glasses is different from the position of the observed object 400 seen when the user does not wear the VR glasses.

Exemplarily, please refer to FIG. 4B and FIG. 2A. In some embodiments, the camera 120 on the VR glasses is located at the lower right of the display screen 110, and the camera 122 is located at the lower left of the display screen 112, so the distance between the two cameras is B 'It is greater than the interocular distance B of the human eye. In simple terms, the camera 122 is further to the left than the person's left eye, and the camera 120 is further to the right than the person's right eye. As shown in FIG. 4B , the triangle on the image 422 collected by the camera 122 is located at (A1, B1,). Since the camera 122 is more to the left than the left eye of the person, the position of the triangle on the image 422 collected by the camera 122 is more to the right than the position of the triangle on the image 401 (see FIG. 4A ) collected by the left eye when not wearing VR glasses, that is, ( A1, B1,) is to the right of (A1, B1). Continuing with FIG. 4B , the triangle on the image 420 collected by the camera 120 is located at the position (A2, B2,). Since the camera 120 is more to the right than the right eye of a person, the position of the triangle on the image 420 collected by the camera 120 is more to the left than the position of the triangle on the image 402 (see FIG. 4A ) collected by the right eye when not wearing VR glasses, that is (A2 , B2,) is more left than (A2, B2). Continue referring to FIG. 4B , assuming that the image 422 is displayed on the display screen 112, and the image 420 is displayed on the display screen 110, the brain sees the position of the triangle based on the image 422 and the image 420 as (A3, B3, L2), that is, the user wearing the VR The distance between the triangle seen behind the glasses and the user's eyes is L2. Since (A1, B1,) is closer to the right than (A1, B1) or (A1, B1) is closer to the center of the image than (A1, B1,), (A2, B2,) is closer to the left than (A2, B2) or ( A2, B2) is closer to the center of the image than (A2, B2,), then, the pixel difference between (A2, B2,) and (A1, B1,) in Figure 4B is greater than that between (A2, B2) and (A1 , B1) pixel difference between. Therefore, the depth L2 of the triangle that the user sees based on the pixel difference between (A1, B1,) and (A2, B2,) is smaller than that seen based on the pixel difference between (A1, B1) and (A2, B2). The depth L1 of the triangle.

That is to say, when the user is at the same location, the object seen when wearing VR glasses is closer to the user than the object seen when not wearing VR glasses. For example, in the real world, the human eye (not wearing VR glasses) sees an object that is 1 meter away from the user, but when the user wears VR glasses, the object seen by the user is 0.7 meters away from the user, which is closer to the user, which is inconsistent with the real situation, and , For objects that are already relatively close to the user in the real world, when wearing a VR wearable device, you will see that these objects are closer to the user, which will make people feel uncomfortable and oppressive. If this happens for a long time, there will be dizziness and a poor experience .

In the above embodiment, one observed object is taken as an example (that is, a triangle), and below, two observed objects are taken as an example, as shown in FIG. 4C , an observed object 400 (triangle) and an observed object 401 (square). To facilitate understanding, take the observed object 402 located at infinite distance as an example, such as the sun. If the VR glasses are not worn, the image 460 should be seen by the left eye, and the image 470 should be seen by the right eye. Since the square is at infinity and close to the left and right eyes, both image 460 and image 470 have the square at the center of the image. In this way, the brain can see the real environment based on the image 460 and the image 470 . When the user wears VR glasses, what the left eye sees is the image 480 collected by the camera 122 , and what the right eye sees is the image 490 collected by the camera 120 . Because the camera 122 is to the left of the human eye, the distance between the triangle and the square on the image 480 observed at the position of the camera 122 is larger than the distance between the triangle and the square on the image 460 observed at the position of the left eye . Similarly, because the camera 120 is to the right of the right eye, the ratio of the distance between the triangle and the square on the image 490 observed at the position of the camera 120 is the distance between the triangle and the square on the image 490 observed at the position of the right eye big. Therefore, in the case of not wearing VR glasses, the triangle seen by the brain based on the image 480 and the image 490 is closer to the user, which is inconsistent with the real situation.

In the above embodiments, the camera 120 is located at the lower right of the display screen 110 and the camera 122 is located at the lower left of the display screen 112 on the VR glasses as an example. It can be understood that, in other embodiments, the camera 120 and the camera 122 can also be located at other positions, for example, the camera 120 is located above the display screen 110, and the camera 122 is located above the display screen 112; If the distance is smaller than the distance between the two display screens, etc., as long as the position of the camera is different from that of the display screen, the distance between the object and the human eye will appear when wearing VR glasses, and the distance between the object and the human eye will be seen when not wearing VR glasses. The distance between the eyes is different.

For the convenience of understanding, an application scenario is taken as an example for description below. The application scenario takes a user wearing VR glasses to play a game at home as an example, as shown in FIG. 5A . In this application scenario, the VR glasses can show the real scene to the user, so when the user wears the VR glasses, he can see the environment at home, such as sofas and tables at home. In some other embodiments, the VR glasses can display real scenes and virtual objects to the user, so when the user wears the VR glasses, what he will see is the environment and virtual objects at home (such as game characters, game interfaces, etc., the virtual Objects are not objects in the real scene), in this way, users can play virtual games in a familiar environment, and the experience is better.

As shown in FIG. 5B , what the user sees when not wearing VR glasses should be the real world 501 shown in (a) in FIG. 5B . When the user wears VR glasses, what the human eyes see is the virtual world 502 shown in (b) in FIG. 5B . It can be seen that all objects in the virtual world 502 are closer to the user, especially objects that are already close to the user in the real world, such as a table. After the user wears VR glasses, the user will see that the table is closer to the user, which is inconsistent with the real situation.

In order to solve this problem, the embodiment of the present application provides a solution, that is, perspective reconstruction. Angle reconstruction can be simply understood as angle adjustment/reconstruction, etc. As mentioned above, because the shooting angle of view of the camera is different from that of the human eye, the scene that the user sees when wearing VR glasses is different from that when not wearing VR glasses. Therefore, in simple terms, perspective reconstruction refers to adjusting the shooting perspective of the camera to the observation perspective of the human eye. However, because the shooting angle of the camera is difficult to adjust, for example, if the camera is fixed at a certain position on the VR glasses, adjusting the shooting angle requires corresponding hardware/mechanical structure, which is not only expensive, but also not conducive to thinning the device. Therefore, in order to avoid increasing hardware costs, the effect of adjusting the shooting angle of view of the camera to that of the human eye can be realized through image post-processing, that is, the image collected by the camera is processed, and when the processed image is displayed on the VR glasses, the user The difference between what you see and what you see when you don't wear VR glasses is reduced. Among them, image processing is called image perspective reconstruction. Simply put, image perspective reconstruction is to adjust the display position of pixels on the image collected by the camera, so that the objects seen by the human eye based on the adjusted image conform to the real situation. Taking Fig. 4B and Fig. 4A as an example, the image perspective reconstruction may include adjusting the position (A1', B1') of the triangle in the image 422 in Fig. 4B to (A1, B1); the position of the triangle in the image 420 in Fig. 4B ( A2', B2') is adjusted to (A2, B2). That is to say, the images before perspective reconstruction are image 422 and image 420 in FIG. 4B , and the images after perspective reconstruction are

images

401 and 402 in FIG. 4A . In this way, the screen of the VR glasses can display the reconstructed image (ie display image 401 and image 402 ), so that the human brain can accurately determine the real position of the object (ie the triangle) based on the image 401 and image 402 .

In an implementation manner, when performing perspective reconstruction on an image, perspective reconstruction may be performed on the entire image (which may be called global perspective reconstruction).

In the following, the scene in FIG. 5A is taken as an example, and the reconstruction of the global viewing angle of an image captured by a camera on the VR glasses is taken as an example. It is assumed that the image collected by the camera is the image of (a) in FIG. 6A . As shown in (b) of FIG. 6A , in some embodiments, the image is divided into four regions, which are region 601 , region 602 , region 603 and region 604 . Assume that the display positions of the

regions

602 and 604 move down after the viewing angle reconstruction, and the display positions of the

regions

601 and 603 move up after the viewing angle reconstruction. The complete image formed by the four regions after perspective reconstruction is shown in (c) in Figure 6A. It can be seen that objects such as walls, sofas, and tables are deformed (or distorted, dislocated, etc.).

It should be noted that Figure 6A is an example of dividing the image into four regions for viewing angle reconstruction. In fact, when reconstructing the global viewing angle, the regions on the image are divided into finer granularity, for example, divided into 9 or 16 regions. Or a larger number of regions; even reconstruct each pixel. It can be understood that the deformation of the object on the image becomes more serious when the perspective is reconstructed for a finer-grained area or for each pixel. For example, as shown in FIG. 6B , the wall surface is distorted (for example, distorted in a wavy line) and the edge of the table is also distorted (for example, distorted in a wavy line) on the image reconstructed from the global perspective. Therefore, the global viewing angle reconstruction solution not only has a huge workload, but also the picture is seriously distorted after the viewing angle reconstruction, which has a great impact on user experience.

In other implementation manners, perspective reconstruction does not need to be performed on the entire image. For example, it is only necessary to reconstruct the viewing angle of the first area on the image (the image captured by the camera), and not perform viewing angle reconstruction on the second area (other areas other than the first area) on the image. Wherein, the first area may be the area where the user's gaze point is located, the user's interest area, the default area, the user-specified area, and the like. For ease of understanding, performing perspective reconstruction on the first region on the image may be referred to as regional perspective reconstruction. Since the viewing angle reconstruction is only performed on the first area and not on the second area, the workload is reduced, and as mentioned above, the viewing angle reconstruction may cause picture distortion. Since the second area does not need to be reconstructed, So the image in the second area will not be distorted. That is to say, the probability or degree of image distortion in regional perspective reconstruction is much lower than that in global perspective reconstruction, which helps to improve the picture distortion in global perspective reconstruction.

Exemplarily, continuing to take the scene in FIG. 5A as an example, assume that the image captured by the VR glasses is the image shown in (a) in FIG. 7 . Assuming that the area where the user's gaze point is located is the area surrounded by the dotted line, then only the area surrounded by the dotted line is reconstructed, and the viewing angle is not reconstructed for other areas, so the display position and/or When the shape changes, the display positions and/or shapes of objects in other areas do not change, as shown in (b) in FIG. 7 . Therefore, the degree of distortion of the object on the image after perspective reconstruction is obviously lower than that of the object on the image after global perspective reconstruction. For example, please compare (b) in Figure 6B and Figure 7, Figure 6B is the reconstructed image from the global perspective, and Figure 7 is the reconstructed image using the technical solution of this application, it can be seen that after reconstruction using the technical solution of this application Other areas of the image (areas other than the area where the gaze point is located) are stable, such as sofas, walls, etc., are not distorted, which significantly reduces the degree and probability of image distortion.

It should be noted that in FIG. 7 , taking the area where the user's gaze is located is an area surrounded by a dotted line as an example, the area surrounded by a dotted line may be the smallest circumscribing rectangle of the table, or be greater than or equal to the smallest circumscribing rectangle of the table. It can be understood that the table may also be the smallest circumscribed square, the smallest circumscribed circle, etc., and the shape is not limited. In some other embodiments, the area where the user's gaze point is located may also be a partial area on the table.

FIG. 7 takes the reconstruction of the regional perspective of an image captured by a camera as an example. It can be understood that when the VR glasses include two cameras, the reconstruction of the regional perspective of the image captured by each camera can be performed separately.

Exemplarily, as shown in FIG. 8 , the camera 122 on the VR glasses captures an image 622 , and the camera 120 captures an image 620 . The VR glasses can reconstruct the area angle of view of the dotted line area on the image 622 to obtain an image 624 . The display position and/or shape of the object in the dotted line area in image 624 and the object in the dotted line area in image 622 are different. For example, the display position of the table in image 624 is to the left of the display position of the table in image 622, and/or the table is deformed to a certain extent. For other areas in the image 622 (areas other than the dotted line area), perspective reconstruction is not performed, so objects (such as tables) in other areas on the image 624 have the same display position and shape as objects in other areas on the image 622 . The VR glasses can also perform regional perspective reconstruction on the dotted line area on the image 620 to obtain an image 626 . Objects within the dotted area in image 626 are displayed in different positions and/or shapes from those within the dotted area in image 620 . For example, the display position of the table in image 626 is to the right of the display position of the table in image 620, and/or the table is deformed to a certain extent. For other areas in the image 620 (areas other than the dotted line area, such as the area where the sofa is located), perspective reconstruction is not performed, so objects in other areas on image 626 have the same display position and shape as objects in other areas on image 620.

The display screen 112 of the VR glasses displays an image 624 , and the display screen 120 displays an image 626 . In this way, after the user wears VR glasses, the left eye sees the image 624, and the right eye sees the image 626. Based on the parallax of the table on the image 624 and the image 626, it is determined that the depth information of the table is accurate, because the display positions of the table on the image 624 and the image 626 After adjustment, the parallax of the table on the two images becomes smaller. Based on the smaller parallax, the determined depth information is larger, so the user will no longer feel that the table is approaching the user, and the scene seen is in line with the real situation. In addition, it should be noted that since the perspective of the area where the dotted line is located has been reconstructed, the table that the user sees after wearing the VR glasses is deformed to a certain extent, but since the perspective of other areas has not been reconstructed, the other Objects within the region are not distorted, and are less distorted/morphological than global view reconstruction. Moreover, since other areas have not been reconstructed for viewing angles, the display positions of objects in other areas seen by the user wearing VR glasses are inaccurate and different from the real world. However, because other areas are not the user's gaze area, the user The degree of attention is low, so when the display position of objects in other areas is inaccurate, it has little impact on user experience, and it saves workload and helps to improve efficiency.

The implementation principle of the above view angle reconstruction will be described below with reference to the accompanying drawings.

First, two coordinate systems, the first coordinate system (X1-O1-Y1) and the second coordinate system (X2-O2-Y2), will be described. Wherein, the first coordinate system (X1-O1-Y1) is a coordinate system established based on the display screen. For example, as shown in FIG. 9 , the first coordinate system takes the center of the display screen 112 as the coordinate origin, and the display direction is the Y-axis direction. It can be understood that the first coordinate system may also be a coordinate system established based on human eyes, for example, the first coordinate system established based on the left eye in FIG. 9 . Considering that it is difficult to establish a coordinate system based on the human eye, it is less difficult to create a coordinate system based on the display screen. Moreover, the position of the display screen is close to the position of the human eye, so the coordinate system based on the display screen is to a certain extent the same as that based on the human eye. The created coordinate system can be considered the same. The second coordinate system (X2-O2-Y2) is established based on the camera 122 . For example, as shown in FIG. 9 , the second coordinate system (X2-O2-Y2) is created based on the camera 122, that is, when the camera 122 shoots an object, it is imaged in the second coordinate system (X2-O2-Y2). Since the image collected by the camera 122 and the image displayed on the display screen 112 are not in the same coordinate system, the viewing angle of the camera is different from the viewing angle of human eyes. Therefore, performing perspective reconstruction on the image captured by the camera 122 can be understood as performing coordinate transformation on the image captured by the camera 122, that is, transforming from the second coordinate system to the first coordinate system.

Transferring from the second coordinate system to the first coordinate system requires the use of an offset, which refers to the offset between the second coordinate system and the first coordinate system, and can also be understood as the center of the camera 122 and the display screen 112 distance between centers. Reconstructing the view angle of the image collected by the camera 122 includes: shifting the pixel points on the image to the target position according to the offset amount. For example, taking the previous Figure 4B as an example, the position of the triangle on the image 422 collected by the camera 122 is (A1', B1'), then (A1', B1')+offset=(A1, B1), that is, The position (A1, B1) of the triangle in FIG. 4A completes the perspective reconstruction of the triangle.

In some embodiments, the offset includes an offset direction and/or an offset distance (the offset distance may also be referred to as a displacement offset).

The offset distance may be the distance from the origin of the second coordinate system to the origin of the first coordinate system. That is to say, the offset distance is related to the distance between the display screen 112 and the camera 122 . For example, when the distance between the display screen 112 and the camera 122 is greater, the distance between the first coordinate system and the second coordinate system is greater, that is, the offset distance is greater. In some embodiments, the offset distance increases as the distance between the camera 122 and the display screen 112 increases, and decreases as the distance between the camera 122 and the display screen 112 decreases. Exemplarily, when the distance between the camera 122 and the display screen 112 is the first distance, the offset distance is the first displacement offset. When the distance between the camera 122 and the display screen 112 is the second distance, the displacement distance is the second displacement offset. If the first distance is greater than or equal to the second distance, the first displacement offset is greater than or equal to the second displacement offset. If the first distance is smaller than the second distance, the first displacement offset is smaller than the second displacement offset. For example, taking the previous FIG. 4B as an example, if the distance between the camera 122 and the display screen 112 increases, the position (A1', B1') of the triangle on the image 422 collected by the camera 122 is the same as that of the triangle in FIG. 4A ( The displacement offset between A1, B1) increases.

The offset direction may be a direction from the origin of the second coordinate system to the origin of the first coordinate system. That is to say, the offset direction is related to the positional relationship between the display screen and the camera. In some embodiments, the offset direction changes as the orientation between the camera and the display screen changes. Exemplarily, when the camera is located in the first direction of the display screen, the offset direction is the first direction. When the camera is located in the second direction of the display screen, the offset direction is the second direction. For example, when the camera 122 is located on the left side of the display screen 112, that is, the second coordinate system is on the left side of the first coordinate system, then the offset direction is leftward offset. For example, taking the previous FIG. 4B as an example, the position (A1', B1') of the triangle on the image 422 captured by the camera 122 is shifted to the left to the position (A1, B1) of the triangle in FIG. 4A. Similarly, if the camera 120 is located on the right side of the display screen 110, then the offset direction is to the right. For example, taking the previous Fig. 4B as an example, the position (A2', B2') of the triangle on the image 420 collected by the camera 120 is shifted to the right to the position (A2, B2) of the triangle in Fig. 4A.

Wherein, the first coordinate system, the second coordinate system, the offset, etc. may be stored in the VR glasses in advance.

In other embodiments, the offset can be changed. In some embodiments, the relative position between the display screen and the camera can be changed. For example, the display screen can be moved on the VR glasses, and/or the camera is on the VR glasses. can be moved. For example, as the position of the display screen on the VR glasses is adjusted, and/or the position of the camera is adjusted or the shooting angle changes, the offset between the first coordinate system corresponding to the display screen and the second coordinate system corresponding to the camera will occur correspondingly Or, with the adjustment of the distance between the two display screens on the VR glasses, and/or the adjustment of the distance between the two cameras, the offset changes accordingly. Exemplarily, the distance between the two display screens and/or the distance between the two cameras can be adjusted along with the distance between the user's left eye pupil and right eye pupil. This solution can be applied to VR glasses with adjustable display screen and/or camera position. This type of VR glasses can be applied to various groups of people. For example, when the VR glasses are used by users with a wider eye distance, the relative distance between the display screen and the camera can be adjusted to be larger; when the VR glasses are used by users with a narrower eye distance, The relative distance between the display screen and the camera can be adjusted to be smaller and so on. Therefore, one VR glasses can be suitable for multiple users, for example, one VR glasses can be used by the whole family. No matter how the position of the display screen and/or camera is adjusted, the offset is adjusted accordingly, and the VR glasses can realize perspective reconstruction based on the adjusted offset.

In some embodiments, the VR glasses can shift all pixels on the image captured by the camera to the target position according to the offset amount (that is, reconstruct the global perspective).

In some other embodiments, the VR glasses may first determine the first area on the image, and shift the pixels in the first area to the target position according to the offset. That is, only the pixels in the first area are offset, and the pixels in other areas may not be moved.

Exemplarily, the first area may be the area where the user's gaze point is located on the image. In some embodiments, the VR glasses include an eye tracking module, through which the user's gaze point can be located. One implementation method is that the VR glasses determine that the user's gaze point is located at a point on an object (such as the table in FIG. 7 ), and then determine the smallest circumscribed rectangle of the object (such as the table) as the first area. It can be understood that the smallest circumscribing rectangle may also be the smallest circumscribing square, the smallest circumscribing circle, and so on. In some other embodiments, when the VR glasses determine that the user's gaze point is located at a point on a certain object (such as a table), it may determine that a rectangle with the point as the center and a preset length as the side length is the first area, or , with the point as the center and a circle with a preset radius as the first area, and so on. Wherein, the preset length, preset radius, etc. may be set by default. In this case, the area where the user's gaze point is located may be a partial area of the object. In still some embodiments, the first area may also be all areas whose depth is located at the depth of the user's gaze point.

Alternatively, the first area may also be the user's interest area on the image. Wherein, the region of interest of the user may be the region where the object of interest of the user is located on the image. Exemplarily, an object of interest to the user (eg, a person, an animal, etc.) may be stored in the VR glasses, and when it is recognized that the object of interest to the user exists on the image captured by the camera, the area where the object is located is determined to be the first area. Among them, the object of interest to the user can be manually stored in the VR glasses by the user, or, in the virtual world, the user can interact with objects in the virtual world, and the object of interest to the user can also be that the number of interactions performed by the user recorded by the VR glasses is greater than Objects that perform interactions a preset number of times and/or take longer than a preset duration, etc.

Alternatively, the first area may also be a default area, such as a central area on the image. Considering that the user generally pays attention to the central area of the image first, the first area is defaulted to the central area.

Alternatively, the first area may also be a user-specified area. For example, the user may set the first area on the VR glasses or set the first area on an electronic device (such as a mobile phone) connected to the VR glasses, and so on.

In some other embodiments, the first area may also be determined according to different scenarios. Taking the VR game scene as an example, if user A participates in the game as a game player, the area where the game character corresponding to user A is located is the first area; or, if user A watches the game played by user B, then the The area where the corresponding game character (that is, the player being watched) is located is the first area. For another example, take the VR driving scene as an example. User A wears VR glasses and sees that he is driving a virtual vehicle on the road. The first area can be the area where the vehicle driven by user A is located, or the steering wheel, The area where the windshield or the like is located, or the area where the vehicle is located in front of the vehicle driven by the user A on the driving road is the first area.

In short, the first area is an area on the image captured by the camera, and specific methods for determining the first area include but are not limited to the above methods, which are not listed in this application.

In some embodiments, the offsets of all pixels in the first area may be the same. For example, the offset distance of all pixels is the distance between the origin of the first coordinate system and the origin of the second coordinate system mentioned above, and the cheap direction of all pixels is from the origin of the second coordinate system to the first coordinate The direction of the origin.

In some other embodiments, the offsets of different pixel points in the first area may be different. For example, as shown in (a) of FIG. 10 , the first area 1000 includes a central area 1010 and an edge area 1020 (area drawn with oblique lines). Wherein, the area of the edge area 1020 may be default, such as the area formed from the edge of the first area to the preset width in the first area. The offset of the pixels in the central area 1010 is greater than the offset of the pixels in the edge area 1020 . For example, taking the above-mentioned distance between the origin of the first coordinate system and the origin of the second coordinate system as an example, the offset distance of the pixels in the central area 1010 is equal to L, and the offset distance of the pixels in the edge area 1020 is equal to L. The moving distance is less than L, such as L/2, L/3 and so on. In this case, the displacement of the pixels at the center of the first area is relatively large, and the displacement of pixels at the edge is small, because the edge is connected to other areas. The connection between them can be relatively smooth, avoiding obvious misalignment at the edge of the area where the fixation point is located.

It can be understood that when the central area offset is large and the edge area offset is small, the degree of deformation (that is, the shape change) of the object in the central area is large, and the degree of sound deformation of the object in the edge area is small. That is to say, the degree of deformation of objects in the first region decreases gradually from the center to the edge.

In some other cases, the offset of the pixels in the first area is greater than the offset of the pixels in the second area, and the second area may be an area outside the first area and surround the outer edge of the first area. The area of the second area is not limited, for example, it may be an area formed by a preset width outward from the outer edge of the first area. Correspondingly, since the offset in the first area is large and the offset in the second area is small, the degree of deformation of the object in the first area is large, and the degree of deformation of the object in the second area is small. That is to say, the degree of deformation of the object from the first region to the second region gradually decreases.

In other cases, the offsets of different pixels on the edge area 1020 may also be different. For example, as shown in (b) in Figure 10, the edge area 1020 includes a first edge area 1022 (hatched part) and a second edge area 1024 (black part), assuming that the offset direction is the direction shown by the arrow in the figure, that is, the first If an area 1000 is shifted to the lower left, then the first edge area 1022 is in the offset direction (ie, the lower left of the first area 1000), and the second edge area is in the direction opposite to the offset direction (ie, the first area 1000 upper right of the ). The offsets of pixels in the two edge regions are different. Continuing with (b) in Figure 10, assuming that the offset direction is the direction shown by the arrow, then the offset of the pixels in the first edge area 1022 (black area)<the offset of the pixels in the central area 1010 amount<the offset of the pixels in the second edge area 1024 (hatched area). That is to say, the objects in the range of offset directions in the first area (ie, the objects in the first edge area 1022) have a large amount of offset, and the objects in the range of directions opposite to the offset direction (ie, the objects in the second edge area 1022) have a large amount of offset. 1024) The offset is small. In this way, when the first region is shifted according to the shifting direction, the edge of the first region opposite to the shifting direction can transition smoothly with other regions.

In some embodiments, the first image information of the first pixel in the edge area of the first area on the image after perspective reconstruction may be an intermediate value between the second image information and the third image information, such as an average value. Wherein, the second image information is the image information of the second pixel in the central area of the first area, and the third image information is the image information of the third pixel in other areas. For example, as shown in FIG. 11 , pixel A is located in the edge area 1020 of the first area 1000 , pixel B is located in other areas, and pixel C is located in the central area 1010 of the first area 1000 . The image information of pixel point A may be the average value of the image information of pixel point B and pixel point C, and the image information includes one or more of resolution, color, color temperature, or brightness. Wherein, the pixel point C and the pixel point B may be pixel points close to the pixel point A. Since the edge area 1020 of the first area is a transition area between the first area and other areas, when the resolution, color, color temperature, brightness, etc. of pixels in the edge area 1020 are intermediate values, the Smooth transitions between other areas.

It can be understood that the above is an example of viewing angle reconstruction of the image captured by the camera 122 in FIG. Do not repeat the details.

In some other embodiments, the present application provides a display method. The method is applicable to an electronic device including at least one camera and at least one display screen, such as VR glasses, where the positions of the camera and the display screen are different. Exemplarily, as shown in FIG. 4C , the position of the camera on the VR glasses is different from the position of the display screen, so when the user wears the VR glasses, there will be a phenomenon that the viewing angle of the human eye is different from the shooting angle of the camera. Exemplarily, FIG. 12 is a schematic flowchart of a display method provided by an embodiment of the present application. The flow of the method includes:

S1, the camera collects a second image.

Wherein, the camera may be any camera on the VR glasses shown in FIG. 4C , such as the left camera 122 or the right camera 120 . Taking the left camera 122 as an example, as shown in FIG. 4C , within the viewing angle of view of the position of the left camera, the triangle is located in front of the left of the square (because the square is at infinity, such as the sun). Since the image taken by the left camera 122 is a plane two-dimensional image, the imaging surface of the left camera 122 includes a triangle and a square, and the triangle is on the left side of the square, as shown in FIG. 13 , which is a schematic diagram of a plane two-dimensional image taken by the camera 122. . In this image the triangle is to the left of the square. It can be understood that an other camera is set at the location of the camera 122 , and the image collected by the other camera is the same as the image collected by the camera 122 . That is to say, the image observed at the location of the camera 122 (observed by a person or captured by other cameras) is the same as the image collected by the camera 122 .

S2. Determine the first area on the second image.

There are multiple ways to determine the first area, please refer to the previous section, and details will not be repeated here. Exemplarily, as shown in FIG. 13 , the first area is the dotted line area on the plane two-dimensional image collected by the camera.

S3. Reconfigure the viewing angle of the first area.

As mentioned above, reconstructing the viewing angle of the first area includes performing coordinate transformation on the image in the first area, that is, transforming from the coordinate system corresponding to the camera to the coordinate system corresponding to the display screen or human eyes. Because the image collected by the camera is a plane two-dimensional image, one implementation of coordinate transformation can be to convert the plane two-dimensional image collected by the camera into a three-dimensional point cloud, and the three-dimensional point cloud can reflect the position of each object in the real environment (including Depth), and then create a virtual camera by simulating the human eye. By shooting a 3D point cloud through the virtual camera, the image seen at the observation angle of the human eye can be obtained, and the observation angle from the position of the camera to the position of the human eye can be realized. Reconstruction of the perspective of observation. Specifically, in step S3, the way to reconstruct the viewing angle of the first region includes the following steps:

The first step is to determine the depth information of the pixels in the first area. Wherein, the manner of determining the depth information of the pixels in the first area includes at least one of manner 1 and manner 2.

Method 1: Determine the depth information of a pixel according to the pixel difference of the same pixel on two images captured by two cameras on the VR glasses. Exemplarily, the depth information of the pixel satisfies the following formula:

Among them, f is the focal length of the camera, B is the distance between the two cameras, disparity is the pixel difference between the same pixel on the two images; d is the depth information of the pixel.

Method 2: Determining the depth information of the pixel point according to the convergence angle between the user's left eye and the right eye, and the corresponding relationship between the convergence angle and the depth information.

Please refer to (a) in FIG. 14 , in a real environment, the angle formed by the sight line of the left eye and the line of sight of the right eye when the two eyes observe an object is called the convergence angle θ. It can be understood that the closer the observed object is to the human eye, the larger the convergence angle θ and the smaller the convergence depth. Correspondingly, the farther the observed object is from the human eye, the smaller the convergence angle θ and the greater the convergence depth. As shown in (b) in Figure 14, when the user wears VR glasses, in the virtual environment presented by the VR glasses, all the scenes seen by the user are displayed on the display screen of the VR glasses. There is no depth difference in the light emitted by the screen, so after adjusting the focal length, the focus of the eyes is fixed on the screen, that is, the angle of convergence θ becomes the angle at which the line of sight of the eyes points to the display screen. However, the depth of objects in the virtual environment that the user actually sees is not the same as the distance from the display screen to the user. Therefore, in the embodiment of the present application, after the user wears VR glasses, the convergence angle θ of the user's eyes is determined. The VR glasses can store a database, which stores the correspondence between the convergence angle θ and the depth information. When the VR glasses determine the convergence angle θ, the corresponding depth information is determined based on the correspondence. Wherein, the database may be obtained based on experience and stored in the VR glasses in advance; or it may be determined based on deep learning.

The second step is to determine the 3D point cloud data corresponding to the first area according to the depth information of the pixels in the first area.

Exemplarily, taking the planar two-dimensional image captured by the camera 122 in FIG. The 3D point cloud of pixels is shown in Figure 15. The three-dimensional point cloud corresponding to the first area can map the position of each pixel in the first area in the real world. For example, in FIG. 15 , the point cloud corresponding to the triangle in the 3D point cloud is in front left of the point cloud corresponding to the square, because the scene observed at the position of the camera 122 is that the triangle is in front left of the square.

The third step is to create a virtual camera.

It can be understood that the image acquisition principle of the human eye is similar to the image capture principle of the camera. In order to simulate the image acquisition process of the human eye, a virtual camera is created. The virtual camera simulates the human eye. For example, the position of the virtual camera is the same as The positions of the human eyes are the same, and/or, the field of view of the virtual camera is the same as that of the human eyes.

For example, generally speaking, the angle of view of the human eye is 110 degrees up and down, and 110 degrees left and right, so the field of view of the virtual camera is 110 degrees up and down, 110 degrees left and right. For another example, the VR glasses can determine the position of the human eye, so the virtual camera is set at the position of the human eye. Among them, there are multiple ways to determine the position of the human eye. For example, in method 1, the position of the display screen is determined first, and then the position of the human eyes can be estimated by adding the distance A to the position of the display screen. The human eye position determined in this way is more accurate. Wherein, the distance A is the distance between the display screen and human eyes, which may be stored in advance. Method 2. The position of the human eye is equal to the position of the display screen. This method is relatively simple, and setting the virtual camera at the display screen can also alleviate the discomfort caused by the difference between the shooting angle of view and the angle of view of the human eye. Exemplarily, as shown in FIG. 16 , the virtual camera is at the position of the human eye.

The fourth step is to use the virtual camera to capture the 3D point cloud data corresponding to the first area to obtain an image, which is an image reconstructed from the angle of view of the first area.

Exemplarily, as shown in FIG. 17, the virtual camera corresponding to the left eye captures a three-dimensional point cloud (a three-dimensional point cloud converted from a two-dimensional image collected by the left camera 122), since the virtual camera corresponding to the left eye is closer to the right than the left camera 122 , so the distance between the triangle and the square on the image captured by the virtual camera corresponding to the left eye is small. For convenience of comparison, as shown in FIG. 18 , image 1701 is a plane two-dimensional image taken by camera 122, and image 1702 is an image taken by a virtual camera corresponding to the left eye (that is, the first area on the plane two-dimensional image taken by camera 122 is regarded as the perspective weight). constructed image). The distance between two objects on image 1702 is smaller than the distance between two objects on image 1701 . Image 1702 corresponds to an image captured by a person's left eye.

The above is an introduction using the camera 122 as an example, and the same principle applies to the camera 120, that is, converting the plane two-dimensional image collected by the camera 120 into a three-dimensional point cloud, and then creating a virtual camera corresponding to the right eye, and using the virtual camera to capture the three-dimensional point cloud. Exemplarily, as shown in FIG. 17 , image 1703 is a planar two-dimensional image captured by camera 120 , and image 1704 is an image captured by a virtual camera corresponding to the right eye (that is, the image after perspective reconstruction of the first region). The distance between two objects on image 1704 is smaller than the distance between two objects on image 1703 . This is because the virtual camera corresponding to the right eye is more left than the camera 120 . Therefore, image 1704 is equivalent to an image captured by a person's right eye.

In this application, only the first area is mapped to the 3D point cloud, and the second area is not mapped to the 3D point cloud, so the image captured by the virtual camera only includes the first area, not the second area, and the workload is small .

S4. Combining the image blocks in the second area on the second image with the image blocks in the first area after perspective reconstruction into the first image. Wherein, the second area is an area other than the first area on the second image.

The second area does not perform perspective reconstruction, and the image captured by the virtual camera is an image after perspective reconstruction of the first area. Therefore, compared with the second image, the first image reconstructs the perspective of the first area, and The perspective of the second area is not reconstructed.

S5. Display the first image.

In some embodiments, the above S2 to S4 can be executed by the processor in the VR glasses, that is, after the camera captures the second image (ie S1), the second image is sent to the processor, and the processor executes S2 to S4 to obtain For the first image, the processor displays the first image through the display screen.

Based on the same idea, FIG. 19 shows an electronic device 1900 provided by this application. The electronic device 1900 may be the aforementioned VR wearable device (eg, VR glasses). As shown in Figure 19, an electronic device 1900 may include: one or more processors 1901; one or more memories 1902; a communication interface 1903, and one or more computer programs 1904, and each of the above devices may communicate through one or more bus 1905 connection. Wherein the one or more computer programs 1904 are stored in the memory 1902 and are configured to be executed by the one or more processors 1901, the one or more computer programs 1904 include instructions, and the instructions can be used to perform the above-mentioned Related steps of the VR wearable device in the corresponding embodiment. The communication interface 1903 is used to implement communication with other devices, for example, the communication interface may be a transceiver.

In the above-mentioned embodiments provided in the present application, the methods provided in the embodiments of the present application are introduced from the perspective of an electronic device (for example, a VR wearable device) as an execution subject. In order to realize the various functions in the method provided by the above embodiments of the present application, the electronic device may include a hardware structure and/or a software module, and realize the above-mentioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

As used in the above embodiments, depending on the context, the terms "when" or "after" may be interpreted to mean "if" or "after" or "in response to determining..." or "in response to detecting ...". Similarly, depending on the context, the phrases "in determining" or "if detected (a stated condition or event)" may be interpreted to mean "if determining..." or "in response to determining..." or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)". In addition, in the above embodiments, relational terms such as first and second are used to distinguish one entity from another, without limiting any actual relationship and order between these entities.

Reference to "one embodiment" or "some embodiments" or the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in this embodiment will be generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)). In the case of no conflict, the solutions of the above embodiments can be used in combination.

It should be pointed out that a part of the patent application documents contains content protected by copyright. Copyright is reserved by the copyright owner other than to make copies of the contents of the patent file or records of the Patent Office.

Claims

A display method, characterized in that it is applied to a wearable device, and the wearable device includes at least one display screen and at least one camera; including:

presenting a first image to a user through the display screen;

The first object on the first image is different from the first object on the second image in at least one of display position or form, and the second object on the first image is different from the first object on the second image The display position and shape of the second object are the same; the second image is an image collected by the camera;

Wherein, the first object is located in an area where the user's gaze point is located, and the second object is located in an area other than the area where the user's gaze point is located.
The method according to claim 1, characterized in that,

The displacement offset between the first display position of the first object on the first image and the second display position of the first object on the second image and the distance between the camera and the display screen Distance is relevant.
The method according to claim 1 or 2, characterized in that,

The offset direction between the first display position of the first object on the first image and the second display position of the first object on the second image and the position between the camera and the display screen Relationship related.
The method according to any one of claims 1-3, characterized in that,

The position offset of the first object on the first image relative to the second object on the second image is a first offset;

The position offset of the third object on the first image relative to the third object on the second image is a second offset;

The third object is located in the area where the user's gaze point is located, and is closer to the edge of the area where the gaze point is located than the first object;

The second offset is smaller than the first offset.
The method according to any one of claims 1-4, characterized in that,

The degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the third object on the second image. The degree of shape change of the object is large;

The third object is located in the region where the gaze point of the user is located, and the third object is closer to an edge of the region where the gaze point is located than the first object.
The method according to any one of claims 1-5, characterized in that,

The position offset of the first object on the first image relative to the first object on the second image is a first offset;

The position offset of the third object on the first image relative to the third object on the second image is a second offset;

The third object is in the area where the user's gaze point is located, and the third object is in a first direction range of the first object, and the first direction range includes the first direction range on the first image. an object relative to the position of the first object on the second image in an offset direction;

The second offset is greater than the first offset.
The method according to any one of claims 1-6, characterized in that,

The first image includes a first pixel point, a second pixel point and a third pixel point, the first pixel point and the second pixel point are located in the area where the user's gaze point is located, and the first pixel point The pixel point is closer to the edge of the area where the user's gaze point is located than the second pixel point; the third image information is related to the area outside the area where the user's gaze point is located;

The image information of the first pixel is located between the image information of the second pixel and the image information of the third pixel.
The method according to claim 7, wherein the image information includes: at least one of resolution, color, brightness, and color temperature.
The method according to any one of claims 1-8, wherein the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; The first display screen is configured to display the image captured by the first camera; the second display screen is configured to display the image of the second camera;

In the case where the positions of the first display screen and the first camera are different, the display of the first object on the image displayed on the first display screen and the first object on the image captured by the first camera At least one of position or form is different, and the display position and form of the second object on the image displayed on the first display screen and the second object on the second image captured by the first camera are the same;

In the case where the positions of the second display screen and the second camera are different, the display of the first object on the image displayed on the second display screen and the first object on the image captured by the second camera At least one of position or form is different, and the display position and form of the second object on the image displayed on the second display screen and the second object on the image captured by the second camera are the same.
The method according to any one of claims 1-9, wherein the shape of the first object on the first image is different from that of the first object on the second image, including:

The edge contour of the first object on the second image is flatter than the edge contour of the first object on the first image.
A display method, characterized in that it is applied to a wearable device, and the wearable device includes at least one display screen, at least one camera and a processor; the camera is configured to transmit the image it collects to the processor, The image is displayed on the display screen via the processor, including:

presenting a first image to a user through the display screen;

The first object on the first image is different from the first object on the second image in at least one of display position or form, and the second object on the first image is different from the first object on the second image The display position and form of the second object are the same; the second image is an image collected by the camera;

Wherein, the first object is located in an area where the user's gaze point is located, and the second object is located in an area other than the area where the user's gaze point is located.
The method according to claim 11, characterized in that,

The position offset of the first object on the first image relative to the second object on the second image is a first offset;

The position offset of the third object on the first image relative to the third object on the second image is a second offset;

The third object is located in the area where the user's gaze point is located, and is closer to the edge of the area where the gaze point is located than the first object;

The second offset is smaller than the first offset.
The method according to any one of claims 11-12, characterized in that,

The degree of morphological change of the first object on the first image relative to the first object on the second image is greater than that of the third object on the first image relative to the third object on the second image. The degree of shape change of the object is large;

The third object is located in the region where the gaze point of the user is located, and the third object is closer to an edge of the region where the gaze point is located than the first object.
The method according to any one of claims 11-13, characterized in that,

The position offset of the first object on the first image relative to the first object on the second image is a first offset;

The position offset of the third object on the first image relative to the third object on the second image is a second offset;

The third object is in the area where the user's gaze point is located, and the third object is in a first direction range of the first object, and the first direction range includes the first direction range on the first image. an object relative to the position of the first object on the second image in an offset direction;

The second offset is greater than the first offset.
The method according to any one of claims 11-14, characterized in that,

The first image includes a first pixel point, a second pixel point and a third pixel point, the first pixel point and the second pixel point are located in the area where the user's gaze point is located, and the first pixel point The pixel point is closer to the edge of the area where the user's gaze point is located than the second pixel point; the third image information is related to the area outside the area where the user's gaze point is located;

The image information of the first pixel is located between the image information of the second pixel and the image information of the third pixel.
The method according to claim 15, wherein the image information includes: at least one of resolution, color, brightness, and color temperature.
The method according to any one of claims 11-16, wherein the at least one camera includes a first camera and a second camera, and the at least one display screen includes a first display screen and a second display screen; the The first display screen is configured to display the image captured by the first camera; the second display screen is configured to display the image of the second camera;

In the case where the positions of the first display screen and the first camera are different, the display of the first object on the image displayed on the first display screen and the first object on the image captured by the first camera At least one of position or form is different, and the display position and form of the second object on the image displayed on the first display screen and the second object on the second image captured by the first camera are the same;

In the case where the positions of the second display screen and the second camera are different, the display of the first object on the image displayed on the second display screen and the first object on the image captured by the second camera At least one of position or form is different, and the display position and form of the second object on the image displayed on the second display screen and the second object on the image captured by the second camera are the same.
An electronic device, characterized in that it comprises:

processor, memory, and, one or more programs;

Wherein, the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the processor, cause the electronic device to perform the functions described in claim 1- 17. The method steps of any one of.
A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, and when the computer program runs on a computer, the computer executes any one of claims 1 to 17. method described in the item.
A computer program product, characterized in that it includes a computer program, and when the computer program is run on a computer, the computer is made to execute the method according to any one of claims 1-17.