WO2018223663A1

WO2018223663A1 - Vr image processing method, device, and apparatus

Info

Publication number: WO2018223663A1
Application number: PCT/CN2017/117554
Authority: WO
Inventors: 邱涛
Original assignee: 歌尔科技有限公司
Priority date: 2017-06-06
Filing date: 2017-12-20
Publication date: 2018-12-13
Also published as: CN107203270A

Abstract

The present invention provides a VR image processing method, device, and apparatus. The method comprises the following steps: identifying, based on eye tracking, the position of a point of interest of a user in a VR display space; dividing, based on the position of the point of interest, the VR display space into at least two image display regions; and dynamically adjusting the resolution of at least one image corresponding to the image display region of the at least two image display regions, such that the resolution of the image of the image display region in an area specified by the position of the point of the interest is greater than the resolution of an image of another image display region, the other image display region being that part of the image display region other than the image display region in the area specified by the position of the point of interest, in the at least two image display regions. The method provided by the present invention can reduce VR apparatus power consumption.

Description

VR image processing method, device and device

cross reference

The present invention is incorporated by reference in its entirety in its entirety in its entirety in its entirety in its entirety in the the the the the the the the the

Technical field

The present invention relates to the field of virtual reality technologies, and in particular, to a VR image processing method, apparatus, and device.

Background technique

Virtual Reality (VR) technology is a computer simulation system that can create and experience virtual worlds. It uses computer to generate a simulation environment. It is a multi-source information fusion, interactive 3D dynamic vision and entity. System simulation of behaviors that immerse users in the environment. At present, VR devices such as VR glasses and VR all-in-ones have brought about a three-dimensional visual experience.

In the process of displaying VR images through these VR devices, developers often need to set a higher resolution for VR images in order to pursue three-dimensional dynamic view, or the definition of VR images. This method makes the power consumption of the VR device too large, which easily causes serious problems such as short standby time, severe heat generation, frequency loss, and nuclear loss of the VR device.

Summary of the invention

Aspects of the present invention provide a VR image processing method, apparatus, and device for reducing power consumption of a VR device.

The present invention provides a VR image processing method, including:

Identifying the location of the user's point of interest in the VR display space based on eye tracking;

Dividing the VR display space into at least two image display areas based on the location of interest;

Dynamically adjusting a resolution of the at least one image display area corresponding image in the at least two image display areas such that an image of the image display area within the specified range of the attention point position has a resolution greater than that of the other image display areas Resolution

The other image display area refers to an image display area other than the image display area within the specified range of the attention point position in the at least two image display areas.

Optionally, dynamically adjusting a resolution of the at least one image display area corresponding image in the at least two image display areas, so that an image of the image display area within the specified range of the attention point position has a resolution greater than other image displays. The resolution of the image of the area, including:

If the default resolution of the image that the VR display space needs to display is less than the first resolution threshold, increase the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position;

If the default resolution of the image that the VR display space needs to display is greater than the second resolution threshold, then the reduction is The resolution of the image currently displayed or to be displayed by the other image display area; wherein the second resolution threshold is greater than the first resolution threshold.

Optionally, the increasing the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position comprises:

And gradually increasing the image currently displayed or to be displayed in the image display area within the specified range of the attention point position from the direction of the edge of the image display area within the specified range of the attention point position Resolution.

Optionally, the image display area within the specified range of the focus point location includes: a first three-dimensional area and a second three-dimensional area centered on the focus point position, the second three-dimensional area including the first a three-dimensional area;

And displaying, from the edge of the image display area within the specified range of the attention point position, the direction of the focus point position, and gradually increasing or temporarily displaying the image display area within the specified range of the attention point position The resolution of the image, including:

Replacing the resource file used by the tag object within the first three-dimensional area with a resource file corresponding to the first resolution;

Replacing the resource file used by the label object between the second three-dimensional area and the first three-dimensional area with a resource file corresponding to the second resolution; wherein the first resolution is greater than the second resolution The second resolution is greater than the first resolution threshold.

Optionally, the image display area within the specified range of the focus point location further includes: a third three-dimensional area centered on the focus point position, the third three-dimensional area including the second three-dimensional area;

And displaying, from the edge of the image display area within the specified range of the attention point position, the direction of the focus point position, and gradually increasing or temporarily displaying the image display area within the specified range of the attention point position The resolution of the image also includes:

Replacing the resource file used by the label object between the third three-dimensional area and the second three-dimensional area with a resource file corresponding to the third resolution; wherein the second resolution is greater than the third resolution The third resolution is greater than the first resolution threshold.

Optionally, the first three-dimensional area is a sphere area having a center of the focus point and a radius of the first distance;

The second three-dimensional region is a sphere region having the focus point as a center of the sphere and the second distance as a radius;

The third three-dimensional region is a sphere region having a center of the focus point and a radius of the third distance; wherein the first distance is smaller than the second distance, and the second distance is smaller than the first Three distances.

Optionally, before the eyeball tracking is performed, identifying a location of a user's point of interest in the VR display space, The method also includes:

Adding a label to at least one image object in the VR display space to obtain at least one label object;

Generating resource files corresponding to the first resolution, the second resolution, and the third resolution, respectively, for the at least one tag object.

Optionally, the method further includes:

A resource file corresponding to the first resolution, the second resolution, and the third resolution is rendered to output a current display or an image to be displayed after dynamically adjusting the resolution.

Optionally, after the dynamically adjusting the resolution of the at least one image display area corresponding image in the at least two image display areas, the method further includes:

Counting the number of patches included in the resource file corresponding to the image of the image display area within the specified range of the attention point position after dynamically adjusting the resolution;

The operating frequency of the CPU and/or GPU is reduced according to the number of patches and the preset frame rate.

The present invention also provides a VR image processing apparatus, comprising:

An identification module for identifying a location of a user's point of interest in the VR display space based on eye tracking;

a dividing module, configured to divide the VR display space into at least two image display areas based on the location of the attention point;

An adjustment module, configured to dynamically adjust a resolution of an image corresponding to at least one image display area of the at least two image display areas, so that an image of an image display area within a specified range of the attention point position has a resolution greater than other images The resolution of the image of the display area;

Optionally, the adjusting module includes:

And an increasing unit, if the default resolution of the image that needs to be displayed in the VR display space is smaller than the first resolution threshold, increasing an image currently displayed or to be displayed in the image display area within the specified range of the attention point position Resolution

a reducing unit, configured to reduce a resolution of an image currently displayed or to be displayed in the other image display area if a default resolution of the image to be displayed by the VR display space is greater than a second resolution threshold; The second resolution threshold is greater than the first resolution threshold.

Optionally, the increasing unit is specifically configured to:

The invention also provides an electronic device comprising a processor and a memory connected to the processor;

The memory is configured to store one or more computer instructions;

The processor is configured to execute the one or more computer instructions to implement the steps in the foregoing method embodiments.

The present invention also provides a computer readable storage medium storing a computer program, the computer program being executed to implement the steps of the above method embodiments.

In the present invention, based on the eyeball tracking, the location of the user's attention point in the VR display space is identified, and the VR display space is divided into at least two image display areas based on the position of the attention point, by dynamically adjusting at least two image display areas. At least one image display area corresponds to a resolution of the image such that the resolution of the image of the image display area within the specified range of the attention point position is greater than the resolution of the image of the other image display area, thereby ensuring the vicinity of the position of the attention point The image has a higher resolution, ensuring that the user can clearly see the image. At the same time, the image of other image display areas only needs to have a lower resolution, which can reduce the data amount of the VR image and reduce the power consumption of the VR device.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the invention, and are intended to be a part of the invention. In the drawing:

1 is a schematic flowchart of a VR image processing method according to an embodiment of the present invention;

2a is a schematic cross-sectional view of a first three-dimensional region and a second three-dimensional region according to an embodiment of the present invention;

2b is a schematic cross-sectional view of a first three-dimensional region, a second three-dimensional region, and a third three-dimensional region according to an embodiment of the present invention;

3 is a block diagram of a VR image processing apparatus according to an embodiment of the present invention;

4 is a block diagram of a VR image processing apparatus according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of an internal configuration structure of a head mounted display device according to an embodiment of the present invention.

detailed description

The technical solutions of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the accompanying drawings. It is apparent that the described embodiments are only a part of the embodiments of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart diagram of a VR image processing method according to an embodiment of the present invention. As shown in Figure 1, the method Includes the following steps:

S101: Identify the location of the user's point of interest in the VR display space based on eye tracking.

S102: Divide the VR display space into at least two image display areas based on the position of the attention point.

S103: Dynamically adjusting a resolution of an image corresponding to at least one image display area in at least two image display areas, so that an image of an image display area within a specified range of the attention point position has a resolution greater than that of another image display area. Wherein, the other image display area refers to an image display area other than the image display area within the specified range of the attention point position in at least two image display areas.

During the operation of the VR device, a VR display space is presented to the user to display images in the VR display space. The image can be displayed in the VR display space or not.

When the user's eyes see the VR display space, they will focus on a certain point, that is, the position of the point of interest. In some embodiments, the point of interest location may be a point of interest location on the image that the user is currently presenting in the VR presentation space.

Among them, the location of the focus can be identified by eye tracking technology. The eyeball tracking technology can extract eye features according to changes in the characteristics of the eyeball and the periphery of the eyeball, changes in the iris angle, or actively projecting infrared rays to the iris, and then determine the user's attention in the VR display space according to the eye features and the VR display space. Point location.

Since the human eye can focus on a limited range of areas of interest, the VR display space can be divided into at least two image display areas based on the position of the attention point.

In some embodiments, the VR display space may be divided into at least two image display regions centered on the location of interest, ie, each of the at least two image display regions is centered on the location of the focus.

Next, an image display area and other image display areas within a specified range of the focus point position are determined. Among them, the specified range can be determined according to the range that the human eye can pay attention to. In an example, the specified range may be set to a range of 0.5 m in the circle of the position of the attention point, and the image display area falling within the range of 0.5 m in the circle of the position of the attention point may be used as the image display area within the specified range of the position of the attention point. .

The other display area refers to an image display area other than the image display area within the specified range of the attention point position in at least two image display areas.

Generally speaking, the image display area within the specified range of the position of the focus point is the area where the eyes of the person are mainly concerned, and the other display areas are relatively far from the position of the point of interest, which is the area of secondary concern. Based on this, the resolution of the image corresponding to the at least one image display area in the at least two image display areas may be dynamically adjusted, so that the resolution of the image of the image display area within the specified range of the attention point position is greater than the image of the other image display area. Resolution.

In some embodiments, dynamically adjusting the resolution of the image corresponding to the at least one image display area comprises increasing or decreasing the resolution of the image corresponding to the at least one image display area.

Of course, the resolution of the image in some image display areas does not have to be adjusted, and the focus position can be specified. The resolution of the image in the image display area inside is larger than the resolution of the image in the other image display area. Specifically, the image display area within the specified range may be the adjusted area or may not be the adjusted area; accordingly, the other display areas may also be the adjusted area or may not be the adjusted area.

In this embodiment, based on the eyeball tracking, the location of the user's attention point in the VR display space is identified, and the VR display space is divided into at least two image display areas based on the position of the attention point, and the at least two image display areas are dynamically adjusted. At least one image display area corresponds to a resolution of the image such that the resolution of the image of the image display area within the specified range of the position of the attention point is greater than the resolution of the image of the other image display area, thereby ensuring an image near the position of the attention point The high resolution ensures that the user can clearly see the image. At the same time, the images in other image display areas only need to have lower resolution, which can reduce the data volume of the VR image and reduce the power consumption of the VR device.

Further, the embodiment dynamically adjusts the resolution of the image corresponding to the at least one image display area in the at least two image display areas, so that the resolution of the image of the image display area within the specified range of the attention point position is higher, in other The image of the image display area has a lower resolution, and the resolution of the entire image is consistent with the prior art, and the embodiment is closer to the picture seen by the human eye in real life, and improves the realism of the image.

In an optional implementation manner, as the VR scene changes, an image of one frame and one frame is generated, and then one frame image can be sequentially displayed in the VR display space according to the image generation order, so that the user can see Smooth VR scene. In this embodiment, the resolution of each frame of the image that the VR display space needs to display may be the default resolution. In some embodiments, the default resolution may be a lower resolution or a higher resolution. Depending on the default resolution, there are two ways to dynamically adjust the resolution of an image.

The first way: if the default resolution of the image to be displayed in the VR display space is smaller than the first resolution threshold (ie, the default resolution is lower resolution), the image display area within the specified range of the attention point position is increased. The resolution of the image currently being displayed or will be displayed.

Specifically, if the default resolution of the image that the VR display space needs to display is less than the first resolution threshold, it means that the resolution of the image is low, and it is difficult to meet the basic visual needs of the user. Then, in order to make the resolution of the image of the image display area within the specified range of the position of the attention point larger than the resolution of the image of the other image display area, the resolution of the image of the image display area within the specified range of the position of the attention point may be increased. .

In some embodiments, when the resolution of the image of the image display area within the specified range of the position of interest is increased, the image may be adjusted on the image currently displayed in the image display area, or may be on the image to be displayed, ie, Adjust on one frame of image.

In an example, if the default resolution of the image to be displayed in the VR display space is 640*360, and the first resolution threshold is 1280*720, it is determined that the default resolution is smaller than the first resolution threshold, and then the focus is increased. The resolution of the image currently displayed or to be displayed in the image display area within the specified range of locations.

In some embodiments, the resolution of the image of the image display area within the specified range of the location of interest may be increased such that the increased resolution is greater than the first resolution threshold. The first resolution threshold can be flexibly customized according to the visual needs of the user.

The second way: if the default resolution of the image to be displayed in the VR display space is greater than the second resolution threshold (ie, the default resolution is a higher resolution), then the current display or upcoming display in other image display areas is reduced. The resolution of the image.

Specifically, if the default resolution of the image to be displayed in the VR display space is greater than the second resolution threshold, it means that the resolution of the image is higher, which can provide a better visual experience for the user. Then, in order to make the resolution of the image of the image display area within the specified range of the attention point position larger than the resolution of the image of the other image display area, the resolution of the image of the other image display area can be reduced.

In an example, if the default resolution of the image to be displayed in the VR display space is 1280*1440 and the second resolution threshold is 1280*720, it is determined that the default resolution is greater than the second resolution threshold, thereby reducing other image display. The resolution of the image of the area.

In some embodiments, the resolution of the images of other image display regions may be reduced such that the reduced resolution is less than the second resolution threshold. The second resolution threshold can also be flexibly customized according to the visual needs of the user.

In real life, because the human perspective is limited, the human eye can distinguish objects near the position of the focus; however, objects farther away from the point of interest are difficult to distinguish clearly.

Based on this, in order to enhance the realism of the image, for the first mode in the above embodiment, when the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the attention point is increased, The edge of the image display area within the specified range of the attention point position points to the direction of the position of the attention point, and gradually increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the attention point.

First, the image display area within the specified range of the position of interest can be divided into a first three-dimensional area and a second three-dimensional area centered on the position of the attention point. Wherein the second three-dimensional area comprises a first three-dimensional area.

In some embodiments, the first three-dimensional region or the second three-dimensional region centered at the position of the focus point may include, but is not limited to, a cubic region centered on the position of the focus point, a cuboid region, a sphere region, a cylinder region, and the like. Preferably, as shown in FIG. 2a, the first three-dimensional area is a sphere area having a focus point as a center of the sphere and a radius of the first distance; and the second three-dimensional area is a center of the focus point, and the second distance is The sphere area of the radius. Wherein the first distance is less than the second distance.

Of course, in addition to dividing the image display area within the specified range of the focus point position into the first three-dimensional area and the second three-dimensional area, the image display area within the specified range of the attention point position may be divided into the focus point position. a first three-dimensional area, a second three-dimensional area, and a third three-dimensional area of the center. Wherein the second three-dimensional region comprises the first three-dimensional region The third dimensional region of the domain includes a second three-dimensional region.

Preferably, as shown in FIG. 2b, the first three-dimensional area is a sphere area having a focus point as a center of the sphere and a radius of the first distance; and the second three-dimensional area is a center of the focus point, and the second distance is The sphere area of the radius; the third three-dimensional area is a sphere area with the focus point as the center of the sphere and the third distance as the radius. Wherein the first distance is smaller than the second distance, and the second distance is smaller than the third distance. For example, the first distance is 0.6 m, the second distance is 0.8 m, and the second distance is 1 m.

Of course, it is also possible to divide the image display area within the specified range of the focus position into more areas centered on the focus point position, and the developer can flexibly set according to the user's visual experience.

After dividing the image display area within the specified range of the attention point position into at least two areas, the edge of the image display area within the specified range of the attention point position may be pointed to the direction of the position of the attention point, and the position of the attention point is gradually increased. The resolution of the image currently displayed or to be displayed in the image display area within the range.

In some embodiments, the edge of the image display area within the specified range of the position of interest is pointed to the direction of the position of the focus point, and the current display of each of the divided image display areas is sequentially increased according to the resolution increasing from small to large. Or the resolution of the image to be displayed. In some embodiments, the increase factor of the resolution of the image currently displayed or to be displayed by the same image display area is the same.

In an example, the resolution increase coefficient of the image currently displayed or to be displayed in the third three-dimensional region may be set to 2, and the resolution increase coefficient of the image currently displayed or to be displayed in the second three-dimensional region may be set to 3. The resolution increase factor of the image currently displayed or to be displayed in the first three-dimensional area is set to four.

In some embodiments, dynamically adjusting the resolution of the at least one image display region corresponding image in the at least two image display regions comprises dynamically adjusting a resolution of the at least one image display region corresponding image object in the at least two image display regions. Based on this, for the first mode in the above embodiment, in some embodiments, increasing the resolution of the image currently displayed or about to be displayed in the image display area within the specified range of the location of interest includes increasing in focus. The resolution of the image object currently displayed or to be displayed in the image display area within the specified range of the point location.

Multiple image objects can be included in the image. For example, when the image is a grassland image, the image object may include an image of a flower, an image of a grass, an image of a stone, an image of a sky, and the like.

Among them, the image object generally includes an image object of a model attribute and an image object of a background attribute. For example, an image of a flower belongs to an image object of a model property, and an image of the sky belongs to an image object of a background property. In general, the user is more concerned with the image object of the model attribute in the image, and the human eye is more sensitive to the image object of the model attribute of different resolution than the image object of the background attribute. Based on this, further, increasing the resolution of the image includes increasing the resolution of the image object of the model attribute.

Similarly, for the second mode in the above embodiment, the resolution of the image of other image display areas is reduced. The rate includes the resolution of the image object that reduces other image display areas. Further, reducing the resolution of the images of the other image display regions includes reducing the resolution of the image objects of the model attributes of the other image display regions.

The resolution of the image object can be the default resolution when the image is initially generated. For the first mode in the above embodiment, the image object of the default resolution in the image display area within the specified range of the point of interest position can be replaced with the image object of the larger resolution. Conversely, for the second mode in the above embodiment, the image object of the default resolution of the other image display areas can be replaced with the image object of the smaller resolution. Further, in order to increase the rate of dynamically adjusting the resolution of the image object, image objects of various resolutions may be generated in advance.

In some embodiments, at least one image object in the VR display space may be tagged to obtain at least one tag object prior to identifying the user's focus position in the VR display space based on eye tracking; then, at least one tag The objects respectively generate resource files corresponding to the first resolution, the second resolution, and the third resolution.

Among them, the label can uniquely identify the image object, which is equivalent to the naming of the image object. After the label is added to the image object, the image object carrying the label, that is, the label object, can be obtained. A resource file corresponding to the first resolution, a resource file corresponding to the second resolution, and a resource file corresponding to the third resolution may be separately generated for each of the tag objects. In some embodiments, different tags may be added to resource files corresponding to different resolutions belonging to the same tag object to distinguish the difference in resolution corresponding to the resource files.

In an example, the image object A adds the label x, the label of the resource file corresponding to the first resolution may be x.1, and the label of the resource file corresponding to the second resolution may be x.2, corresponding to the third The resolution resource file can be labeled x.3.

The first resolution is greater than the second resolution, and the second resolution is greater than the third resolution.

In some embodiments, the resource file includes image models, pictures, and the like that are required to synthesize image objects of corresponding resolution. The resolution of the image model and the resolution of the image are the same as the resolution corresponding to the resource file to which it belongs.

For example, a resource file corresponding to a resolution of 1280*1440 includes an image model with a resolution of 1280*1440 and a resolution of 1280*1440.

After the resource files corresponding to the plurality of resolutions are generated in advance, the pre-generated resource files may be directly replaced with the resource files used by the currently displayed or upcoming image objects.

In some embodiments, for the first mode of the above embodiment, the edge of the image display area within the specified range of the attention point position points to the direction of the focus point position, and the image within the specified range of the focus point position is gradually increased. When the resolution of the currently displayed or upcoming image of the display area is different, the method of dividing the image display area within the specified range of the position of the attention point may be different, including the following two cases.

In the first case, if the image display area within the specified range of the attention point position is divided into the first three-dimensional area and the second three-dimensional area, the resource file used by the label object within the first three-dimensional area is replaced with the corresponding One resolution a resource file; replacing a resource file used by the tag object between the second three-dimensional area and the first three-dimensional area with a resource file corresponding to the second resolution. The three-dimensional area between the second three-dimensional area and the first three-dimensional area is as shown in FIG. 2a.

In some embodiments, the first resolution and the second resolution are both greater than the first resolution threshold, such that the resolution of the image of the replaced image display area within the specified range of the attention point position is greater than the first resolution threshold. .

The second case: if the image display area within the specified range of the focus point position is divided into the first three-dimensional area, the second three-dimensional area, and the third three-dimensional area, the resource file used by the label object within the first three-dimensional area may be used. Replaced with a resource file corresponding to the first resolution; a resource file used by the label object between the second three-dimensional area and the first three-dimensional area is replaced with a resource file corresponding to the second resolution; the third three-dimensional area and the third The resource file used by the tag object between the two three-dimensional regions is replaced with a resource file corresponding to the third resolution.

Wherein, the three-dimensional region between the second three-dimensional region and the first three-dimensional region, and the three-dimensional region between the third three-dimensional region and the second three-dimensional region are as shown in FIG. 2b.

In some embodiments, the first resolution, the second resolution, and the third resolution are both greater than the first resolution threshold, such that the resolution of the image of the image display area within the specified range of the position of interest is replaced is greater than First resolution threshold.

For the second mode of the above embodiment, the resource file used by the tag object within the other image display area may be replaced with the resource file corresponding to the third resolution.

In some embodiments, the third resolution may be less than the second resolution threshold such that the resolution of the image of the replaced other image display area is less than the second resolution threshold.

In some embodiments, when the user's viewing angle is in the image display area within the specified range of the attention point position, generally the image of the other image display area is blurred. Based on this, the resource files within other image display areas can be blurred after the replacement operation to further enhance the realism of the image. In some embodiments, a shader can be used to fuzzify resource files within other image display areas.

Then, the resource files corresponding to the first resolution, the second resolution, and the third resolution are rendered to output the current display or the image to be displayed after the dynamic adjustment of the resolution. In this way, the current display or the image to be displayed after the resolution is adjusted can be displayed on the VR display device.

If the resolution of the currently displayed image is adjusted, the currently displayed image after dynamically adjusting the resolution may be re-output after rendering the resource files corresponding to the first resolution, the second resolution, and the third resolution. If the resolution of the image to be displayed is adjusted, after the resource files corresponding to the first resolution, the second resolution, and the third resolution are rendered, the image to be displayed after the dynamic adjustment resolution is continuously output, as follows: One frame of image.

In an optional implementation, the first resolution and the second resolution are respectively generated for the at least one label object. And a resource file of a third resolution, in some embodiments, a patch file may be used to generate a resource file corresponding to the first resolution, a resource file corresponding to the second resolution, and a third resolution corresponding to the third resolution resource. The number of patches used to generate the resource file corresponding to the first resolution is greater than the number of patches used to generate the resource file corresponding to the second resolution, and the surface used by the resource file corresponding to the second resolution is generated. The number of slices is greater than the number of patches used to generate the resource file corresponding to the third resolution.

In one example, the number of patches used to generate the table model of the first resolution is 500, and the number of patches used to generate the table model of the second resolution is 400, which is used to generate the table model of the third resolution. The patch is 300 pieces.

In some embodiments, the shape of the patch includes, but is not limited to, a triangle, a rectangle, a diamond, or the like. Preferably, the shape of the patch is triangular.

In an optional implementation, after dynamically adjusting the resolution of the at least one image display area corresponding image in the at least two image display areas, the method further includes: displaying the image display area within the specified range of the attention point position after dynamically adjusting the resolution The number of patches included in the image; in turn, the CPU and/or GPU operating frequency is reduced based on the number of patches and the preset frame rate.

The number of patches included in the image display area within the specified range of the point of interest location is the number of patches used by the resource file corresponding to the image.

On the one hand, the fewer the patches used by the resource files, the lower the CPU and/or GPU required to render the resource files. On the other hand, the operating frequency of the CPU and/or GPU should not be too low, and the frame rate of the image should be kept above the preset frame rate to ensure the consistency of the image display. In some embodiments, the preset frame rate should be greater than 16 fps. Based on this, the correspondence between the operating frequency of the CPU and/or the GPU and the number of patches and the preset frame rate can be established in advance.

Since the resolution of the image of the image display area within the specified range of the attention point position is higher after the resolution is dynamically adjusted, the number of patches used by the resource file corresponding to the image is increased. Then, the number of patches used by the resource file corresponding to the image of the image display area within the specified range of the attention point position after the dynamic adjustment of the resolution can be counted. Of course, it is also possible to count the number of patches used by the resource file corresponding to the entire image.

After dynamically adjusting the resolution, the number of patches is reduced compared to the number of patches used when the resolution of the entire image is higher. Based on this, the operating frequency of the CPU and/or GPU can be reduced according to the pre-established correspondence between the operating frequency of the CPU and/or GPU and the number of patches and the preset frame rate.

After reducing the operating frequency of the CPU and/or the GPU, the resource files corresponding to the first resolution, the second resolution, and the third resolution may be rendered by the CPU and/or GPU after the operating frequency is reduced to output a dynamic adjustment resolution. After the image.

In some embodiments, after the current operating frequency of the CPU and/or GPU is reduced, if the next dynamic adjustment is performed The image of the image display area within the specified range of the attention point position after the resolution is increased by the number of patches, which can increase the operating frequency of the CPU and/or GPU to reduce the delay time during the rendering operation.

The embodiment of the present invention further provides a VR image processing apparatus 300. As shown in FIG. 3, the VR image processing apparatus 300 includes an identification module 301, a partitioning module 302, and an adjustment module 303.

The identification module 301 is configured to identify a location of a user's point of interest in the VR display space based on eye tracking.

The dividing module 302 is configured to divide the VR display space into at least two image display areas based on the position of the point of interest identified by the identification module 301.

The adjusting module 303 is configured to dynamically adjust a resolution of the image corresponding to the image display area of the at least two image display areas, so that the resolution of the image of the image display area within the specified range of the attention point position is greater than that of the other image display areas. The resolution of the image; wherein the other image display area refers to an image display area other than the image display area within the specified range of the attention point position in the at least two image display areas.

In some embodiments, as shown in FIG. 4, the adjustment module 303 includes an increase unit 3031 and a decrease unit 3032.

The increasing unit 3031 is configured to increase the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position if the default resolution of the image to be displayed in the VR display space is smaller than the first resolution threshold. .

The reducing unit 3032 is configured to reduce the resolution of the image currently displayed or to be displayed in other image display areas if the default resolution of the image that the VR display space needs to display is greater than the second resolution threshold.

The second resolution threshold is greater than the first resolution threshold.

In some embodiments, when the increasing unit 3031 increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position, specifically for: displaying the image within the specified range of the attention point position The edge of the region points to the direction of the position of the point of interest, gradually increasing the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the point of interest.

In some embodiments, the image display area within the specified range of the focus point location includes a first three-dimensional area centered on the point of interest location and a second three-dimensional area, the second three-dimensional area including the first three-dimensional area.

The increasing unit 3031 points to the position of the attention point at the edge of the image display area within the specified range of the position of the attention point of interest The direction is gradually increased when the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the focus position is further used to: replace the resource file used by the label object within the first three-dimensional area with Corresponding to the resource file of the first resolution; replacing the resource file used by the label object between the second three-dimensional area and the first three-dimensional area with a resource file corresponding to the second resolution; wherein the first resolution The rate is greater than the second resolution, and the second resolution is greater than the first resolution threshold.

In some embodiments, the image display area within the specified range of the focus point location further includes: a third three-dimensional area centered on the point of interest location, the third three-dimensional area including the second three-dimensional area.

The increasing unit 3031 gradually points the direction of the position of the attention point from the edge of the image display area within the specified range of the position of the attention point, and gradually increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the attention point. Specifically, the method further includes: replacing, by using a resource file used by the label object between the three-dimensional area and the second three-dimensional area, a resource file corresponding to the third resolution; wherein, the second resolution is greater than the third resolution, and the third The resolution is greater than the first resolution threshold.

In some embodiments, the first three-dimensional region is a sphere region having a focus point as a center of the sphere and a radius of the first distance; and the second three-dimensional region is a sphere having a focus point as a center of the sphere and a second distance as a radius The third three-dimensional region is a sphere region having a focal point position as a center of the sphere and a third distance as a radius.

Wherein the first distance is smaller than the second distance, and the second distance is smaller than the third distance.

In some embodiments, as shown in FIG. 4, the VR image processing apparatus 300 further includes an adding module 304 and a generating module 305.

The adding module 304 is configured to add a label to at least one image object in the VR display space to obtain at least one label object.

The generating module 305 is configured to respectively generate resource files corresponding to the first resolution, the second resolution, and the third resolution for the at least one label object obtained by the adding module 304.

In some embodiments, VR image processing device 300 also includes a rendering module 306.

The rendering module 306 is configured to render resource files corresponding to the first resolution, the second resolution, and the third resolution to output a current display or an image to be displayed after dynamically adjusting the resolution.

In some embodiments, as shown in FIG. 4, the VR image processing apparatus 300 further includes a statistics module 307 and an adjustment module 308.

The statistics module 307 is configured to count the number of patches included in the resource file corresponding to the image of the image display area within the specified range of the attention point position after dynamically adjusting the resolution.

The adjustment module 308 is configured to reduce the operating frequency of the CPU and/or the GPU according to the number of patches counted by the statistics module 307 and the preset frame rate.

An embodiment of the present invention further provides an electronic device. As shown in FIG. 5, the electronic device 400 includes a processor 401 and a memory 402 connected to the processor 401.

The memory 402 is configured to store one or more computer instructions. The processor 401 is configured to execute one or more computer instructions stored in the memory 402 to implement the steps in the foregoing method embodiments.

The processor 401 is configured to execute one or more computer instructions stored in the memory 402, to: identify, according to eyeball tracking, a location of a user's point of interest in the VR display space; and divide the VR display space into at least two based on the location of the attention point. Image display area; dynamically adjusting the resolution of the image corresponding to at least one of the image display areas in at least two image display areas, so that the image of the image display area within the specified range of the attention point position has a resolution greater than that of the other image display areas The resolution of the image display area refers to an image display area other than the image display area within the specified range of the attention point position in the at least two image display areas.

In an optional implementation manner, the processor 401 dynamically adjusts the resolution of the image corresponding to the at least one image display area in the at least two image display areas, so that the resolution of the image of the image display area within the specified range of the attention point position is When the resolution of the image is larger than that of the other image display area, it is specifically used to: if the default resolution of the image to be displayed in the VR display space is smaller than the first resolution threshold, increase the current image display area within the specified range of the attention point position. The resolution of the image displayed or to be displayed; if the default resolution of the image to be displayed in the VR display space is greater than the second resolution threshold, the resolution of the image currently displayed or to be displayed in other image display areas is reduced; The second resolution threshold is greater than the first resolution threshold.

In an optional implementation manner, when the processor 401 increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position, the processor 401 is specifically configured to: image within the specified range of the attention point position. The edge of the display area points to the direction of the position of the point of interest, and gradually increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the point of interest.

In an optional implementation, the image display area within the specified range of the focus point location includes: a first three-dimensional area centered on the point of interest location and a second three-dimensional area, the second three-dimensional area including the first three-dimensional area. Based on this, the processor 401 points the direction of the position of the attention point position from the edge of the image display area within the specified range of the attention point position, and gradually increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the attention point. The rate is specifically used to: replace the resource file used by the label object within the first three-dimensional area with the resource file corresponding to the first resolution; and use the label object between the second three-dimensional area and the first three-dimensional area The resource file is replaced with a resource file corresponding to the second resolution; wherein the first resolution is greater than the second resolution, and the second resolution is greater than the first resolution threshold.

In an optional implementation manner, the image display area within the specified range of the focus point location further includes: a third three-dimensional area centered on the focus point position, and the third three-dimensional area includes the second three-dimensional area. Based on this, the processor 401 When the edge of the image display area within the specified range of the position of the attention point points to the direction of the position of the attention point, and gradually increases the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the position of the attention point, specifically for Replacing the resource file used by the label object between the third three-dimensional area and the second three-dimensional area with a resource file corresponding to the third resolution; wherein the second resolution is greater than the third resolution, and the third resolution is greater than the third A resolution threshold.

In an optional embodiment, the first three-dimensional region is a sphere region having a focal point position as a center of the sphere and a first distance as a radius. The second three-dimensional region is a sphere region having a focal point position as a center of the sphere and a second distance as a radius; and the third three-dimensional region is a sphere region having a focal point position as a center of the sphere and a third distance as a radius. Wherein the first distance is smaller than the second distance, and the second distance is smaller than the third distance.

In an optional implementation, the processor 401 is further configured to: add a label to the at least one image object in the VR display space to obtain at least one label, before the location of the user is in the VR display space based on the eye tracking. An object; a resource file corresponding to the first resolution, the second resolution, and the third resolution is generated for each of the at least one label object.

In an optional implementation, the processor 401 is further configured to: render a resource file corresponding to the first resolution, the second resolution, and the third resolution to output a current display or a display to be displayed after dynamically adjusting the resolution. image.

In an optional implementation, after dynamically adjusting the resolution of the at least one image display area corresponding image in the at least two image display areas, the processor 401 is further configured to: after the dynamic adjustment of the resolution, the specified range of the attention point position The image display area image corresponds to the number of patches contained in the resource file; according to the number of patches and the preset frame rate, the operating frequency of the CPU and/or GPU is reduced.

The embodiment of the present invention further provides a computer storage medium, which stores one or more computer instructions, and when the one or more computer instructions are executed by a computer, the steps in the foregoing method embodiments may be implemented.

The electronic device provided by another embodiment of the present invention may be an external head mounted display device or an integrated head mounted display device, wherein the external head mounted display device needs to be used in conjunction with an external processing system (eg, a computer processing system).

FIG. 6 is a schematic diagram of an internal configuration structure of a head mounted display device 500 according to another embodiment of the present invention.

The display unit 501 may include a display panel disposed on a side surface of the head mounted display device 500 facing the user's face, and may be a one-piece panel or left and right panels respectively corresponding to the left and right eyes of the user. The display panel may be an electroluminescence (EL) element, a liquid crystal display or a microdisplay having a similar structure, or a laser-scanned display in which the retina may be directly displayed or similar.

The virtual image optical unit 502 photographs the image displayed by the display unit 501 in an enlarged manner and allows the user to observe the displayed image in the enlarged virtual image. As the display image outputted to the display unit 501, it may be an image of a virtual scene supplied from a content reproduction device (a Blu-ray disc or a DVD player) or a streaming media server, or an external use An image of a real scene taken by the camera 510. In some embodiments, virtual image optical unit 502 can include a lens unit, such as a spherical lens, an aspheric lens, a Fresnel lens, and the like.

The input operation unit 503 includes at least one operation member for performing an input operation, such as a button, a button, a switch, or other similarly functioned component, receives a user instruction through the operation member, and outputs an instruction to the control unit 507.

The status information acquisition unit 504 is configured to acquire status information of the user wearing the head mounted display device 500. The status information acquisition unit 504 may include various types of sensors for detecting status information by itself, and may acquire status information from an external device such as a smartphone, a wristwatch, and other multi-function terminals worn by the user through the communication unit 505. The status information acquisition unit 504 can acquire location information and/or posture information of the user's head. The status information acquisition unit 504 may include one or more of a gyro sensor, an acceleration sensor, a global positioning system (GPS) sensor, a geomagnetic sensor, a Doppler effect sensor, an infrared sensor, and a radio frequency field intensity sensor. Further, the state information acquisition unit 504 acquires state information of the user wearing the head-mounted display device 500, for example, acquires, for example, an operation state of the user (whether the user wears the head-mounted display device 500), an action state of the user (such as standing, walking, running) And the state of movement such as the state of the hand or fingertip, the open or closed state of the eye, the direction of the line of sight, the size of the pupil, the mental state (whether the user is immersed in observing the displayed image, and the like), or even the physiological state.

The communication unit 505 performs communication processing with the external device, modulation and demodulation processing, and encoding and decoding processing of the communication signal. In addition, the control unit 507 can transmit transmission data from the communication unit 505 to an external device. The communication method may be wired or wireless, such as mobile high-definition link (MHL) or universal serial bus (USB), high-definition multimedia interface (HDMI), wireless fidelity (Wi-Fi), Bluetooth communication, or low-power Bluetooth communication. And the mesh network of the IEEE802.11s standard. Additionally, communication unit 505 can be a cellular wireless transceiver that operates in accordance with Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), and the like.

In some embodiments, the head mounted display device 500 can also include a storage unit, the storage unit 506 being a mass storage device configured to have a solid state drive (SSD) or the like. In some embodiments, storage unit 506 can store applications or various types of data. For example, content viewed by the user using the head mounted display device 500 may be stored in the storage unit 506.

In some embodiments, the head mounted display device 500 can also include a control unit, and the control unit 507 can include a computer processing unit (CPU) or other device having similar functionality. In some embodiments, control unit 507 can be used to execute an application stored by storage unit 506, or control unit 507 can also be used to perform the methods, functions, and operations disclosed in some embodiments of the present invention.

The image processing unit 508 is for performing signal processing such as image quality correction related to the image signal output from the control unit 507, and converting its resolution into a resolution according to the screen of the display unit 501. Then, the display drive The moving unit 509 sequentially selects each line of pixels of the display unit 501, and sequentially scans each line of pixels of the display unit 501 line by line, thereby providing a pixel signal based on the signal processed image signal.

In some embodiments, the head mounted display device 500 can also include an external camera. The external camera 510 may be disposed on the front surface of the body of the head mounted display device 500, and the external camera 510 may be one or more. The external camera 510 can acquire three-dimensional information and can also be used as a distance sensor. Additionally, a position sensitive detector (PSD) or other type of distance sensor that detects reflected signals from the object can be used with the external camera 510. The external camera 510 and the distance sensor can be used to detect the body position, posture, and shape of the user wearing the head mounted display device 500. In addition, under certain conditions, the user can directly view or preview the real scene through the external camera 510.

In some embodiments, the head mounted display device 500 may further include a sound processing unit 511 that may perform sound quality correction or sound amplification of the sound signal output from the control unit 507, signal processing of the input sound signal, and the like. Then, the sound input/output unit 512 outputs the sound to the outside and the sound from the microphone after the sound processing.

It should be noted that the structure or component shown by the dashed line in FIG. 6 may be independent of the head mounted display device 500, for example, may be disposed in an external processing system (eg, a computer system) for use with the head mounted display device 500; or The structure or components shown in dashed boxes may be disposed inside or on the surface of the head mounted display device 500.

The above description is only an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and changes can be made in the present invention. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the invention are intended to be included within the scope of the appended claims.

Claims

A VR image processing method, comprising:

Identifying the location of the user's point of interest in the VR display space based on eye tracking;

Dividing the VR display space into at least two image display areas based on the location of interest;

Dynamically adjusting a resolution of the at least one image display area corresponding image in the at least two image display areas such that an image of the image display area within the specified range of the attention point position has a resolution greater than that of the other image display areas Resolution

The other image display area refers to an image display area other than the image display area within the specified range of the attention point position in the at least two image display areas.
The method according to claim 1, wherein dynamically adjusting a resolution of at least one image display area corresponding image in the at least two image display areas to cause an image display area within a specified range of the attention point position The resolution of the image is greater than the resolution of the image in other image display areas, including:

If the default resolution of the image that the VR display space needs to display is less than the first resolution threshold, increase the resolution of the image currently displayed or to be displayed in the image display area within the specified range of the attention point position;

If the default resolution of the image that the VR display space needs to display is greater than the second resolution threshold, reducing the resolution of the image currently displayed or to be displayed in the other image display area;

The second resolution threshold is greater than the first resolution threshold.
The method according to claim 2, wherein the increasing the resolution of an image currently displayed or to be displayed in an image display area within a specified range of the location of interest location comprises:

And gradually increasing the image currently displayed or to be displayed in the image display area within the specified range of the attention point position from the direction of the edge of the image display area within the specified range of the attention point position Resolution.
The method according to claim 3, wherein the image display area within the specified range of the focus point position comprises: a first three-dimensional area and a second three-dimensional area centered on the focus point position, The second three-dimensional area includes the first three-dimensional area;

And displaying, from the edge of the image display area within the specified range of the attention point position, the direction of the focus point position, and gradually increasing or temporarily displaying the image display area within the specified range of the attention point position The resolution of the image, including:

Replacing the resource file used by the tag object within the first three-dimensional area with a resource file corresponding to the first resolution;

Replacing the resource file used by the label object between the second three-dimensional area and the first three-dimensional area with a resource file corresponding to the second resolution;

The first resolution is greater than the second resolution, and the second resolution is greater than the first resolution threshold.
The method according to claim 4, wherein the image display area within the specified range of the focus point position further comprises: a third three-dimensional area centered on the focus point position, the third three-dimensional The area includes the second three-dimensional area;

And displaying, from the edge of the image display area within the specified range of the attention point position, the direction of the focus point position, and gradually increasing or temporarily displaying the image display area within the specified range of the attention point position The resolution of the image also includes:

Replacing the resource file used by the label object between the third three-dimensional area and the second three-dimensional area with a resource file corresponding to the third resolution;

The second resolution is greater than the third resolution, and the third resolution is greater than the first resolution threshold.
The method according to claim 5, wherein the first three-dimensional area is a sphere area having a center of the focus point and a radius of the first distance;

The second three-dimensional region is a sphere region having the focus point as a center of the sphere and the second distance as a radius;

The third three-dimensional area is a sphere area with the focus point as the center of the sphere and the third distance as the radius;

The first distance is smaller than the second distance, and the second distance is smaller than the third distance.
The method according to claim 5 or 6, wherein before the recognizing the position of the user in the VR display space based on the eyeball tracking, the method further comprises:

Adding a label to at least one image object in the VR display space to obtain at least one label object;

Generating resource files corresponding to the first resolution, the second resolution, and the third resolution, respectively, for the at least one tag object.
The method of any of claims 5-7, further comprising:

A resource file corresponding to the first resolution, the second resolution, and the third resolution is rendered to output a current display or an image to be displayed after dynamically adjusting the resolution.
The method according to any one of claims 1-8, wherein after the dynamically adjusting the resolution of the at least one image display area corresponding image in the at least two image display areas, the method further comprises :

Counting the number of patches included in the resource file corresponding to the image of the image display area within the specified range of the attention point position after dynamically adjusting the resolution;

The operating frequency of the CPU and/or GPU is reduced according to the number of patches and the preset frame rate.
A VR image processing apparatus, comprising:

An identification module for identifying a location of a user's point of interest in the VR display space based on eye tracking;

a dividing module, configured to divide the VR display space into at least two image display areas based on the location of the attention point;

An adjustment module, configured to dynamically adjust a resolution of an image corresponding to at least one image display area of the at least two image display areas, so that an image of an image display area within a specified range of the attention point position has a resolution greater than other images The resolution of the image of the display area;

The other image display area refers to an image display area other than the image display area within the specified range of the attention point position in the at least two image display areas.
The apparatus according to claim 10, wherein the adjustment module comprises:

And an increasing unit, if the default resolution of the image that needs to be displayed in the VR display space is smaller than the first resolution threshold, increasing an image currently displayed or to be displayed in the image display area within the specified range of the attention point position Resolution

a reducing unit, configured to reduce a resolution of an image currently displayed or to be displayed in the other image display area if a default resolution of the image that the VR display space needs to display is greater than a second resolution threshold;

The second resolution threshold is greater than the first resolution threshold.
The device according to claim 11, wherein the increasing unit is specifically configured to:

And gradually increasing the image currently displayed or to be displayed in the image display area within the specified range of the attention point position from the direction of the edge of the image display area within the specified range of the attention point position Resolution.
An electronic device, comprising: a processor, and a memory coupled to the processor;

The memory is configured to store one or more computer instructions;

The processor is configured to execute the one or more computer instructions to implement the method of any one of claims 1-9.
A computer readable storage medium storing a computer program, wherein the computer program, when executed, is capable of implementing the steps of the method of any of claims 1-9.