CN114418857A

CN114418857A - Image display method and device, head-mounted display equipment and storage medium

Info

Publication number: CN114418857A
Application number: CN202210089192.2A
Authority: CN
Inventors: 章锦腾
Original assignee: Shanghai Jinsheng Communication Technology Co ltd; Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Shanghai Jinsheng Communication Technology Co ltd; Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2022-01-25
Filing date: 2022-01-25
Publication date: 2022-04-29

Abstract

The application discloses an image display method, an image display device, a head-mounted display device and a storage medium, wherein the image display method is applied to the head-mounted display device, and the method comprises the following steps: acquiring an interested area corresponding to human eyes in an image to be displayed as a first area; determining a second area and a third area in the image to be displayed based on the first area; respectively performing super-resolution reconstruction on a first region, a second region and a third region in an image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is reduced in sequence; and displaying the image to be displayed after the super-resolution reconstruction. The method can improve the image quality of the display image of the head-mounted display device and reduce the processing amount of the head-mounted display device.

Description

Image display method and device, head-mounted display equipment and storage medium

Technical Field

The present disclosure relates to the field of display technologies, and in particular, to an image display method and apparatus, a head-mounted display device, and a storage medium.

Background

With the progress of science and technology, technologies such as Virtual Reality (VR), Augmented Reality (AR), etc. have gradually become hot spots of research at home and abroad. Head-mounted display devices (such as AR glasses and the like) based on virtual reality and augmented reality are increasing, and people can watch display contents and play of audio by using the head-mounted display devices, so that the head-mounted display devices are popular among people. However, when the head-mounted display device displays content, it is difficult to achieve both image quality and processing efficiency.

Disclosure of Invention

In view of the above, the present application proposes an image display method, an apparatus, a head-mounted display device, and a storage medium.

In a first aspect, an embodiment of the present application provides an image display method, which is applied to a head-mounted display device, and the method includes: acquiring an interested area corresponding to human eyes in an image to be displayed as a first area; determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed; respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is reduced in sequence; and displaying the image to be displayed after the super-resolution reconstruction.

In a second aspect, an embodiment of the present application provides an image display apparatus applied to a head-mounted display device, the apparatus including: the display device comprises a first determining module, a second determining module, an image processing module and an image display module, wherein the first determining module is used for acquiring a region of interest corresponding to human eyes in an image to be displayed as a first region; the second determining module is configured to determine a second area and a third area in the image to be displayed based on the first area, where the second area is adjacent to the first area, and the third area is an area other than the first area and the second area in the image to be displayed; the image processing module is used for respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced; the image display module is used for displaying the image to be displayed after super-resolution reconstruction.

In a third aspect, an embodiment of the present application provides a head-mounted display device, including: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs being configured to perform the image display method provided by the first aspect above.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, and the program code may be called by a processor to execute the image display method provided in the first aspect.

According to the scheme, when the head-mounted display device displays the image, the region of interest corresponding to human eyes in the image to be displayed is acquired and used as the first region, then based on the first region, the second region and the third region in the image to be displayed are determined, the second region is adjacent to the first region, the third region is a region except the first region and the second region in the image to be displayed, then different super-resolution reconstruction is carried out on the first region, the second region and the third region in the image to be displayed respectively, the image quality of the first region, the second region and the third region after the super-resolution reconstruction is reduced in sequence, and the image to be displayed is displayed after the super-resolution reconstruction. Therefore, super-resolution reconstruction can be performed on different regions in the image to be displayed according to the region of interest of human eyes in the image to be displayed, so that the image quality of the different regions is improved to different degrees, the display quality of the content watched by a user is ensured, the processing amount of the head-mounted display equipment is reduced, the display delay and the blockage are avoided, and the display effect of the head-mounted display equipment is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 shows a scene schematic diagram provided in an embodiment of the present application.

Fig. 2 shows another schematic view of a scenario provided in an embodiment of the present application.

Fig. 3 shows a schematic view of another scenario provided in the embodiment of the present application.

FIG. 4 shows a flow diagram of an image display method according to one embodiment of the present application.

Fig. 5 shows a schematic view of still another scenario provided in an embodiment of the present application.

Fig. 6 shows a flow chart of an image display method according to another embodiment of the present application.

Fig. 7 shows a flowchart of an image display method according to yet another embodiment of the present application.

Fig. 8 shows a flowchart of an image display method according to still another embodiment of the present application.

Fig. 9 shows a flowchart of an image display method according to yet another embodiment of the present application.

FIG. 10 shows a block diagram of an image display device according to an embodiment of the present application.

Fig. 11 is a block diagram of a head mounted display device for executing an image display method according to an embodiment of the present application.

Fig. 12 is a storage unit for storing or carrying a program code implementing an image display method according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

Virtual Reality technology (VR) is a computer simulation system that creates and experiences a Virtual world, using a computer to create a simulated environment into which a user is immersed. The virtual reality technology is to combine electronic signals generated by computer technology with data in real life to convert the electronic signals into phenomena which can be felt by people, wherein the phenomena can be true and true objects in reality or substances which can not be seen by the naked eyes, and the phenomena are expressed by a three-dimensional model.

Augmented Reality (AR) is a technology that increases the user's perception of the real world through information provided by a computer system, which superimposes content objects such as computer-generated virtual objects, scenes, or system cues into the real scene to enhance or modify the perception of the real world environment or data representing the real world environment.

In the field of image processing, Super Resolution (SR) is an image processing task for processing a Low Resolution (LR) image or an image sequence to recover a High Resolution (HR) image. In recent years, SR shows broad application prospects in the fields of monitoring equipment, satellite image remote sensing, digital high definition, microscopic imaging, video coding communication, video restoration, medical imaging and the like, and becomes a research hotspot in the field of computer vision. SR technology is also increasingly used in display technologies such as VR and AR.

Among them, SR technology mainly considers realizing two-part functions on a head-mounted display device.

The first part of the function is to recover the distortion caused by the resolution scaling, for example, since the low resolution video cannot effectively display the image details, and thus the user experience is limited, the distortion from the 1K resolution scaling to the 4K resolution can be realized by the SR technology.

The second part is used for reconstructing lost information, wherein the video coding and decoding causes information loss due to compression, especially some high-frequency detail information, and Peak signal-to-noise ratio (PSNR) performance can be further improved by over-separating and reconstructing the lost information in the cyclic process of a decoding end or a coding end. Because the compression is a one-body two-system, the image quality is improved under a certain code rate, or the code rate is saved under a certain image quality. If the lost information can be reconstructed under a certain code rate through the SR, the image quality can be further improved, and the code rate and the bandwidth can be further saved on the premise of keeping the same video quality.

With the development of VR and AR technologies, head-mounted display devices (e.g., VR helmets, AR glasses, AR helmets, etc.) are increasing. In order to achieve the lightness of the head-mounted display device, the head-mounted display device is gradually changed from an integrated device to an external (wireless) device. The integrated head-mounted display device has stronger processing capacity and can realize all processing in VR or AR display; the wireless head-mounted display device is connected with an external terminal, for example, a smart phone, and places main processing functions on the connected terminal, and the terminal processes data and transmits the processed data to the head-mounted display device for output.

The inventor has found that, after a long time of research, rendering of a head-mounted display device requires a large amount of computing power, and when a display scene with extremely high resolution of 4K/8K required for the presence is executed, the image quality of the display content generally needs to be improved by the SR technology, so that a large amount of computation is required. However, the wireless head-mounted display device has limited computing power, and thus is prone to problems such as reduced frame rate, delay, reduced resolution, and increased aliasing, which are not favorable for the overall user experience.

In view of the above problems, the inventor proposes an image display method, an apparatus, a head-mounted display device, and a storage medium provided in this embodiment of the present application, which can perform different super-resolution reconstructions on different regions in an image to be displayed according to an area of interest of human eyes in the image to be displayed, so that the image quality in the different regions is improved to different degrees, thereby ensuring the display quality of content viewed by a user, reducing the processing amount of the head-mounted display device, avoiding display delay and jamming, and improving the display effect of the head-mounted display device. Specific image display methods are described in detail in the following examples.

The following first describes a head-mounted display device according to an image display method provided in an embodiment of the present application. The head-mounted display device may be AR glasses, an AR helmet, VR glasses, a VR helmet, MR (Mixed Reality) glasses, an MR helmet, or the like, which is not limited herein. The head mounted display device in the embodiment of the present application is described below with glasses.

As shown in FIG. 1, FIG. 1 shows a schematic view of a head mounted display device. As shown in fig. 1, the head-mounted display device 100 includes a display screen 110, a frame 120, an imaging device 130, and an eye tracking device 140.

The frame 120 includes a front surface 121 on which the display screen 110 is mounted, a side surface 122, and a rear surface 123, and the imaging device 130 is capable of displaying an image of a virtual object on the display screen 110. For example, the imaging device 130 may be a diffractive light guide capable of projecting an image onto a display screen.

The eye tracking device 140 may be a camera or the like, and is configured to collect an eye image of the user wearing the head-mounted display apparatus 100, track an eye line of the user according to the eye image, and determine a gazing point position of a human eye on the display screen 110 based on the eye line, so as to perform super-resolution reconstruction on the image to be displayed after determining an area of interest in the image to be displayed according to the gazing point position.

As an embodiment, the display screen 110 may be a lens of the AR glasses, and the display screen 110 may also transmit light, that is, the display screen 110 may be a transflective lens, when the user wears the head-mounted display device, when an image is displayed on the display screen 110, the user can see the image displayed on the display screen 110 and can also see objects in the real world in the surrounding environment through the display screen 110. Then, through the semi-transparent and semi-reflective lens, the user can superimpose the image displayed on the lens with the surrounding environment, thereby realizing the visual effect of augmented reality.

When the user wears the head-mounted display device, the display screen 110 is located in front of the eyes of the user, that is, the front surface 121 is located in front of the eyes of the user, the rear surface 123 is located behind the eyes of the user, and the side surface 122 is located at the side of the eyes of the user. In addition, a front camera may be disposed on the front surface 121, and the front environmental information is sensed by the front camera, so as to implement instant positioning and Mapping (SLAM), and further implement a visual effect of augmented reality or mixed reality.

In some embodiments, the head-mounted display device 100 may be an integrated (access) head-mounted display device, or may be a wireless (external/access) head-mounted display device. When the head-mounted display device 100 is a wireless head-mounted display device, an intelligent terminal such as a mobile phone connected to the head-mounted display device 100 may be used as a processing and storage device of the head-mounted display device, and may be inserted or connected to an external head-mounted display device to process data. When the head-mounted display device 100 is an integral head-mounted display device, the head-mounted display device may include a processor and a memory for processing data.

Referring to fig. 2, the head-mounted display device 100 may be a wireless device, and when the head-mounted display device displays content, the head-mounted display device 100 may be connected to the electronic terminal 200. The electronic terminal 200 sends the image to be displayed, which is subjected to scaling and encoding in resolution, to the head-mounted display device 100, and after the image to be displayed is decoded by the head-mounted display device 100, the image to be displayed is displayed by the image display method provided in the embodiment of the present application. Therefore, the data transmission amount between the head-mounted display device 100 and the electronic terminal 200 can be reduced, the display effect can be ensured while the transmission bandwidth is saved, and the processing amount of the head-mounted display device 100 is reduced. The electronic terminal 200 may be a smart phone, a tablet computer, or the like, which is not limited herein.

Referring to fig. 3, the head-mounted display device 100 may be a wireless device, when the head-mounted display device displays content, the head-mounted display device 100 may communicate with the server 300 through a network, the server 300 may send an image to be displayed, which is scaled and encoded in resolution, to the head-mounted display device 100, and after the image to be displayed is decoded by the head-mounted display device 100, the image to be displayed is displayed by using the image display method provided in the embodiment of the present application. The server 300 may be a cloud server, a conventional server, etc., and is not limited herein.

The following describes an image display method provided in an embodiment of the present application in detail with reference to the accompanying drawings.

Referring to fig. 4, fig. 4 is a schematic flowchart illustrating an image display method according to an embodiment of the present application. In a specific embodiment, the image display method is applied to the image display apparatus 400 shown in fig. 10 and the head mounted display device 100 (fig. 11) equipped with the image display apparatus 400. The following will describe a specific flow of the present embodiment by taking the head mounted display device as an example. As will be described in detail with respect to the flow shown in fig. 4, the image display method may specifically include the following steps:

step S110: and acquiring a region of interest corresponding to the human eye in the image to be displayed as a first region.

When the head-mounted display device displays content, the image to be displayed can be acquired. The image to be displayed may be a video image in a video stream, or may be a single picture, and the like, which is not limited herein. Optionally, the image to be displayed may be transmitted by an external device, for example, the head-mounted display device is a wireless head-mounted display device, the wireless head-mounted display device is connected to the mobile terminal, the mobile terminal may transmit the image to be displayed after resolution scaling and encoding compression to the head-mounted display device, and after receiving the image to be displayed, the head-mounted display device performs the image display method provided in the embodiment of the present application.

The region of interest (ROI) refers to a region to be processed, which is delineated from a processed image in a manner of a box, a circle, an ellipse, an irregular polygon, or the like in machine vision and image processing. In this embodiment of the application, the region of interest corresponding to the human eye in the image to be displayed may be a region focused by the human eye in the image to be displayed, that is, a region focused by a user when viewing the displayed image to be displayed by using a head-mounted display device.

In some embodiments, the head-mounted display device may determine a region of interest corresponding to a human eye in an image to be displayed based on a gaze point position of the human eye, and use the region of interest as the first region. The fixation means that the center position of the eyes of the user is aligned with an object for a time longer than a predetermined time, for example, 100 milliseconds, and the object to be fixed is imaged at the center position during the time, and is more sufficiently processed to form a clear image. The point of regard (gaze) refers to a point on an object at which the eye is looking during the process of viewing the object. In this embodiment, the position at which the line of sight is aligned may be a display area of the head mounted display device, that is, a display area of the display screen, that is, the head mounted display device may acquire a gaze point position at which a human eye is currently located in the display area of the head mounted display device. Wherein the gaze point position may be a coordinate, such as a screen coordinate of the gaze point in the display area. The head-mounted display device can determine that the gazing point position corresponds to an area in the image to be displayed as an interested area corresponding to human eyes.

In other embodiments, the head-mounted display device may acquire a target region in the image to be displayed as a region of interest corresponding to the human eye in the image to be displayed. Alternatively, the target region may be a region where the center position in the image to be displayed is located, and it is understood that when the user views the content through the head-mounted display device, the probability that the user focuses on the middle region is generally high, and therefore, based on this, the region of interest corresponding to the human eye in the image to be displayed may be determined. Optionally, the target area may also be determined by the gaze point position when the user historically views the content through the head-mounted display device, where a position where the number of times that the gaze point position of the human eye appears meets a preset number condition may be determined as the target position according to the gaze point position historically by the user, and then an area corresponding to the target position is determined from the image to be displayed as the region of interest. Of course, the specific target area may not be limited.

In still other embodiments, the head-mounted display device may identify an object of interest in the image to be displayed, and determine a region in which the object of interest is located in the image to be displayed, as a region of interest corresponding to a human eye in the image to be displayed. The object of interest may be a person, an animal, or other object that is likely to attract the attention of the user, and the specific object of interest may not be limited, and may be set according to the requirement or habit of the user. When the object of interest in the image to be displayed is identified, a plurality of objects of interest may be identified, and optionally, when the plurality of objects of interest are identified in the image to be displayed, a region with the largest number of objects of interest may be determined according to the distribution of the objects of interest in the image to be displayed, and the region is used as a region of interest corresponding to human eyes in the image to be displayed; optionally, when a plurality of interested objects are identified in the image to be displayed, the interested object corresponding to the position with the highest probability may be determined according to the position of each interested object in the image and the probability of each position being focused by the user, and then the region where the interested object is located may be determined as the interested region from the image to be displayed, where the probability of each position being focused by the user may be determined according to the gazing habit of the user when the user views the content using the head-mounted display device.

Step S120: and determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed.

In this embodiment of the application, after the head-mounted display device determines the first region in the image to be displayed, segmentation of the image to be displayed is achieved, and the head-mounted display device may further segment the image to be displayed based on the first region, and determine the second region and the third region in the image to be displayed. The second region is adjacent to the first region, and the third region is a region of the image to be displayed except the first region and the second region, that is, the second region adjacent to the first region can be determined based on the first region, and the image to be displayed can be divided into the first region, the second region, and the third region.

For example, referring to fig. 5, the head-mounted display device may divide the image to be displayed into a first region, a second region and a third region as shown in fig. 5 through the above steps, wherein the first region is a region of interest corresponding to a human eye, the second region is adjacent to (around) the first region, and the third region is other regions of the image to be displayed except the first region and the second region.

Step S130: respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

In the embodiment of the application, after the head-mounted display device divides the image to be displayed into the first region, the second region and the third region based on the gazing point position, super-resolution reconstruction of the image to be displayed can be performed. Different super-resolution reconstruction can be respectively carried out on the first region, the second region and the third region in the image to be displayed, so that the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

As can be understood, the super-resolution reconstruction algorithm has a positive correlation between the improvement of the image quality and the computation amount when processing the image, and therefore, when the head-mounted display device performs the super-resolution reconstruction on the first region, the second region, and the third region, the computation amount consumed by the first region, the second region, and the third region is different. In addition, since the first region is a region where an image to be displayed corresponds to a gaze point position of the display region, the second region is a region adjacent to the first region, and the third region is a region other than the first region and the second region, the first region may be regarded as a region of attention of a user, the second region may be regarded as a transition region, and the third region may be regarded as a region of non-attention; after the super-resolution reconstruction is carried out on the first region, the second region and the third region, the improvement degree of the image quality of the region concerned by the user is large, so that the display effect of the image to be displayed watched by the user can be ensured, and the same super-resolution reconstruction is not carried out on the image to be displayed due to the fact that the improvement degrees of the image quality of different regions are different, so that the calculation amount can be reduced, the calculation resource of the head-mounted display device is saved, and the display blockage is avoided.

In some embodiments, image quality may be evaluated by Peak signal-to-noise ratio (PSNR). The head-mounted display device can respectively perform super-resolution reconstruction on the first region, the second region and the third region by using super-resolution reconstruction algorithms with different PSNR lifting degrees, wherein the lifting degree of the super-resolution reconstruction algorithm corresponding to the first region to the PSNR, the lifting degree of the super-resolution reconstruction algorithm corresponding to the second region to the PSNR and the lifting degree of the super-resolution reconstruction algorithm corresponding to the third region to the PSNR are sequentially reduced, so that the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

Step S140: and displaying the image to be displayed after the super-resolution reconstruction.

In this embodiment of the application, after the head-mounted display device performs the super-resolution reconstruction on the first region, the second region, and the third region in the image to be displayed, the image to be displayed after the super-resolution reconstruction may be displayed through the display screen.

Illustratively, referring to fig. 6, the image display scene is a video playing scene, and the head-mounted display device is a wireless head-mounted display device and can be connected with an external device. The external device can perform video encoder (for example, h.265encoder) on the low-resolution video, and then transmit the low-resolution video to the video decoder (for example, h.265decoder) of the head-mounted display device through the network, and then the head-mounted display device performs super-resolution reconstruction on the video image through the above method, and performs video enhancement, so as to obtain the high-resolution video with higher resolution, thereby ensuring the video display effect and reducing the data transmission amount between the head-mounted display device and the external device.

The image display method provided by the embodiment of the application realizes super-resolution reconstruction of the image to be displayed according to the region of interest corresponding to human eyes, and divides the image to be displayed into three regions, so that the improvement degrees of the image quality are different for different regions, the whole operation amount can be greatly reduced, the operation complexity is reduced, the time delay is reduced, the display delay and the blockage are avoided, and the display effect of the head-mounted display device is improved. Moreover, the resolution improvement degree aiming at the interested area is the highest, so that the user can perceive that the resolution improvement of the displayed image is obvious.

Referring to fig. 7, fig. 7 is a schematic flowchart illustrating an image display method according to another embodiment of the present application. The image display method is applied to the head-mounted display device. As will be described in detail with respect to the flow shown in fig. 7, the image display method may specifically include the following steps:

step S210: and acquiring a region of interest corresponding to the human eye in the image to be displayed as a first region.

In the embodiment of the present application, step S210 may refer to the contents of other embodiments, which are not described herein again.

Step S220: and judging whether the first area meets a preset condition, wherein the preset condition is a condition when the watching probability of the area adjacent to the first area is greater than a preset probability.

In this embodiment of the application, after the head-mounted display device determines the first region, before the second region and the third region are further divided from the image to be displayed, it may be further determined whether the first region meets a preset condition. The preset condition is a condition when the probability that the area adjacent to the first area is watched is greater than the preset probability, the preset condition is used as a basis for judging whether the head-mounted display device further divides the image to be displayed, if the first area meets the preset condition, the probability that the area adjacent to the first area is watched is greater than the preset probability, namely the probability that the second area is watched is higher, and therefore the head-mounted display device can further divide the image to be displayed; if the first area does not satisfy the preset condition, the probability that the area adjacent to the first area is watched is not greater than (less than or equal to) the preset probability, that is, the probability that the second area is watched is low, so that the head-mounted display device does not need to further divide the image to be displayed.

In some embodiments, the head-mounted display device may determine whether a target object is present in a first area when determining whether the first area satisfies a preset condition; if the target object exists, determining that the first area meets a preset condition; and if the target object does not exist, determining that the first area does not meet the preset condition. The target object may be an object that easily draws the attention of the user to the second area, if the target object exists in the first area, the probability that the user pays attention to the content in the second area is greater than a preset probability, that is, the probability that the user pays attention to the second area is higher, and if the target object does not exist in the first area, the probability that the user pays attention to the content in the second area is lower.

As a possible implementation, the target object may include: and at least one of a texture having a contrast greater than a preset contrast and a texture having a sharpness greater than a preset sharpness. It is understood that if there is a high contrast texture, a hard line of texture (e.g., a shadow with high sharpness), a text with high sharpness, etc. in the first area, the user will notice the second area adjacent to the first area with high probability; if there is no high contrast texture, hard lines of texture (e.g., high-sharpness shadows), high-sharpness text, etc. in the first area, the user will notice a second area adjacent to the first area with a low probability. Therefore, the texture with the contrast greater than the preset contrast and the texture with the sharpness greater than the preset sharpness may be used as the preset object to determine whether the first area satisfies the preset condition.

In other embodiments, after the head-mounted display device determines the first area, a main object (for example, an existing object or the like) in the first area may be identified, and if the area where the main object is located exceeds the first area, the user may easily pay attention to an area adjacent to the first area, that is, observe a portion of the main object outside the first area when observing the image to be displayed. Therefore, the head-mounted display device may also determine that the first area satisfies the preset condition when recognizing that the area where the main object is located exceeds the first area; and when the area where the main body object is located is identified not to exceed the first area, determining that the first area does not meet the preset condition.

Of course, the specific manner of determining whether the first area satisfies the preset condition in the embodiment of the present application may not be limited.

Step S230: and if the first area meets the preset condition, determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed.

In the embodiment of the present application, if the head mounted display device determines that the first region satisfies the preset condition, it indicates that the probability of paying attention to the region adjacent to the first region is high, and therefore, the image to be displayed may be further divided based on the first region to determine the second region adjacent to the first region, and a region other than the first region and the second region in the image to be displayed, that is, a third region may be obtained.

Step S240: respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

In some embodiments, the degree of PSNR improvement by the super-resolution reconstruction algorithm corresponding to the first region, the degree of PSNR improvement by the super-resolution reconstruction algorithm corresponding to the second region, and the degree of PSNR improvement by the super-resolution reconstruction algorithm corresponding to the third region are sequentially decreased.

In one possible embodiment, a first super-resolution reconstruction strategy may be employed to perform a super-resolution reconstruction on the first region; performing super-resolution reconstruction on the second region by adopting a second super-resolution reconstruction strategy; and performing super-resolution reconstruction on the third region by adopting a third super-resolution reconstruction strategy.

The super-resolution reconstruction of the first region can be realized in different ways by adopting a first super-resolution reconstruction strategy. Optionally, for the first region, a more advanced hyper-differentiation algorithm, such as laprn or DCSCN, may be used to achieve a substantial improvement in PSNR performance. Optionally, after the first region is processed by using a Fixed Focused Rendering (FFR) algorithm, a very advanced super-partition algorithm is used, such as a laprn or DCSCN super-partition algorithm, or after the first region is processed by using the laprn or DCSCN algorithm, the first region is processed by using an FFR algorithm. Optionally, the hyper-resolution neural network may be trained in advance by using images after FFR, and the trained FFR-based super-resolution network is used to improve the PSNR of the first region, so as to implement super-resolution reconstruction of the first region.

And performing super-resolution reconstruction on the second region by adopting a second super-resolution reconstruction strategy, wherein a super-resolution algorithm with low algorithm complexity but advanced performance, such as an FSRCNN-s or C-DCSCN algorithm, is adopted.

The third super-resolution reconstruction strategy is adopted to perform super-resolution reconstruction on the third area, which can be a super-resolution algorithm with general performance but ultra-fast speed, such as an SR-LUT, and the image quality of the third area after super-resolution reconstruction is the lowest, but the effect is still better than that of the traditional interpolation method.

The super-resolution algorithm described above is described below.

The traditional most common super-resolution amplification method adopts interpolation methods, such as Bicubic, nerest, bilinear and the like, and has the advantages of high speed and obvious defects, namely, after the image is amplified, the image has the phenomena of blurring and detail loss.

SR-LUT visual effect is better than traditional interpolation method, but not as good as FSRCNN. The SR-LUT trains the hyper-division network by a small receptive field through a lookup table and transfers the output value of the hyper-division network to the lookup table; in the test phase, the calculated HR output is indexed from the LUT according to the input. The proposed method is very fast in computation since a large number of floating point calculations are not required. The method has faster and better visual effect than bicubic.

FSRCNN (accumulating the Super-Resolution Neural Network) is an improvement on SRCNN, mainly in three aspects: one is that a deconvolution layer is used to enlarge the size at the end, so the original low resolution image can be directly input into the network, rather than requiring the upsizing by the bicubic method as in the previous SRCNN. The second is to change the feature dimension, use smaller convolution kernel and use more mapping layers. And thirdly, the mapping layers can be shared, and if models with different up-sampling multiplying powers need to be trained, only the final deconvolution layer needs to be finely adjusted. FSRCNN provides a greater speed increase compared to SRCNN because it does not require an operation to enlarge the size of the wafer outside the network, while replacing one large layer with a few small layers by adding a shrink layer and an expansion layer. FSRCNN can only define the last deconvolution layer during training, so the training speed is faster.

DCSCN is an efficient, fast single-image super-resolution (SISR) model using a Deep convolutional neural network (Deep CNN). Deep CNN has shown significant reconstruction performance on SR. The current trend is to use deeper CNN layers to improve performance. However, the deep model requires more computing resources and is not suitable for mobile, tablet and internet of things devices. DCSCN achieves the most advanced reconstruction performance with a deep CNN with residual, skipped connections and net-in-net (DCSCN), reducing the computational cost by at least a factor of 10. Meanwhile, the number of layers of each CNN and the number of filters are optimized, and the calculation cost is greatly reduced. Thus, the algorithm not only achieves very advanced performance, but also achieves faster, more efficient computations.

In the LapSRN algorithm, a Laplacian pyramid super-resolution network (LapSRN) is proposed, the LapSRN takes an LR image as input and gradually predicts subband residual errors in a coarse-to-fine mode; at each level, firstly, a cascade convolution layer is applied to extract a feature map; then using the transposed convolutional layer to up-sample the feature map to a finer level; finally, the convolutional layer is used to predict the sub-band residual. Since the LapSRN does not need bicubic interpolation as a preprocessing step, the computational complexity is greatly reduced. And, the LapSRN can be trained under deep supervision by using a strong Charbonnier loss function, so that the trained LapSRN can realize high-quality reconstruction. Furthermore, the laprn network generates multi-scale predictions in one feed-forward pass through progressive reconstruction, thereby facilitating resource-aware applications. A large number of quantitative and qualitative assessments of the reference data set indicate that the lapssrn algorithm is superior to the most advanced methods in terms of speed and accuracy. Therefore, the lapssrn algorithm has the following advantages: the accuracy is high, the LapSRN improves the reconstruction quality by using Charbonier loss, and can reduce artifacts; the method is high in speed, and can perform super-resolution reconstruction in real time on most test sets as the FSRCNN; and (4) gradually reconstructing, wherein the model uses a Laplacian pyramid structure, and a plurality of intermediate results are generated in the process of one-time prediction and can be used as the over-scoring results of different multiples.

In the super-resolution reconstruction algorithm, the direction of the texture is judged based on the texture of the image in the traditional super-resolution reconstruction algorithm, and enhancement is performed on the basis. And the neural network structure based on machine learning can better learn the characteristics of residual errors between low-resolution and high-resolution images, repair distortion and restore the resolution to a higher first-grade resolution. The super-resolution technology based on machine learning, such as SR-LUT, FSRCNN, LapSRN or DCSCN network models, has incomparable advantages in super-resolution effect.

Step S250: and if the first area does not meet the preset condition, determining an area except the first area in the image to be displayed as a fourth area.

In the embodiment of the present application, if the head-mounted display device determines that the first region does not satisfy the preset condition, it indicates that the probability of paying attention to the region adjacent to the first region is low, and therefore, it is not necessary to further determine the region adjacent to the first region, that is, it is not necessary to further divide the image to be displayed. At this time, the head-mounted display device may directly acquire a region other than the first region in the image to be displayed, and take it as a fourth region.

Step S260: respectively performing super-resolution reconstruction on the first region and the fourth region in the image to be displayed, wherein the image quality of the first region after super-resolution reconstruction is higher than that of the fourth region after super-resolution reconstruction.

In the embodiment of the present application, if the head-mounted display device only segments the first region and the fourth region from the image to be displayed, different super-resolution reconstructions can be performed for the first region and the fourth region, and the image quality of the first region after the super-resolution reconstruction is higher than that of the fourth region after the super-resolution reconstruction. As can be understood, the super-resolution reconstruction algorithm has a positive correlation between the improvement of the image quality and the computation amount when processing the image, and therefore, when the head-mounted display device performs the super-resolution reconstruction on the first region and the fourth region, the computation amount consumed by the first region and the fourth region is different. In addition, because the first region is the region in which the image to be displayed corresponds to the gazing point position of the display region, and the fourth region is the region other than the first region (the region in which the user does not pay attention with a high probability), after the super-resolution reconstruction is performed on the first region and the fourth region, the improvement degree of the image quality of the region in which the user pays attention to the first region and the fourth region is high, so that the display effect of the image to be displayed viewed by the user can be ensured, and because the improvement degrees of the image quality of different regions are different, rather than performing the same super-resolution reconstruction on the image to be displayed, the calculation amount can be reduced, the calculation resource of the head-mounted display device can be saved, and the display blockage can be avoided. The method for performing different super-resolution reconstruction on the first region and the fourth region by the head-mounted display device can refer to the above-mentioned method for performing different super-resolution reconstruction on the first region and the third region, that is, the fourth region can be used as the above-mentioned third region, and only the highest image quality corresponding to the first region needs to be ensured, for example, the super-resolution reconstruction algorithm with the highest PSNR improvement can be adopted to perform the super-resolution reconstruction on the first region, and the super-resolution reconstruction algorithm with the smaller PSNR improvement is adopted for the fourth region to perform the super-resolution reconstruction, so that the computation workload of the head-mounted display device can be further reduced.

Step S270: and displaying the image to be displayed after the super-resolution reconstruction.

In the embodiment of the present application, step S270 may refer to the contents of other embodiments, which are not described herein again.

The image display method provided by the embodiment of the application can realize that different super-resolution reconstruction is carried out on different regions in the image to be displayed by determining the region of interest corresponding to human eyes in the image to be displayed and taking the region of interest as a first region, then determining the segmentation mode of the image to be displayed based on the first region, further segmenting the image to be displayed and then carrying out different super-resolution reconstruction on different regions in the image to be displayed, thereby ensuring the display quality of the content watched by a user, reducing the processing amount of the head-mounted display device, avoiding display delay and blockage, and further improving the display effect of the head-mounted display device.

Referring to fig. 8, fig. 8 is a flowchart illustrating an image display method according to another embodiment of the present application. The image display method is applied to the head-mounted display device. As will be described in detail with respect to the flow shown in fig. 8, the image display method may specifically include the following steps:

step S310: and acquiring the current gazing point position of the human eyes.

In this embodiment of the application, the head-mounted display device may acquire the gazing Point position based on a principle of a gazing Point Rendering (gaze Rendering) technology. The gaze Point Rendering (visualization) technology is a selective image Rendering technology, and can select a gaze area of a human eye to perform full-resolution Rendering based on an eyeball tracking technology, and perform fuzzy Rendering in an area other than the gaze area of the human eye, so as to display an image with a clear gaze area and a fuzzy non-gaze area. The gaze point rendering technology achieves the purpose of saving the amount of transmitted data by rendering only the image of the gaze area, thereby saving computer computing resources and reducing power consumption.

In some embodiments, the head-mounted display device may acquire an eye image of the human eye using an eye movement (Eyetracking) tracking technique and determine a gaze direction and a gaze point of the human eye using the eye image and a gaze estimation technique. The eye tracking technology is also called eyeball tracking technology, and is a technology for analyzing the movement information of eyeballs by collecting eyeball images of the eyes of a person and determining the position of a fixation point on a screen which is watched by the eyes of the person currently based on the movement information of the eyeballs.

In one possible embodiment, an eye tracking device, such as a camera or the like, may be provided in the head-mounted display device, and the camera may be used to capture an eye image of a human eye, and in particular, the eye image may be obtained by directing near-infrared light to a pupil of the eye to cause detectable reflected light in the pupil and a cornea. The detectable reflected light forms a vector with the cornea and the Pupil, and the vector is tracked and photographed by the Pupil camera, and the vector is optical tracking of corneal reflection, which is called Pupil-corneal reflection (Pupil-CR). By continuously recording corneal reflections of the human eye; and then, an integral image of eyeball movement can be obtained by utilizing an image processing technology, and the aim of tracking the sight can be fulfilled after data processing, so that the position of the gaze of human eyes at a certain moment can be determined.

Optionally, the head-mounted display device may be provided with eye tracking devices corresponding to the left eye and the right eye of the human eye, respectively. As one mode, the head-mounted display device may detect a gaze point position corresponding to a left eye by using an eye tracking device corresponding to the left eye, as a current gaze point position of the human eye.

Alternatively, the eye tracking device for the right eye may be used to detect the gaze point position for the right eye as the current gaze point position of the human eye.

As another mode, the eye tracking device corresponding to the left eye and the eye tracking device corresponding to the right eye may be used to detect the gaze position corresponding to the left eye and the gaze position corresponding to the right eye, and then the current gaze position of the human eye may be determined according to the gaze position corresponding to the left eye and the gaze position corresponding to the right eye. For example, the position coordinates of the gazing point position corresponding to the left eye and the position coordinates of the gazing point position corresponding to the right eye may be averaged to obtain new position coordinates as the current gazing point position of the human eye; for another example, the position coordinate of the gaze point position corresponding to the left eye and the position coordinate of the gaze point position corresponding to the right eye may be weighted and summed according to the weight corresponding to the left eye and the weight corresponding to the right eye, so as to obtain a new position coordinate as the current gaze point position of the human eye, where the sum of the weight corresponding to the left eye and the weight corresponding to the right eye is 1, and the weight corresponding to the left eye and the weight corresponding to the right eye may be preset, for example, according to the eyesight corresponding to the left eye and the eyesight corresponding to the right eye.

As still another way, the head-mounted display device may also detect a dominant eye among the left and right eyes of the human eye; and then, the fixation point position corresponding to the dominant eye is used as the current fixation point position of the human eye. Therefore, the fixation point position for performing super-resolution reconstruction on the image to be displayed subsequently is determined based on the dominant eye, the image to be displayed can be processed more accurately, and the display effect is improved. The head-mounted display equipment can determine the dominant eye by displaying the test image and according to dominant eye confirmation information input by a user according to the test image; the head-mounted display device may also obtain the dominant eye set by the user in advance, or determine the dominant eye of the user according to the user portrait of the user, and of course, the specific way of obtaining the dominant eye by the head-mounted display device may not be limited.

Step S320: and acquiring a region where the gazing point position is located in a display region of the head-mounted display equipment as a gazing region.

In the embodiment of the application, when the head-mounted display device determines the first area based on the gazing point position, an area where the gazing point position is located in the display area of the head-mounted display device may be acquired as the gazing area. When the head-mounted display equipment determines the fixation point position, the fixation point position coordinates in the display area can be obtained; based on the gaze point position coordinates, a region including the gaze point position coordinates may be acquired from the display region as a gaze region.

In some embodiments, the head-mounted display device may acquire a region of a preset size centered on the gaze point location from the display region as the gaze region. That is, in the gazing area, the gazing point position is located at the center of the gazing area, and the size of the gazing area is a preset size. The specific size of the preset size may not be limited, and may be, for example, 64 × 64 pixels, 32 × 32 pixels, and 16 × 16 pixels.

Step S330: and determining a region of interest in the image to be displayed as a first region based on the gazing region.

In some embodiments, since the head-mounted display device needs to map an image into the display screen when displaying the image, an area corresponding to the gazing area may be determined from the image to be displayed based on a mapping relationship between the display content and the display area, so as to obtain an area of interest corresponding to human eyes in the image to be displayed, and the area of interest may be used as the first area.

Step S340: and determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed.

In some embodiments, after the first region is determined, a region of the target size range may be taken from the regions adjacent to the first region as the second region; and then acquiring the areas except the obtained first area and the second area from the image to be displayed, thereby obtaining a third area. The target size range may be 64 × 64 pixels or 32 × 32 pixels around the first region, and the specific size range may not be limited.

Step S350: respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

In some embodiments, the sizes of the first region, the second region, and the third region sequentially increase, and since the degree of improvement of the image quality by the super-resolution reconstruction corresponding to the first region, the degree of improvement of the image quality by the super-resolution reconstruction corresponding to the second region, and the degree of improvement of the image quality by the super-resolution reconstruction corresponding to the third region sequentially decrease, the amount of computation consumed by the first region, the second region, and the third region sequentially decreases, and thus the amount of computation of the head-mounted display device can be further reduced.

For example, the size relationship among the first region, the second region, and the third region, the size relationship among the degree of improvement in image quality by the SR algorithm, and the size relationship among the amounts of calculation required are shown in the following table,

step S360: and displaying the image to be displayed after the super-resolution reconstruction.

In the embodiment of the present application, step S350 and step S360 may refer to the contents of other embodiments, and are not described herein again.

The image display method provided by the embodiment of the application can realize that the region corresponding to the gazing point position is divided from the image to be displayed to be used as the first region according to the gazing point position of the current gazing point position of human eyes and the gazing region in the display region according to the gazing point position, then the image to be displayed is further divided based on the first region, and then different super-resolution reconstruction is carried out on different regions in the image to be displayed, so that the display quality of the content watched by a user is ensured, the processing amount of the head-mounted display equipment is reduced, the display delay and the blockage are avoided, and the display effect of the head-mounted display equipment is improved.

Referring to fig. 9, fig. 9 is a schematic flowchart illustrating an image display method according to still another embodiment of the present application. The image display method is applied to the head-mounted display equipment, the head-mounted display equipment comprises a first display unit and a second display unit, and the image to be displayed comprises a first display image and a second display image which respectively correspond to eyes of human eyes. As will be described in detail with respect to the flow shown in fig. 9, the image display method may specifically include the following steps:

step S410: and acquiring a region of interest corresponding to the human eye in the image to be displayed as a first region.

Step S420: and determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed.

Step S430: respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced.

In this embodiment, the first display unit and the second display unit may be respectively configured to display a left-eye image and a right-eye image, where the first display image corresponds to the first display unit and the second display image corresponds to the second display unit. That is, the first display unit is used for displaying a first display image corresponding to a left eye, and the second display unit is used for displaying a second display image corresponding to a right eye. Therefore, after the first display image and the second display image are subsequently displayed, the user can realize the corresponding display effect according to the observed first display image and the observed second display image through brain fusion. For example, if the first display image and the second display image are display images corresponding to three-dimensional content, a parallax exists between the first display image and the second display image, and after the first display image and the second display image are displayed, a user can obtain a three-dimensional display effect through brain fusion according to the first display image and the second display image observed.

When the head-mounted display device processes the image to be displayed, the processes of step S410 to step S430 may be performed for the first display image and the second display image respectively, so as to realize super-resolution reconstruction of the first display image and the second display image. That is, a region of interest corresponding to the human eye in the first display image may be determined as the first region; determining a second area and a third area in the first display image based on the first area; then, different super-resolution reconstructions are respectively carried out on the first region, the second region and the third region in the first display image. In addition, a region of interest corresponding to the human eye in the second display image can be determined as a second region; determining a second area and a third area in a second display image based on the first area; then, different super-resolution reconstructions are respectively carried out on the first region, the second region and the third region in the second display image.

In some embodiments, the head-mounted display device may acquire a dominant eye of a human eye when performing different super-resolution reconstructions on a first region, a second region, and a third region of a first display image and a second display image with respect to the first display image and the second display image; and based on the dominant eye, performing different super-resolution reconstruction on the first region, the second region and the third region in the first display image, and performing different super-resolution reconstruction on the first region, the second region and the third region in the second display image, wherein the image quality of the first region, the second region and the third region in the same display image after the super-resolution reconstruction is sequentially reduced, and the image quality of the display image corresponding to the dominant eye is higher than that of the other display image.

The other display image is another image other than the display image corresponding to the dominant eye among the first display image and the second display image. The manner of obtaining the dominant eye may refer to the contents of the foregoing embodiments, and is not described herein again. As can be understood, because the dominant eye has stronger sensory ability, super-resolution reconstruction with better image quality improvement can be performed on the display image corresponding to the dominant eye, and super-resolution reconstruction with relatively lower image quality can be performed on the other display image.

Optionally, if the image quality is evaluated through the PSNR, for the first display image, after the super-resolution reconstruction, the PSNR of the first region, the PSNR of the second region, and the PSNR of the third region are respectively: for the second display image, after super-resolution reconstruction, the PSNR of the first region, the PSNR of the second region, and the PSNR of the third region are respectively: a fourth PSNR value, a fifth PSNR value, and a sixth PSNR value. Wherein, the first PSNR value, the second PSNR value and the third PSNR value are reduced in sequence, and the fourth PSNR value, the fifth PSNR value and the sixth PSNR value are reduced in sequence. And if the first display image is a display image corresponding to the primary eye, the first PSNR value is higher than the fourth PSNR value, the second PSNR value is higher than the fifth PSNR value, and the third PSNR value is higher than the sixth PSNR value. That is, for the first display image and the second display image, the PSNR value of the region in the display image corresponding to the leading eye is increased to a greater extent for the same region.

In some embodiments, when the head-mounted display device determines the first region, the second region, and the third region for the first display image and the second display image, respectively, for a display image corresponding to the dominant eye, the size of the first region may be larger than the size of the first region in another display image, the size of the second region may be larger than the size of the second region in another display image, and the size of the third region may be larger than the size of the third region in another display image. Thus, the calculation amount of the non-dominant-eye display image for super-resolution reconstruction is smaller than that of the dominant-eye display image, and the calculation amount of the head-mounted display device can be further reduced.

Step S440: and displaying the first display image after super-resolution reconstruction through the first display unit.

Step S450: and displaying the second display image after the super-resolution reconstruction through the second display unit.

In this embodiment of the application, after performing super-resolution reconstruction on the first region, the second region, and the third region in the first display image and the second display image, the first display image after super-resolution reconstruction may be displayed by the first display unit, and the second display image after super-resolution reconstruction may be displayed by the second display unit, so as to achieve a three-dimensional display effect.

The embodiments provided in the examples of the present application may be combined with other examples. For example, in the second embodiment, if the first region does not satisfy the preset condition, after the fourth region is determined, different super-resolution reconstructions are performed on the first region and the fourth region in the image to be displayed, the super-resolution reconstructions may be performed on the first region and the fourth region in the first display image and the first region and the fourth region in the second display image based on the dominant eye by acquiring the dominant eye, where image qualities of the first region and the fourth region in the same display image after the super-resolution reconstruction are sequentially reduced, and an image quality of a display image corresponding to the dominant eye is higher than an image quality of another display image after the super-resolution reconstruction is performed.

The image display method provided by the embodiment of the application can be used for segmenting a first region corresponding to a gazing point position respectively for a first display image and a second display image aiming at an interested region corresponding to human eyes, further segmenting the display images based on the first region, performing different super-resolution reconstruction aiming at different regions in the first display image and the second display image, and respectively displaying the first display image and the second display image after the super-resolution reconstruction through different display units, so that when three-dimensional display content is displayed, the display quality of the content watched by a user can be ensured, the processing amount of head-mounted display equipment is reduced, display delay and blockage are avoided, and the display effect of the head-mounted display equipment is improved.

Referring to fig. 10, a block diagram of an image display apparatus 400 according to an embodiment of the present disclosure is shown. The image display apparatus 400 is applied to a head-mounted display device. The image display apparatus 400 includes: a first determination module 410, a second determination module 420, an image processing module 430, and an image display module 440. The first determining module 410 is configured to obtain a region of interest corresponding to a human eye in an image to be displayed as a first region; the second determining module 420 is configured to determine, based on the first region, a second region and a third region in the image to be displayed, where the second region is adjacent to the first region, and the third region is a region of the image to be displayed other than the first region and the second region; the image processing module 430 is configured to perform super-resolution reconstruction on the first region, the second region, and the third region in the image to be displayed, respectively, where image qualities of the first region, the second region, and the third region after super-resolution reconstruction are sequentially reduced; the image display module 440 is configured to display the image to be displayed after super-resolution reconstruction.

In some embodiments, the second determining module 420 may be specifically configured to: and if the first area meets the preset condition, determining a second area and a third area in the image to be displayed based on the first area, wherein the preset condition is a condition when the probability that the area adjacent to the first area is watched is greater than the preset probability.

As a possible implementation, the image display apparatus 400 may further include a third determining module. The third determining module is used for determining an area except the first area in the image to be displayed as a fourth area if the first area does not meet the preset condition; the image processing module 430 may be further configured to perform super-resolution reconstruction on the first region and the fourth region in the image to be displayed, respectively, where an image quality of the first region after super-resolution reconstruction is higher than an image quality of the fourth region after super-resolution reconstruction.

As a possible implementation, the image display apparatus 400 may further include a region determination module. The region determination module may be configured to: judging whether a target object exists in the first area or not; if the target object exists, determining that the first area meets a preset condition; and if the target object does not exist, determining that the first area does not meet the preset condition.

Optionally, the target object includes: and at least one of a texture having a contrast greater than a preset contrast and a texture having a sharpness greater than a preset sharpness.

In some embodiments, the first determining module 410 may be specifically configured to: acquiring the current gazing point position of human eyes; acquiring a region where the gazing point position is located in a display region of the head-mounted display device as a gazing region; and determining a region of interest in the image to be displayed as a first region based on the gazing region.

As a possible implementation, the first determining module 410 may be further configured to: and acquiring a region with a preset size by taking the gazing point position as a center from the display region to serve as a gazing region.

In some embodiments, the head-mounted display device includes a first display unit and a second display unit, and the image to be displayed includes a first display image and a second display image respectively corresponding to both eyes of a human eye. The image display module 440 may be specifically configured to: displaying the first display image after super-resolution reconstruction through the first display unit; and displaying the second display image after the super-resolution reconstruction through the second display unit.

As a possible implementation, the image processing module 430 may be specifically configured to: acquiring a dominant eye in human eyes; respectively performing super-resolution reconstruction on the first region, the second region and the third region in the first display image and respectively performing super-resolution reconstruction on the first region, the second region and the third region in the second display image based on the dominant eye, wherein the image quality of the first region, the second region and the third region in the same display image after super-resolution reconstruction is sequentially reduced, and the image quality of the display image corresponding to the dominant eye is higher than that of the other display image.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other type of coupling.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

To sum up, the scheme that this application provided, wear display device when showing the image, through obtaining the region of interest that the people's eye corresponds in waiting to show the image as first region, again based on first region, confirm second region and third region in waiting to show the image, and the second region borders on with first region, the third region is except that first region and second region in waiting to show the image, then treats first region, second region and the third region in the image that shows respectively and carries out different super resolution and rebuilds, and the image quality of first region, second region and third region after the super resolution is rebuilt reduces in proper order, rebuild the back to the super resolution wait to show the image shows. Therefore, super-resolution reconstruction can be performed on different regions in the image to be displayed according to the region of interest of human eyes in the image to be displayed, so that the image quality of the different regions is improved to different degrees, the display quality of the content watched by a user is ensured, the processing amount of the head-mounted display equipment is reduced, the display delay and the blockage are avoided, and the display effect of the head-mounted display equipment is improved.

Referring to fig. 11, a block diagram of a head-mounted display device according to an embodiment of the present disclosure is shown. The head-mounted display device 100 may be AR glasses, AR helmet, VR glasses, VR helmet, MR glasses, MR helmet, or other display device capable of running an application. The head mounted display device 100 in the present application may include one or more of the following components: a processor 101, a memory 102, and one or more applications, wherein the one or more applications may be stored in the memory 102 and configured to be executed by the one or more processors 101, the one or more applications configured to perform a method as described in the aforementioned method embodiments.

Processor 101 may include one or more processing cores. The processor 101 connects various parts within the overall head mounted display device 100 using various interfaces and lines, performs various functions of the head mounted display device 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 102, and calling data stored in the memory 102. Alternatively, the processor 101 may be implemented in hardware using at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 101 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing display content; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 101, but may be implemented by a communication chip.

The Memory 102 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). The memory 102 may be used to store instructions, programs, code sets, or instruction sets. The memory 102 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The stored data area may also store data created by the head mounted display device 100 during use (e.g., phone book, audio-video data, chat log data), etc.

Referring to fig. 12, a block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable medium 800 has stored therein a program code that can be called by a processor to execute the method described in the above-described method embodiments.

The computer-readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 800 includes a non-volatile computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 to perform any of the method steps of the method described above. The program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. An image display method applied to a head-mounted display device, the method comprising:

acquiring an interested area corresponding to human eyes in an image to be displayed as a first area;

determining a second area and a third area in the image to be displayed based on the first area, wherein the second area is adjacent to the first area, and the third area is an area except the first area and the second area in the image to be displayed;

respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is reduced in sequence;

and displaying the image to be displayed after the super-resolution reconstruction.

2. The method according to claim 1, wherein the determining a second region and a third region in the image to be displayed based on the first region comprises:

and if the first area meets a preset condition, determining a second area and a third area in the image to be displayed based on the first area, wherein the preset condition is a condition when the probability that the area adjacent to the first area is watched is greater than a preset probability.

3. The method according to claim 2, wherein after the acquiring a region of interest corresponding to a human eye in the image to be displayed as the first region, the method further comprises:

if the first area does not meet the preset condition, determining an area except the first area in the image to be displayed as a fourth area;

respectively performing super-resolution reconstruction on the first region and the fourth region in the image to be displayed, wherein the image quality of the first region after super-resolution reconstruction is higher than that of the fourth region after super-resolution reconstruction.

4. The method according to claim 2, wherein before determining the second area and the third area in the image to be displayed based on the first area if the first area satisfies the predetermined condition, the method further comprises:

judging whether a target object exists in the first area or not;

if the target object exists, determining that the first area meets a preset condition;

and if the target object does not exist, determining that the first area does not meet the preset condition.

5. The method of claim 4, wherein the target object comprises:

and at least one of a texture having a contrast greater than a preset contrast and a texture having a sharpness greater than a preset sharpness.

6. The method according to claim 1, wherein the acquiring a region of interest corresponding to a human eye in the image to be displayed as the first region comprises:

acquiring the current gazing point position of human eyes;

acquiring a region where the gazing point position is located in a display region of the head-mounted display device as a gazing region;

and determining a region of interest in the image to be displayed as a first region based on the gazing region.

7. The method according to claim 6, wherein the obtaining, as the gaze area, an area of a display area of the head-mounted display device where the gaze point location is located comprises:

and acquiring a region with a preset size by taking the gazing point position as a center from the display region to serve as a gazing region.

8. The method according to any one of claims 1 to 7, wherein the head-mounted display device comprises a first display unit and a second display unit, the image to be displayed comprises a first display image and a second display image respectively corresponding to eyes of a human eye, and the displaying the image to be displayed after super-resolution reconstruction comprises:

displaying the first display image after super-resolution reconstruction through the first display unit;

and displaying the second display image after the super-resolution reconstruction through the second display unit.

9. The method according to claim 8, wherein the performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed respectively comprises:

acquiring a dominant eye in human eyes;

respectively performing super-resolution reconstruction on the first region, the second region and the third region in the first display image and respectively performing super-resolution reconstruction on the first region, the second region and the third region in the second display image based on the dominant eye, wherein the image quality of the first region, the second region and the third region in the same display image after super-resolution reconstruction is sequentially reduced, and the image quality of the display image corresponding to the dominant eye is higher than that of the other display image.

10. An image display apparatus, applied to a head-mounted display device, the apparatus comprising: a first determining module, a second determining module, an image processing module, and an image display module, wherein,

the first determining module is used for acquiring an interested area corresponding to human eyes in an image to be displayed as a first area;

the second determining module is configured to determine a second area and a third area in the image to be displayed based on the first area, where the second area is adjacent to the first area, and the third area is an area other than the first area and the second area in the image to be displayed;

the image processing module is used for respectively performing super-resolution reconstruction on the first region, the second region and the third region in the image to be displayed, wherein the image quality of the first region, the second region and the third region after the super-resolution reconstruction is sequentially reduced;

the image display module is used for displaying the image to be displayed after super-resolution reconstruction.

11. A head-mounted display device, comprising:

one or more processors;

a memory;

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-9.

12. A computer-readable storage medium, having stored thereon program code that can be invoked by a processor to perform the method according to any one of claims 1 to 9.