WO2021169853A1 - 一种显示方法、装置、终端设备及存储介质 - Google Patents

一种显示方法、装置、终端设备及存储介质 Download PDF

Info

Publication number
WO2021169853A1
WO2021169853A1 PCT/CN2021/076919 CN2021076919W WO2021169853A1 WO 2021169853 A1 WO2021169853 A1 WO 2021169853A1 CN 2021076919 W CN2021076919 W CN 2021076919W WO 2021169853 A1 WO2021169853 A1 WO 2021169853A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
depth plane
information
target
remaining
Prior art date
Application number
PCT/CN2021/076919
Other languages
English (en)
French (fr)
Inventor
路伟成
张朕
韦余伟
Original Assignee
北京七鑫易维信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京七鑫易维信息技术有限公司 filed Critical 北京七鑫易维信息技术有限公司
Priority to JP2022551634A priority Critical patent/JP2023515205A/ja
Publication of WO2021169853A1 publication Critical patent/WO2021169853A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment

Definitions

  • the embodiments of the present invention relate to the field of computer technology, and in particular, to a display method, device, terminal device, and storage medium.
  • VR technology is a computer simulation system that can create and experience a virtual world.
  • VR technology uses a computer to generate a virtual environment, which is an interactive three-dimensional dynamic view and entity that integrates multi-source information
  • the system simulation of behavior makes the user immersed in the environment.
  • Augmented Reality (AR) technology is a technology that ingeniously integrates virtual information with the real world. It uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing.
  • the generated text, image, 3D model, music, video and other virtual information are simulated and applied to the real world, and the two kinds of information complement each other, thus realizing the "enhancement" of the real world.
  • At least some embodiments of the present invention provide a display method, device, terminal device, and storage medium, which are used to enhance a user's sense of depth when viewing a current image.
  • a display method which includes: acquiring the gaze information of the user on the current image; determining the corresponding target depth plane based on the gaze information; adjusting the display parameters of the remaining depth planes, The display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are depth planes other than the target depth plane on the current image.
  • the gaze information includes gaze point information; based on the gaze information, determining the corresponding target depth plane includes: determining the target object corresponding to the gaze point information on the current image; The depth plane where the target object is located is determined as the target depth plane.
  • the display parameter includes a blur radius.
  • the blur radius of each remaining depth plane is proportional to the distance of the remaining depth plane from the target depth plane.
  • the distance between the remaining depth plane and the target depth plane is determined by the difference between the distance information of the remaining depth plane and the distance information of the target depth plane.
  • the method further includes: determining the depth plane and corresponding distance information contained in each frame of the virtual reality or augmented reality video, and the current image is the current in the virtual reality or augmented reality video.
  • the distance information is absolute distance information between the corresponding depth plane and the user.
  • determining the depth plane contained in each frame of the virtual reality or augmented reality video includes: acquiring a target image in the virtual reality or augmented reality video frame by frame; acquiring the target image The depth information of the object included in the above; the target image is segmented based on each of the depth information to obtain at least one depth plane, and the distance information of each depth plane obtained by the segmentation is determined according to the depth information.
  • the depth information of the objects included in the same depth plane is the same.
  • a display device including: an acquisition module, configured to acquire gaze information of the user on the current image; and a determining module, configured to determine the corresponding target depth based on the gaze information Plane; the adjustment module is set to adjust the display parameters of the remaining depth planes, the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are on the current image A depth plane other than the target depth plane.
  • a terminal device including: one or more processors; a storage device configured to store one or more programs; the one or more programs are controlled by the one or more Execution by multiple processors, so that the one or more processors implement the method provided in the embodiment of the present invention.
  • a computer-readable storage medium is also provided, on which a computer program is stored, and when the program is executed by a processor, the method provided in the embodiment of the present invention is implemented.
  • At least some embodiments of the present invention provide a display method, device, terminal device, and storage medium. First, obtain the user's gaze information on the current image; then determine the corresponding target depth plane based on the gaze information; finally adjust the remaining depths The display parameters of the plane, the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are the depth planes on the current image other than the target depth plane . With the above technical solutions, the sense of depth of the current image viewed by the user can be enhanced.
  • Fig. 1 is a flowchart of a display method according to the first embodiment of the present invention.
  • Fig. 2 is a flowchart of a display method according to the second embodiment of the present invention.
  • Fig. 3a is a schematic diagram of an image preprocessing effect according to the second embodiment of the present invention.
  • Fig. 3b is a schematic diagram of a scene including multiple depth planes according to the second embodiment of the present invention.
  • FIG. 4 is a schematic structural diagram of a display device according to the third embodiment of the present invention.
  • Fig. 5 is a schematic structural diagram of a terminal device according to the fourth embodiment of the present invention.
  • Fig. 1 is a flowchart of a display method according to the first embodiment of the present invention.
  • the method can be applied to the situation of enhancing the sense of depth of the image.
  • the method can be executed by a display device, wherein the device can be implemented by software and/or hardware.
  • the terminal device includes but is not limited to: a device capable of displaying a virtual reality scene, such as a VR device; or a device capable of displaying an augmented reality scene, such as an AR device.
  • the display method provided in this embodiment can be considered as a depth perception enhancement method for a three-dimensional virtual scene.
  • Depth perception refers to the process by which the visual system of the human eye judges the distance of different objects.
  • the sources of clues for the visual system's perception of depth can be divided into two categories.
  • One type is monocular clues, which can be obtained through the visual information of one eye.
  • the other type is binocular clues, which must have the cooperation of two eyes.
  • Focus and defocus are one of the main monocular cues for the visual system to perceive depth.
  • the image at the same depth plane around the object will be relatively clear (focus), while the images at different depth planes will be relatively blurred (out of focus).
  • the degree of blur is affected by the absolute difference between the depth planes. Influence of distance difference.
  • Binocular disparity is one of the main binocular cues for the visual system to perceive depth. The closer the object is to the observer, the greater the difference between the objects seen by the two eyes, which forms binocular parallax. The brain can use the measurement of this parallax to estimate the distance from the object to the eye.
  • the display method provided in this embodiment uses eye tracking technology to enhance the user's sense of depth when viewing images.
  • the eye tracking technology can estimate the gaze point through an image recognition algorithm with the aid of an eye tracker.
  • Eye tracking can also be called gaze tracking, which can estimate the line of sight and/or gaze point of the eye by measuring eye movement.
  • the line of sight can be understood as a three-dimensional vector
  • the gaze point can be understood as the two-dimensional coordinates of the above-mentioned three-dimensional vector on a certain plane, such as the plane being looked at.
  • the display method provided by this embodiment can realize eye tracking through the pupil-corneal reflection method in the optical recording method, or a method that is not based on eye images, for example, based on contact/non-contact sensors (such as electrodes, capacitors, etc.).
  • the sensor calculates the movement of the eye.
  • the optical recording method uses a camera or video camera to record the subject's eye movement, that is, obtains eye images that reflect the eye movement, and extracts eye features from the obtained eye images to establish a model for line of sight/gaze point estimation .
  • the eye features may include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, spot (also called Purkin spot) position, and the like.
  • the working principle of the pupil-corneal reflection method can be summarized as follows: Obtain the eye image; estimate the line of sight/gaze point based on the eye image.
  • the hardware requirements of the pupil-corneal reflection method are:
  • Light source generally infrared light source, because infrared light does not affect the vision of the eyes; and can be multiple infrared light sources arranged in a predetermined manner, such as a fringe shape and/or a line shape, etc.;
  • Image acquisition equipment such as infrared camera equipment, infrared image sensors, cameras or video cameras, etc.
  • the specific implementation of the pupil-corneal reflection method can be:
  • Eye image acquisition The light source shines on the eyes, and the image acquisition device takes pictures of the eyes.
  • the corresponding reflection point of the light source on the cornea is the light spot (also called Purkin spot), thereby acquiring the eye image with the light spot.
  • Sight line/gaze point estimation As the eyeball rotates, the relative position relationship between the pupil center and the light spot changes accordingly, and the correspondingly collected several eye images with light spots reflect this position change relationship.
  • the line of sight/gaze point estimation is performed according to the position change relationship.
  • the display method provided by this embodiment includes the following steps:
  • the scene of the present invention may be a scene where a user watches an augmented reality image, or a scene where a user watches a virtual reality video.
  • a scene where a user watches a virtual reality video through a VR device may be a scene where a user watches a virtual reality video through a VR device.
  • the user may be a person currently viewing images.
  • the current image may be the image that the user is currently looking at.
  • the gaze information can be understood as the information of the eyes when the user is gazing at the current image.
  • the gaze information includes, but is not limited to, line of sight information and point of gaze information.
  • the line of sight information may be information indicating the line of sight of the user, such as a direction.
  • the gaze point information may be information indicating the gaze point of the user, such as coordinates.
  • the gaze information can be obtained through a gaze tracking device, which can be installed on a device that displays the current image, such as a VR or AR device.
  • the present invention can obtain the user's gaze information on the current image through the pupil-corneal reflection method, and can also use other methods to obtain the user's gaze information.
  • the eye tracking device may be a MEMS microelectromechanical system, including a MEMS infrared scanning mirror and an infrared light source. , Infrared receiver; in another embodiment, the eye tracking device can also be a capacitive sensor, which detects eye movement through the capacitance value between the eyeball and the capacitor plate; in another embodiment, the eye tracking device It can also be a myoelectric current detector, for example, by placing electrodes on the bridge of the nose, forehead, ear or earlobe, and detecting eye movement through the detected myoelectric signal pattern. There is no limitation here.
  • S120 Determine a corresponding target depth plane based on the gaze information.
  • the target depth plane can be understood as the depth plane corresponding to the gaze information in the current image.
  • the target depth plane may be considered as the depth plane where the target object corresponding to the gaze point information on the current image is located.
  • the current image may include multiple objects, and each object is preset with object information, and the object information can be used to identify the object.
  • Object information includes position information and depth information, and depth information can be considered as information indicating the depth of the object in the current image.
  • Each depth information can correspond to a depth plane, so that each object can correspond to a depth plane.
  • the target depth plane can be considered as the depth plane of the object that the user is currently looking at.
  • the target object can be considered as the object that the user is currently looking at.
  • the gaze information can be matched with the position information in the object information of the object included in the current image, the object information corresponding to the gaze information can be determined, and the target depth plane can be determined based on the depth plane in the object information.
  • the gaze point information when the gaze information is gaze point information, the gaze point information may be compared with the position information in the object information of the object included in the current image, such as coordinate comparison.
  • the object whose position information is equal to the gaze point information or the deviation within the set range in the current image is taken as the target object, and the depth plane of the target object is taken as the target depth plane.
  • the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are the current image divided by the target depth The depth plane outside the plane.
  • the present invention can adjust the display parameters of the remaining depth planes after determining the target depth plane.
  • the number of the remaining depth planes may be at least one.
  • the size of the display parameters of each remaining depth plane after adjustment may be the same or different.
  • the display parameters can be considered as parameters that determine the display effect.
  • Display parameters include but are not limited to pixel value and blur radius.
  • Different display parameters can have different adjustment methods, which are not limited here, as long as the definition of the remaining depth planes is lower than the definition of the target depth plane.
  • the display parameters of the remaining depth planes can be determined based on the distance between the remaining depth planes and the target depth plane.
  • the display parameter as the blur radius as an example, the greater the distance between the remaining depth planes and the target depth plane, the larger the blur radius of the remaining depth planes; the smaller the distance between the remaining depth planes and the target depth plane, the blur radius of the remaining depth planes It can be smaller.
  • the specific value of the blur radius of the remaining depth planes is not limited here, as long as it is ensured that the distance between the remaining depth plane and the target depth plane is proportional to the blur radius.
  • the display parameter is the pixel value
  • the distance between the remaining depth plane and the target depth plane is inversely proportional to the pixel value.
  • the distance between the remaining depth planes and the target depth plane can be directly determined by depth analysis of the current image, or can be determined based on the absolute distance information between the remaining depth planes and the user and the absolute distance information between the target depth plane and the user.
  • the display parameters of the remaining depth planes are different from the display parameters of the target depth plane, which improves the sense of depth of the current image.
  • the display parameter includes a blur radius.
  • the blur radius is proportional to the blur degree of the image.
  • the present invention can be realized by using Gaussian blur algorithm when adjusting the blur radius.
  • the display method provided by this embodiment, first obtain the user's gaze information on the current image; then based on the gaze information, determine the corresponding target depth plane; finally adjust the display parameters of the remaining depth planes, the display of the remaining depth planes
  • the parameter is determined according to the distance between the remaining depth plane and the target depth plane, and the remaining depth plane is a depth plane other than the target depth plane on the current image.
  • the user's sense of depth in viewing the current image can be enhanced.
  • a modified embodiment of the above-mentioned embodiment is proposed. It should be noted here that, in order to make the description concise, only the differences from the above-mentioned embodiment are described in the modified embodiment.
  • the blur radius of each remaining depth plane is proportional to the distance of the remaining depth plane from the target depth plane. In the case where the blur radius of each remaining depth plane is proportional to the distance between the remaining depth planes and the target depth plane, it is ensured that the further the remaining depth planes away from the target depth plane are blurred, the three-dimensional sense of the current image is improved and A sense of depth.
  • the distance between the remaining depth plane and the target depth plane is determined by the difference between the distance information of the remaining depth plane and the distance information of the target depth plane.
  • the distance information of the remaining depth planes can be understood as the absolute distance information between the remaining depth planes and the user.
  • the distance information of the target depth plane can be understood as the absolute distance information between the target depth plane and the user.
  • the difference between the distance information of the remaining depth planes and the distance information of the target depth plane may be used as the distance between the remaining depth planes and the target depth plane.
  • Fig. 2 is a flowchart of a display method according to the second embodiment of the present invention.
  • the second embodiment is optimized on the basis of the above-mentioned embodiments.
  • the gaze information specifically includes gaze point information, and accordingly, determining the corresponding target depth plane based on the gaze information includes:
  • the depth plane where the target object is located is determined as the target depth plane. Since the object itself is not necessarily flat, but may also be three-dimensional, the depth plane of the three-dimensional object can be determined by the plane where the object is closest to the user as the depth plane; or the plane where the center of the object is located as the depth plane; and Or take any side of the three-dimensional object as the depth plane, and there is no restriction here.
  • the above method may further include: determining the depth plane and corresponding distance information contained in each frame of the virtual reality or augmented reality video, where the current image is the one in the virtual reality or augmented reality video In the currently displayed image, the distance information is absolute distance information between the corresponding depth plane and the user.
  • the display method provided by this embodiment includes the following steps:
  • S210 Determine the depth plane and corresponding distance information included in each frame of the virtual reality or augmented reality video, where the current image is the image currently displayed in the virtual reality or augmented reality video, and the distance information is corresponding Absolute distance information between the depth plane and the user.
  • the current image may be a frame of image in virtual reality or augmented reality video.
  • the present invention can process each frame of the virtual reality or augmented reality video to determine the object information contained in each image.
  • the object information may be pre-set information in the image, such as each image Included depth plane and corresponding distance information.
  • Virtual reality video can be considered as a video presented in virtual reality technology.
  • Augmented reality video can be considered as a video presented with augmented reality technology.
  • the depth plane contained in the image can be determined by the depth information of the objects contained in the image.
  • the depth information of each object can be obtained by processing the image, or the depth information of each object in the image obtained by the depth camera can be directly read. There is no limitation here, as long as the depth information of each object in the image can be read.
  • the plane corresponding to each different depth information can be used as a depth plane, so that the image is split to include multiple depth planes.
  • the distance information can be the absolute distance information between the depth plane and the user. How to determine the absolute distance between the depth plane and the user is not limited here.
  • the information can be determined according to the depth information of each depth plane and the size of the display device. For example, the distance information of each depth plane is determined based on the distance between the plane on which the current image is displayed by the display device and the user's eyes and the depth information of each depth plane.
  • the present invention can determine the target object corresponding to the gaze point information on the current image by means of coordinate comparison. For example, traverse each object in the current image, and use the object whose coordinates are the same as the gaze information or within a certain range as the target object.
  • S240 Determine the depth plane where the target object is located as the target depth plane.
  • the present invention can use the depth plane where the target object is located as the target depth plane, that is, the depth plane that the user is currently looking at.
  • the user In the development of three-dimensional virtual scenes (such as VR videos), it is necessary to make full use of the perception of depth by the visual system to create a stronger sense of three-dimensionality and depth.
  • the user In the existing three-dimensional virtual scenes, the user’s visual system mainly relies on binocular parallax to perceive depth. When observing distant objects in the scene, the visual axis is close to parallel and the binocular parallax is zero. This is a clue to the perception of depth. The source is useless. At this time, the user can only perceive the depth through experience such as the relative size and perspective of the object, which greatly affects the three-dimensional sense and the sense of depth of the three-dimensional virtual scene.
  • the user In the existing three-dimensional virtual scene, since the scene presents a fixed-focus image, the user (ie, the user) cannot obtain depth clues for focusing and out-of-focus at different depth planes. At this time, if the user is unable to perceive depth with binocular parallax due to problems such as the absolute distance of the object and the too small angle of view, it will seriously affect the experience of gaming and interaction in the three-dimensional virtual scene.
  • This embodiment preprocesses the scene screen by marking the absolute distance information of different depth planes in the three-dimensional virtual scene, and then obtains the user's gaze point information based on eye tracking technology, and obtains the absolute distance of the gaze point on the depth plane according to the position of the gaze point.
  • the distance information can provide users with depth clues about focusing and out-of-focus, effectively making up for the lack and lack of existing depth clues, and greatly enhancing the user's sense of three-dimensionality and depth in a three-dimensional virtual scene.
  • the display method provided in this embodiment may be aimed at depth perception of a three-dimensional virtual scene, and the method may include the following steps:
  • Step 1 Image preprocessing of 3D virtual scene
  • the image regions at different depth planes are segmented frame by frame. Then, according to the depth of the plane where the image is located, the absolute distance information is marked on each image area.
  • the depth information of the specific object in each area of the image can be included in the image in advance.
  • Fig. 3a is a schematic diagram of the image preprocessing effect according to the second embodiment of the present invention. Referring to Fig. 3a, after the image is segmented, the first object 1, the second object 2, and the third object 3 located in different depth planes are obtained. Among them, “first”, “second” and “third” are only used to distinguish the corresponding content, not to limit the order or interdependence.
  • Fig. 3b is a schematic diagram of a scene containing multiple depth planes according to the second embodiment of the present invention.
  • the distance information of the depth plane corresponding to the first object 1 is the absolute distance information a between the first object 1 and the user 4
  • the second The distance information of the depth plane corresponding to the object 2 is the absolute distance information b of the second object 2 and the user 4
  • the distance information of the depth plane corresponding to the third object 3 is the absolute distance information c of the third object 3 and the user 4. It can be seen from Fig. 3b that c>b>a, that is, the absolute distance between the third object 3 and the user 4 is the farthest, and the absolute distance between the first object 1 and the user 4 is the shortest.
  • the depth plane of the first object 1 is the target depth plane
  • the display parameters of the depth plane of the second object 2 can be based on the distance of the second object 2 from the target depth plane. adjust.
  • the display parameters of the depth plane of the third object 3 can be adjusted according to the distance of the third object 3 from the target depth plane. Because the distance between the depth plane of the second object 2 and the target depth plane is less than the distance between the depth plane of the third object 3 and the target depth plane, the adjustment size of the display parameters of the second object 2 is smaller than the adjustment size of the display parameters of the third object 3 Therefore, when the user looks at the first object 1, the second object 2 is clearer than the third object 3.
  • the clarity of the figure can be characterized based on the density of the filler.
  • the denser the filling represents the higher the definition, and the thinner the filling the lower the definition.
  • the distance between the depth plane of the first object 1 and the depth plane of the second object 2 is less than the distance between the depth plane of the first object 1 and the depth plane of the third object 3, so the first object 1 is clear when the user looks at the second object 2
  • the degree of clarity is higher than that of the first object 1 when the user looks at the third object 3.
  • Step 2 Obtaining gaze point information When the user experiences a three-dimensional virtual scene, the real-time gaze point information of the user can be obtained through the eye tracker, and then the depth plane of the image area being looked at is determined.
  • the eye tracker can be located on the VR device.
  • Step 3 Presenting the focusing and out-of-focus effects of different depth planes.
  • the real-time image of the three-dimensional virtual scene is focused on the depth plane where the user's gaze point is. At this time, in the three-dimensional virtual scene in front of the user, only the depth plane corresponding to the gaze object is clear. Objects in other depth planes appear different according to the absolute distance difference from the "gazing depth plane" The degree of fuzzy state.
  • the human visual system can obtain the depth cues of focus and out-of-focus when observing the three-dimensional virtual scene.
  • the three-dimensional virtual scene it provides users with depth cues for focusing and out-of-focus, which effectively compensates for the lack and lack of depth cues caused by the use of fixed-focus images in the existing scene, and greatly enhances the user’s three-dimensional sense of the three-dimensional virtual scene. A sense of depth.
  • the display method provided in this embodiment describes the operation of determining the target depth plane and the operation of determining the depth plane and the corresponding distance information. Using this method can enhance the three-dimensional sense and depth sense of virtual reality or augmented reality video.
  • determining the depth plane included in each frame of the image in the virtual reality or augmented reality video includes:
  • the target image is segmented based on each of the depth information to obtain at least one depth plane, and the distance information of each depth plane obtained by segmentation is determined according to the depth information.
  • images in the virtual reality or augmented reality video can be obtained frame by frame as the target image, and for each target image, the depth information of the objects included in the target image can be obtained, and each object can correspond to a piece of depth information.
  • the present invention can segment the target image based on each depth information to obtain at least one depth plane, and the number of depth planes can be determined based on the number of depth information. When the values of multiple pieces of depth information are the same, the number of multiple pieces of depth information may be determined as 1.
  • the target image is divided into multiple depth planes according to depth information, and the distance information of each depth plane can be determined by the depth information.
  • the distance information of the depth plane is determined by the difference of the depth information corresponding to the depth plane.
  • the depth information of the objects included in the same depth plane is the same.
  • Fig. 4 is a schematic structural diagram of a display device according to the third embodiment of the present invention.
  • the device can be used to enhance the image depth perception.
  • the device can be implemented by software and/or hardware, and is generally integrated on a terminal device.
  • the device includes: an acquisition module 31, a determination module 32, and an adjustment module 33; the acquisition module 31 is configured to acquire the user's gaze information on the current image; the determination module 32 is configured to be based on the gaze information, Determine the corresponding target depth plane; the adjustment module 33 is set to adjust the display parameters of the remaining depth planes, the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes Is a depth plane other than the target depth plane on the current image.
  • the device first obtains the user's gaze information on the current image through the obtaining module 31; secondly, the determination module 32 determines the corresponding target depth plane based on the gaze information; finally, the adjustment module 33 adjusts the remaining depth planes
  • the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are depth planes other than the target depth plane on the current image.
  • the display device provided in this embodiment can enhance the sense of depth of the current image viewed by the user.
  • the gaze information includes gaze point information; and determining the corresponding target depth plane based on the gaze information includes:
  • the display parameter includes a blur radius.
  • the blur radius of each remaining depth plane is proportional to the distance of the remaining depth plane from the target depth plane.
  • the distance between the remaining depth plane and the target depth plane is determined by the difference between the distance information of the remaining depth plane and the distance information of the target depth plane.
  • the device further includes: an information determination module configured to:
  • the information determining module is set to:
  • the target image is segmented based on each of the depth information to obtain at least one depth plane, and the distance information of each depth plane obtained by the segmentation is determined according to the depth information.
  • the depth information of the objects included in the same depth plane is the same.
  • the above-mentioned display device can execute the display method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
  • Fig. 5 is a schematic structural diagram of a terminal device according to the fourth embodiment of the present invention.
  • the terminal device provided by this embodiment includes: one or more processors 41 and a storage device 42; the processor 41 in the terminal device may be one or more, and in FIG. 41 as an example;
  • the storage device 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement any one of the embodiments of the present invention. The method described in the item.
  • the terminal device may further include: an input device 43 and an output device 44.
  • the processor 41, the storage device 42, the input device 43, and the output device 44 in the terminal device may be connected through a bus or other methods.
  • the connection through a bus is taken as an example.
  • the storage device 42 in the terminal device is used as a computer-readable storage medium and can be used to store one or more programs.
  • the programs can be software programs, computer-executable programs, and modules, as described in the first or second embodiment of the present invention.
  • the program instructions/modules corresponding to the provided method include: an acquisition module 31, a determination module 32, and an adjustment module 33).
  • the processor 41 executes various functional applications and data processing of the terminal device by running the software programs, instructions, and modules stored in the storage device 42, that is, implements the display method in the foregoing method embodiment.
  • the storage device 42 may include a storage program area and a storage data area.
  • the storage program area may store an operating system and an application program required by at least one function; the storage data area may store data created according to the use of the terminal device, and the like.
  • the storage device 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the storage device 42 may further include a memory remotely provided with respect to the processor 41, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the input device 43 may be configured to receive input digital or character information, and to generate key signal input related to user settings and function control of the terminal device.
  • the output device 44 may include a display device such as a display screen. Moreover, when one or more programs included in the foregoing terminal device are executed by the one or more processors 41, the programs perform the following operations:
  • the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are on the current image except the target depth plane The depth plane.
  • the fifth embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, it is used to execute the display method provided by the present invention, and the method includes:
  • the display parameters of the remaining depth planes are determined according to the distance between the remaining depth planes and the target depth plane, and the remaining depth planes are on the current image except the target depth plane The depth plane.
  • the program when executed by the processor, it may also be used to execute the display method provided in any embodiment of the present invention.
  • the computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above.
  • Computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (Read Only Memory, ROM), Erasable Programmable Read Only Memory (EPROM), flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above .
  • the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • the computer program code used to perform the operations of the present invention can be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to Connect via the Internet).
  • LAN local area network
  • WAN wide area network

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Processing Or Creating Images (AREA)
  • Position Input By Displaying (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种显示方法、装置、终端设备及存储介质。所述方法包括：获取用户在当前图像上的注视信息（110）；基于所述注视信息，确定对应的目标深度平面（120）；调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面（130）。利用该方法能够增强用户观看的当前图像的深度感。

Description

一种显示方法、装置、终端设备及存储介质 技术领域
本发明实施例涉及计算机技术领域，尤其涉及一种显示方法、装置、终端设备及存储介质。
背景技术
虚拟现实(Virtual Reality，VR)技术是一种可以创建和体验虚拟世界的计算机仿真系统，VR技术利用计算机生成一种虚拟环境，是一种多源信息融合的交互式的三维动态视景和实体行为的系统仿真，使用户沉浸到该环境中。
增强现实(Augmented Reality，AR)技术是一种将虚拟信息与真实世界巧妙融合的技术，广泛运用了多媒体、三维建模、实时跟踪及注册、智能交互、传感等多种技术手段，将计算机生成的文字、图像、三维模型、音乐、视频等虚拟信息模拟仿真后，应用到真实世界中，两种信息互为补充，从而实现对真实世界的“增强”。
在利用VR和AR技术进行虚拟场景开发的过程中，需要充分利用用户的视觉系统对深度的感知来营造更强的立体感与深度感。然而，在某些场景(如视场角过小的场景或包含较远物体的场景)下仅凭用户的视觉系统无法实现虚拟场景中的立体感与深度感。
发明内容
本发明至少部分实施例提供了一种显示方法、装置、终端设备及存储介质，利用该方法增强用户观看当前图像的深度感。
在本发明其中一实施例中提供了一种显示方法，包括：获取用户在当前图像上的注视信息；基于所述注视信息，确定对应的目标深度平面；调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
在一个可选实施例中，所述注视信息包括注视点信息；基于所述注视信息，确定对应的目标深度平面包括：确定所述注视点信息在所述当前图像上对应的目标物体；将所述目标物体所在的深度平面确定为目标深度平面。
在一个可选实施例中，所述显示参数包括模糊半径。
在一个可选实施例中，在所述其余深度平面的个数为至少两个的情况下，每个其余深度平面的模糊半径与该其余深度平面距所述目标深度平面的距离成正比。
在一个可选实施例中，所述其余深度平面与所述目标深度平面的距离由所述其余深度平面的距离信息和所述目标深度平面的距离信息的差值确定。
在一个可选实施例中，该方法还包括：确定虚拟现实或增强现实视频中每帧图像所包含的深度平面及对应的距离信息，所述当前图像为所述虚拟现实或增强现实视频中当前显示的图像，所述距离信息为对应的深度平面与所述用户的绝对距离信息。
在一个可选实施例中，确定所述虚拟现实或增强现实视频中每帧图像所包含的深度平面，包括：逐帧获取所述虚拟现实或增强现实视频中的目标图像；获取所述目标图像上所包括物体的深度信息；基于各所述深度信息将所述目标图像进行分割，得到至少一个深度平面，分割得到的各深度平面的距离信息根据所述深度信息确定。
在一个可选实施例中，同一深度平面所包括物体的深度信息相同。
在本发明其中一实施例中，还提供了一种显示装置，包括：获取模块，设置为获取用户在当前图像上的注视信息；确定模块，设置为基于所述注视信息，确定对应的目标深度平面；调节模块，设置为调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
在本发明其中一实施例中，还提供了一种终端设备，包括：一个或多个处理器；存储装置，设置为存储一个或多个程序；所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现本发明实施例提供的方法。
在本发明其中一实施例中，还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本发明实施例提供的方法。
本发明至少部分实施例提供了一种显示方法、装置、终端设备及存储介质，首先获取用户在当前图像上的注视信息；然后基于所述注视信息，确定对应的目标深度平面；最后调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。利用上述技术方案，能够增强用户观看的当前图像的深度感。
附图说明
图1是根据本发明实施例一的一种显示方法的流程图。
图2是根据本发明实施例二的一种显示方法的流程图。
图3a是根据本发明实施例二的图像预处理效果的示意图。
图3b是根据本发明实施例二的包含多个深度平面场景的示意图。
图4是根据本发明实施例三的一种显示装置的结构示意图。
图5是根据本发明实施例四的一种终端设备的结构示意图。
具体实施方式
下面结合附图和实施例对本发明作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本发明，而非对本发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本发明相关的部分而非全部结构。
在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将各项操作(或步骤)描述成顺序的处理，但是其中的许多操作可以被并行地、并发地或者同时实施。此外，各项操作的顺序可以被重新安排。当其操作完成时所述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。所述处理可以对应于方法、函数、规程、子例程、子程序等等。此外，在不冲突的情况下，本发明中的实施例及实施例中的特征可以相互组合。
本发明使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”。
实施例一
图1是根据本发明实施例一的一种显示方法的流程图，该方法可适用于提升图像的深度感的情况，该方法可以由显示装置来执行，其中该装置可由软件和/或硬件实现，并一般集成在终端设备上，在本实施例中终端设备包括但不限于：能够实现虚拟现实场景显示的设备，如VR设备；或者能够实现增强现实场景显示的设备，如AR设备。
该实施例所提供的显示方法可以认为是一种三维虚拟场景的深度感知增强方法，深度感知指的是人眼的视觉系统判断不同物体远近的过程。一般来讲，视觉系统感知深度的线索来源可以分为两大类。一类是单眼线索，这类线索通过一只眼睛的视觉信息就可以获得。另一类是双眼线索，必须有两只眼睛的配合。
对焦和失焦是视觉系统感知深度的主要单眼线索之一。当观察者注视某一物体时，该物体周边处于同一深度平面的画面会相对清晰(对焦)，而处于不同深度平面的画面 则会相对模糊(失焦)，模糊的程度受深度平面间的绝对距离差值影响。
双眼视差是视觉系统感知深度的主要双眼线索之一。物体离观察者越近，两只眼睛所看到的物体差别也越大，这就形成了双眼视差。大脑可以利用对这种视差的测量，估计出物体到眼睛的距离。
该实施例所提供的显示方法利用眼动追踪技术增强用户观看图像时的深度感，眼动追踪技术可以借助眼动仪通过图像识别算法估算注视点。眼动追踪也可称为视线追踪，可以通过测量眼睛运动情况来估计眼睛的视线和/或注视点。视线可以理解为是一个三维矢量，注视点可以理解为上述三维矢量在某个平面，如所注视的平面上的二维坐标。
该实施例所提供的显示方法可以通过该光学记录法中的瞳孔-角膜反射法实现眼动追踪，也可以不基于眼部图像的方法，例如基于接触/非接触式的传感器(例如电极、电容传感器)推算眼睛的运动。
光学记录法即用照相机或摄像机记录被试者的眼睛运动情况，即获取反映眼睛运动的眼部图像，以及从获取到的眼部图像中提取眼部特征用于建立视线/注视点估计的模型。其中，眼部特征可以包括：瞳孔位置、瞳孔形状、虹膜位置、虹膜形状、眼皮位置、眼角位置、光斑(也称为普尔钦斑)位置等。
瞳孔-角膜反射法的工作原理可以概括为：获取眼部图像；根据眼部图像估计视线/注视点。
瞳孔-角膜反射法的硬件要求为：
光源：一般为红外光源，因为红外光线不会影响眼睛的视觉；并且可以为多个红外光源，以预定的方式排列，例如品字形和/或一字形等；
图像采集设备：例如红外摄像设备、红外图像传感器、照相机或摄像机等。
瞳孔-角膜反射法的具体实施可以为：
眼部图像获取：光源照向眼睛，由图像采集设备对眼部进行拍摄，相应拍摄光源在角膜上的反射点即光斑(也称普尔钦斑)，由此获取带有光斑的眼部图像。
视线/注视点估计：随着眼球转动时，瞳孔中心与光斑的相对位置关系随之发生变化，相应采集到的带有光斑的若干眼部图像反映出这样的位置变化关系。
根据所述位置变化关系进行视线/注视点估计。
如图1所示，该实施例所提供的显示方法包括如下步骤：
S110、获取用户在当前图像上的注视信息。
本发明的场景可以为用户进行增强现实图像观看的场景，或者用户对虚拟现实视频进行观看的场景。如用户通过VR设备进行虚拟现实视频进行观看的场景。
在本实施例中，用户可以为当前进行图像观看的人。当前图像可以为用户当前注视的图像。注视信息可以理解为表示用户注视当前图像时眼部的信息。注视信息包括但不限于视线信息和注视点信息，视线信息可以为表示用户视线的信息，如方向。注视点信息可以为表示用户注视点的信息，如坐标。注视信息可以通过视线追踪设备获取，视线追踪设备可以安装在显示当前图像的设备，如VR或者AR设备。
本发明可以通过瞳孔-角膜反射法获取用户在当前图像上的注视信息，也可以使用其他方法获取用户的注视信息，例如眼球追踪装置可以是MEMS微机电系统，包括MEMS红外扫描反射镜、红外光源、红外接收器；在其他另一个实施例中，眼球追踪装置还可以是电容传感器，其通过眼球与电容极板之间的电容值来检测眼球运动；在其他又一个实施例中，眼球追踪装置还可以是肌电流检测器，例如通过在鼻梁、额头、耳朵或耳垂处放置电极，通过检测的肌电流信号模式来检测眼球运动。此处不作限定。
S120、基于所述注视信息，确定对应的目标深度平面。
目标深度平面可以理解为当前图像中注视信息所对应的深度平面。示例性的，在注视信息为注视点信息的情况下，目标深度平面可以认为是注视点信息在当前图像上所对应的目标物体所在的深度平面。
可以理解的是，当前图像中可以包括多个物体，每个物体预设有物体信息，物体信息可以用于标识该物体。物体信息包括位置信息和深度信息，深度信息可以认为是表示物体在当前图像中深度的信息。每个深度信息可以对应有一个深度平面，从而，每个物体可以对应一个深度平面。目标深度平面可以认为是用户当前所注视的物体的深度平面。目标物体可以认为是用户当前所注视的物体。
在确定目标深度平面时，可以将注视信息与当前图像中所包括物体的物体信息中的位置信息进行匹配，确定对应注视信息的物体信息，并基于物体信息中的深度平面，确定目标深度平面。
在一个可选实施例中，在注视信息为注视点信息的情况下，可以将注视点信息与当前图像中所包括物体的物体信息中的位置信息进行比较，如进行坐标比对。将当前图像中位置信息等于注视点信息或者偏差在设定范围的物体作为目标物体，并将该目标物体的深度平面作为目标深度平面。
S130、调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
为了提升当前图像的深度感，本发明确定目标深度平面后，可以调节其余深度平面的显示参数。其余深度平面的个数可以为至少一个，在包括至少两个其余深度平面的情况下，调节后的每个其余深度平面的显示参数的大小可以相同，也可以不同。
显示参数可以认为是决定显示效果的参数。显示参数包括但不限于像素值和模糊半径。不同的显示参数可以具有不同的调节手段，此处不作限定，只要满足其余深度平面的清晰度低于目标深度平面的清晰度即可。
具体的，可以基于其余深度平面与目标深度平面的距离，确定其余深度平面的显示参数。以显示参数为模糊半径时为例，其余深度平面与目标深度平面的距离越大，其余深度平面的模糊半径可以越大；其余深度平面与目标深度平面的距离越小，其余深度平面的模糊半径可以越小。此处不限定其余深度平面的模糊半径的具体数值，只要保证其余深度平面与目标深度平面的距离，与模糊半径成正比即可。在显示参数为像素值时，其余深度平面与目标深度平面的距离与像素值成反比。
其余深度平面与目标深度平面的距离可以直接对当前图像进行深度分析确定，也可以基于其余深度平面与用户的绝对距离信息和目标深度平面与用户的绝对距离信息确定。
在调节其余深度平面的显示参数后，其余深度平面的显示参数和目标深度平面的显示参数存在差异，提升了当前图像的深度感。
在一个可选实施例中，显示参数包括模糊半径。模糊半径与图像的模糊度成正比。本发明在进行模糊半径调节时可以采用高斯模糊算法实现。
通过该实施例提供的显示方法，首先获取用户在当前图像上的注视信息；然后基于所述注视信息，确定对应的目标深度平面；最终调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。利用上述方法，能够增强用户观看当前图像的深度感。在上述实施例的基础上，提出了上述实施例的变型实施例，在此需要说明的是，为了使描述简要，在变型实施例中仅描述与上述实施例的不同之处。在一个可选实施例中，在所述其余深度平面的个数为至少两个的情况下，每个其余深度平面的模糊半径与该其余深度平面距所述目标深度平面的距离成正比。在每个其余深度平面的模糊半径与其余深度平面距所述目标深度平面的距 离成正比的情况下，保证了距目标深度平面越远的其余深度平面越模糊，提升了当前图像的立体感和深度感。
在一个可选实施例中，所述其余深度平面与所述目标深度平面的距离由所述其余深度平面的距离信息和所述目标深度平面的距离信息的差值确定。
其余深度平面的距离信息可以理解为其余深度平面与用户的绝对距离信息。目标深度平面的距离信息可以理解为目标深度平面与用户的绝对距离信息。
在调整其余深度平面的显示参数时，可以将其余深度平面的距离信息和所述目标深度平面的距离信息的差值作为其余深度平面与目标深度平面的距离。
实施例二
图2是根据本发明实施例二的一种显示方法的流程图，本实施例二在上述各实施例的基础上进行优化。在本实施例中，所述注视信息具体包括注视点信息，相应的，基于所述注视信息，确定对应的目标深度平面，包括：
确定所述注视点信息在所述当前图像上对应的目标物体；
将所述目标物体所在的深度平面确定为目标深度平面。由于物体本身并不一定是平面的，也可能是立体的，所以立体物体的深度平面可以以物体距离用户的最近距离所在的平面确定为深度平面；或者以物体中心所在的平面作为深度平面；又或者以立体物体的任意一面作为深度平面，这里不做限制。
在一个可选实施例中，上述方法还可以包括：确定虚拟现实或增强现实视频中每帧图像所包含的深度平面及对应的距离信息，所述当前图像为所述虚拟现实或增强现实视频中当前显示的图像，所述距离信息为对应的深度平面与所述用户的绝对距离信息。
本实施例尚未详尽的内容请参考实施例一。
如图2所示，该实施例所提供的显示方法包括如下步骤：
S210、确定虚拟现实或增强现实视频中每帧图像所包含的深度平面及对应的距离信息，所述当前图像为所述虚拟现实或增强现实视频中当前显示的图像，所述距离信息为对应的深度平面与所述用户的绝对距离信息。
当前图像可以为虚拟现实或增强现实视频中的一帧图像。在显示当前图像之前，本发明可以先对虚拟现实或增强现实视频中各帧图像进行处理，确定各图像所包含的物体信息，该物体信息可以是预先在图像中设置好的信息，如各图像所包含的深度平 面及对应的距离信息。
虚拟现实视频可以认为是以虚拟现实技术呈现的视频。增强现实视频可以认为是以增强现实技术呈现的视频。图像所包含的深度平面可以由图像中所包括物体的深度信息确定。每个物体的深度信息可以通过对图像进行处理得到，或者直接读取由深度相机获取的图像中各物体的深度信息，此处不作限定，只要能够读取图像中各物体的深度信息即可。该实施例可以将每个不同深度信息对应的平面作为一个深度平面，从而将图像拆分包括多个深度平面。
确定图像所包含的深度平面后，可以为每个深度平面确定一个对应的距离信息，该距离信息可以为该深度平面与用户的绝对距离信息，此处不限定如何确定深度平面与用户的绝对距离信息，如可以根据各深度平面的深度信息和显示设备的尺寸确定。如基于显示设备显示当前图像的平面距用户眼睛的距离和各深度平面的深度信息，确定各深度平面的距离信息。
S220、获取用户在当前图像上的注视信息。
S230、确定所述注视点信息在所述当前图像上对应的目标物体。
确定目标深度平面时，本发明可以通过坐标比对的方式，确定注视点信息在当前图像上对应的目标物体。如，遍历当前图像中各物体，将坐标与注视信息相同或偏差在一定范围的物体作为目标物体。
S240、将所述目标物体所在的深度平面确定为目标深度平面。
在确定目标物体后，本发明可以将目标物体所在的深度平面作为目标深度平面，即用户当前所注视的深度平面。
S250、调节其余深度平面的显示参数。以下对本发明进行示例性描述：
在三维虚拟场景(如VR视频)的开发中，需要充分利用视觉系统对深度的感知，从而营造更强的立体感与深度感。在现有的三维虚拟场景中，使用者的视觉系统主要依赖双眼视差来感知深度，而在观察场景中较远的物体时，由于视轴接近平行、双眼视差为零，这种感知深度的线索来源便失去了作用。此时使用者仅能通过物体的相对大小、透视等图像信息，凭借经验来感知深度，极大影响了三维虚拟场景的立体感与深度感。
同时，在三维虚拟场景的实际应用中(如VR头显)，往往存在视场角过小的问题，导致单眼视觉范围内显示的东西少。此时，如果场景中的一个物体，使用者只有一只眼睛看到了它，而另一只眼睛没有，使用者就很可能难以判定这个物体的深度，进而 影响在场景中的体验。
在现有的三维虚拟场景中，由于场景呈现出来的是定焦画面，因此使用者(即用户)无法获取不同深度平面对焦&失焦的深度线索。此时，如果使用者因为物体的绝对距离远、视场角过小等问题，无法凭借双眼视差感知深度，就会严重影响在三维虚拟场景中游戏、交互等体验。
该实施例通过标记三维虚拟场景中不同深度平面的绝对距离信息，对场景画面进行预处理，再基于眼动追踪技术获取使用者的注视点信息，根据注视点位置获知注视点所在深度平面的绝对距离信息，就可以为使用者提供对焦和失焦的深度线索，有效弥补现有深度线索的不足和缺失，极大增强使用者在三维虚拟场景中的立体感与深度感。
该实施例所提供的显示方法可以针对三维虚拟场景的深度感知，该方法可以包括如下步骤：
步骤一、三维虚拟场景图像预处理
在三维虚拟场景中，逐帧将处于不同深度平面的图像区域进行分割。而后根据该图像所处平面的深度，在每一个图像区域上标记绝对距离信息。
图像内每个区域的具体物体的深度信息可以预先包含在图像中。
图3a是根据本发明实施例二的图像预处理效果的示意图，参见图3a，对图像进行分割后，得到位于不同深度平面的第一物体1、第二物体2和第三物体3。其中，“第一”、“第二”和“第三”等仅用于对相应内容进行区分，并非用于限定顺序或者相互依存关系。
各物体的深度信息可以预先包含在图像中，基于各物体的深度信息可以确定各物体相对于用户的绝对距离信息。图3b是根据本发明实施例二的包含多个深度平面场景的示意图，参见图3b，第一物体1对应的深度平面的距离信息为第一物体1与用户4的绝对距离信a，第二物体2对应的深度平面的距离信息为第二物体2与用户4的绝对距离信息b，第三物体3对应的深度平面的距离信息为第三物体3与用户4的绝对距离信息c。从图3b可以看出，c＞b＞a，即第三物体3与用户4的绝对距离最远，第一物体1与用户4的绝对距离最近。
以图3b为例，当用户4注视第一物体1时，第一物体1的深度平面为目标深度平面，第二物体2的深度平面的显示参数可以根据第二物体2距目标深度平面的距离调节。第三物体3的深度平面的显示参数可以根据第三物体3距目标深度平面的距离调 节。因为第二物体2的深度平面距目标深度平面的距离小于第三物体3的深度平面距目标深度平面的距离，所以第二物体2显示参数的调节的大小小于第三物体3显示参数调节的大小，从而在用户注视第一物体1时，第二物体2相比于第三物体3更清晰。
参见图3b，图中可以基于填充物的稀密表征清晰程度。图3b中填充物越密集代表清晰度越高，填充物越稀疏代表清晰度越低。第一物体1的深度平面距第二物体2的深度平面的距离小于第一物体1的深度平面距第三物体3深度平面的距离，故在用户注视第二物体2时第一物体1的清晰度比用户注视第三物体3时第一物体1的清晰度高。步骤二、获取注视点信息当使用者体验三维虚拟场景时，通过眼动仪可以获取使用者实时的注视点信息，进而判断出被注视的图像区域所在的深度平面。
眼动仪可以位于VR设备上。
步骤三、呈现不同深度平面的对焦和失焦效果三维虚拟场景的实时图像对焦于使用者注视点所在的深度平面，处于其他深度平面的图像区域根据绝对距离的差值呈现不同的失焦状态。此时，在使用者面前的三维虚拟场景中，只有其注视物体所对应的深度平面是清晰的，处于其他深度平面的物体，根据与“被注视深度平面”的绝对距离差值，呈现出不同程度的模糊状态。
在一个可选实施例中，距离注视点所在深度平面的绝对距离越近越清晰，越远越模糊。
该实施例通过对三维虚拟场景图像的分割和标记，结合眼动追踪技术，使人眼视觉系统在观察三维虚拟场景时，获得对焦和失焦的深度线索。在三维虚拟场景中为使用者提供对焦和失焦的深度线索，有效弥补现有场景由于使用定焦画面导致的深度线索的不足和缺失，极大增强使用者在三维虚拟场景中的立体感与深度感。
该实施例所提供的显示方法描述了确定目标深度平面的操作和确定深度平面和对应的距离信息的操作。利用该方法能够提升虚拟现实或增强现实视频的立体感和深度感。
本发明实施例在上述各实施例的技术方案的基础上，提供了几种具体的实施方式。
在一个可选实施例中，确定所述虚拟现实或增强现实视频中每帧图像所包含的深度平面，包括：
逐帧获取所述虚拟现实或增强现实视频中的目标图像；获取所述目标图像上所包括物体的深度信息；
基于各所述深度信息将所述目标图像进行分割，得到至少一个深度平面，分割得 到的各深度平面的距离信息根据所述深度信息确定。
当确定深度平面时，可以逐帧获取虚拟现实或增强现实视频中的图像作为目标图像，针对每个目标图像，获取目标图像所包括物体的深度信息，每个物体可以对应一个深度信息。确定深度信息后，本发明可以基于各深度信息对目标图像进行分割，得到至少一个深度平面，深度平面的个数可以基于深度信息的个数确定。当多个深度信息的数值相同时，可以将该多个深度信息的个数确定为1。
将目标图像按照深度信息划分包括多个深度平面，各深度平面的距离信息可以由深度信息确定。如深度平面的距离信息为深度平面对应的深度信息的差值确定。
在一个可选实施例中，同一深度平面所包括物体的深度信息相同。
实施例三
图4是根据本发明实施例三的一种显示装置的结构示意图，该装置可适用于提升图像深度感的情况，其中该装置可由软件和/或硬件实现，并一般集成在终端设备上。
如图4所示，该装置包括：获取模块31、确定模块32和调节模块33；获取模块31，设置为获取用户在当前图像上的注视信息；确定模块32，设置为基于所述注视信息，确定对应的目标深度平面；调节模块33，设置为调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
在本实施例中，该装置首先通过获取模块31获取用户在当前图像上的注视信息；其次通过确定模块32基于所述注视信息，确定对应的目标深度平面；最后通过调节模块33调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
该实施例所提供的显示装置，能够增强用户观看的当前图像的深度感。
在一个可选实施例中，所述注视信息包括注视点信息；基于所述注视信息确定对应的目标深度平面包括：
确定所述注视点信息在所述当前图像上对应的目标物体；将所述目标物体所在的深度平面确定为目标深度平面。在一个可选实施例中，所述显示参数包括模糊半径。
在一个可选实施例中，在所述其余深度平面的个数为至少两个的情况下，每个其余深度平面的模糊半径与该其余深度平面距所述目标深度平面的距离成正比。
在一个可选实施例中，所述其余深度平面与所述目标深度平面的距离由所述其余深度平面的距离信息和所述目标深度平面的距离信息的差值确定。
在一个可选实施例中，该装置还包括：信息确定模块，设置为：
确定虚拟现实或增强现实视频中每帧图像所包含的深度平面及对应的距离信息，所述当前图像为所述虚拟现实或增强现实视频中当前显示的图像，所述距离信息为对应的深度平面与所述用户的绝对距离信息。
在一个可选实施例中，信息确定模块，设置为：
逐帧获取所述虚拟现实或增强现实视频中的目标图像；获取所述目标图像上所包括物体的深度信息；
基于各所述深度信息将所述目标图像进行分割，得到至少一个深度平面，分割得到的各深度平面的距离信息根据所述深度信息确定。
在一个可选实施例中，同一深度平面所包括物体的深度信息相同。
上述显示装置可执行本发明任意实施例所提供的显示方法，具备执行方法相应的功能模块和有益效果。
实施例四
图5是根据本发明实施例四的一种终端设备的结构示意图。如图5所示，该实施例所提供的终端设备包括：一个或多个处理器41和存储装置42；该终端设备中的处理器41可以是一个或多个，图5中以一个处理器41为例；
存储装置42用于存储一个或多个程序；所述一个或多个程序被所述一个或多个处理器41执行，使得所述一个或多个处理器41实现如本发明实施例中任一项所述的方法。
所述终端设备还可以包括：输入装置43和输出装置44。
终端设备中的处理器41、存储装置42、输入装置43和输出装置44可以通过总线或其他方式连接，图5中以通过总线连接为例。
该终端设备中的存储装置42作为一种计算机可读存储介质，可用于存储一个或多个程序，所述程序可以是软件程序、计算机可执行程序以及模块，如本发明实施例一或二所提供的方法对应的程序指令/模块(例如，附图4所示的显示装置中的模块，包括：获取模块31、确定模块32和调节模块33)。处理器41通过运行存储在存储装置42中的软件程序、指令以及模块，从而执行终端设备的各种功能应用以及数据处理， 即实现上述方法实施例中的显示方法。
存储装置42可包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据终端设备的使用所创建的数据等。此外，存储装置42可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储装置42可进一步包括相对于处理器41远程设置的存储器，这些远程存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置43可设置为接收输入的数字或字符信息，以及产生与终端设备的用户设置以及功能控制有关的键信号输入。输出装置44可包括显示屏等显示设备。并且，当上述终端设备所包括一个或者多个程序被所述一个或者多个处理器41执行时，程序进行如下操作：
获取用户在当前图像上的注视信息；
基于所述注视信息，确定对应的目标深度平面；
调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
实施例五
本发明实施例五提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时用于执行本发明提供的显示方法，该方法包括：
获取用户在当前图像上的注视信息；
基于所述注视信息，确定对应的目标深度平面；
调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
可选的，该程序被处理器执行时还可以用于执行本发明任意实施例所提供的显示方法。
本发明实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机 可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、可擦式可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、闪存、光纤、便携式CD-ROM、光存储器件、磁存储器件、或者上述的任意合适的组合。计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于：电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、无线电频率(Radio Frequency，RF)等等，或者上述的任意合适的组合。可以以一种或多种程序设计语言或其组合来编写用于执行本发明操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)——连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
注意，上述仅为本发明的较佳实施例及所运用技术原理。本领域技术人员会理解，本发明不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代而不会脱离本发明的保护范围。因此，虽然通过以上实施例对本发明进行了较为详细的说明，但是本发明不仅仅限于以上实施例，在不脱离本发明构思的情况下，还可以包括更多其他等效实施例，而本发明的范围由所附的权利要求范围决定。

Claims (11)

  1. 一种显示方法，包括：
    获取用户在当前图像上的注视信息；
    基于所述注视信息，确定对应的目标深度平面；
    调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
  2. 根据权利要求1所述的方法，其中，所述注视信息包括注视点信息；基于所述注视信息，确定对应的目标深度平面包括：
    确定所述注视点信息在所述当前图像上对应的目标物体；将所述目标物体所在的深度平面确定为目标深度平面。
  3. 根据权利要求1所述的方法，其中，所述显示参数包括模糊半径。
  4. 根据权利要求3所述的方法，其中，在所述其余深度平面的个数为至少两个的情况下，每个其余深度平面的模糊半径与该其余深度平面距所述目标深度平面的距离成正比。
  5. 根据权利要求1所述的方法，其中，所述其余深度平面与所述目标深度平面的距离由所述其余深度平面的距离信息和所述目标深度平面的距离信息的差值确定。
  6. 根据权利要求1所述的方法，其中，还包括：
    确定虚拟现实或增强现实视频中每帧图像所包含的深度平面及对应的距离信息，所述当前图像为所述虚拟现实或增强现实视频中当前显示的图像，所述距离信息为对应的深度平面与所述用户的绝对距离信息。
  7. 根据权利要求6所述的方法，其中，确定所述虚拟现实或增强现实视频中每帧图像所包含的深度平面包括：
    逐帧获取所述虚拟现实或增强现实视频中的目标图像；获取所述目标图像上所包括物体的深度信息；
    基于各所述深度信息将所述目标图像进行分割，得到至少一个深度平面，分割得到的各深度平面的距离信息根据所述深度信息确定。
  8. 根据权利要求7所述的方法，其中，同一深度平面所包括物体的深度信息相同。
  9. 一种显示装置，包括：
    获取模块，设置为获取用户在当前图像上的注视信息；
    确定模块，设置为基于所述注视信息，确定对应的目标深度平面；
    调节模块，设置为调节其余深度平面的显示参数，所述其余深度平面的显示参数根据所述其余深度平面与所述目标深度平面的距离确定，所述其余深度平面为所述当前图像上除所述目标深度平面外的深度平面。
  10. 一种终端设备，包括：一个或多个处理器；存储装置，设置为存储一个或多个程序；当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-8中任一所述的方法。
  11. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-8中任一所述的方法。
PCT/CN2021/076919 2020-02-28 2021-02-19 一种显示方法、装置、终端设备及存储介质 WO2021169853A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2022551634A JP2023515205A (ja) 2020-02-28 2021-02-19 表示方法、装置、端末機器及びコンピュータプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010130618.5 2020-02-28
CN202010130618.5A CN113325947A (zh) 2020-02-28 2020-02-28 一种显示方法、装置、终端设备及存储介质

Publications (1)

Publication Number Publication Date
WO2021169853A1 true WO2021169853A1 (zh) 2021-09-02

Family

ID=77412782

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/076919 WO2021169853A1 (zh) 2020-02-28 2021-02-19 一种显示方法、装置、终端设备及存储介质

Country Status (3)

Country Link
JP (1) JP2023515205A (zh)
CN (1) CN113325947A (zh)
WO (1) WO2021169853A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115562497A (zh) * 2022-11-04 2023-01-03 浙江舜为科技有限公司 增强现实信息交互方法、增强现实设备和存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116850012B (zh) * 2023-06-30 2024-03-12 广州视景医疗软件有限公司 一种基于双眼分视的视觉训练方法及系统
CN117880630B (zh) * 2024-03-13 2024-06-07 杭州星犀科技有限公司 对焦深度获取方法、对焦深度获取系统及终端

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106537219A (zh) * 2014-05-30 2017-03-22 奇跃公司 用于在虚拟和增强现实中产生焦平面的方法和系统
CN108369325A (zh) * 2015-12-08 2018-08-03 欧库勒斯虚拟现实有限责任公司 焦点调整虚拟现实耳机
CN110663246A (zh) * 2017-05-24 2020-01-07 深圳市大疆创新科技有限公司 用于处理图像的方法和系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016115872A1 (zh) * 2015-01-21 2016-07-28 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法
US10698215B2 (en) * 2016-03-25 2020-06-30 Magic Leap, Inc. Virtual and augmented reality systems and methods
CN110679147B (zh) * 2017-03-22 2022-03-08 奇跃公司 用于显示系统的基于深度的凹式渲染
CN110555873A (zh) * 2018-05-30 2019-12-10 Oppo广东移动通信有限公司 控制方法、控制装置、终端、计算机设备和存储介质
CN110727111A (zh) * 2019-10-23 2020-01-24 深圳惠牛科技有限公司 一种头戴显示光学系统及头戴显示设备

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106537219A (zh) * 2014-05-30 2017-03-22 奇跃公司 用于在虚拟和增强现实中产生焦平面的方法和系统
CN108369325A (zh) * 2015-12-08 2018-08-03 欧库勒斯虚拟现实有限责任公司 焦点调整虚拟现实耳机
CN110663246A (zh) * 2017-05-24 2020-01-07 深圳市大疆创新科技有限公司 用于处理图像的方法和系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115562497A (zh) * 2022-11-04 2023-01-03 浙江舜为科技有限公司 增强现实信息交互方法、增强现实设备和存储介质
CN115562497B (zh) * 2022-11-04 2024-04-05 浙江舜为科技有限公司 增强现实信息交互方法、增强现实设备和存储介质

Also Published As

Publication number Publication date
CN113325947A (zh) 2021-08-31
JP2023515205A (ja) 2023-04-12

Similar Documents

Publication Publication Date Title
WO2021169853A1 (zh) 一种显示方法、装置、终端设备及存储介质
JP6759371B2 (ja) ３ｄプレノプティックビデオ画像を作成するためのシステムおよび方法
CN110187855B (zh) 近眼显示设备的避免全息影像阻挡视线的智能调节方法
CN111415422B (zh) 虚拟对象调整方法、装置、存储介质与增强现实设备
CN109086726B (zh) 一种基于ar智能眼镜的局部图像识别方法及系统
JP7094266B2 (ja) 単一深度追跡型の遠近調節－両眼転導ソリューション
US10382699B2 (en) Imaging system and method of producing images for display apparatus
US10241329B2 (en) Varifocal aberration compensation for near-eye displays
EP2994812B1 (en) Calibration of eye location
WO2017183346A1 (ja) 情報処理装置、情報処理方法、及びプログラム
KR101788452B1 (ko) 시선 인식을 이용하는 콘텐츠 재생 장치 및 방법
WO2015035822A1 (en) Pickup of objects in three-dimensional display
WO2020215960A1 (zh) 注视区域的确定方法、装置及可穿戴设备
US11983310B2 (en) Gaze tracking apparatus and systems
CN103517060A (zh) 一种终端设备的显示控制方法及装置
WO2022267573A1 (zh) 裸眼3d显示模式的切换控制方法、介质和系统
WO2019109323A1 (zh) 图像显示方法、穿戴式智能设备及存储介质
US20210382316A1 (en) Gaze tracking apparatus and systems
CN112655202A (zh) 用于头戴式显示器的鱼眼镜头的减小带宽立体失真校正
EP3945401A1 (en) Gaze tracking system and method
CN106708249B (zh) 交互方法、交互装置及用户设备
RU2815753C1 (ru) Способ и устройство отображения, оконечное устройство и носитель для хранения данных
CN115914603A (zh) 图像渲染方法、头戴显示设备和可读存储介质
CN115202475A (zh) 显示方法、装置、电子设备及计算机可读存储介质
US11934571B2 (en) Methods and systems for a head-mounted device for updating an eye tracking model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21760940

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022551634

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21760940

Country of ref document: EP

Kind code of ref document: A1