WO2023169283A1 - 双目立体全景图像的生成方法、装置、设备、存储介质和产品 - Google Patents

双目立体全景图像的生成方法、装置、设备、存储介质和产品 Download PDF

Info

Publication number
WO2023169283A1
WO2023169283A1 PCT/CN2023/079064 CN2023079064W WO2023169283A1 WO 2023169283 A1 WO2023169283 A1 WO 2023169283A1 CN 2023079064 W CN2023079064 W CN 2023079064W WO 2023169283 A1 WO2023169283 A1 WO 2023169283A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinate
panoramic
image
panoramic image
depth
Prior art date
Application number
PCT/CN2023/079064
Other languages
English (en)
French (fr)
Inventor
苏坦
Original Assignee
影石创新科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 影石创新科技股份有限公司 filed Critical 影石创新科技股份有限公司
Publication of WO2023169283A1 publication Critical patent/WO2023169283A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present application relates to the field of image processing technology, and in particular to a method, device, equipment, storage medium and product for generating a binocular stereoscopic panoramic image.
  • VR virtual reality
  • electronic devices such as VR glasses
  • binocular stereoscopic panoramic images or videos can be displayed to the user, and the images of the left and right eyes are displayed on the left and right eye screens respectively.
  • a method for generating a binocular stereoscopic panoramic image includes:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • mapping the panoramic image into a left eye panoramic picture and a right eye panoramic picture according to the preset interpupillary distance and depth image includes:
  • the left eye mapping relationship and the right eye mapping relationship are obtained;
  • the left eye mapping relationship includes the first coordinate of the pixel point in the panoramic image and the second coordinate of the pixel point in the left eye panoramic picture.
  • the right-eye mapping relationship includes the correspondence between the first coordinate and the third coordinate of the pixel point in the right-eye panoramic picture;
  • the panoramic images are mapped and projected respectively to generate a left eye panoramic picture and a right eye panoramic picture.
  • the left eye mapping relationship and the right eye mapping relationship are obtained according to the preset interpupillary distance and depth image, including:
  • the preset interpupillary distance and the first coordinate obtain the second coordinate and the third coordinate
  • the corresponding relationship between the first coordinate and the second coordinate is determined as the left eye mapping relationship; and the corresponding relationship between the first coordinate and the third coordinate is determined as the right eye mapping relationship.
  • obtaining the second coordinates based on the depth information, the preset interpupillary distance and the first coordinates includes:
  • ⁇ is the longitude coordinate in the first coordinate
  • ⁇ is the latitude coordinate in the first coordinate
  • D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image
  • L ⁇ ( ⁇ , ⁇ ) is the first The longitude coordinate in the second coordinate corresponding to the coordinate
  • p is the preset interpupillary distance.
  • obtaining the third coordinate based on the depth information, the preset interpupillary distance, and the longitude coordinate in the first coordinate includes:
  • ⁇ is the longitude coordinate in the first coordinate
  • ⁇ is the latitude coordinate in the first coordinate
  • D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image
  • R ⁇ ( ⁇ , ⁇ ) is the first The longitude coordinate in the third coordinate corresponding to the coordinate
  • p is the preset interpupillary distance.
  • the above method further includes:
  • the training samples include panoramic sample images and sample depth images corresponding to the panoramic sample images;
  • the panoramic sample image is used as the reference input of the initial depth estimation model
  • the sample depth image is used as the reference output of the initial depth estimation model
  • the initial depth estimation model is trained according to the preset loss function to obtain the depth estimation model.
  • a method for generating a binocular stereoscopic panoramic video includes:
  • corresponding binocular stereoscopic panoramic images are generated according to each panoramic image in the panoramic video;
  • a binocular stereoscopic panoramic video is generated.
  • a device for generating a binocular stereoscopic panoramic image includes:
  • the acquisition module is used to input the panoramic image into a preset depth estimation model to obtain the depth image corresponding to the panoramic image; the depth image includes depth information corresponding to each pixel in the panoramic image;
  • the mapping module is used to map the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture based on the preset interpupillary distance and depth image;
  • the generation module is used to generate a binocular stereoscopic panoramic image based on the left-eye panoramic image and the right-eye panoramic image.
  • this application also provides a computer device.
  • Computer equipment includes a memory and a processor.
  • the memory stores computer programs.
  • the processor executes the computer program, it implements the following steps:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • this application also provides a computer-readable storage medium.
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • this application also provides a computer program product.
  • a computer program product includes a computer program that, when executed by a processor, performs the following steps:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • the electronic device inputs the panoramic image into a preset depth estimation model to obtain a depth image corresponding to the panoramic image; then, according to the depth image and the preset interpupillary distance , mapping the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture; generating a binocular stereoscopic panoramic image according to the left-eye panoramic picture and the right-eye panoramic picture; wherein the above-mentioned depth image includes each pixel point in the panoramic image corresponding depth information.
  • the electronic device can obtain the depth image of the panoramic image, it can map the above-mentioned panoramic image into a left-eye panoramic picture and a right-eye panoramic picture respectively according to the depth image and the preset interpupillary distance, and obtain a binocular stereoscopic panoramic image, so that the electronic device can complete
  • the mapping conversion between panoramic images and binocular stereoscopic panoramic images does not require professional multi-lens panoramic shooting equipment to complete the collection of binocular panoramic stereoscopic images, which reduces the cost of electronic equipment and is simple to operate.
  • Figure 1 is an application environment diagram of a method for generating a binocular stereoscopic panoramic image in one embodiment
  • Figure 2 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in one embodiment
  • Figure 3 is a schematic diagram of a method for generating a binocular stereoscopic panoramic image in one embodiment
  • Figure 4 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 5 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 6 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 7 is a structural block diagram of a device for generating a binocular stereoscopic panoramic image in one embodiment
  • Figure 8 is a structural block diagram of a device for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 9 is a structural block diagram of a device for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 10 is a structural block diagram of a device for generating a binocular stereoscopic panoramic image in another embodiment
  • Figure 11 is an internal structural diagram of an electronic device in one embodiment.
  • the method for generating a binocular stereoscopic panoramic image provided by this application can be applied to electronic devices.
  • the electronic device can process the panoramic image to obtain a binocular stereoscopic panoramic image corresponding to the panoramic image.
  • the above-mentioned electronic devices may be, but are not limited to, various personal computers, laptops, smartphones, tablets and portable wearable devices.
  • the above-mentioned electronic devices can also be imaging devices such as cameras and camcorders; the above-mentioned cameras can be, but are not limited to, ordinary cameras, pocket cameras, anti-shake cameras, virtual reality (Virtual Reality, VR) panoramic cameras, action cameras, and consumer-grade or professional-grade cameras. Panoramic camera etc.
  • a method for generating a binocular stereoscopic panoramic image is provided.
  • the application of this method to an electronic device is used as an example to illustrate, including:
  • the above-mentioned panoramic image may be obtained by shooting with an electronic device, or may be an image stored in the electronic device, which is not limited here.
  • the panoramic image captured by the electronic device may be an image captured by the electronic device through a panoramic camera, or it may be an image frame in a video captured by the electronic device, which is not limited here.
  • the above panoramic image is an image stored in an electronic device, it may be stored in the electronic device in a picture format, or may be a video frame in a stored video.
  • the electronic device may be VR glasses, and the panoramic image may be a panoramic image input to the VR glasses to be played.
  • the camera of the above-mentioned electronic device may be a dual fish-eye panoramic camera.
  • the electronic device captures a panoramic image, any angle may be covered by the field of view of one of the lenses of the dual fish-eye panoramic camera.
  • Electronic equipment can stitch images captured by different lenses to obtain a panoramic image.
  • the depth estimation model may be a neural network model, and the depth estimation model may be used to extract the depth information of each pixel in the panoramic image, and generate a depth image corresponding to the panoramic image based on the depth information corresponding to each pixel.
  • the above-mentioned depth information refers to the distance between the object represented by the pixels in the image and the center of the camera when shooting a panoramic image.
  • the electronic device may input the panoramic image into the above-mentioned depth estimation model, or may preprocess the panoramic image and then input it into the depth estimation model, which is not limited here.
  • the preprocessing operations of the panoramic image by the electronic device may include downsampling the panoramic image, changing the projection method of the panoramic image, changing the brightness or contrast of the panoramic image, and converting the panoramic image into a single-channel grayscale image, etc.
  • the above-mentioned depth estimation model can output a depth image corresponding to the panoramic image.
  • the size of the above-mentioned depth image can be equal to the panoramic image or smaller than the panoramic image, which is not limited here.
  • the above-mentioned depth image and the above-mentioned panoramic image can adopt the same panoramic projection method.
  • the above-mentioned panoramic projection method can be spherical projection or equidistant cylindrical projection, which is not limited here.
  • the electronic device can map the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture, so that the parallax generated by the left-eye panoramic picture and the right-eye panoramic picture corresponds to the above-mentioned depth image.
  • parallax When the user views the left-eye panoramic image through the left eye and the right-eye panoramic image through the right eye at the same time, there will be a position gap between the left eye and the right eye for the same object, which is called parallax.
  • the electronic device maps the panoramic image into the corresponding left-eye panoramic picture and right-eye panoramic picture
  • the distance perceived by the user through the parallax generated by the above-mentioned binocular stereoscopic panoramic image corresponds to the above-mentioned depth image.
  • the above-mentioned panoramic image includes object A, and the depth information corresponding to object A in the depth image obtained through the depth estimation model is H; based on the above-mentioned depth image, the electronic device maps the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture. Afterwards, the user can perceive the distance between the object A and the user through the left-eye panoramic screen and the right-eye panoramic screen, and the distance corresponds to the depth information H.
  • the electronic device can use the Omni-directional stereo (ODS) projection method to map the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture.
  • ODS Omni-directional stereo
  • the electronic device can combine the left-eye panoramic image and the right-eye panoramic image into a binocular stereoscopic panoramic image.
  • the electronic device inputs the panoramic image into a preset depth estimation model to obtain a depth image corresponding to the panoramic image; then, based on the depth image and the preset interpupillary distance, the panoramic image is mapped into a left eye panorama and a right-eye panoramic picture; a binocular stereoscopic panoramic image is generated according to the left-eye panoramic picture and the right-eye panoramic picture; wherein the above-mentioned depth image includes depth information corresponding to each pixel in the panoramic image.
  • the electronic device can obtain the depth image of the panoramic image, it can map the above-mentioned panoramic image into a left-eye panoramic picture and a right-eye panoramic picture respectively according to the depth image and the preset interpupillary distance, and obtain a binocular stereoscopic panoramic image, so that the electronic device can complete
  • the mapping conversion between panoramic images and binocular stereoscopic panoramic images does not require professional multi-lens panoramic shooting equipment to complete the collection of binocular panoramic stereoscopic images, which reduces the cost of electronic equipment and is simple to operate.
  • FIG. 2 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in one embodiment.
  • This embodiment relates to a way for an electronic device to map a panoramic image into a left-eye panoramic picture and a right-eye panoramic picture.
  • the above-mentioned S102 includes:
  • the left eye mapping relationship includes the first coordinate of the pixel point in the panoramic image and the second coordinate of the pixel point in the left eye panoramic picture.
  • the corresponding relationship between the right eye mapping relationship includes the corresponding relationship between the first coordinate and the third coordinate of the pixel point in the right eye panoramic picture.
  • the above-mentioned interpupillary distance can be used to characterize the distance between the pupil of the user's left eye and the pupil of the right eye.
  • a preset value of the interpupillary distance can be stored in the electronic device, and the preset value is used to map the above panoramic image.
  • the electronic device can adopt different interpupillary distances for different users; the electronic device can preset the corresponding relationship between different user accounts and the interpupillary distance, and the interpupillary distance in the above corresponding relationship can be input by the user. It can also be selected by the user from multiple preset values, or it can be obtained by the electronic device based on the user's image extraction. There is no limitation on the method of obtaining the above-mentioned interpupillary distance.
  • the user can collect images through the electronic device or a terminal such as a mobile phone connected to the electronic device. The above image collection process can be during the user registration process or the login process, and is not limited here.
  • different types of electronic devices may correspond to different interpupillary distances.
  • the above-mentioned electronic device may be VR glasses or a smart helmet, etc.
  • Different interpupillary distances may be used for different electronic devices to meet the mapping requirements of the binocular stereoscopic panoramic image of the electronic device.
  • the electronic device can obtain the left eye mapping relationship and the right eye mapping relationship corresponding to the panoramic image based on the ODS mapping method.
  • the above mapping relationship is a coordinate correspondence relationship.
  • the pixels in the panoramic image can be mapped to the left-eye panoramic screen and the right-eye panoramic screen respectively.
  • the coordinates of the above-mentioned pixel points in the panoramic image may be the first coordinates
  • the coordinates in the left-eye panoramic picture may be the second coordinates
  • the coordinates in the right-eye panoramic picture may be the third coordinates, as shown in Figure 3 .
  • the above-mentioned left eye mapping relationship is the correspondence between the first coordinate and the second coordinate of each pixel in the panoramic image
  • the above-mentioned right eye mapping relationship is the correspondence between the first coordinate and the third coordinate of each pixel in the panoramic image. relation.
  • the electronic device can determine which position to map the pixels in the panoramic image, and then determine the coordinates of each pixel in the left-eye panoramic picture and the right-eye panoramic picture. . After associating each of the above second coordinates with the corresponding pixel values, a left-eye panoramic image is obtained. After correlating each of the above third coordinates with the corresponding pixel values, a right-eye panoramic image is obtained.
  • the electronic device obtains the left eye mapping relationship and the right eye mapping relationship through the interpupillary distance and depth images, and can accurately map the panoramic image into a binocular stereoscopic panoramic image, making the binocular stereoscopic image
  • the panoramic image can present a stereoscopic effect corresponding to the depth information of the panoramic image.
  • Figure 4 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in one embodiment. This embodiment involves a way for an electronic device to obtain a left eye mapping relationship and a right eye mapping relationship. Based on the above embodiment, as shown in Figure 4 As shown, the above S201 includes:
  • the above-mentioned first coordinate, second coordinate and third coordinate may be spherical coordinates or three-dimensional plane coordinates, which are not limited here.
  • the electronic device can perform coordinate mapping according to a preset formula and calculate second coordinates and third coordinates corresponding to each first coordinate.
  • each pixel point in the above-mentioned panoramic image and the above-mentioned depth image can be represented by spherical coordinates; that is, the coordinates of each pixel point can be composed of longitude coordinates and latitude coordinates.
  • the above preset formula may include a longitude coordinate calculation formula and a latitude coordinate calculation formula.
  • the longitude coordinates in the second coordinates and the third coordinates may be related to the depth information, the preset interpupillary distance, and the longitude coordinates in the first coordinates.
  • the longitude coordinate of the corresponding second coordinate is different from the longitude coordinate of the third coordinate.
  • the difference between the longitude coordinate of the second coordinate and the longitude coordinate of the third coordinate can be obtained by the ratio of the interpupillary distance and the depth information corresponding to the coordinate. Due to the parallax generated by the left-eye panoramic picture and the right-eye panoramic picture, the distance information used to generate is mainly related to the longitude coordinate. Therefore, the electronic device can directly determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate. and the latitude coordinate in the third coordinate.
  • the electronic device can be based on the formula Calculate the longitude coordinate in the second coordinate and determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate.
  • ⁇ is the longitude coordinate in the first coordinate
  • ⁇ is the latitude coordinate in the first coordinate
  • D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image
  • L ⁇ ( ⁇ , ⁇ ) is the first The longitude coordinate in the second coordinate corresponding to the coordinate
  • p is the preset interpupillary distance.
  • the third coordinate the electronic device can be based on the formula Calculate the longitude coordinate in the third coordinate, and determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate.
  • ⁇ is the longitude coordinate in the first coordinate
  • ⁇ is the latitude coordinate in the first coordinate
  • D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image
  • R ⁇ ( ⁇ , ⁇ ) is the first
  • p is the preset interpupillary distance.
  • the formula used to calculate latitude coordinates in the above preset formula can be:
  • L ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the second coordinate corresponding to the first coordinate
  • R ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the third coordinate corresponding to the first coordinate
  • the electronic device can determine the corresponding relationship between the first coordinates and the second coordinates of each pixel point as the left eye mapping relationship, and combine the first coordinates with the third coordinates. The corresponding relationship between them is determined as the right eye mapping relationship.
  • the above method for generating a binocular stereoscopic panoramic image uses electronic equipment to complete coordinate mapping using spherical coordinates, and can be applied to panoramic images of any projection method, thereby improving the applicability of mapping panoramic images to binocular stereoscopic panoramic images.
  • Figure 5 is a schematic flowchart of a method for generating a binocular stereoscopic panoramic image in one embodiment. This embodiment relates to an implementation of a depth estimation model. Based on the above embodiment, as shown in Figure 5, the above method also includes :
  • training samples include panoramic sample images and sample depth images corresponding to the panoramic sample images.
  • the electronic device can acquire a binocular stereoscopic panoramic sample image, and then extract depth information from the binocular stereoscopic panoramic sample image to obtain a sample depth image corresponding to the binocular stereoscopic panoramic sample image; further, the electronic device can extract the above-mentioned binocular stereoscopic panoramic sample image.
  • the image is processed monocularly to obtain a panoramic sample image corresponding to the binocular stereoscopic panoramic sample image.
  • the above panoramic sample images and their corresponding sample depth images constitute the training samples.
  • a binocular stereo panoramic camera and a monocular panoramic camera can be used to shoot the same scene at the same time, to obtain binocular stereo panoramic sample images and panoramic sample images respectively, and then generate samples based on the binocular stereo panoramic sample images. After depth images, the above training samples are obtained.
  • the electronic device can use the panoramic sample image as the reference input of the initial depth estimation model, use the sample depth image as the reference output of the initial depth estimation model, and train the initial depth estimation model according to the preset loss function. , obtain the depth estimation model.
  • the above-mentioned binocular stereo panoramic image generation method can obtain a depth estimation model through sample training, so that the depth estimation can be
  • the model obtains the depth image of the panoramic image, which provides a data basis for mapping from the panoramic image to the binocular stereoscopic panoramic image.
  • a method for generating a binocular stereoscopic panoramic image includes:
  • S505. Determine the corresponding relationship between the first coordinate and the second coordinate as the left eye mapping relationship; and determine the corresponding relationship between the first coordinate and the third coordinate as the right eye mapping relationship;
  • a method for generating a binocular stereoscopic panoramic video is provided.
  • the electronic device can use the above method for generating a binocular stereoscopic panoramic image to generate a binocular stereoscopic panoramic image according to each panoramic image in the panoramic video; then, Based on each binocular stereoscopic panoramic image, a binocular stereoscopic panoramic video is generated.
  • embodiments of the present application also provide a binocular stereoscopic panoramic image generating device for implementing the above-mentioned binocular stereoscopic panoramic image generating method.
  • the solution to the problem provided by this device is similar to the solution recorded in the above method. Therefore, the specific limitations in the embodiments of the device for generating one or more binocular stereoscopic panoramic images provided below can be found in the above description of the binocular stereoscopic panoramic image. The limitations of the method for generating a stereoscopic panoramic image will not be described again here.
  • a device for generating a binocular stereoscopic panoramic image including:
  • the acquisition module 10 is used to input the panoramic image into a preset depth estimation model to obtain a depth image corresponding to the panoramic image; the depth image includes depth information corresponding to each pixel in the panoramic image;
  • the mapping module 20 is used to map the panoramic image into a left-eye panoramic picture and a right-eye panoramic picture according to the preset interpupillary distance and depth image;
  • the generation module 30 is configured to generate a binocular stereoscopic panoramic image based on the left-eye panoramic image and the right-eye panoramic image.
  • the above mapping module 20 includes:
  • the acquisition unit 201 is used to obtain the left eye mapping relationship and the right eye mapping relationship according to the preset interpupillary distance and depth image;
  • the left eye mapping relationship includes the first coordinate of the pixel point in the panoramic image and the position of the pixel point in the left eye panoramic picture.
  • the right-eye mapping relationship includes the corresponding relationship between the first coordinates and the third coordinates of the pixel point in the right-eye panoramic picture;
  • the mapping unit 202 is configured to respectively map and project the panoramic image according to the left-eye mapping relationship and the right-eye mapping relationship to generate a left-eye panoramic picture and a right-eye panoramic picture.
  • the above acquisition unit 201 includes:
  • the acquisition subunit 2011 is used to acquire the second coordinates and the third coordinates according to the depth information, the preset interpupillary distance and the first coordinates;
  • the determination subunit 2012 is used to determine the corresponding relationship between the first coordinate and the second coordinate as the left eye mapping relationship; and, determine the first sitting position The corresponding relationship between the target and the third coordinate is determined as the right eye mapping relationship.
  • the above-mentioned acquisition subunit 2011 is specifically used to: according to the formula Calculate the longitude coordinate in the second coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, L ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the second coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the above-mentioned acquisition subunit 2011 is specifically used to: according to the formula Calculate the longitude coordinate in the third coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the third coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, R ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the third coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the above device also includes a training module 40 for: obtaining training samples; the training samples include panoramic sample images, and sample depths corresponding to the panoramic sample images Image; use the panoramic sample image as the reference input of the initial depth estimation model, use the sample depth image as the reference output of the initial depth estimation model, train the initial depth estimation model according to the preset loss function, and obtain the depth estimation model.
  • a training module 40 for: obtaining training samples; the training samples include panoramic sample images, and sample depths corresponding to the panoramic sample images Image; use the panoramic sample image as the reference input of the initial depth estimation model, use the sample depth image as the reference output of the initial depth estimation model, train the initial depth estimation model according to the preset loss function, and obtain the depth estimation model.
  • Each module in the above-mentioned binocular stereoscopic panoramic image generating device may be implemented in whole or in part by software, hardware, or a combination thereof.
  • Each of the above modules may be embedded in or independent of the processor of the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • an electronic device is provided, the internal structure diagram of which can be shown in Figure 11.
  • the electronic device includes a processor, memory, communication interface, display screen and input device connected through a system bus.
  • the processor of the electronic device is used to provide computing and control capabilities.
  • the memory of the electronic device includes non-volatile storage media and internal memory.
  • the non-volatile storage medium stores operating systems and computer programs.
  • This internal memory provides an environment for the execution of operating systems and computer programs in non-volatile storage media.
  • the communication interface of the electronic device is used for wired or wireless communication with external terminals.
  • the wireless mode can be implemented through WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies.
  • the display screen of the electronic device may be a liquid crystal display or an electronic ink display.
  • the input device of the electronic device may be a touch layer covered on the display screen, or may be a button, trackball or touch pad provided on the housing of the electronic device. , it can also be an external keyboard, trackpad or mouse, etc.
  • Figure 11 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied.
  • Specific computer equipment can May include more or fewer parts than shown, or combine certain parts, or have a different arrangement of parts.
  • an electronic device including a memory and a processor.
  • a computer program is stored in the memory.
  • the processor executes the computer program, it implements the following steps:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • the processor executes the computer program, the following steps are also implemented: according to the preset interpupillary distance and depth image, obtain the left eye mapping relationship and the right eye mapping relationship; the left eye mapping relationship includes the first of the pixels in the panoramic image. The corresponding relationship between the coordinates and the second coordinate of the pixel point in the left eye panoramic picture; the right eye mapping relationship includes the corresponding relationship between the first coordinate and the third coordinate of the pixel point in the right eye panoramic picture; according to the left eye The mapping relationship and the right-eye mapping relationship map and project the panoramic images respectively to generate a left-eye panoramic picture and a right-eye panoramic picture.
  • the processor when the processor executes the computer program, the following steps are also implemented: obtaining the second coordinates and the third coordinates according to the depth information, the preset interpupillary distance and the first coordinates; and converting the distance between the first coordinates and the second coordinates.
  • the corresponding relationship is determined as the left eye mapping relationship; and the corresponding relationship between the first coordinate and the third coordinate is determined as the right eye mapping relationship.
  • the processor also implements the following steps when executing the computer program: According to the formula Calculate the longitude coordinate in the second coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, L ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the second coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the processor also implements the following steps when executing the computer program: According to the formula Calculate the longitude coordinate in the third coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the third coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, R ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the third coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the processor also implements the following steps when executing the computer program: obtaining training samples; the training samples include panoramic sample images, and sample depth images corresponding to the panoramic sample images; and using the panoramic sample images as reference inputs for the initial depth estimation model. , use the sample depth image as the reference output of the initial depth estimation model, train the initial depth estimation model according to the preset loss function, and obtain the depth estimation model.
  • the processor also implements the following steps when executing the computer program: executing the steps of the method for generating a binocular stereoscopic panoramic image, and generating a binocular stereoscopic panoramic image based on each panoramic image in the panoramic video; and then, based on each binocular stereoscopic panoramic image.
  • Binocular stereoscopic panoramic images are generated to generate binocular stereoscopic panoramic videos.
  • a computer-readable storage medium is provided with a computer program stored thereon.
  • the computer program is executed by a processor, the following steps are implemented:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • the following steps are also implemented: according to the preset interpupillary distance and depth image, obtain the left eye mapping relationship and the right eye mapping relationship; the left eye mapping relationship includes the third pixel point in the panoramic image. The corresponding relationship between the first coordinate and the second coordinate of the pixel point in the left-eye panoramic picture; the right-eye mapping relationship includes the corresponding relationship between the first coordinate and the third coordinate of the pixel point in the right-eye panoramic picture; according to the left-eye mapping relationship The eye mapping relationship and the right eye mapping relationship map and project the panoramic images respectively to generate a left eye panoramic picture and a right eye panoramic picture.
  • the following steps are also implemented: obtaining the second coordinates and the third coordinates based on the depth information, the preset interpupillary distance and the first coordinates; converting the first coordinates to the second coordinates.
  • the corresponding relationship between the first coordinate and the third coordinate is determined as the left eye mapping relationship; and the corresponding relationship between the first coordinate and the third coordinate is determined as the right eye mapping relationship.
  • the computer program also implements the following steps when executed by the processor: according to the formula Calculate the longitude coordinate in the second coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, L ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the second coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the computer program also implements the following steps when executed by the processor: according to the formula Calculate the longitude coordinate in the third coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the third coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, R ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the third coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the following steps are also implemented: obtain training samples; the training samples include panoramic sample images, and sample depth images corresponding to the panoramic sample images; use the panoramic sample images as a reference for the initial depth estimation model Input, use the sample depth image as the reference output of the initial depth estimation model, train the initial depth estimation model according to the preset loss function, and obtain the depth estimation model.
  • the following steps are also implemented: execute the steps of the method for generating a binocular stereoscopic panoramic image, and generate a binocular stereoscopic panoramic image based on each panoramic image in the panoramic video; and then, based on each panoramic image, Binocular stereoscopic panoramic images, generating binocular stereoscopic panoramic videos.
  • a computer program product comprising a computer program that when executed by a processor implements the following steps:
  • the depth image includes the depth information corresponding to each pixel in the panoramic image
  • the panoramic image is mapped into a left-eye panoramic picture and a right-eye panoramic picture;
  • a binocular stereoscopic panoramic image is generated based on the left eye panoramic image and the right eye panoramic image.
  • the following steps are also implemented: according to the preset interpupillary distance and depth image, obtain the left eye mapping relationship and the right eye mapping relationship; the left eye mapping relationship includes the third pixel point in the panoramic image. The corresponding relationship between the first coordinate and the second coordinate of the pixel point in the left-eye panoramic picture; the right-eye mapping relationship includes the corresponding relationship between the first coordinate and the third coordinate of the pixel point in the right-eye panoramic picture; according to the left-eye mapping relationship The eye mapping relationship and the right eye mapping relationship map and project the panoramic images respectively to generate a left eye panoramic picture and a right eye panoramic picture.
  • the following steps are also implemented: obtaining the second coordinates and the third coordinates based on the depth information, the preset interpupillary distance and the first coordinates; converting the first coordinates to the second coordinates.
  • the corresponding relationship between the first coordinate and the third coordinate is determined as the left eye mapping relationship; and the corresponding relationship between the first coordinate and the third coordinate is determined as the right eye mapping relationship.
  • the computer program also implements the following steps when executed by the processor: according to the formula Calculate the longitude coordinate in the second coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the second coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, L ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the second coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the computer program also implements the following steps when executed by the processor: according to the formula Calculate the longitude coordinate in the third coordinate; determine the latitude coordinate in the first coordinate as the latitude coordinate in the third coordinate; where ⁇ is the longitude coordinate in the first coordinate; ⁇ is the latitude coordinate in the first coordinate, D ( ⁇ , ⁇ ) is the depth information corresponding to the first coordinate in the depth image, R ⁇ ( ⁇ , ⁇ ) is the longitude coordinate in the third coordinate corresponding to the first coordinate, and p is the preset interpupillary distance.
  • the following steps are also implemented: obtain training samples; the training samples include panoramic sample images, and sample depth images corresponding to the panoramic sample images; use the panoramic sample images as a reference for the initial depth estimation model Input, use the sample depth image as the reference output of the initial depth estimation model, train the initial depth estimation model according to the preset loss function, and obtain the depth estimation model.
  • the following steps are also implemented: execute the steps of the method for generating a binocular stereoscopic panoramic image, and generate a binocular stereoscopic panoramic image based on each panoramic image in the panoramic video; and then, based on each panoramic image, Binocular stereoscopic panoramic images, generating binocular stereoscopic panoramic videos.
  • the computer program can be stored in a non-volatile computer-readable storage.
  • the computer program when executed, may include the processes of the above method embodiments.
  • Any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory.
  • Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory (MRAM), ferroelectric memory (Ferroelectric Random Access Memory (FRAM)), phase change memory (Phase Change Memory, PCM), graphene memory, etc.
  • Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory.
  • RAM Random Access Memory
  • RAM Random Access Memory
  • RAM random access memory
  • RAM Random Access Memory
  • RAM random access memory
  • RAM Random Access Memory
  • RAM random access memory
  • RAM Random Access Memory
  • SRAM static random access memory
  • DRAM Dynamic Random Access Memory
  • the database involved in the example may include at least one of a relational database and a non-relational database.
  • Non-relational databases may include blockchain-based distributed databases, etc., but are not limited thereto.
  • the processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to this.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请涉及一种双目立体全景图像的生成方法、装置、设备、存储介质和产品，方法包括：将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；然后，根据深度图像以及预设瞳距，将全景图像映射成左眼全景画面和右眼全景画面；根据左眼全景画面和右眼全景画面生成双目立体全景图像；其中，上述深度图像中包括全景图像中各个像素点对应的深度信息。采用上述方法可以将全景图像映射成双目立体全景图像，降低电子设备的成本。

Description

双目立体全景图像的生成方法、装置、设备、存储介质和产品 技术领域
本申请涉及图像处理技术领域，特别是涉及一种双目立体全景图像的生成方法、装置、设备、存储介质和产品。
背景技术
随着虚拟现实(Virtual Reality，VR)等技术的发展，用户对图像的要求越来越高。在VR眼镜等电子设备中，可以向用户展示双目立体全景图像或者视频，在左右眼屏幕分别显示左右眼的图像，用户获取这种带有差异的信息后在脑海中产生立体感。
传统方法中，电子设备可以通过多个镜头同时拍摄同一物体，将多个镜头采集到的图像拼接成左眼全景画面和右眼全景画面，组合得到双目立体全景图像。但是，采用上述方法一般需要专业级多镜头的全景拍摄设备，操作复杂且成本昂贵。
目前普通非立体全景图像/视频的拍摄设备已经非常普遍，且操作简单成本低，亟需一种简单快速的方法，直接通过普通非立体全景图像/视频生成立体全景图像/视频。
发明内容
基于此，有必要针对上述技术问题，提供一种直接通过普通非立体全景图像/视频生成立体全景图像/视频的生成方法、装置、设备、存储介质和产品。
第一方面，提供一种双目立体全景图像的生成方法，上述方法包括：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
在其中一个实施例中，根据预设瞳距以及深度图像，将全景图像映射为左眼全景画面和右眼全景画面，包括：
根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系；
根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在其中一个实施例中，根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系，包括：
根据深度信息、预设瞳距以及第一坐标，获取第二坐标以及第三坐标；
将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
在其中一个实施例中，根据深度信息、预设瞳距以及第一坐标，获取第二坐标，包括：
根据公式计算第二坐标中的经度坐标；
将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标；
其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为预设瞳距。
在其中一个实施例中，根据深度信息、预设瞳距以及第一坐标中的经度坐标，获取第三坐标，包括：
根据公式计算第三坐标中的经度坐标；
将第一坐标中的纬度坐标确定为第三坐标中的纬度坐标；
其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为预设瞳距。
在其中一个实施例中，上述方法还包括：
获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像；
将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
第二方面，提供一种双目立体全景视频的生成方法，上述方法包括：
采用第一方面中所述的双目立体全景图像的生成方法，根据全景视频中的各个全景图像分别生成对应的双目立体全景图像；
基于各双目立体全景图像，生成双目立体全景视频。
第三方面，提供一种双目立体全景图像的生成装置，上述装置包括：
获取模块，用于将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；深度图像中包括全景图像中各个像素点对应的深度信息；
映射模块，用于根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
生成模块，用于根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
第四方面，本申请还提供了一种计算机设备。计算机设备包括存储器和处理器，存储器存储有计算机程序，处理器执行计算机程序时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
第五方面，本申请还提供了一种计算机可读存储介质。计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
第六方面，本申请还提供了一种计算机程序产品。计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
上述双目立体全景图像的生成方法、装置、设备、存储介质和产品，电子设备将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；然后，根据深度图像以及预设瞳距，将全景图像映射成左眼全景画面和右眼全景画面；根据所述左眼全景画面和所述右眼全景画面生成双目立体全景图像；其中，上述深度图像中包括全景图像中各个像素点对应的深度信息。由于电子设备可以获取全景图像的深度图像，从而可以根据深度图像及预设瞳距将上述全景图像分别映射成左眼全景画面和右眼全景画面，获得双目立体全景图像，使得电子设备可以完成全景图像至双目立体全景图像之间的映射转换，而不需要通过专业的多镜头全景拍摄设备完成双目全景立体图像的采集，降低了电子设备的成本，且操作简单。
附图说明
图1为一个实施例中双目立体全景图像的生成方法的应用环境图；
图2为一个实施例中双目立体全景图像的生成方法的流程示意图；
图3为一个实施例中双目立体全景图像的生成方法的示意图；
图4为另一个实施例中双目立体全景图像的生成方法的流程示意图；
图5为另一个实施例中双目立体全景图像的生成方法的流程示意图；
图6为另一个实施例中双目立体全景图像的生成方法的流程示意图；
图7为一个实施例中双目立体全景图像的生成装置的结构框图；
图8为另一个实施例中双目立体全景图像的生成装置的结构框图；
图9为另一个实施例中双目立体全景图像的生成装置的结构框图；
图10为另一个实施例中双目立体全景图像的生成装置的结构框图；
图11为一个实施例中电子设备的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供的双目立体全景图像的生成方法，可以应用于电子设备，电子设备可以对全景图像进行处理，获得全景图像对应的双目立体全景图像。上述电子设备可以但不限于是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备。上述电子设备还可以是相机、摄像机等影像设备；上述相机可以但不限于是普通相机、口袋相机、防抖相机、虚拟现实(Virtual Reality，简称VR)全景相机、运动相机以及消费级或专业级全景相机等。
在一个实施例中，如图1所示，提供了一种双目立体全景图像的生成方法，以该方法应用于电子设备为例进行说明，包括：
S101、将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；深度图像中包括全景图像中各个像素点对应的深度信息。
其中，上述全景图像可以是电子设备通过拍摄获取的，也可以是电子设备中存储的图像，在此不做限定。电子设备拍摄获取的全景图像，可以是电子设备通过全景相机拍摄的图像，也可以是电子设备拍摄的视频中的图像帧，在此不做限定。上述全景图像为电子设备中存储的图像时，可以以图片格式存储在电子设备中，也可以为存储的视频中的视频帧。例如，上述电子设备可以为VR眼镜，上述全景图像可以是输入至VR眼镜中待播放的全景图像。
上述电子设备的相机可以为双鱼眼全景相机，电子设备在拍摄全景图像时，任一个角度可以被双鱼眼全景相机中的其中一个镜头的视野覆盖。电子设备可以将不同镜头拍摄到的图像进行拼接，获得全景图像。
上述深度估计模型可以是神经网络模型，上述深度估计模型可以用于提取全景图像中各个像素点的深度信息，并根据每个像素点对应的深度信息生成该全景图像对应的深度图像。其中，上述深度信息是指拍摄全景图像时，图像中像素点所代表的物体与相机中心之间的距离。
电子设备可以将全景图像输入上述深度估计模型，也可以对全景图像进行预处理之后再输入至深度估计模型，在此不做限定。例如，电子设备对全景图像的预处理操作可以包括对全景图像进行下采样、改变全景图像的投影方式、改变全景图像的亮度或对比度，以及将全景图像转换成单通道灰度图等。上述深度估计模型可以输出全景图像对应的深度图像，上述深度图像的尺寸可以等于全景图像，也可以小于全景图像，在此不做限定。
上述深度图像与上述全景图像可以采用相同的全景投影方式，上述全景投影方式可以是球面投影，也可以是等距圆柱投影，在此不做限定。
S102、根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面。
在获得深度图像的基础上，电子设备可以将全景图像映射成左眼全景画面和右眼全景画面，使得上述左眼全景画面和右眼全景画面产生的视差与上述深度图像对应。
当用户通过左眼观看左眼全景画面，同时通过右眼观看右眼全景画面时，同一个物体在左眼和右眼中出现位置差距，也就是视差。上述视差越大，用户可以感知到该物体的距离越近；上述视差越小，用户感知该物体距离越远。
电子设备将全景图像映射成对应的左眼全景画面和右眼全景画面之后，使得用户通过上述双目立体全景图像产生的视差而感知到的距离，与上述深度图像对应。例如，上述全景图像中包括物体A，通过深度估计模型获得的深度图像中该物体A对应的深度信息为H；电子设备基于上述深度图像，将全景图像映射成左眼全景画面和右眼全景画面之后，用户通过左眼全景画面和右眼全景画面可以感知到该物体A距离用户的距离，该距离与深度信息H对应。
具体地，电子设备可以采用全向立体(Omni-directional stereo，简称ODS)投影方法，将全景图像映射成左眼全景画面和右眼全景画面。
S103、根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
电子设备在获得上述左眼全景画面和右眼全景画面的基础上，可以将上述左眼全景画面和右眼全景画面组成双目立体全景图像。
上述双目立体全景图像的生成方法，电子设备将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；然后，根据深度图像以及预设瞳距，将全景图像映射成左眼全景画面和右眼全景画面；根据所述左眼全景画面和所述右眼全景画面生成双目立体全景图像；其中，上述深度图像中包括全景图像中各个像素点对应的深度信息。由于电子设备可以获取全景图像的深度图像，从而可以根据深度图像及预设瞳距将上述全景图像分别映射成左眼全景画面和右眼全景画面，获得双目立体全景图像，使得电子设备可以完成全景图像至双目立体全景图像之间的映射转换，而不需要通过专业的多镜头全景拍摄设备完成双目全景立体图像的采集，降低了电子设备的成本，且操作简单。
图2为一个实施例中双目立体全景图像的生成方法的流程示意图。本实施例涉及电子设备将全景图像映射为左眼全景画面和右眼全景画面的一种方式，在上述实施例的基础上，如图2所示，上述S102包括：
S201、根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系。
其中，上述瞳距可以用于表征用户左眼瞳孔与右眼瞳孔之间的距离。在一种实现方式中，电子设备中可以存储一个瞳距的预设值，采用预设值对上述全景图像进行映射。
在另一种实现方式中，电子设备可以针对不同的用户采用不同的瞳距；电子设备中可以预设不同用户账号与瞳距的对应关系，上述对应关系中的瞳距可以是用户输入的，也可以是用户在多个预设值中选择的，还可以是电子设备基于用户图像提取获得的，对于上述瞳距的获取方式在此不做限定。例如，用户在使用电子设备时，可以通过电子设备或与电子设备连接的手机等终端进行图像采集，上述图像采集过程可以用户注册过程中，也可以登录过程中，在此不做限定。
在另一种实现方式中，不同类型的电子设备可以对应不同的瞳距。例如，上述电子设备可以是VR眼镜，也可以是智能头盔等，对于不同的电子设备可以采用不同的瞳距，以满足电子设备的双目立体全景图像的映射需求。
电子设备在获得瞳距以及深度图像的基础上，可以基于ODS映射方法获得该全景图像对应的左眼映射关系和右眼映射关系。
上述映射关系为坐标对应关系。对于全景图像中的像素点，可以分别被映射至左眼全景画面和右眼全景画面。上述像素点在全景图像中的坐标可以为第一坐标，在左眼全景画面中的坐标可以为第二坐标，在右眼全景画面中的坐标可以为第三坐标，如图3所示。上述左眼映射关系为全景图像中每个像素点的第一坐标与第二坐标的对应关系，上述右眼映射关系为全景图像中每个像素点的第一坐标与第三坐标之间的对应关系。
S202、根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在获得上述左眼映射关系以及右眼映射关系之后，电子设备可以确定将全景图像中的像素点映射至哪一个位置，进而确定了各个像素点在左眼全景画面和右眼全景画面中的坐标。将上述各个第二坐标与对应的像素值对应之后，获得左眼全景画面。将上述各个第三坐标与对应的像素值对应之后，获得右眼全景画面。
上述双目立体全景图像的生成方法，电子设备电子设备通过瞳距和深度图像，获得左眼映射关系和右眼映射关系，可以准确地将全景图像映射成双目立体全景图像，使得双目立体全景图像可以呈现与该全景图像的深度信息对应的立体效果。
图4为一个实施例中双目立体全景图像的生成方法的流程示意图，本实施例涉及电子设备获得左眼映射关系和右眼映射关系一种方式，在上述实施例的基础上，如图4所示，上述S201包括：
S301、根据所述深度信息、所述预设瞳距以及所述第一坐标，获取所述第二坐标以及所述第三坐标。
上述第一坐标、第二坐标以及第三坐标可以是球面坐标，也可以是三维平面坐标，在此不做限定。电子设备可以根据预设公式进行坐标映射，计算各个第一坐标对应的第二坐标和第三坐标。
在一种实现方式中，上述全景图像和上述深度图像中的各个像素点可以采用球面坐标进行表示；也就是说，每个像素点的坐标可以由经度坐标和纬度坐标构成。
上述预设公式可以包括经度坐标计算公式和纬度坐标计算公式。其中，上述第二坐标以及第三坐标中的经度坐标可以与深度信息、预设瞳距以及第一坐标中的经度坐标有关。对于同一个第一坐标，其对应的第二坐标的经度坐标与第三坐标的经度坐标不同。上述第二坐标的经度坐标与第三坐标的经度坐标之间的差值可以由瞳距与该坐标对应的深度信息的比值获得。由于左眼全景画面和右眼全景画面产生的视差，用于产生距离信息的主要与经度坐标有关，因此，电子设备可以直接将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标，以及第三坐标中的纬度坐标。
针对第二坐标，电子设备可以根据公式计算第二坐标中的经度坐标，并将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标。其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为所述预设瞳距。
第三坐标，电子设备可以根据公式计算第三坐标中的经度坐标，并将所述第一坐标中的纬度坐标确定为所述第二坐标中的纬度坐标。其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为所述预设瞳距。
也就是说，上述预设公式中用于计算纬度坐标的公式可以为：
Rθ(φ，θ)＝Lθ(φ，θ)＝θ
其中，Lθ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，Rθ(φ，θ)为第一坐标对应的第三坐标中的经度坐标。
S302、将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
在获得第二坐标以及第三坐标的基础上，电子设备可以将每个像素点的第一坐标与第二坐标之间的对应关系确定为左眼映射关系，并且将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
上述双目立体全景图像的生成方法，电子设备采用球面坐标完成坐标映射，可以应用于任何投影方式的全景图像中，提高了全景图像至双目立体全景图像进行映射的适用性。
图5为一个实施例中双目立体全景图像的生成方法的流程示意图，本实施例涉及深度估计模型的一种实现方式，在上述实施例的基础上，如图5所示，上述方法还包括：
S401、获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像。
电子设备可以获取双目立体全景样本图像，然后对双目立体全景样本图像进行深度信息提取，获得双目立体全景样本图像对应的样本深度图像；进一步地，电子设备可以对上述双目立体全景样本图像进行单目化处理，获得双目立体全景样本图像对应的全景样本图像。上述全景样本图像及其对应的样本深度图像构成了训练样本。
在另一种实现方式中，可以采用双目立体全景相机和单目全景相机同时针对同一场景进行拍摄，分别获得双目立体全景样本图像和全景样本图像，然后根据双目立体全景样本图像生成样本深度图像后，获得上述训练样本。
S402、将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
在获取训练样本的基础上，电子设备可以将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
上述双目立体全景图像的生成方法，通过样本训练可以获得深度估计模型，从而可以根据深度估计 模型获得全景图像的深度图像，为从全景图像向双目立体全景图像的映射提供了数据基础。
在一个实施例中，提供一种双目立体全景图像的生成方法，如图6所示，上述方法包括：
S501、将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；
S502、根据公式计算全景图像中像素点在左眼全景画面中第二坐标中的经度坐标；
S503、根据公式计算全景图像中像素点在右眼全景画面中第三坐标中的经度坐标；
S504、将像素点在单目全景画面中第一坐标中的纬度坐标确定为第二坐标中的纬度坐标，以及第三坐标中的纬度坐标；
S505、将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系；
S506、根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
S507、根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
上述双目立体全景图像的生成方法，其技术原理和实现效果可以参见上述各实施例，在此不做赘述。
应该理解的是，虽然如上所述的各实施例所涉及的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，如上所述的各实施例所涉及的流程图中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，提供一种双目立体全景视频的生成方法，电子设备可以采用上述双目立体全景图像的生成方法，根据全景视频中的各个全景图像分别生成双目立体全景图像；然后，基于各双目立体全景图像，生成双目立体全景视频。
上述双目立体全景视频的生成方法，其实现原理和技术效果参见上述双目立体全景图像的生成方法的实施例，在此不做赘述。
基于同样的发明构思，本申请实施例还提供了一种用于实现上述所涉及的双目立体全景图像的生成方法的双目立体全景图像的生成装置。该装置所提供的解决问题的实现方案与上述方法中所记载的实现方案相似，故下面所提供的一个或多个双目立体全景图像的生成装置实施例中的具体限定可以参见上文中对于双目立体全景图像的生成方法的限定，在此不再赘述。
在一个实施例中，如图7所示，提供了一种双目立体全景图像的生成装置，包括：
获取模块10，用于将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像；深度图像中包括全景图像中各个像素点对应的深度信息；
映射模块20，用于根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
生成模块30，用于根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
在一个实施例中，在上述实施例的基础上，如图8所示，上述映射模块20包括：
获取单元201，用于根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系；
映射单元202，用于根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在一个实施例中，在上述实施例的基础上，如图9所示，上述获取单元201包括：
获取子单元2011，用于根据深度信息、预设瞳距以及第一坐标，获取第二坐标以及第三坐标；
确定子单元2012，用于将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐 标与第三坐标之间的对应关系确定为右眼映射关系。
在一个实施例中，在上述实施例的基础上，上述获取子单元2011具体用于：根据公式 计算第二坐标中的经度坐标；将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为预设瞳距。
在一个实施例中，在上述实施例的基础上，上述获取子单元2011具体用于：根据公式 计算第三坐标中的经度坐标；将第一坐标中的纬度坐标确定为第三坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为预设瞳距。
在一个实施例中，在上述实施例的基础上，如图10所示，上述装置还包括训练模块40，用于：获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像；将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
上述双目立体全景图像的生成装置，其技术原理和实现效果可以参见上述方法实施例，在此不做赘述。
上述双目立体全景图像的生成装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种电子设备，其内部结构图可以如图11所示。该电子设备包括通过系统总线连接的处理器、存储器、通信接口、显示屏和输入装置。其中，该电子设备的处理器用于提供计算和控制能力。该电子设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该电子设备的通信接口用于与外部的终端进行有线或无线方式的通信，无线方式可通过WIFI、移动蜂窝网络、NFC(近场通信)或其他技术实现。该计算机程序被处理器执行时以实现一种双目立体全景图像的生成方法。该电子设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该电子设备的输入装置可以是显示屏上覆盖的触摸层，也可以是电子设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图11中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种电子设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系；根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：根据深度信息、预设瞳距以及第一坐标，获取第二坐标以及第三坐标；将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：根据公式计算第二坐标中的经度坐标；将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为预设瞳距。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：根据公式计算第三坐标中的经度坐标；将第一坐标中的纬度坐标确定为第三坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为预设瞳距。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像；将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：执行双目立体全景图像的生成方法的步骤，根据全景视频中的各个全景图像分别生成双目立体全景图像；然后，基于各双目立体全景图像，生成双目立体全景视频。
本实施例提供的电子设备，其实现原理和技术效果与上述方法实施例类似，在此不再赘述。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系；根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根根据深度信息、预设瞳距以及第一坐标，获取第二坐标以及第三坐标；将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据公式 计算第二坐标中的经度坐标；将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为预设瞳距。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据公式 计算第三坐标中的经度坐标；将第一坐标中的纬度坐标确定为第三坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为预设瞳距。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像；将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：执行双目立体全景图像的生成方法的步骤，根据全景视频中的各个全景图像分别生成双目立体全景图像；然后，基于各双目立体全景图像，生成双目立体全景视频。
本实施例提供的计算机可读存储介质，其实现原理和技术效果与上述方法实施例类似，在此不再赘述。
在一个实施例中，提供了一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现以下步骤：
将全景图像输入预设的深度估计模型，获得全景图像对应的深度图像，深度图像中包括全景图像中各个像素点对应的深度信息；
根据预设瞳距以及深度图像，将全景图像映射成左眼全景画面和右眼全景画面；
根据左眼全景画面和右眼全景画面，生成双目立体全景图像。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据预设瞳距以及深度图像，获得左眼映射关系和右眼映射关系；左眼映射关系包括全景图像中像素点的第一坐标与像素点在左眼全景画面中的第二坐标之间的对应关系；右眼映射关系包括第一坐标与像素点在右眼全景画面中的第三坐标之间的对应关系；根据左眼映射关系和右眼映射关系，将全景图像分别映射投影，生成左眼全景画面和右眼全景画面。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根根据深度信息、预设瞳距以及第一坐标，获取第二坐标以及第三坐标；将第一坐标与第二坐标之间的对应关系确定为左眼映射关系；以及，将第一坐标与第三坐标之间的对应关系确定为右眼映射关系。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据公式 计算第二坐标中的经度坐标；将第一坐标中的纬度坐标确定为第二坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Lφ(φ，θ)为第一坐标对应的第二坐标中的经度坐标，p为预设瞳距。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：根据公式 计算第三坐标中的经度坐标；将第一坐标中的纬度坐标确定为第三坐标中的纬度坐标；其中，φ为第一坐标中的经度坐标；θ为第一坐标中的纬度坐标，D(φ，θ)为深度图像中第一坐标对应的深度信息，Rφ(φ，θ)为第一坐标对应的第三坐标中的经度坐标，p为预设瞳距。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：获取训练样本；训练样本包括全景样本图像，以及全景样本图像对应的样本深度图像；将全景样本图像作为初始深度估计模型的参考输入，将样本深度图像作为初始深度估计模型的参考输出，根据预设的损失函数对初始深度估计模型进行训练，获得深度估计模型。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：执行双目立体全景图像的生成方法的步骤，根据全景视频中的各个全景图像分别生成双目立体全景图像；然后，基于各双目立体全景图像，生成双目立体全景视频。
本实施例提供的计算机程序产品，其实现原理和技术效果与上述方法实施例类似，在此不再赘述。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory，ROM)、磁带、软盘、闪存、光存储器、高密度嵌入式非易失性存储器、阻变存储器(ReRAM)、磁变存储器(Magnetoresistive Random Access Memory，MRAM)、铁电存储器(Ferroelectric Random Access Memory，FRAM)、相变存储器(Phase Change Memory，PCM)、石墨烯存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或外部高速缓冲存储器等。作为说明而非局限，RAM可以是多种形式，比如静态随机存取存储器(Static Random Access Memory，SRAM)或动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。本申请所提供的各实施 例中所涉及的数据库可包括关系型数据库和非关系型数据库中至少一种。非关系型数据库可包括基于区块链的分布式数据库等，不限于此。本申请所提供的各实施例中所涉及的处理器可为通用处理器、中央处理器、图形处理器、数字信号处理器、可编程逻辑器、基于量子计算的数据处理逻辑器等，不限于此。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请的保护范围应以所附权利要求为准。

Claims (11)

  1. 一种双目立体全景图像的生成方法，其特征在于，所述方法包括：
    将全景图像输入预设的深度估计模型，获得所述全景图像对应的深度图像，所述深度图像中包括所述全景图像中各个像素点对应的深度信息；
    根据预设瞳距以及所述深度图像，将所述全景图像映射成左眼全景画面和右眼全景画面；
    根据所述左眼全景画面和所述右眼全景画面，生成双目立体全景图像。
  2. 根据权利要求1所述的方法，其特征在于，所述根据预设瞳距以及所述深度图像，将所述全景图像映射为左眼全景画面和右眼全景画面，包括：
    根据预设瞳距以及所述深度图像，获得左眼映射关系和右眼映射关系；所述左眼映射关系包括所述全景图像中像素点的第一坐标与所述像素点在所述左眼全景画面中的第二坐标之间的对应关系；所述右眼映射关系包括所述第一坐标与所述像素点在所述右眼全景画面中的第三坐标之间的对应关系；
    根据所述左眼映射关系和所述右眼映射关系，将所述全景图像分别映射投影，生成所述左眼全景画面和所述右眼全景画面。
  3. 根据权利要求2所述的方法，其特征在于，所述根据预设瞳距以及所述深度图像，获得左眼映射关系和右眼映射关系，包括：
    根据所述深度信息、所述预设瞳距以及所述第一坐标，获取所述第二坐标以及所述第三坐标；
    将所述第一坐标与所述第二坐标之间的对应关系确定为所述左眼映射关系；以及，将所述第一坐标与所述第三坐标之间的对应关系确定为所述右眼映射关系。
  4. 根据权利要求3所述的方法，其特征在于，根据所述深度信息、所述预设瞳距以及所述第一坐标，获取所述第二坐标，包括：
    根据公式计算所述第二坐标中的经度坐标；
    将所述第一坐标中的纬度坐标确定为所述第二坐标中的纬度坐标；
    其中，φ为所述第一坐标中的经度坐标；θ为所述第一坐标中的纬度坐标，D(φ，θ)为所述深度图像中所述第一坐标对应的深度信息，Lφ(φ，θ)为所述第一坐标对应的第二坐标中的经度坐标，p为所述预设瞳距。
  5. 根据权利要求3所述的方法，其特征在于，根据所述深度信息、所述预设瞳距以及所述第一坐标中的经度坐标，获取所述第三坐标，包括：
    根据公式计算所述第三坐标中的经度坐标；
    将所述第一坐标中的纬度坐标确定为所述第三坐标中的纬度坐标；
    其中，φ为所述第一坐标中的经度坐标；θ为所述第一坐标中的纬度坐标，D(φ，θ)为所述深度图像中所述第一坐标对应的深度信息，Rφ(φ，θ)为所述第一坐标对应的第三坐标中的经度坐标，p为所述预设瞳距。
  6. 根据权利要求1-5任一项所述的方法，其特征在于，所述方法还包括：
    获取训练样本；所述训练样本包括全景样本图像，以及所述全景样本图像对应的样本深度图像；
    将所述全景样本图像作为初始深度估计模型的参考输入，将所述样本深度图像作为所述初始深度估计模型的参考输出，根据预设的损失函数对所述初始深度估计模型进行训练，获得所述深度估计模型。
  7. 一种双目立体全景视频的生成方法，其特征在于，所述方法包括：
    采用权利要求1-6中任一项所述的双目立体全景图像的生成方法，根据全景视频中的各全景图像分别生成对应的双目立体全景图像；
    基于各所述双目立体全景图像，生成双目立体全景视频。
  8. 一种双目立体全景图像的生成装置，其特征在于，所述装置包括：
    获取模块，用于将全景图像输入预设的深度估计模型，获得所述全景图像对应的深度图像；所述深度图像中包括所述全景图像中各个像素点对应的深度信息；
    映射模块，用于根据预设瞳距以及所述深度图像，将所述全景图像映射成左眼全景画面和右眼全景画面；
    生成模块，用于根据所述左眼全景画面和所述右眼全景画面，生成双目立体全景图像。
  9. 一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器 执行所述计算机程序时实现权利要求1至6中任一项所述的方法的步骤。
  10. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至6中任一项所述的方法的步骤。
  11. 一种计算机程序产品，包括计算机程序，其特征在于，该计算机程序被处理器执行时实现权利要求1至6中任一项所述的方法的步骤。
PCT/CN2023/079064 2022-03-11 2023-03-01 双目立体全景图像的生成方法、装置、设备、存储介质和产品 WO2023169283A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210242661.XA CN114742703A (zh) 2022-03-11 2022-03-11 双目立体全景图像的生成方法、装置、设备和存储介质
CN202210242661.X 2022-03-11

Publications (1)

Publication Number Publication Date
WO2023169283A1 true WO2023169283A1 (zh) 2023-09-14

Family

ID=82275095

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/079064 WO2023169283A1 (zh) 2022-03-11 2023-03-01 双目立体全景图像的生成方法、装置、设备、存储介质和产品

Country Status (2)

Country Link
CN (1) CN114742703A (zh)
WO (1) WO2023169283A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117201705A (zh) * 2023-11-07 2023-12-08 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114742703A (zh) * 2022-03-11 2022-07-12 影石创新科技股份有限公司 双目立体全景图像的生成方法、装置、设备和存储介质
CN117974742B (zh) * 2022-10-19 2024-10-18 摩尔线程智能科技(北京)有限责任公司 双目图像的生成方法、装置、设备、存储介质和程序产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002073003A (ja) * 2000-08-28 2002-03-12 Namco Ltd 立体視画像生成装置及び情報記憶媒体
CN107666606A (zh) * 2016-07-29 2018-02-06 东南大学 双目全景图像获取方法及装置
CN111105347A (zh) * 2019-11-19 2020-05-05 贝壳技术有限公司 一种生成带深度信息的全景图的方法、装置及存储介质
CN111429501A (zh) * 2020-03-25 2020-07-17 贝壳技术有限公司 深度图预测模型生成方法和装置、深度图预测方法和装置
CN114742703A (zh) * 2022-03-11 2022-07-12 影石创新科技股份有限公司 双目立体全景图像的生成方法、装置、设备和存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1353518A1 (en) * 2002-04-09 2003-10-15 STMicroelectronics S.r.l. Process and system for generating stereoscopic images from monocular images
JP5059922B2 (ja) * 2010-09-15 2012-10-31 シャープ株式会社 立体画像生成装置、立体画像表示装置、立体画像調整方法、立体画像調整方法をコンピュータに実行させるためのプログラム、及びそのプログラムを記録した記録媒体
WO2014130019A1 (en) * 2013-02-20 2014-08-28 Intel Corporation Real-time automatic conversion of 2-dimensional images or video to 3-dimensional stereo images or video
CN107545537A (zh) * 2017-08-21 2018-01-05 嘀拍信息科技南通有限公司 一种从稠密点云生成3d全景图片的方法
CN108616746A (zh) * 2017-10-12 2018-10-02 叠境数字科技（上海）有限公司 基于深度学习的2d全景图像转3d全景图像的方法
CN109788270B (zh) * 2018-12-28 2021-04-09 南京美乐威电子科技有限公司 3d-360度全景图像生成方法及装置
CN111556304B (zh) * 2020-04-22 2021-12-31 浙江未来技术研究院(嘉兴) 一种全景影像处理方法、装置及系统
TWI800828B (zh) * 2020-05-11 2023-05-01 美商尼安蒂克公司 用於自單目影像產生立體影像資料之電腦實施方法及電腦可讀儲存媒體

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002073003A (ja) * 2000-08-28 2002-03-12 Namco Ltd 立体視画像生成装置及び情報記憶媒体
CN107666606A (zh) * 2016-07-29 2018-02-06 东南大学 双目全景图像获取方法及装置
CN111105347A (zh) * 2019-11-19 2020-05-05 贝壳技术有限公司 一种生成带深度信息的全景图的方法、装置及存储介质
CN111429501A (zh) * 2020-03-25 2020-07-17 贝壳技术有限公司 深度图预测模型生成方法和装置、深度图预测方法和装置
CN114742703A (zh) * 2022-03-11 2022-07-12 影石创新科技股份有限公司 双目立体全景图像的生成方法、装置、设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
WANG XINHUA, LI DAYU, ZHANG GUANG: "Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision", SENSORS, vol. 21, no. 6, 10 March 2021 (2021-03-10), pages 1944, XP093089688, DOI: 10.3390/s21061944 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117201705A (zh) * 2023-11-07 2023-12-08 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质
CN117201705B (zh) * 2023-11-07 2024-02-02 天津云圣智能科技有限责任公司 一种全景图像的获取方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN114742703A (zh) 2022-07-12

Similar Documents

Publication Publication Date Title
WO2023169283A1 (zh) 双目立体全景图像的生成方法、装置、设备、存储介质和产品
CN106251403B (zh) 一种虚拟立体场景实现的方法、装置和系统
CN104641633B (zh) 用于组合来自多个深度相机的数据的系统和方法
WO2017113681A1 (zh) 一种基于虚拟现实技术的视频图像处理方法及装置
KR20170017700A (ko) 360도 3d 입체 영상을 생성하는 전자 장치 및 이의 방법
US9813693B1 (en) Accounting for perspective effects in images
WO2019237745A1 (zh) 人脸图像处理方法、装置、电子设备及计算机可读存储介质
US10074343B2 (en) Three-dimensional image output apparatus and three-dimensional image output method
US10074216B2 (en) Information processing to display information based on position of the real object in the image
JP7184748B2 (ja) 場面の積層深度データを生成するための方法
CN104205825B (zh) 图像处理装置及方法以及摄像装置
WO2018121401A1 (zh) 一种全景视频图像的拼接方法及全景相机
CN109495733B (zh) 三维影像重建方法、装置及其非暂态电脑可读取储存媒体
GB2554925A (en) Display of visual data with a virtual reality headset
US20150326847A1 (en) Method and system for capturing a 3d image using single camera
JP2018033107A (ja) 動画の配信装置及び配信方法
KR101632514B1 (ko) 깊이 영상 업샘플링 방법 및 장치
KR102534449B1 (ko) 이미지 처리 방법, 장치, 전자 장치 및 컴퓨터 판독 가능 저장 매체
US11451743B2 (en) Control of image output
TWM630947U (zh) 立體影像播放裝置
CN108335336B (zh) 超声成像方法和装置
US20240236288A9 (en) Method And Apparatus For Generating Stereoscopic Display Contents
CN117729320B (zh) 图像显示方法、设备及存储介质
WO2018000610A1 (zh) 一种基于图像类型判断的自动播放方法和电子设备
CN116091572B (zh) 获取图像深度信息的方法、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23765861

Country of ref document: EP

Kind code of ref document: A1