WO2023201771A1 - 基于神经辐射场的计算全息场生成方法及装置 - Google Patents

基于神经辐射场的计算全息场生成方法及装置 Download PDF

Info

Publication number
WO2023201771A1
WO2023201771A1 PCT/CN2022/089982 CN2022089982W WO2023201771A1 WO 2023201771 A1 WO2023201771 A1 WO 2023201771A1 CN 2022089982 W CN2022089982 W CN 2022089982W WO 2023201771 A1 WO2023201771 A1 WO 2023201771A1
Authority
WO
WIPO (PCT)
Prior art keywords
viewing angle
computational
dimensional image
dimensional
amplitude
Prior art date
Application number
PCT/CN2022/089982
Other languages
English (en)
French (fr)
Inventor
于涛
邬京耀
戴琼海
Original Assignee
清华大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华大学 filed Critical 清华大学
Publication of WO2023201771A1 publication Critical patent/WO2023201771A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
    • G03H1/0866—Digital holographic imaging, i.e. synthesizing holobjects from holograms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • the present application relates to the technical field of image data processing or generation, and in particular to a computational holographic field generation method and device based on neural radiation fields.
  • the neural radiation field can model a complex scene using a neural network, construct an implicit expression corresponding to the complex scene, and use the trained neural radiation field network to render the complex scene from any angle. It is a new technology in the field of 3D vision. Emerging research areas.
  • Computational holographic display technology is a technology derived from the development of digital computers and holographic imaging technology that uses digital computers to simulate optical processes and generate holograms. Compared with other three-dimensional display technologies, holographic display contains the amplitude and amplitude of the target. Phase information can accurately present the three-dimensional image of the target, so it is considered the best means to achieve three-dimensional display.
  • This application provides a computational holographic field generation method and device based on neural radiation fields to solve the problem that related technologies are limited by factors such as observation angles and hologram calculation rates, and cannot quickly and effectively observe targets at corresponding angles from multiple perspectives.
  • the first embodiment of the present application provides a computational holographic field generation method based on neural radiation fields, which includes the following steps:
  • point cloud data at the corresponding viewing angle is generated through the color image and depth map, and the holographic amplitude map and phase map at the corresponding viewing angle are calculated to obtain a true value image;
  • the computational hologram at the corresponding viewing angle is obtained from any observation viewpoint, wherein the computational holographic field network is trained by the computational hologram at the corresponding viewing angle.
  • the computational hologram under the corresponding perspective from the arbitrary observation perspective through the pre-trained computational holographic field network it also includes:
  • Construct an initial neural radiation field network train the initial neural radiation field network based on the set loss function, computational holograms at different viewing angles, and corresponding true-value images to obtain the computational holographic field network.
  • using a preset neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle includes:
  • the angle information and the multiple two-dimensional images are input into the neural network, and the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle are obtained.
  • calculating the complex amplitude distribution of each two-dimensional image and superimposing it to obtain a calculated hologram at the corresponding viewing angle includes:
  • P n is the complex amplitude of the n-th two-dimensional image
  • a n is the amplitude distribution of the n-th two-dimensional image output by the neural network
  • j represents the imaginary part of the imaginary number, which is essentially the phase sign
  • the computational hologram at the corresponding viewing angle is calculated according to the complex amplitude distribution of each two-dimensional image, where the calculation formula of the computational hologram is:
  • H m is the calculated hologram under the mth free viewing angle
  • N is the total number of two-dimensional images under the corresponding viewing angle.
  • the loss function includes:
  • the computational hologram under the corresponding perspective is obtained from any observation viewpoint, including:
  • the corresponding complex amplitude distribution is calculated according to the amplitude and phase distribution of the two-dimensional image at each depth, and superimposed to obtain the calculated hologram at the corresponding viewing angle.
  • the second embodiment of the present application provides a computational holographic field generation device based on a neural radiation field, including:
  • the acquisition module is used to obtain the three-dimensional model of the target scene
  • a recording module used to collect color images and depth maps of the target scene from multiple viewing angles, and record the corresponding angle information and internal and external parameters of the camera;
  • the first calculation module is used to generate point cloud data at the corresponding viewing angle through the color image and depth map according to the internal and external parameters, calculate the holographic amplitude map and phase map at the corresponding viewing angle, and obtain a true value image;
  • a sampling module for performing two-dimensional sampling of the three-dimensional model within a preset depth range based on different angles to obtain multiple mutually parallel two-dimensional images at different depths;
  • the second calculation module is used to use a preset neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image at the corresponding viewing angle, calculate the complex amplitude distribution of each two-dimensional image, and superimpose it to obtain the calculation at the corresponding viewing angle.
  • the generation module is used to obtain the computational hologram under the corresponding perspective from any observation viewpoint through the pre-trained computational holographic field network, wherein the computational holographic field network is trained by the computational hologram under the corresponding perspective.
  • a training module is used to train the initial neural radiation field network based on the set loss function, computational holograms at different viewing angles, and corresponding true-value images to obtain the computational holographic field network.
  • the second calculation module includes:
  • a construction unit used to construct the relevant functions and network parameters of the preset neural network
  • An acquisition unit is configured to input the angle information and the plurality of two-dimensional images into the neural network, and acquire the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle.
  • the second calculation module includes:
  • P n is the complex amplitude of the n-th two-dimensional image
  • a n is the amplitude distribution of the n-th two-dimensional image output by the neural network
  • j represents the imaginary part of the imaginary number, which is essentially the phase sign
  • the computational hologram at the corresponding viewing angle is calculated according to the complex amplitude distribution of each two-dimensional image, where the calculation formula of the computational hologram is:
  • H m is the calculated hologram under the mth free viewing angle
  • N is the total number of two-dimensional images under the corresponding viewing angle.
  • the loss function includes:
  • the generation module includes:
  • a depth determination unit configured to determine the two-dimensional image sampling depth range of the scene under the observation viewpoint based on the angle of the observation viewpoint;
  • a sampling unit used to perform two-dimensional image sampling within the sampling depth range to obtain the amplitude and phase distribution of the two-dimensional image at each depth;
  • a superposition unit is used to calculate the corresponding complex amplitude distribution according to the amplitude and phase distribution of the two-dimensional image at each depth, and superpose to obtain the calculated hologram at the corresponding viewing angle.
  • a third embodiment of the present application provides an electronic device, including:
  • a memory a processor, and a computer program stored on the memory and executable on the processor.
  • the processor executes the program to implement the computational holographic field based on the neural radiation field as described in the above embodiments. Generate method.
  • a fourth embodiment of the present application provides a computer-readable storage medium that stores computer instructions, and the computer instructions are used to cause the computer to execute the neural radiation field-based method as described in the above embodiments. Computational holographic field generation methods.
  • Embodiments of the present application can use neural networks to model scenes and obtain implicit expressions of the complex amplitude characteristics of the scene.
  • the amplitude and phase distribution of two-dimensional sampling images of the scene at different depths in that direction can be obtained, and then The computational hologram of the scene at the corresponding observation angle is calculated, which can quickly and effectively realize the holographic reproduction of the scene at a free viewing angle without the need for a large amount of calculations, making up for the limited observation angle and low calculation rate of existing computational holographic displays. defect.
  • This solves the technical problem that related technologies are limited by factors such as observation angle and hologram calculation rate, and cannot quickly and effectively observe three-dimensional images of targets at corresponding angles from multiple viewing angles.
  • Figure 1 is a flow chart of a computational holographic field generation method based on neural radiation fields provided according to an embodiment of the present application
  • Figure 2 is a flow chart of a computational holographic field generation method based on neural radiation fields according to an embodiment of the present application
  • Figure 3 is a schematic structural diagram of a computational holographic field generation device based on neural radiation fields provided according to an embodiment of the present application
  • FIG. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • this application provides a Computational holographic field generation method based on neural radiation field.
  • the neural network can be used to model the scene and obtain the implicit expression of the complex amplitude characteristics of the scene. By inputting any observation angle, the scene at different depths in that direction can be obtained.
  • the amplitude and phase distribution of the two-dimensional sampling image is processed, and then the computational hologram of the scene at the corresponding observation angle is calculated, which can quickly and effectively realize the holographic reproduction of the scene at a free viewing angle without the need for a large amount of calculations, making up for the existing computational holography. It shows the existing defects such as limited observation angle and low calculation speed. This solves the technical problem that related technologies are limited by factors such as observation angle and hologram calculation rate, and cannot quickly and effectively observe three-dimensional images of targets at corresponding angles from multiple viewing angles.
  • FIG. 1 is a schematic flowchart of a method for generating a computational holographic field based on a neural radiation field provided by an embodiment of the present application.
  • the computational holographic field generation method based on neural radiation fields includes the following steps:
  • step S101 a three-dimensional model of the target scene is obtained.
  • the embodiments of the present application can obtain the three-dimensional model of the scene in different ways.
  • the scene can be three-dimensionally rendered and modeled through modeling software, or the scene can be three-dimensionally scanned using lidar.
  • the embodiments of the present application are only for The scene 3D model is therefore not limited by the scene 3D model construction method.
  • step S102 color images and depth maps of the target scene from multiple viewing angles are collected, and corresponding angle information and internal and external parameters of the camera are recorded.
  • the embodiments of the present application can collect color images and depth maps of scenes from multiple viewing angles in different ways.
  • the scene can be captured by an RGB-D (RGB-Depth Map, depth image) camera or used
  • RGB-D RGB-Depth Map, depth image
  • the software renders the scene, etc.
  • the embodiments of this application only focus on the color image and depth image of the scene, as well as the angle and camera internal and external parameter information involved in the acquisition process. Therefore, it is not limited by the scene color image and depth image acquisition method.
  • step S103 based on the internal and external parameters, the point cloud data at the corresponding viewing angle is generated through the color image and the depth map, and the holographic amplitude map and phase map at the corresponding viewing angle are calculated to obtain the true value image.
  • embodiments of the present application can generate point cloud data at corresponding viewing angles through color images and depth maps at different viewing angles based on internal and external parameters of the camera, and calculate holographic data at corresponding viewing angles through the point cloud data.
  • the amplitude image and phase image are used as true value images.
  • the embodiment of the present application can directly perform calculations through point cloud data, or first render the point cloud data into discrete patches and then perform the calculation.
  • the embodiment of the present application focuses on the scene
  • the holographic amplitude truth map and phase truth map under the corresponding viewing angle are therefore not limited by the calculation methods of the scene holographic amplitude truth map and phase truth map.
  • step S104 two-dimensional sampling is performed on the three-dimensional model within a preset depth range based on different angles to obtain multiple mutually parallel two-dimensional images at different depths.
  • embodiments of the present application can perform two-dimensional sampling on a three-dimensional model within a preset depth range based on different angles, thereby obtaining multiple parallel two-dimensional images at different depths, where the sampled two-dimensional
  • the image normal direction is consistent with the observation angle direction and the two-dimensional image resolution is the same as the calculated true image resolution.
  • preset depth range can be set by those skilled in the art according to actual conditions, and is not specifically limited here.
  • step S105 a preset neural network is used to obtain the amplitude distribution and phase distribution of each two-dimensional image at the corresponding viewing angle, the complex amplitude distribution of each two-dimensional image is calculated, and superimposed to obtain the calculated hologram at the corresponding viewing angle.
  • embodiments of the present application can use a preset neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image at the corresponding viewing angle, and calculate the complex amplitude distribution of each two-dimensional image, and then obtain the amplitude distribution and phase distribution at the corresponding viewing angle through superposition. of computational holograms.
  • Embodiments of the present application can use neural networks to establish the observation angle, the connection between the two-dimensional sampling image of the scene along the observation angle and the output sampling image amplitude and phase distribution, and then construct the corresponding neural radiation field, that is, the calculated holographic field, The calculation of the hologram under the corresponding viewing angle is completed through the complex amplitude distribution of each two-dimensional sampling image, thereby realizing the rapid generation of holograms of the scene under a free viewing angle, effectively improving the practicality and applicability of computational holographic display.
  • a preset neural network is used to obtain the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle, including: constructing the correlation function and network parameters of the preset neural network; Input the angle information and multiple two-dimensional images into the neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle.
  • embodiments of the present application can construct the relevant functions and network parameters of the preset neural network, and input the recorded observation angle information and each two-dimensional image at the corresponding perspective into the neural network, thereby obtaining the amplitude of each two-dimensional image at the corresponding perspective. distribution and phase distribution.
  • Embodiments of the present application can connect the two-dimensional sampling image of the scene along the observation angle and the amplitude and phase distribution of the output sampling image, and then construct the corresponding neural radiation field, that is, calculate the holographic field, which is beneficial to the subsequent processing of the corresponding viewing angle.
  • the calculation of holograms under the conditions enables rapid generation of holograms of scenes under free viewing angles, effectively improving the practicality and applicability of computational holographic displays.
  • calculating the complex amplitude distribution of each two-dimensional image and superimposing it to obtain a computational hologram under the corresponding viewing angle includes: calculating the complex amplitude distribution of each two-dimensional image, where, The formula for calculating the complex amplitude distribution is:
  • P n is the complex amplitude of the n-th two-dimensional image
  • a n is the amplitude distribution of the n-th two-dimensional image output by the neural network
  • j represents the imaginary part of the imaginary number, which is essentially the phase sign
  • the computational hologram at the corresponding viewing angle is calculated based on the complex amplitude distribution of each two-dimensional image.
  • the calculation formula of the computational hologram is:
  • H m is the calculated hologram under the mth free viewing angle
  • N is the total number of two-dimensional images under the corresponding viewing angle.
  • step S106 through a pre-trained computational holographic field network, a computational hologram under a corresponding viewing angle is obtained from any observation viewpoint, where the computational holographic field network is trained by a computational hologram under the corresponding viewing angle.
  • the embodiments of the present application can obtain the computational holographic field network from the computational hologram training at the corresponding viewing angle, and through the pre-trained computational holographic field network, the computational hologram under the corresponding viewing angle can be obtained from any observation viewpoint.
  • the application embodiment can construct the corresponding neural radiation field, that is, calculate the holographic field, and complete the calculation of the hologram under the corresponding viewing angle through the complex amplitude distribution of each two-dimensional sampling image, thereby realizing the rapid generation of holograms of the scene under a free viewing angle, effectively Improving the practicality and applicability of computational holographic displays.
  • the computational hologram before obtaining the computational hologram at the corresponding perspective from any observation viewpoint through the pre-trained computational holographic field network, it also includes: constructing an initial neural radiation field network; setting-based loss Functions, computational holograms under different viewing angles and corresponding true-value images are used to train the initial neural radiation field network, and the computational holographic field network is obtained.
  • the embodiment of the present application can construct an initial neural radiation field network and train the initial neural radiation field network based on the set loss function, calculated holograms from different viewing angles and corresponding ground-truth images, and train the initial neural radiation field network.
  • the completed neural radiation field network is the computational holographic field network in the corresponding scene.
  • the loss function includes:
  • the neural radiation field network in the embodiment of this application is a function f( ⁇ ) that takes a two-dimensional vector and a two-dimensional image as input, where the two-dimensional vector is the viewing angle direction ( ⁇ , ⁇ ), and the two-dimensional image is the viewing angle.
  • the output of function f( ⁇ ) is the amplitude distribution A and phase distribution of the corresponding two-dimensional image
  • the preset loss functions include:
  • the computational hologram under the corresponding perspective is obtained from any observation viewpoint, including: based on the angle of the observation viewpoint, determining the two elements of the scene under the observation viewpoint. 2D image sampling depth range; perform 2D image sampling within the sampling depth range to obtain the amplitude and phase distribution of the 2D image at each depth; calculate the corresponding complex amplitude distribution based on the amplitude and phase distribution of the 2D image at each depth, and The calculated hologram at the corresponding viewing angle is obtained by superposition.
  • the embodiments of this application can obtain the computational hologram under the corresponding perspective from any observation viewpoint through the pre-trained computational holographic field network.
  • the specific steps are as follows:
  • Step S201 Obtain the three-dimensional model of the scene.
  • the embodiments of the present application can obtain the three-dimensional model of the scene in different ways.
  • the scene can be three-dimensionally rendered and modeled through modeling software, or the scene can be three-dimensionally scanned using lidar.
  • the embodiments of the present application are only for The scene 3D model is therefore not limited by the scene 3D model construction method.
  • Step S202 Collect color images and depth maps of the scene from multiple viewing angles, and record the corresponding angle information and internal and external parameters of the camera.
  • the embodiments of the present application can collect color images and depth maps of the scene from multiple perspectives in different ways.
  • the scene can be captured by an RGB-D camera or the scene can be rendered using software.
  • This application The embodiment only focuses on the color image and depth image of the scene, as well as the angles and camera internal and external parameter information involved in the acquisition process, and is therefore not limited by the acquisition methods of the scene color image and depth image.
  • Step S203 According to the internal and external parameters of the camera, point cloud data at corresponding viewing angles is generated through color images and depth maps at different viewing angles.
  • embodiments of the present application can generate point cloud data at corresponding viewing angles based on the internal and external parameters of the camera through color images and depth maps at different viewing angles.
  • Step S204 Calculate the holographic amplitude image and phase image under the corresponding viewing angle as the true value image through the point cloud data.
  • the embodiment of the present application can directly perform calculations based on point cloud data, or first render the point cloud data into discrete patches and then perform calculations.
  • the embodiment of the present application focuses on the holographic amplitude true value map and the holographic amplitude true value map of the scene at the corresponding viewing angle.
  • the phase truth map is therefore not limited by the scene holographic amplitude truth map and phase truth map calculation methods.
  • Step S205 Perform two-dimensional sampling on the three-dimensional scene model within a certain depth range based on different angles to obtain multiple mutually parallel two-dimensional images at different depths.
  • embodiments of the present application can perform two-dimensional sampling on a three-dimensional model within a preset depth range based on different angles, thereby obtaining multiple parallel two-dimensional images at different depths, where the sampled two-dimensional The image normal direction is consistent with the observation angle direction and the two-dimensional image resolution is the same as the calculated true image resolution.
  • preset depth range can be set by those skilled in the art according to actual conditions, and is not specifically limited here.
  • Step S206 Use a preset neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle. Furthermore, embodiments of the present application can construct the relevant functions and network parameters of the preset neural network, and input the recorded observation angle information and each two-dimensional image at the corresponding perspective into the neural network, thereby obtaining the amplitude of each two-dimensional image at the corresponding perspective. distribution and phase distribution.
  • the complex amplitude distribution of each two-dimensional image is calculated and superimposed to obtain the computational hologram under the corresponding viewing angle, including: calculating the complex amplitude distribution of each two-dimensional image, where the calculation formula of the complex amplitude distribution is:
  • P n is the complex amplitude of the n-th two-dimensional image
  • a n is the amplitude distribution of the n-th two-dimensional image output by the neural network
  • j represents the imaginary part of the imaginary number, which is essentially the phase sign
  • H m is the calculated hologram under the mth free viewing angle
  • N is the total number of two-dimensional images under the corresponding viewing angle.
  • Step S207 Calculate the complex amplitude distribution of each two-dimensional image and superimpose it to obtain a calculated hologram under the corresponding viewing angle.
  • embodiments of the present application can obtain a computational holographic field network from computational hologram training at a corresponding viewing angle, and obtain a computational hologram at a corresponding viewing angle from any observation viewpoint through the pre-trained computational holographic field network.
  • Step S208 Construct an initial neural radiation field network, and train the initial neural radiation field network based on the set loss function, calculated holograms at different viewing angles, and corresponding true-value images.
  • the neural radiation field network after training is the corresponding scene Computational holographic field network under.
  • the embodiment of the present application can construct an initial neural radiation field network and train the initial neural radiation field network based on the set loss function, calculated holograms from different viewing angles and corresponding ground-truth images, and train the initial neural radiation field network.
  • the completed neural radiation field network is the computational holographic field network in the corresponding scene.
  • the neural radiation field network in the embodiment of this application is a function f( ⁇ ) that takes a two-dimensional vector and a two-dimensional image as input, where the two-dimensional vector is the viewing angle direction ( ⁇ , ⁇ ), and the two-dimensional image is the viewing angle.
  • the output of function f( ⁇ ) is the amplitude distribution A and phase distribution of the corresponding two-dimensional image
  • the preset loss functions include:
  • Step S209 By calculating the holographic field network, given any observation point, the calculated hologram at that point of view can be obtained. Furthermore, the embodiments of this application can obtain the computational hologram under the corresponding perspective from any observation viewpoint through the pre-trained computational holographic field network. The specific steps are as follows:
  • the neural network can be used to model the scene, and the complex amplitude characteristics of the scene can be implicitly expressed.
  • the scene in that direction can be obtained.
  • the amplitude and phase distribution of two-dimensional sampling images at different depths are then calculated to obtain the computational hologram of the scene at the corresponding observation angle, which can quickly and effectively realize the holographic reproduction of the scene at a free viewing angle without the need for a large amount of calculations, making up for the existing Computational holographic displays have shortcomings such as limited observation angle and low calculation rate. This solves the technical problem that related technologies are limited by factors such as observation angle and hologram calculation rate, and cannot quickly and effectively observe three-dimensional images of targets at corresponding angles from multiple viewing angles.
  • Figure 3 is a block schematic diagram of a computational holographic field generation device based on a neural radiation field according to an embodiment of the present application.
  • the neural radiation field-based computational holographic field generation device 10 includes: an acquisition module 100 , a recording module 200 , a first calculation module 300 , a sampling module 400 , a second calculation module 500 and a generation module 600 .
  • the acquisition module 100 is used to acquire a three-dimensional model of the target scene.
  • the recording module 200 is used to collect color images and depth maps of the target scene from multiple viewing angles, and record the corresponding angle information and internal and external parameters of the camera.
  • the first calculation module 300 is used to generate point cloud data at corresponding viewing angles through color images and depth maps based on internal and external parameters, calculate holographic amplitude images and phase images at corresponding viewing angles, and obtain true value images.
  • the sampling module 400 is used to perform two-dimensional sampling of the three-dimensional model within a preset depth range based on different angles to obtain multiple parallel two-dimensional images at different depths.
  • the second calculation module 500 is used to use a preset neural network to obtain the amplitude distribution and phase distribution of each two-dimensional image at the corresponding viewing angle, calculate the complex amplitude distribution of each two-dimensional image, and superimpose it to obtain the calculated hologram at the corresponding viewing angle. picture.
  • the generation module 600 is used to obtain the computational hologram at the corresponding viewing angle from any observation viewpoint through the pre-trained computational holographic field network, where the computational holographic field network is trained by the computational hologram at the corresponding visual angle.
  • the neural radiation field-based computational holographic field generation device 10 further includes: a building module and a training module.
  • the building module is used to build the initial neural radiation field network.
  • the training module is used to train the initial neural radiation field network based on the set loss function, computational holograms from different viewing angles, and corresponding true-value images to obtain the computational holographic field network.
  • the second computing module 500 includes: a construction unit and an acquisition unit.
  • the construction unit is used to construct the relevant functions and network parameters of the preset neural network.
  • the acquisition unit is used to input the angle information and multiple two-dimensional images into the neural network, and obtain the amplitude distribution and phase distribution of each two-dimensional image under the corresponding viewing angle.
  • the second calculation module 500 includes: calculating the complex amplitude distribution of each two-dimensional image, where the calculation formula of the complex amplitude distribution is:
  • P n is the complex amplitude of the n-th two-dimensional image
  • a n is the amplitude distribution of the n-th two-dimensional image output by the neural network
  • j represents the imaginary part of the imaginary number, which is essentially the phase sign
  • the computational hologram at the corresponding viewing angle is calculated based on the complex amplitude distribution of each two-dimensional image.
  • the calculation formula of the computational hologram is:
  • H m is the calculated hologram under the mth free viewing angle
  • N is the total number of two-dimensional images under the corresponding viewing angle.
  • the loss function includes:
  • the neural network can be used to model the scene and obtain the implicit expression of the complex amplitude characteristics of the scene.
  • the scene in that direction can be obtained.
  • the amplitude and phase distribution of two-dimensional sampling images at different depths are then calculated to obtain the computational hologram of the scene at the corresponding observation angle, which can quickly and effectively realize the holographic reproduction of the scene at a free viewing angle without the need for a large amount of calculations, making up for the existing Computational holographic displays have shortcomings such as limited observation angle and low calculation rate. This solves the technical problem that related technologies are limited by factors such as observation angle and hologram calculation rate, and cannot quickly and effectively observe three-dimensional images of targets at corresponding angles from multiple viewing angles.
  • FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device may include:
  • Memory 401 Memory 401, processor 402, and a computer program stored on memory 401 and executable on processor 402.
  • the processor 402 executes the program, it implements the computational holographic field generation method based on the neural radiation field provided in the above embodiment.
  • electronic equipment also includes:
  • Communication interface 403 is used for communication between the memory 401 and the processor 402.
  • Memory 401 is used to store computer programs that can run on the processor 402.
  • the memory 401 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
  • the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc.
  • ISA Industry Standard Architecture
  • PCI Peripheral Component
  • EISA Extended Industry Standard Architecture
  • the bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 4, but it does not mean that there is only one bus or one type of bus.
  • the memory 401, the processor 402 and the communication interface 403 are integrated on one chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through the internal interface.
  • the processor 402 may be a central processing unit (Central Processing Unit, referred to as CPU), or a specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), or one or more processors configured to implement the embodiments of the present application. integrated circuit.
  • CPU Central Processing Unit
  • ASIC Application Specific Integrated Circuit
  • This embodiment also provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the above computational holographic field generation method based on the neural radiation field is implemented.
  • references to the terms “one embodiment,” “some embodiments,” “an example,” “specific examples,” or “some examples” or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include at least one of these features. In the description of this application, “N” means at least two, such as two, three, etc., unless otherwise clearly and specifically limited.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Non-exhaustive list of computer readable media include the following: electrical connections with one or N wires (electronic device), portable computer disk cartridge (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.
  • N steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: discrete logic gate circuits with logic functions for implementing data signals; Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
  • the program can be stored in a computer-readable storage medium.
  • the program can be stored in a computer-readable storage medium.
  • each functional unit in various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module.
  • the above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
  • the storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Graphics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Geometry (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Holo Graphy (AREA)

Abstract

本申请涉及图像数据处理或产生技术领域，特别涉及一种基于神经辐射场的计算全息场生成方法及装置，其中，方法包括：获取目标场景的三维模型；采集多个视角下目标场景的彩色图像和深度图，记录相应的角度信息和相机的内外参数，生成点云数据，计算全息振幅图和相位图，得到真值图像；基于不同角度对三维模型进行二维采样，得到多个二维图像；利用预设的神经网络获取每个二维图像的振幅分布和相位分布，计算并叠加得到对应视角下的计算全息图；通过预先训练的计算全息场网络，得到对应视角下的计算全息图。由此，解决了相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。

Description

基于神经辐射场的计算全息场生成方法及装置
相关申请的交叉引用
本申请基于申请号为202210407271.3，申请日为2022年04月19日申请的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及图像数据处理或产生技术领域，特别涉及一种基于神经辐射场的计算全息场生成方法及装置。
背景技术
神经辐射场可以通过对一个复杂场景利用神经网络建模，构建复杂场景对应的隐式表达，并利用训练好的神经辐射场网络可从任意角度对复杂场景进行场景渲染，是3D视觉领域里一个新兴的研究领域。
计算全息显示技术是随着数字计算机和全息成像技术的发展所衍生的一项利用数字计算机模拟光学过程并生成全息图的技术，与其他三维显示技术相比，全息显示由于包含了目标的振幅和相位信息，可以精确呈现目标的三维图像，因此被认为是实现三维显示的最佳手段。
然而，相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像，有待改善。
发明内容
本申请提供一种基于神经辐射场的计算全息场生成方法及装置，以解决相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。
本申请第一方面实施例提供一种基于神经辐射场的计算全息场生成方法，包括以下步骤：
获取目标场景的三维模型；
采集多个视角下所述目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数；
根据所述内外参数，通过所述彩色图像和深度图生成相应视角下的点云数据，计算对 应视角下的全息振幅图和相位图，得到真值图像；
基于不同角度对所述三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像；
利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图；以及
通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，所述计算全息场网络由所述对应视角下的计算全息图训练得到。
可选地，在所述通过预先训练的计算全息场网络，由所述任意观察视点得到所述对应视角下的计算全息图之前，还包括：
构建初始神经辐射场网络；基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练所述初始神经辐射场网络，得到所述计算全息场网络。
可选地，所述利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，包括：
构建所述预设的神经网络的相关函数与网络参数；
将所述角度信息和所述多个二维图像输入所述神经网络，获取所述对应视角下每个二维图像的振幅分布和相位分布。
可选地，所述计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图，包括：
计算所述每个二维图像的复振幅分布，其中，所述复振幅分布的计算公式为：
Figure PCTCN2022089982-appb-000001
其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
Figure PCTCN2022089982-appb-000002
为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
根据所述每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，所述计算全息图的计算公式为：
Figure PCTCN2022089982-appb-000003
其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
可选地，所述损失函数包括：
Figure PCTCN2022089982-appb-000004
Figure PCTCN2022089982-appb-000005
其中，
Figure PCTCN2022089982-appb-000006
和
Figure PCTCN2022089982-appb-000007
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000008
和
Figure PCTCN2022089982-appb-000009
分别为场景在第m个视角下的振幅真值图和相位真值图。
可选地，所述通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，包括：
基于观察视点的角度，确定所述观察视点下场景的二维图像采样深度范围；
在所述采样深度范围内进行二维图像采样，得到各个深度下二维图像的振幅和相位分布；
根据所述各个深度下二维图像的振幅和相位分布计算相应的复振幅分布，并叠加得到所述对应视角下的计算全息图。
本申请第二方面实施例提供一种基于神经辐射场的计算全息场生成装置，包括：
获取模块，用于获取目标场景的三维模型；
记录模块，用于采集多个视角下所述目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数；
第一计算模块，用于根据所述内外参数，通过所述彩色图像和深度图生成相应视角下的点云数据，计算对应视角下的全息振幅图和相位图，得到真值图像；
采样模块，用于基于不同角度对所述三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像；
第二计算模块，用于利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图；以及
生成模块，用于通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，所述计算全息场网络由所述对应视角下的计算全息图训练得到。
可选地，还包括：
构建模块，用于构建初始神经辐射场网络；
训练模块，用于基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练所述初始神经辐射场网络，得到所述计算全息场网络。
可选地，所述第二计算模块包括：
构建单元，用于构建所述预设的神经网络的相关函数与网络参数；
获取单元，用于将所述角度信息和所述多个二维图像输入所述神经网络，获取所述对应视角下每个二维图像的振幅分布和相位分布。
可选地，所述第二计算模块包括：
计算所述每个二维图像的复振幅分布，其中，所述复振幅分布的计算公式为：
Figure PCTCN2022089982-appb-000010
其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分 布，
Figure PCTCN2022089982-appb-000011
为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
根据所述每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，所述计算全息图的计算公式为：
Figure PCTCN2022089982-appb-000012
其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
可选地，所述损失函数包括：
Figure PCTCN2022089982-appb-000013
Figure PCTCN2022089982-appb-000014
其中，
Figure PCTCN2022089982-appb-000015
和
Figure PCTCN2022089982-appb-000016
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000017
和
Figure PCTCN2022089982-appb-000018
分别为场景在第m个视角下的振幅真值图和相位真值图。
可选地，所述生成模块包括：
深度确定单元，用于基于观察视点的角度，确定所述观察视点下场景的二维图像采样深度范围；
采样单元，用于在所述采样深度范围内进行二维图像采样，得到各个深度下二维图像的振幅和相位分布；
叠加单元，用于根据所述各个深度下二维图像的振幅和相位分布计算相应的复振幅分布，并叠加得到所述对应视角下的计算全息图。
本申请第三方面实施例提供一种电子设备，包括：
存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述程序，以实现如上述实施例所述的基于神经辐射场的计算全息场生成方法。
本申请第四方面实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储计算机指令，所述计算机指令用于使所述计算机执行如上述实施例所述的基于神经辐射场的计算全息场生成方法。
本申请实施例可以利用神经网络对场景进行建模，得到场景的复振幅特性隐式表达，通过输入任意观察角度，得到该方向下场景在不同深度处二维采样图像的振幅和相位分布，进而计算得到对应观测角下场景的计算全息图，可在无需进行大量计算的情况下快速有效地实现自由视角下场景的全息再现，弥补现有计算全息显示存在的观测角度受限和计算速率低等缺陷。由此，解决了相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1为根据本申请实施例提供的一种基于神经辐射场的计算全息场生成方法的流程图；
图2为根据本申请一个实施例的基于神经辐射场的计算全息场生成方法的流程图；
图3为根据本申请实施例提供的一种基于神经辐射场的计算全息场生成装置的结构示意图；
图4为根据本申请实施例提供的电子设备的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
下面参考附图描述本申请实施例的基于神经辐射场的计算全息场生成方法及装置。针对上述背景技术中心提到的相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题，本申请提供了一种基于神经辐射场的计算全息场生成方法，在该方法中，可以利用神经网络对场景进行建模，得到场景的复振幅特性隐式表达，通过输入任意观察角度，得到该方向下场景在不同深度处二维采样图像的振幅和相位分布，进而计算得到对应观测角下场景的计算全息图，可在无需进行大量计算的情况下快速有效地实现自由视角下场景的全息再现，弥补现有计算全息显示存在的观测角度受限和计算速率低等缺陷。由此，解决了相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。
具体而言，图1为本申请实施例所提供的一种基于神经辐射场的计算全息场生成方法的流程示意图。
如图1所示，该基于神经辐射场的计算全息场生成方法包括以下步骤：
在步骤S101中，获取目标场景的三维模型。
可以理解的是，本申请实施例可通过不同方式获取场景的三维模型，例如可以通过建 模软件对场景进行三维渲染建模，或者利用激光雷达对场景进行三维扫描等，本申请实施例仅针对场景三维模型，因此不受场景三维模型构建方法的局限。
在步骤S102中，采集多个视角下目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数。
在实际执行过程中，本申请实施例可通过不同方式采集多个视角下场景的彩色图像和深度图，例如可以通过RGB-D(RGB-Depth Map，深度图像)相机对场景进行实拍或者利用软件对场景进行渲染等，本申请实施例仅针对场景的彩色图像、深度图以及采集过程中涉及到的角度和相机内外参数信息，因此不受场景彩色图像和深度图像采集方法的局限。
在步骤S103中，根据内外参数，通过彩色图像和深度图生成相应视角下的点云数据，计算对应视角下的全息振幅图和相位图，得到真值图像。
作为一种可能实现的方式，本申请实施例可以根据相机的内外参数，通过不同视角下的彩色图像和深度图，生成相应视角下的点云数据，并通过点云数据计算对应视角下的全息振幅图和相位图作为真值图像，举例而言，本申请实施例可以直接通过点云数据进行计算，或者先将点云数据渲染成离散面片后再进行计算，本申请实施例针对场景在对应视角下的全息振幅真值图和相位真值图，因此不受场景全息振幅真值图和相位真值图计算方法的局限。
在步骤S104中，基于不同角度对三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像。
在实际执行过程中，本申请实施例可以基于不同角度，对三维模型在预设深度范围内进行二维采样，进而获得多个不同深度下，相互平行的二维图像，其中，采样的二维图像法线方向与观测角方向一致且二维图像分辨率和计算的真值图像分辨率相同。
需要注意的是，预设深度范围可以由本领域技术人员根据实际情况进行设置，在此不做具体限制。
在步骤S105中，利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图。
具体地，本申请实施例可以利用预设的神经网络获取对应视角下的每个二维图像的振幅分布和相位分布，并计算每个二维图像的复振幅分布，进而通过叠加得到对应视角下的计算全息图。本申请实施例可以利用神经网络建立观测角度、沿着观测角度对场景进行二维采样的图像与输出的采样图像振幅和相位分布之间的联系，进而构建相应的神经辐射场即计算全息场，并通过各个二维采样图像的复振幅分布完成相应视角下全息图的计算，从而实现自由视角下场景的全息图快速生成，有效提高计算全息显示的实用性和适用性。
可选地，在本申请的一个实施例中，利用预设的神经网络获取对应视角下每个二维图 像的振幅分布和相位分布，包括：构建预设的神经网络的相关函数与网络参数；将角度信息和多个二维图像输入神经网络，获取对应视角下每个二维图像的振幅分布和相位分布。
进一步地，本申请实施例可以构建预设的神经网络的相关函数与网络参数，将记录的观测角度信息和对应视角下各个二维图像输入神经网络，从而获取相应视角下各个二维图像的振幅分布和相位分布。本申请实施例可以沿着观测角度对场景进行二维采样的图像与输出的采样图像振幅和相位分布之间的联系，进而构建相应的神经辐射场，即计算全息场，有利于后续进行相应视角下的全息图的计算，从而实现自由视角下场景的全息图快速生成，有效提高计算全息显示的实用性和适用性。
可选地，在本申请的一个实施例中，计算每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图，包括：计算每个二维图像的复振幅分布，其中，复振幅分布的计算公式为：
Figure PCTCN2022089982-appb-000019
其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
Figure PCTCN2022089982-appb-000020
为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
根据每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，计算全息图的计算公式为：
Figure PCTCN2022089982-appb-000021
其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
在步骤S106中，通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，计算全息场网络由对应视角下的计算全息图训练得到。
在实际执行过程中，本申请实施例可以由对应视角下的计算全息图训练得到计算全息场网络，并通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，本申请实施例可以通过构建相应的神经辐射场，即计算全息场，并通过各个二维采样图像的复振幅分布完成相应视角下全息图的计算，从而实现自由视角下场景的全息图快速生成，有效提高计算全息显示的实用性和适用性。
可选地，在本申请的一个实施例中，在通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图之前还包括：构建初始神经辐射场网络；基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练初始神经辐射场网络，得到计算全息场网络。
作为一种可能实现的方式，本申请实施例可以通过构建初始神经辐射场网络，并基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练初始神经辐射场网络，训练完成后的神经辐射场网络即为对应场景下的计算全息场网络。
可选地，在本申请的一个实施例中，损失函数包括：
Figure PCTCN2022089982-appb-000022
Figure PCTCN2022089982-appb-000023
其中，
Figure PCTCN2022089982-appb-000024
和
Figure PCTCN2022089982-appb-000025
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000026
和
Figure PCTCN2022089982-appb-000027
分别为场景在第m个视角下的振幅真值图和相位真值图。
具体地，本申请实施例的神经辐射场网络为一个以二维向量和二维图像作为输入的函数f(·)，其中，二维向量为视角方向(θ,ψ)，二维图像为视角方向(θ,ψ)下某一深度的切割图I，函数f(·)的输出为对应二维图像的振幅分布A和相位分布
Figure PCTCN2022089982-appb-000028
即
Figure PCTCN2022089982-appb-000029
表示各个二维图像的振幅和相位分布，并且预设的损失函数，包括：
Figure PCTCN2022089982-appb-000030
Figure PCTCN2022089982-appb-000031
其中，
Figure PCTCN2022089982-appb-000032
和
Figure PCTCN2022089982-appb-000033
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000034
和
Figure PCTCN2022089982-appb-000035
分别为场景在第m个视角下的振幅真值图和相位真值图。
可选地，在本申请的一个实施例中，通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，包括：基于观察视点的角度，确定观察视点下场景的二维图像采样深度范围；在采样深度范围内进行二维图像采样，得到各个深度下二维图像的振幅和相位分布；根据各个深度下二维图像的振幅和相位分布计算相应的复振幅分布，并叠加得到对应视角下的计算全息图。
进一步地，本申请实施例可以通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其具体步骤如下：
1、给定观察视点的角度
Figure PCTCN2022089982-appb-000036
确定指定视点下场景的二维图像采样深度范围；
2、在采样深度范围内进行二维图像采样，利用计算全息场网络得到各个深度下二维图像的振幅和相位分布；
3、根据各个采样图像的振幅和相位分布计算相应的复振幅分布；
4、对各个采样图像的复振幅分布叠加得到相应观察视点下的计算全息图。
下面结合图2所示，以一个具体实施例对本申请实施例的基于神经辐射场的计算全息场生成方法进行详细阐述。
如图2所示，本申请实施例的步骤如下：
步骤S201：获取场景的三维模型。可以理解的是，本申请实施例可通过不同方式获取 场景的三维模型，例如可以通过建模软件对场景进行三维渲染建模，或者利用激光雷达对场景进行三维扫描等，本申请实施例仅针对场景三维模型，因此不受场景三维模型构建方法的局限。
步骤S202：采集多个视角下场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数。在实际执行过程中，本申请实施例可通过不同方式采集多个视角下场景的彩色图像和深度图，例如可以通过RGB-D相机对场景进行实拍或者利用软件对场景进行渲染等，本申请实施例仅针对场景的彩色图像、深度图以及采集过程中涉及到的角度和相机内外参数信息，因此不受场景彩色图像和深度图像采集方法的局限。
步骤S203：根据相机的内外参数，通过不同视角下的彩色图像和深度图生成相应视角下的点云数据。作为一种可能实现的方式，本申请实施例可以根据相机的内外参数，通过不同视角下的彩色图像和深度图，生成相应视角下的点云数据。
步骤S204：通过点云数据计算对应视角下的全息振幅图和相位图作为真值图像。举例而言，本申请实施例可以直接通过点云数据进行计算，或者先将点云数据渲染成离散面片后再进行计算，本申请实施例针对场景在对应视角下的全息振幅真值图和相位真值图，因此不受场景全息振幅真值图和相位真值图计算方法的局限。
步骤S205：基于不同角度对场景三维模型在一定深度范围内进行二维采样得到多个不同深度下相互平行的二维图像。在实际执行过程中，本申请实施例可以基于不同角度，对三维模型在预设深度范围内进行二维采样，进而获得多个不同深度下，相互平行的二维图像，其中，采样的二维图像法线方向与观测角方向一致且二维图像分辨率和计算的真值图像分辨率相同。
需要注意的是，预设深度范围可以由本领域技术人员根据实际情况进行设置，在此不做具体限制。
步骤S206：利用预设的神经网络获取对应视角下各个二维图像的振幅分布和相位分布。进一步地，本申请实施例可以构建预设的神经网络的相关函数与网络参数，将记录的观测角度信息和对应视角下各个二维图像输入神经网络，从而获取相应视角下各个二维图像的振幅分布和相位分布。
其中，计算每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图，包括：计算每个二维图像的复振幅分布，其中，复振幅分布的计算公式为：
Figure PCTCN2022089982-appb-000037
其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
Figure PCTCN2022089982-appb-000038
为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
根据每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，计算全息图的 计算公式为：
Figure PCTCN2022089982-appb-000039
其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
步骤S207：计算各个二维图像的复振幅分布并叠加得到对应视角下的计算全息图。在实际执行过程中，本申请实施例可以由对应视角下的计算全息图训练得到计算全息场网络，并通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图。
步骤S208：构建初始神经辐射场网络，并基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练初始神经辐射场网络，训练完成后的神经辐射场网络即为对应场景下的计算全息场网络。
作为一种可能实现的方式，本申请实施例可以通过构建初始神经辐射场网络，并基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练初始神经辐射场网络，训练完成后的神经辐射场网络即为对应场景下的计算全息场网络。
具体地，本申请实施例的神经辐射场网络为一个以二维向量和二维图像作为输入的函数f(·)，其中，二维向量为视角方向(θ,ψ)，二维图像为视角方向(θ,ψ)下某一深度的切割图I，函数f(·)的输出为对应二维图像的振幅分布A和相位分布
Figure PCTCN2022089982-appb-000040
即
Figure PCTCN2022089982-appb-000041
表示各个二维图像的振幅和相位分布，并且预设的损失函数，包括：
Figure PCTCN2022089982-appb-000042
Figure PCTCN2022089982-appb-000043
其中，
Figure PCTCN2022089982-appb-000044
和
Figure PCTCN2022089982-appb-000045
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000046
和
Figure PCTCN2022089982-appb-000047
分别为场景在第m个视角下的振幅真值图和相位真值图。
步骤S209：通过计算全息场网络，给定任意观察视点即可得到该视角下的计算全息图。进一步地，本申请实施例可以通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其具体步骤如下：
1、给定观察视点的角度
Figure PCTCN2022089982-appb-000048
确定指定视点下场景的二维图像采样深度范围；
2、在采样深度范围内进行二维图像采样，利用计算全息场网络得到各个深度下二维图像的振幅和相位分布；
3、根据各个采样图像的振幅和相位分布计算相应的复振幅分布；
4、对各个采样图像的复振幅分布叠加得到相应观察视点下的计算全息图。
根据本申请实施例提出的基于神经辐射场的计算全息场生成方法，可以利用神经网络对场景进行建模，得到场景的复振幅特性隐式表达，通过输入任意观察角度，得到该方向 下场景在不同深度处二维采样图像的振幅和相位分布，进而计算得到对应观测角下场景的计算全息图，可在无需进行大量计算的情况下快速有效地实现自由视角下场景的全息再现，弥补现有计算全息显示存在的观测角度受限和计算速率低等缺陷。由此，解决了相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。
其次参照附图描述根据本申请实施例提出的基于神经辐射场的计算全息场生成装置。
图3是本申请实施例的基于神经辐射场的计算全息场生成装置的方框示意图。
如图3所示，该基于神经辐射场的计算全息场生成装置10包括：获取模块100、记录模块200、第一计算模块300、采样模块400、第二计算模块500和生成模块600。
具体地，获取模块100，用于获取目标场景的三维模型。
记录模块200，用于采集多个视角下目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数。
第一计算模块300，用于根据内外参数，通过彩色图像和深度图生成相应视角下的点云数据，计算对应视角下的全息振幅图和相位图，得到真值图像。
采样模块400，用于基于不同角度对三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像。
第二计算模块500，用于利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图。
生成模块600，用于通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，计算全息场网络由对应视角下的计算全息图训练得到。
可选地，在本申请的一个实施例中，基于神经辐射场的计算全息场生成装置10还包括：构建模块和训练模块。
其中，构建模块，用于构建初始神经辐射场网络。
训练模块，用于基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练初始神经辐射场网络，得到计算全息场网络。
可选地，在本申请的一个实施例中，第二计算模块500包括：构建单元和获取单元。
其中，构建单元，用于构建预设的神经网络的相关函数与网络参数。
获取单元，用于将角度信息和多个二维图像输入神经网络，获取对应视角下每个二维图像的振幅分布和相位分布。
可选地，在本申请的一个实施例中，第二计算模块500包括：计算每个二维图像的复振幅分布，其中，复振幅分布的计算公式为：
Figure PCTCN2022089982-appb-000049
其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
Figure PCTCN2022089982-appb-000050
为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
根据每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，计算全息图的计算公式为：
Figure PCTCN2022089982-appb-000051
其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
可选地，在本申请的一个实施例中，损失函数包括：
Figure PCTCN2022089982-appb-000052
Figure PCTCN2022089982-appb-000053
其中，
Figure PCTCN2022089982-appb-000054
和
Figure PCTCN2022089982-appb-000055
分别为场景在第m个视角下的计算全息振幅图和相位图，
Figure PCTCN2022089982-appb-000056
和
Figure PCTCN2022089982-appb-000057
分别为场景在第m个视角下的振幅真值图和相位真值图。
需要说明的是，前述对基于神经辐射场的计算全息场生成方法实施例的解释说明也适用于该实施例的基于神经辐射场的计算全息场生成装置，此处不再赘述。
根据本申请实施例提出的基于神经辐射场的计算全息场生成装置，可以利用神经网络对场景进行建模，得到场景的复振幅特性隐式表达，通过输入任意观察角度，得到该方向下场景在不同深度处二维采样图像的振幅和相位分布，进而计算得到对应观测角下场景的计算全息图，可在无需进行大量计算的情况下快速有效地实现自由视角下场景的全息再现，弥补现有计算全息显示存在的观测角度受限和计算速率低等缺陷。由此，解决了相关技术受限于观察角度和全息图计算速率等因素，无法快速有效地从多个视角观测目标在对应角度下的三维图像的技术问题。
图4为本申请实施例提供的电子设备的结构示意图。该电子设备可以包括：
存储器401、处理器402及存储在存储器401上并可在处理器402上运行的计算机程序。
处理器402执行程序时实现上述实施例中提供的基于神经辐射场的计算全息场生成方法。
进一步地，电子设备还包括：
通信接口403，用于存储器401和处理器402之间的通信。
存储器401，用于存放可在处理器402上运行的计算机程序。
存储器401可能包含高速RAM存储器，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
如果存储器401、处理器402和通信接口403独立实现，则通信接口403、存储器401和处理器402可以通过总线相互连接并完成相互间的通信。总线可以是工业标准体系结构(Industry Standard Architecture，简称为ISA)总线、外部设备互连(Peripheral Component，简称为PCI)总线或扩展工业标准体系结构(Extended Industry Standard Architecture，简称为EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图4中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
可选地，在具体实现上，如果存储器401、处理器402及通信接口403，集成在一块芯片上实现，则存储器401、处理器402及通信接口403可以通过内部接口完成相互间的通信。
处理器402可能是一个中央处理器(Central Processing Unit，简称为CPU)，或者是特定集成电路(Application Specific Integrated Circuit，简称为ASIC)，或者是被配置成实施本申请实施例的一个或多个集成电路。
本实施例还提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上的基于神经辐射场的计算全息场生成方法。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或N个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“N个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更N个用于实现定制逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或N个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，N个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。如，如果用硬件来实现和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。
上述提到的存储介质可以是只读存储器，磁盘或光盘等。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (14)

  1. 一种基于神经辐射场的计算全息场生成方法，其特征在于，包括以下步骤：
    获取目标场景的三维模型；
    采集多个视角下所述目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数；
    根据所述内外参数，通过所述彩色图像和深度图生成相应视角下的点云数据，计算对应视角下的全息振幅图和相位图，得到真值图像；
    基于不同角度对所述三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像；
    利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图；以及
    通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，所述计算全息场网络由所述对应视角下的计算全息图训练得到。
  2. 根据权利要求1所述的方法，其特征在于，在所述通过预先训练的计算全息场网络，由所述任意观察视点得到所述对应视角下的计算全息图之前，还包括：
    构建初始神经辐射场网络；
    基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练所述初始神经辐射场网络，得到所述计算全息场网络。
  3. 根据权利要求1所述的方法，其特征在于，所述利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，包括：
    构建所述预设的神经网络的相关函数与网络参数；
    将所述角度信息和多个二维图像输入所述神经网络，获取所述对应视角下每个二维图像的振幅分布和相位分布。
  4. 根据权利要求1所述的方法，其特征在于，所述计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图，包括：
    计算所述每个二维图像的复振幅分布，其中，所述复振幅分布的计算公式为：
    Figure PCTCN2022089982-appb-100001
    其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
    Figure PCTCN2022089982-appb-100002
    为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
    根据所述每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，所述计算全息图的计算公式为：
    Figure PCTCN2022089982-appb-100003
    其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
  5. 根据权利要求2所述的方法，其特征在于，所述损失函数包括：
    Figure PCTCN2022089982-appb-100004
    Figure PCTCN2022089982-appb-100005
    其中，
    Figure PCTCN2022089982-appb-100006
    和
    Figure PCTCN2022089982-appb-100007
    分别为场景在第m个视角下的计算全息振幅图和相位图，
    Figure PCTCN2022089982-appb-100008
    和
    Figure PCTCN2022089982-appb-100009
    分别为场景在第m个视角下的振幅真值图和相位真值图。
  6. 根据权利要求1-5任一项所述的方法，其特征在于，所述通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，包括：
    基于观察视点的角度，确定所述观察视点下场景的二维图像采样深度范围；
    在所述采样深度范围内进行二维图像采样，得到各个深度下二维图像的振幅和相位分布；
    根据所述各个深度下二维图像的振幅和相位分布计算相应的复振幅分布，并叠加得到所述对应视角下的计算全息图。
  7. 一种基于神经辐射场的计算全息场生成装置，其特征在于，包括：
    获取模块，用于获取目标场景的三维模型；
    记录模块，用于采集多个视角下所述目标场景的彩色图像和深度图，并记录相应的角度信息和相机的内外参数；
    第一计算模块，用于根据所述内外参数，通过所述彩色图像和深度图生成相应视角下的点云数据，计算对应视角下的全息振幅图和相位图，得到真值图像；
    采样模块，用于基于不同角度对所述三维模型在预设深度范围内进行二维采样，得到多个不同深度下相互平行的二维图像；
    第二计算模块，用于利用预设的神经网络获取对应视角下每个二维图像的振幅分布和相位分布，计算所述每个二维图像的复振幅分布，并叠加得到对应视角下的计算全息图；以及
    生成模块，用于通过预先训练的计算全息场网络，由任意观察视点得到对应视角下的计算全息图，其中，所述计算全息场网络由所述对应视角下的计算全息图训练得到。
  8. 根据权利要求7所述的装置，其特征在于，还包括：
    构建模块，用于构建初始神经辐射场网络；
    训练模块，用于基于设置的损失函数、不同视角下的计算全息图和相对应的真值图像训练所述初始神经辐射场网络，得到所述计算全息场网络。
  9. 根据权利要求7所述的装置，其特征在于，所述第二计算模块包括：
    构建单元，用于构建所述预设的神经网络的相关函数与网络参数；
    获取单元，用于将所述角度信息和多个二维图像输入所述神经网络，获取所述对应视角下每个二维图像的振幅分布和相位分布。
  10. 根据权利要求7所述的装置，其特征在于，所述第二计算模块包括：
    计算所述每个二维图像的复振幅分布，其中，所述复振幅分布的计算公式为：
    Figure PCTCN2022089982-appb-100010
    其中，P n为第n个二维图像的复振幅，A n为神经网络输出的第n个二维图像的振幅分布，
    Figure PCTCN2022089982-appb-100011
    为神经网络输出的第n个二维图像的相位分布，j代表虚数的虚部，本质为相位符号；
    根据所述每个二维图像的复振幅分布计算对应视角下的计算全息图，其中，所述计算全息图的计算公式为：
    Figure PCTCN2022089982-appb-100012
    其中，H m为第m个自由视角下的计算全息图，N为对应视角下的二维图像总数。
  11. 根据权利要求8所述的装置，其特征在于，所述损失函数包括：
    Figure PCTCN2022089982-appb-100013
    Figure PCTCN2022089982-appb-100014
    其中，
    Figure PCTCN2022089982-appb-100015
    和
    Figure PCTCN2022089982-appb-100016
    分别为场景在第m个视角下的计算全息振幅图和相位图，
    Figure PCTCN2022089982-appb-100017
    和
    Figure PCTCN2022089982-appb-100018
    分别为场景在第m个视角下的振幅真值图和相位真值图。
  12. 根据权利要求7-11任一项所述的装置，其特征在于，所述生成模块包括：
    深度确定单元，用于基于观察视点的角度，确定所述观察视点下场景的二维图像采样深度范围；
    采样单元，用于在所述采样深度范围内进行二维图像采样，得到各个深度下二维图像的振幅和相位分布；
    叠加单元，用于根据所述各个深度下二维图像的振幅和相位分布计算相应的复振幅分布，并叠加得到所述对应视角下的计算全息图。
  13. 一种电子设备，其特征在于，包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述程序，以实现如权利要求1-6任一项所述的基于神经辐射场的计算全息场生成方法。
  14. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行，以用于实现如权利要求1-6任一项所述的基于神经辐射场的计算全息场生成方法。
PCT/CN2022/089982 2022-04-19 2022-04-28 基于神经辐射场的计算全息场生成方法及装置 WO2023201771A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210407271.3 2022-04-19
CN202210407271.3A CN114529679B (zh) 2022-04-19 2022-04-19 基于神经辐射场的计算全息场生成方法及装置

Publications (1)

Publication Number Publication Date
WO2023201771A1 true WO2023201771A1 (zh) 2023-10-26

Family

ID=81628337

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/089982 WO2023201771A1 (zh) 2022-04-19 2022-04-28 基于神经辐射场的计算全息场生成方法及装置

Country Status (2)

Country Link
CN (1) CN114529679B (zh)
WO (1) WO2023201771A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117372602A (zh) * 2023-12-05 2024-01-09 成都索贝数码科技股份有限公司 一种异构三维多对象融合渲染方法、设备及系统

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114972924A (zh) * 2022-06-10 2022-08-30 北京理工大学 无监督的深度学习全息成像方法及装置
CN115082639B (zh) * 2022-06-15 2023-06-27 北京百度网讯科技有限公司 图像生成方法、装置、电子设备和存储介质
CN115035252B (zh) * 2022-06-20 2023-05-09 北京市燃气集团有限责任公司 一种基于神经辐射场的燃气厂站三维重建方法及装置
CN115061348B (zh) * 2022-08-01 2024-05-17 世优（北京）科技有限公司 基于物点的三维动态全息显示方法及装置
CN116645370B (zh) * 2023-07-27 2023-10-10 山东顺发重工有限公司 基于三维点云数据的高温锻件表面缺陷实时检测方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113269862A (zh) * 2021-05-31 2021-08-17 中国科学院自动化研究所 场景自适应的精细三维人脸重建方法、系统、电子设备
US20210264628A1 (en) * 2020-02-25 2021-08-26 Electronics And Telecommunications Research Institute System and method for digital hologram synthesis and process using deep learning

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11514325B2 (en) * 2018-03-21 2022-11-29 The Regents Of The University Of California Method and system for phase recovery and holographic image reconstruction using a neural network
CN109459923B (zh) * 2019-01-02 2020-10-27 西北工业大学 一种基于深度学习的全息重建算法
CN113099208B (zh) * 2021-03-31 2022-07-29 清华大学 基于神经辐射场的动态人体自由视点视频生成方法和装置
CN113591854B (zh) * 2021-08-12 2023-09-26 中国海洋大学 一种浮游生物全息图的低冗余快速重建方法
CN114004941B (zh) * 2022-01-04 2022-08-16 苏州浪潮智能科技有限公司 一种基于神经辐射场的室内场景三维重建系统及方法

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210264628A1 (en) * 2020-02-25 2021-08-26 Electronics And Telecommunications Research Institute System and method for digital hologram synthesis and process using deep learning
CN113269862A (zh) * 2021-05-31 2021-08-17 中国科学院自动化研究所 场景自适应的精细三维人脸重建方法、系统、电子设备

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
WANG, YUNFENG: "Dynamic Holographic Three-Dimensional Display based on Quantum Dot Doped Liquid Crystal", INFORMATION & TECHNOLOGY, CHINA MASTER'S THESES FULL-TEXT DATABASE, 1 January 2020 (2020-01-01), CN, pages 1 - 78, XP009549805, DOI: 10.27307/d.cnki.gsjtu.2020.001470 *
ZHAO, LINMIN: "Real-Time Generation Method of Hologram Based on Layered Rendering", INFORMATION & TECHNOLOGY, CHINA MASTER'S THESES FULL-TEXT DATABASE, 10 April 2019 (2019-04-10), CN, pages 1 - 66, XP009549812 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117372602A (zh) * 2023-12-05 2024-01-09 成都索贝数码科技股份有限公司 一种异构三维多对象融合渲染方法、设备及系统
CN117372602B (zh) * 2023-12-05 2024-02-23 成都索贝数码科技股份有限公司 一种异构三维多对象融合渲染方法、设备及系统

Also Published As

Publication number Publication date
CN114529679B (zh) 2022-09-16
CN114529679A (zh) 2022-05-24

Similar Documents

Publication Publication Date Title
WO2023201771A1 (zh) 基于神经辐射场的计算全息场生成方法及装置
JP7403528B2 (ja) シーンの色及び深度の情報を再構成するための方法及びシステム
US20140300941A1 (en) Method and apparatus for generating hologram based on multi-view image
CN112733820B (zh) 障碍物信息生成方法、装置、电子设备和计算机可读介质
Shibuya et al. Privacy preserving visual SLAM
JP7479729B2 (ja) 三次元表現方法及び表現装置
JP2022174085A (ja) 場面の積層深度データを生成するための方法
CN112330709A (zh) 一种前景图像提取方法、装置、可读存储介质及终端设备
JPWO2019244944A1 (ja) 三次元再構成方法および三次元再構成装置
Ali et al. Single image Façade segmentation and computational rephotography of House images using deep learning
US20130202221A1 (en) Systems and methods for modifying stereoscopic images
EP3906530B1 (en) Method for 3d reconstruction of an object
CN109816791B (zh) 用于生成信息的方法和装置
Musialski et al. Interactive Multi-View Facade Image Editing.
Ke et al. Towards real-time 3D visualization with multiview RGB camera array
CN112634439B (zh) 一种3d信息展示方法及装置
CN112652056B (zh) 一种3d信息展示方法及装置
JP2004258775A (ja) 光線空間データ処理方法、空間データ処理装置
CN107845108A (zh) 一种光流值计算方法、装置及电子设备
CN115421509A (zh) 一种无人机飞行拍摄规划方法、装置及存储介质
Ke et al. Towards real-time, multi-view video stereopsis
KR20220085369A (ko) 의미론적 객체 매칭을 통한 파노라마 텍스처 매핑 방법 및 시스템
CN116310408B (zh) 一种建立事件相机与帧相机数据关联的方法及装置
JP2019522909A (ja) ピクセルビームを表すデータを生成する装置および方法
EP4350632A2 (en) Method and appratus with neural rendering based on view augmentation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22938006

Country of ref document: EP

Kind code of ref document: A1