WO2022193104A1 - Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé - Google Patents

Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé Download PDF

Info

Publication number
WO2022193104A1
WO2022193104A1 PCT/CN2021/080893 CN2021080893W WO2022193104A1 WO 2022193104 A1 WO2022193104 A1 WO 2022193104A1 CN 2021080893 W CN2021080893 W CN 2021080893W WO 2022193104 A1 WO2022193104 A1 WO 2022193104A1
Authority
WO
WIPO (PCT)
Prior art keywords
distance
cube
small
small cube
light field
Prior art date
Application number
PCT/CN2021/080893
Other languages
English (en)
Chinese (zh)
Inventor
郑凯
韩磊
李琳
李选富
林天鹏
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202180095331.6A priority Critical patent/CN117015966A/zh
Priority to PCT/CN2021/080893 priority patent/WO2022193104A1/fr
Publication of WO2022193104A1 publication Critical patent/WO2022193104A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • the present application relates to the field of computer technology, and in particular, to a method for generating a light field prediction model and a related device.
  • the embodiments of the present application disclose a method for generating a light field prediction model and a related device, which can improve the generation efficiency of the light field prediction model.
  • an embodiment of the present application discloses a method for generating a light field prediction model.
  • the method includes: establishing a cube model surrounding a captured scene according to respective shooting orientations of multiple sample images, wherein the cube model includes multiple a plurality of small cube voxels; then, according to the plurality of sample images, respectively calculate a plurality of cut-off distances for each small cube in the plurality of small cubes, wherein one of each small cube calculated according to the first sample image
  • the truncation distance includes: determining the truncation distance according to the distance from the camera to each small cube and the distance from the camera to the surface of the object in the scene when the first sample image is captured, the first sample image being the any one of the multiple sample images; then sample spatial points from the small cubes according to the multiple cut-off distances of each small cube, wherein each sampled spatial point corresponds to a spatial coordinate; then, according to the sampling The spatial coordinates of the spatial point train a light field prediction
  • the voxel sampling points used for training the deep learning network are obtained based on the depth information of the image, specifically based on the depth information and the voxel to the camera.
  • a truncation distance is calculated from the distance, and then differentiated sampling is performed according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling to key areas and improve the sampling efficiency; on the other hand, the voxel sampled by this sampling method Basically, they are concentrated near the surface of the object. Therefore, the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors.
  • the method further includes: predicting the light field prediction model by using the light field prediction model.
  • the light field of the scene That is to say, after training the pre-stored light field model, the model training device will also predict the light field through the light field prediction model.
  • the Sampling space points in the cube includes: performing fusion calculation on the plurality of cut-off distances of each small cube to obtain the fusion cut-off distance of each small cube; The sample space points in each small cube are described.
  • the fusion cutoff distance is calculated for each small cube, and then sampling is performed according to the fusion cutoff distance.
  • the Sampling space points in a cube includes: determining at least one first small cube whose absolute value of cutoff distance is smaller than a preset threshold in each of the small cubes; and fusing the plurality of cutoff distances of the first small cube Calculate to obtain the fusion cut-off distance of the first small cube; and sample spatial points from the first small cube according to the fusion cut-off distance of the first small cube.
  • the fusion cut-off distance is not calculated for all small cubes, but only for the small cubes whose absolute value of cut-off distance is less than the preset threshold, because when the cut-off distance is large, the corresponding The farther the cube is from the surface of the object, the less necessary for subsequent sampling. Therefore, this application does not perform fusion calculation on the truncation distance of such small cubes, which is equivalent to excluding these small cubes from the sampling category in advance. Without reducing the effect of subsequent sampling, the amount of calculation is reduced, and the generation efficiency of the light field prediction model is improved.
  • the longer the fusion cutoff distance is. Smaller cubes sample more points in space. It can be understood that the smaller the fusion truncation distance is, the closer the cube is to the surface of the object. Compared with the spatial points in other small cubes on the same camera ray, the spatial points in such small cubes can better reflect the pixel information. Therefore, training the light field prediction model based more on the spatial points in such small cubes is beneficial to the subsequent prediction of more accurate images by the light field prediction model.
  • the fusion calculation includes weighted average calculation. It can be understood that the fusion cutoff distance calculated by the weighted average can more accurately reflect the distance between the small cube and the surface of the object.
  • the cutoff distance of the second small cube is calculated based on the first sample image.
  • the weight value occupied in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and/or positively correlated with the first included angle, so
  • the first angle is the angle between the field of view camera ray where the second small cube is located and the normal vector of the surface of the object closest to the second small cube
  • the second small cube is the cube Any small cube in the model.
  • the weight value calculated based on each sample image is used to calculate the fusion cutoff distance, because the weight value is negatively correlated with the distance to the camera, and/or is positive with the first included angle Therefore, when the weight value is combined to calculate the fusion cut-off distance, the influence of different orientations on the fusion cut-off distance can be more accurately reflected.
  • the method further includes: when capturing each sample image in the plurality of sample images, collecting depth information from the shooting angle of each sample image, wherein the depth information is used to characterize the camera The distance to the surface of the object in the scene.
  • the weight value occupied by the cut-off distance of the second small cube calculated from a sample image in the weighted average calculation which may also be called the weight value w of the second small cube calculated according to the first sample image (p)
  • the weight value w(p) satisfies the following relationship:
  • ⁇ is the first included angle
  • distance(v) is the distance from the second small cube to the camera when the first sample image is captured.
  • the weight value of the second small cube calculated according to the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or, it is positively correlated with the first included angle
  • the expression of w(p) here is an optional expression of this idea, so when the weight value is combined to calculate the fusion cutoff distance, it can reflect more accurately The effect of different orientations on the fusion cutoff distance.
  • the calculated value of the second small cube based on the first sample image is The cutoff distance d(p) satisfies the following relationship:
  • sdf(p) is the difference between the distance from the camera to the first small cube when the first sample image is captured, and the distance from the camera to the surface of the object in the scene, and u is a preset threshold.
  • d(p) is only an optional calculation formula for the cut-off distance, and other expressions may also be used in practical applications.
  • the truncation distance of the small cubes in one range is assigned as 1, and the truncation distance of the small cubes in another range is assigned as -1, which is conducive to the subsequent processing of the two types of small cubes. Thereby improving computational efficiency.
  • an embodiment of the present application provides an apparatus for generating a light field prediction model, the apparatus comprising:
  • a establishing unit configured to establish a cube model surrounding the scene to be shot according to the respective shooting orientations of the multiple sample images, wherein the cube model includes a plurality of small cube voxels;
  • a first calculation unit configured to calculate a plurality of cutoff distances of each small cube in the plurality of small cubes according to the plurality of sample images, wherein one of each small cube calculated according to the first sample image
  • the truncation distance includes: determining the truncation distance according to the distance from the camera to each small cube and the distance from the camera to the surface of the object in the scene when the first sample image is captured, the first sample image being the any one of the multiple sample images;
  • a sampling unit configured to sample spatial points from the small cubes according to a plurality of cutoff distances of each small cube, wherein each sampled spatial point corresponds to a spatial coordinate
  • the second computing unit is configured to train a light field prediction model according to the sampled spatial coordinates of the spatial point, wherein the light field prediction model is used to predict the light field of the scene.
  • the voxel sampling points used for training the deep learning network are obtained based on the depth information of the image, specifically based on the depth information and the voxel to the camera.
  • a truncation distance is calculated from the distance, and then differentiated sampling is performed according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling to key areas and improve the sampling efficiency; on the other hand, the voxel sampled by this sampling method Basically, they are concentrated near the surface of the object. Therefore, the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors.
  • the device further includes:
  • a prediction unit configured to predict the light field of the scene by using the light field prediction model. That is to say, after training the pre-stored light field model, the model training device will also predict the light field through the light field prediction model.
  • the sampling unit is specifically used to: perform fusion calculation on the plurality of cut-off distances of each small cube to obtain the fusion cut-off distance of each small cube; according to each small cube The fusion cut-off distance samples spatial points from each of the small cubes.
  • the fusion cutoff distance is calculated for each small cube, and then sampling is performed according to the fusion cutoff distance.
  • the sampling unit is specifically configured to: determine at least one first small cube whose absolute value of the truncation distance is less than a preset threshold in each small cube; A fusion calculation is performed on a plurality of cutoff distances to obtain a fusion cutoff distance of the first small cube; spatial points are sampled from the first small cube according to the fusion cutoff distance of the first small cube.
  • the fusion cut-off distance is not calculated for all small cubes, but only for the small cubes whose absolute value of cut-off distance is less than the preset threshold, because when the cut-off distance is large, the corresponding The farther the cube is from the surface of the object, the less necessary for subsequent sampling. Therefore, this application does not perform fusion calculation on the truncation distance of such small cubes, which is equivalent to excluding these small cubes from the sampling category in advance. Without reducing the effect of subsequent sampling, the amount of calculation is reduced, and the generation efficiency of the light field prediction model is improved.
  • the fusion cutoff distance is longer. Smaller cubes sample more points in space. It can be understood that the smaller the fusion truncation distance is, the closer the cube is to the surface of the object. Compared with the spatial points in other small cubes on the same camera ray, the spatial points in such small cubes can better reflect the pixel information. Therefore, training the light field prediction model based more on the spatial points in such small cubes is beneficial to the subsequent prediction of more accurate images by the light field prediction model.
  • the fusion calculation includes weighted average calculation. It can be understood that the fusion cutoff distance calculated by the weighted average can more accurately reflect the distance between the small cube and the surface of the object.
  • the cutoff distance of the second small cube is calculated based on the first sample image.
  • the weight value occupied in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and/or positively correlated with the first included angle, so
  • the first angle is the angle between the field of view camera ray where the second small cube is located and the normal vector of the surface of the object closest to the second small cube
  • the second small cube is the cube Any small cube in the model.
  • the weight value calculated based on each sample image is used to calculate the fusion cutoff distance, because the weight value is negatively correlated with the distance to the camera, and/or is positive with the first included angle Therefore, when the weight value is combined to calculate the fusion cut-off distance, the influence of different orientations on the fusion cut-off distance can be more accurately reflected.
  • the device further includes:
  • an acquisition unit configured to acquire depth information from a shooting angle of view of each sample image when shooting each sample image in the plurality of sample images, wherein the depth information is used to represent the distance from the camera to the scene distance from the surface of the object.
  • the weight value of the cut-off distance calculated from a sample image in the weighted average calculation can also be called the weight value w(p) of the second small cube calculated according to the first sample image.
  • the weight value w(p) satisfies the following relationship:
  • ⁇ is the first included angle
  • distance(v) is the distance from the second small cube to the camera when the first sample image is captured.
  • the weight value of the second small cube calculated according to the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or, it is positively correlated with the first included angle
  • the expression of w(p) here is an optional expression of this idea, so when the weight value is combined to calculate the fusion cutoff distance, it can reflect more accurately The effect of different orientations on the fusion cutoff distance.
  • the second small cube calculated according to the first sample image is The cutoff distance d(p) satisfies the following relationship:
  • sdf(p) is the difference between the distance from the camera to the first small cube when the first sample image is captured, and the distance from the camera to the surface of the object in the scene, and u is a preset threshold.
  • d(p) is only an optional calculation formula for the cut-off distance, and other expressions may also be used in practical applications.
  • the truncation distance of the small cubes in one range is assigned as 1, and the truncation distance of the small cubes in another range is assigned as -1, which is conducive to the subsequent processing of the two types of small cubes. Thereby improving computational efficiency.
  • an embodiment of the present application provides a device for generating a light field prediction model, including a processor and a memory, wherein the memory is used to store a computer program, and when the computer program runs on the processor, the first The method described in one aspect or any optional solution of the first aspect.
  • an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program runs on a processor, the first aspect or the first aspect is implemented Any of the alternatives of the method described.
  • the voxel sampling points used for training a deep learning network are obtained based on the depth information of the image, specifically based on the depth information and the voxel to
  • the distance of the camera calculates a truncation distance, and then performs differential sampling according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling in key areas and improve the sampling efficiency;
  • the voxel is basically concentrated near the surface of the object, so the subsequent deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors .
  • FIG. 1 is a schematic diagram of a scenario for acquiring NeRF provided by an embodiment of the present application.
  • FIG. 2A is a schematic diagram of a sampling voxel scenario provided by an embodiment of the present application.
  • 2B is a schematic diagram of a three-dimensional light field and RGB information changing with depth information according to an embodiment of the present application
  • FIG. 3 is a schematic diagram of the architecture of a model training provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of a method for determining a three-dimensional light field of a scene provided by an embodiment of the present application
  • FIG. 5 is a schematic diagram of the distance from a camera to the voxel and the surface of an object provided by an embodiment of the present application;
  • FIG. 6 is a schematic diagram of a scenario of a cut-off distance provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of the distribution of a truncation distance provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram illustrating a comparison of prediction effects of a light field prediction model provided by an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of an apparatus for generating a light field prediction model provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of another apparatus for generating a light field prediction model provided by an embodiment of the present application.
  • FIG. 1 is a schematic diagram of a scene for obtaining Neural Radiance Fields (NeRF).
  • NeRF Neural Radiance Fields
  • a sparse image data set is used to synthesize a three-dimensional light field of a complex scene, which specifically includes :
  • the deep learning network reconstructs (i.e.
  • the corresponding RGB information which can be expressed as RGB ⁇ , includes density and color.
  • the deep learning network After training the deep learning network with the 5D coordinates of the spatial points collected on the camera ray according to the above (a), (b), (c), and (d) processes, the deep learning network can predict the new 5D coordinates. Therefore, for a scene represented by a 5D coordinate system, the deep learning network can predict the view of viewing the scene from any angle, and the view set of all viewing angles of the scene is the three-dimensional light field of the scene.
  • the inventor of the present application found that for a certain field of view (camera ray, also called viewing ray), if it passes through the surface of the object (surface) point, the RGB information (or pixel) corresponding to the camera ray is mainly characterized by the depth and color of the point on the surface of the object.
  • camera ray also called viewing ray
  • the RGB information (or pixel) corresponding to the camera ray is mainly characterized by the depth and color of the point on the surface of the object.
  • parts (e) and (f) are uniformly sampled, and the sampling granularity of part (f) is smaller; then, for Each sampling point is trained and analyzed to obtain the sampling points near the surface of the object.
  • the approximate range of the "surface of the object” can be determined. Therefore, based on the approximate range, further training and analysis can be used to obtain the object.
  • the weights and RGB information of the deep learning network for predicting 3D light fields have high coincidence in terms of being affected by depth.
  • the depth information can reflect the distance from the camera to the surface of the object. Therefore, it can be considered that the weight and RGB information of the deep learning network used to predict the three-dimensional light field have a high coincidence in terms of being affected by the surface of the object.
  • the depth information can reflect the distance from the camera to the surface of the object, so importing the depth information into the entire deep learning network can reduce the invalid computation of the deep learning network and improve the convergence speed and efficiency of the deep learning network.
  • the depth information is used to focus on sampling and training near the depth value of each camera ray, so as to ensure that the deep learning network quickly converges to the surface of the object on the camera ray in the early stage of training, and concentrates computing power to represent the texture details of the object. , to avoid blur and structural errors.
  • FIG. 3 is a schematic diagram of a model training architecture provided by an embodiment of the present application.
  • the architecture includes a model training device 301 and one or more model using devices 302.
  • the model training device 301 and the model using device 302 Communication between them is carried out by wire or wireless, so the model training device 301 can send the trained deep learning network (or light field prediction model) for predicting the three-dimensional light field to the model using device 302; accordingly, the model
  • the device 302 is used to predict the three-dimensional light field in a particular scene through the received deep learning network.
  • the model training device itself predicts the three-dimensional light field in the scene based on the deep learning network obtained by training.
  • the model using device 302 can feed back the result predicted based on the model to the above-mentioned model training device 301, so that the model training device 301 can further train the model based on the prediction result of the model using device 302; retraining;
  • the good model can be sent to the model usage device 302 to update the original model.
  • the model training device 301 may be a device with relatively strong computing power, for example, a server, or a server cluster composed of multiple servers.
  • the model-using device 302 is a device that needs to acquire a three-dimensional light field of a specific scene, such as a handheld device (eg, a mobile phone, a tablet computer, a PDA, etc.), a vehicle-mounted device (eg, a car, a bicycle, an electric vehicle, an airplane, a ship, etc.) ), wearable devices (such as smart watches (such as iWatch, etc.), smart bracelets, pedometers, etc.), smart home devices (such as refrigerators, TVs, air conditioners, electricity meters, etc.), smart robots, workshop equipment, etc.
  • a handheld device eg, a mobile phone, a tablet computer, a PDA, etc.
  • a vehicle-mounted device eg, a car, a bicycle, an electric vehicle, an airplane, a ship, etc.
  • wearable devices such as smart watches (such as iWatch, etc.), smart bracelets, pedometers, etc.), smart home devices (such as refrigerators, TVs,
  • the following is an example of the model using device 302 being a car and a mobile phone, respectively.
  • navigation maps play a very critical role in improving the efficiency of cars on the road; on some complex roads, it is often difficult for users to obtain comprehensive information on the road.
  • using the method of this embodiment of the present application can predict the three-dimensional light field of a specific scene, so the user can be presented with the effect of viewing complex roads from various directions, which is beneficial for the user to control driving and improve the efficiency of vehicle traffic.
  • 3D light fields need to be predicted, such as VR house viewing, VR movies, games, street scene production, etc.
  • FIG. 4 is a schematic flowchart of a method for determining a three-dimensional light field of a scene provided by an embodiment of the present application.
  • the method may be implemented based on the architecture shown in FIG. 3, or may be implemented based on other architectures.
  • steps S400 - S405 may be implemented by the model training device 301
  • step S406 may be implemented by the model using device 302 .
  • steps S400-S406 can be completed by one device, or can be completed by multiple devices in cooperation.
  • the application field of the one device or the multiple devices is not limited here, and corresponding devices can be provided. Computing power, and/or communication capabilities are sufficient. Steps S400-S406 are as follows:
  • Step S400 Inputting a plurality of sample images and information of shooting orientations of the plurality of sample images into the deep learning network.
  • the multiple sample images are images of the same scene captured by the camera at different orientations.
  • the orientation includes position coordinates (x, y, z) and a viewing angle direction.
  • x, y, and z in the position coordinates (x, y, z) represent longitude, latitude, and altitude respectively.
  • ⁇ in represent the horizontal and vertical angles, respectively.
  • the orientation can also be expressed in other ways.
  • Step S401 establishing a cube model surrounding the captured scene according to the shooting orientation of the multiple sample images when shooting.
  • a cube model that can surround the scene can be constructed based on multiple different orientations when the camera takes the multiple photos.
  • the length, width, and height of the cube model are calculated from the multiple orientations.
  • the scene here is not limited, for example, it can be a scene with people as the main object, another example, it can be a scene with a tree as the main object, another example, it can be a scene with the internal structure of the house as the main scene, etc. .
  • the multiple different orientations mentioned here may be four orientations of east, west, south, and north, or may be different orientations under other reference objects.
  • the cube model is divided into a plurality of small cubes (grid voxel), referred to as voxel.
  • the voxel can be obtained by dividing the cube model into N equal parts, and its size can be set according to actual needs. It can be understood that when The smaller the voxel, the better the training accuracy of the subsequent deep learning network, but the smaller the voxel will also lead to the computational pressure during deep learning network training.
  • the size of the voxel is considered in combination with the accuracy and the computing power of the device.
  • the voxel can be The size is set to 2 centimeters (cm).
  • centimeters Generally speaking, for any voxel, it is considered that it is either located on the surface of the object in the above scene, or it is not located on the surface of the object (it can be considered to be located in an open position in the scene).
  • a GPU thread can process one.
  • a voxel on (x, y), that is, a GPU process scans a lattice column at (x, y) coordinates.
  • the coordinates of the center point of the voxel are generally used as the coordinates of the voxel.
  • the coordinates of other points in the voxel can also be used as the coordinates of the voxel, for example, the upper left corner of the voxel and the upper right corner of the voxel. Wait.
  • Step S402 Calculate a plurality of cut-off distances for each small cube in the plurality of small cubes according to the plurality of sample images.
  • the first sample image is taken as
  • the first sample image is any one of the plurality of sample images.
  • the cut-off distance of each small cube calculated according to the first sample image is calculated according to the distance between the camera and each small cube when the first sample image was captured and the distance between the camera and the surface of the object in the scene. The distance is determined, and an example is given below.
  • the camera will also collect the depth information of the scene when shooting in the first orientation.
  • the depth information reflects the distance from the camera to the surface of the object in the scene, so the camera is in the In the image captured in one direction, each pixel x corresponds to a depth value, and the depth value (x) corresponding to the pixel x reflects the voxel to the surface of the object on the camera ray corresponding to the pixel x.
  • the distance value(x) of this camera is also collect the depth information of the scene when shooting in the first orientation.
  • the depth information reflects the distance from the camera to the surface of the object in the scene, so the camera is in the In the image captured in one direction, each pixel x corresponds to a depth value, and the depth value (x) corresponding to the pixel x reflects the voxel to the surface of the object on the camera ray corresponding to the pixel x. The distance value(x) of this camera.
  • the voxel near the surface of the object should be highlighted. Assuming that the distance from the surface of the object does not exceed the preset threshold u, it is considered to be located near the surface of the object. Then, whether the voxel is located nearby and how close it is can be determined by the truncation distance d(p) represents, wherein, the calculation method of the cut-off distance d(p) of the second small cube voxel can be as follows:
  • FIG. 7 illustrates the distribution of the cutoff distance d(p) of some small cube voxels in the cube model corresponding to the scene captured by the camera.
  • the weight value of each voxel can also be calculated.
  • the distance of the camera is negatively correlated, and/or, is positively correlated with the first included angle, and the first included angle is the line of view camera ray where the second small cube is located and the closest distance to the second small cube.
  • This weight value is used for the subsequent fusion calculation of the cutoff distance of the second small cube.
  • the reason for determining the weight value of each voxel in this way is: when the camera shoots in the first orientation, for any voxel in the above-mentioned cube model, its pixel information (including density, color, etc. (color, etc.) is affected by many aspects, for example, the closer it is to the camera, the more pixel information, and the smaller the corresponding first angle (equivalent to the smaller the angle deviated from the camera), the more pixel information.
  • the expression of the weight value w(p) of the second small cube voxel can be as follows:
  • ⁇ is the first included angle
  • distance(v) is the distance from the second small cube to the camera when the first sample image is captured.
  • the cut-off distance d(p) and weight value w(p) of each voxel in the cube model can be calculated.
  • This set of cut-off distance d(p) and weight value w(p) is a parameter relative to the first orientation.
  • the truncation distance d(p) and weight value w(p) of each voxel in the cube model can also be calculated.
  • This set of truncation distance d(p) and weight value w( p) is a parameter relative to another orientation.
  • the coordinates of the voxel and the camera can be unified into one coordinate system to facilitate the calculation.
  • the position g of each voxel in the cube model can be converted into the position point p in the world coordinate system, and then the position point p in the world coordinate system can be determined according to the camera pose matrix.
  • the mapping point v in the camera coordinate system, and the corresponding pixel point x in the depth image is determined according to the camera internal parameter matrix and the mapping point v point; then, the depth value of the pixel point x is obtained, and the depth value of the pixel point x is in the cube model.
  • the distance value(x) from the voxel on the surface of the object to the camera, and the distance from the mapping point v to the origin of the camera coordinate system is recorded as distance(v); obtained value(x) and distance(v) can then be used to calculate the truncation distance d(p) and the weight value w(p).
  • depth information is collected from the shooting angle of the each sample image, wherein the depth information is used to represent the distance from the camera to all the sample images.
  • the distance to the surface of the object in the scene that is, the above value(x).
  • the depth information may be specifically collected by a sensor, for example, the sensor may be a radar sensor, an infrared sensor or the like, and the sensor may be deployed on or near the camera.
  • Step S403 Perform fusion calculation on the cut-off distances calculated according to the plurality of sample images respectively, to obtain the fusion cut-off distance of the small cubes in the cube model.
  • the fusion calculation is performed on the multiple truncation distances of each small cube to obtain the fusion truncation distance of each small cube; in subsequent sampling, it is based on the The fusion cutoff distance of the small cubes samples the spatial points from each of the small cubes.
  • each small cube has at least one first small cube whose absolute value of the truncation distance is less than a preset threshold;
  • the truncation distance of the first small cube is fused to calculate the fusion truncation distance of the first small cube in the cube model. For example, if multiple truncation distances of a voxel are calculated based on multiple sample images, then if the If the smallest cutoff distance among the plurality of cutoff distances is smaller than the preset threshold, the one voxel is considered to be the first small cube.
  • the preset threshold may be set to 1. During subsequent sampling, spatial points are sampled from the first small cube according to the fusion cutoff distance of the first small cube.
  • the principle of fusion calculation may be that the fusion cut-off distance of the second small cube is obtained by performing a weighted average calculation of cut-off distances calculated based on different sample images, and the second small cube is the one in the cube model. Any small cube.
  • the weight value occupied by the cut-off distance of the second small cube calculated based on the first sample image in the weighted average calculation is equal to the previous "The second small cube calculated based on the first sample image”. Weight value w(p)".
  • the following operations are performed for multiple sample images: the cutoff distance d(p) calculated based on one of the sample images is used as the initial fusion cutoff distance D(p), and the weight calculated based on the one sample image is used as the initial fusion cutoff distance D(p) The value w(p) is used as the initial fusion weight value W(p); then, for other sample images, the following operations are performed in turn, and the truncation distance d(p) calculated based on the current sample image is integrated into the existing fusion truncation distance D( p) to update the fusion cut-off distance D(p), and integrate the weight value w(p) calculated based on the current sample image into the existing fusion weight value W(p) to the fusion weight value W(p) Update until the truncation distance d(p) and weight value w(p) calculated based on each sample image are fused.
  • the expression of fusion calculation can be as follows:
  • D(p) is the fusion cut-off distance of the second small cube
  • W(p) is the fusion weight value of the second small cube
  • d(p) is the cut-off distance of the second small cube calculated based on the current sample image
  • w(p) is the weight value of the second small cube calculated based on the current sample image.
  • the final fusion cut-off distance D(p) and fusion weight value W(p) of all voxels in the cube model can be calculated in this way.
  • the final fusion cut-off distance D(p) and fusion weight value W(p) of all voxels in the cube model can be input into Marching Cube, and the triangular faces can be calculated to present the cut-off distance field of the cube model.
  • Step S404 Sample spatial points from the small cubes according to the fusion cutoff distance of the small cubes in the cube model.
  • the spatial point is sampled from the voxel in the cube model, and for the first optional solution in step S403, the fusion cut-off distance of each voxel can be based on each voxel.
  • the spatial points are sampled in the voxel, for the second optional solution in step S403, the spatial points can be sampled from the first small cube based on the fusion cutoff distance of the first small cube (ie, the first voxel).
  • sampling is that in the cube model, the smaller the fusion cutoff distance is, the more space points are sampled by the small cube; for example, for any phase plane, on the camera ray emitted by the phase plane, the fusion cutoff distance D(p ), the closer the voxel is to 0, the greater the sampling density and the more sampling times. If the fusion cut-off distance D(p) is closer to 1 or -1, the sampling density is smaller and the sampling times less for such voxels.
  • the number of samples of cubes whose fusion cut-off distance is greater than or equal to a preset threshold in the cube model is zero. For example, for a voxel whose fusion cut-off distance D(p) is more equal to 1 or equal to -1, it may not be performed at all. sampling.
  • sampling times of the voxel on the camera ray and the fusion cutoff distance D(p) of the voxel satisfy the following relationship:
  • D(p) is the fusion cutoff distance of the voxel
  • n is the sampling times of the voxel.
  • each voxel is not a point in the cube space, but contains a large number of spatial points. Therefore, after calculating the sampling times of each voxel through the above method, it can be obtained from the large number of spatial points contained in the voxel.
  • Sampling a part of the spatial points for example, take 10 voxels in all voxels as an example, if each voxel includes 1000 spatial points, these 10 voxels are respectively expressed as voxel-1, voxel-2, voxel-3, voxel -4, voxel-5, voxel-6, voxel-7, voxel-8, voxel-9, voxel-10, the 10 voxel fusion truncation distances D(p) are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, then the number of spatial points sampled from voxel-1 is 90, the number of spatial points sampled from voxel-2 is 80, and the number of spatial points sampled from voxel-3 is 90.
  • the number is 70, the number of spatial points sampled from voxel-4 is 60, the number of spatial points sampled from voxel-5 is 50, the number of spatial points sampled from voxel-6 is 40, and the number of spatial points sampled from voxel-7 is 40.
  • the number of spatial points sampled from voxel-8 is 30, the number of spatial points sampled from voxel-8 is 20, the number of spatial points sampled from voxel-9 is 10, and the number of spatial points sampled from voxel-10 is 0.
  • Each spatial point sampled has a spatial coordinate (x, y, z).
  • the distance between some voxels on the camera ray and the camera cannot be detected by the relevant sensors, so the cutoff distance cannot be obtained for such voxels, so sampling based on the cutoff distance cannot be performed.
  • traditional uniform sampling or average sampling
  • Step S405 Train the deep learning network by using the spatial coordinates of the sampled spatial points and multiple sample images.
  • the multiple sample images correspond to multiple orientations respectively, and the deep learning network will reconstruct (or calculate) the image of each orientation in the multiple orientations according to the spatial coordinates of the sampled spatial points, until the reconstructed image of each orientation
  • the loss (such as RGB information difference) from the original sample image of the azimuth is less than a preset value, and the loss between the reconstructed image of each azimuth and the original sample image of the azimuth is less than the preset value, then for the depth
  • the training of the learning network is over.
  • the sample image 1 is taken from the orientation 1, then, add the pixel value of the pixel point A in the image 1 to the spatial coordinates of the sampled spatial point on the camera ray corresponding to the pixel point A. , and perform similar operations on other pixels in the image 1. After this operation is performed on all the pixels in the image 1, an image effect can be presented.
  • This image is the reconstructed image A of the orientation 1.
  • the ideal reconstruction The effect is that the loss (eg, RGB information difference) between the image A reconstructed by the deep learning network in orientation 1 and the sample image 1 is less than the preset value, and the sample images and reconstructed images captured in other orientations also satisfy this relationship.
  • the deep learning network after the training can also be called a light field prediction model.
  • Step S406 Predict the image when the above scene is shot from a new orientation through the deep learning network (light field prediction model).
  • the new orientation here can be other orientations except the above-mentioned multiple orientations, and the orientation can be represented by five-dimensional (5D) coordinates, for example, orientation 1 where (x, y, z) represents the position, Indicates the viewing angle direction.
  • orientation 1 where (x, y, z) represents the position, Indicates the viewing angle direction.
  • the predicted image may be an RGB image.
  • Table 1 is the prediction effect of the well-trained deep learning network in the prior art illustrated in Fig. 1, and the contrast situation of the prediction effect of the well-trained deep learning network in the application, the PSNR comparison result can be output by simulation:
  • PSNR-Coarse 33 PSNR-Fine 36 37.2 PSNR-Test 35.5 36.5
  • PSNR peak signal-to-noise ratio
  • PSNR-Coarse represents the coarse-grained prediction of the existing sample images corresponding to the azimuth image. Effect, PSNR-Fine indicates the effect of fine-grained prediction on the captured image of the azimuth corresponding to the existing sample image, and PSNR-Test indicates the effect of predicting the captured image of a new azimuth. The larger the value of PSNR, the better the prediction effect of the trained deep learning network.
  • Figure 8 illustrates the prediction effect of the deep learning network trained in the above prior art, and the comparison of the prediction effect of the deep learning network trained in the application, this result is presented in the form of an image
  • part (g) is an image of a scene taken
  • part (h) is the effect of reconstructing the image through a deep learning network in the prior art
  • part (i) is an image obtained by using an embodiment of the present application The effect after the deep learning network reconstructs the image.
  • the embodiments of the present application express more and more clearly the texture of the surface of the object, and can better reflect the level of the scene.
  • the voxel sampling points used for training the deep learning network are obtained based on the depth information of the image, specifically calculated based on the depth information and the distance from the voxel to the camera A truncation distance, and then perform differential sampling according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling to key areas and improve the sampling efficiency; on the other hand, the voxels sampled by this sampling method are basically concentrated.
  • the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors.
  • FIG. 9 is a schematic structural diagram of an apparatus 90 for generating a light field prediction model provided by an embodiment of the present application.
  • the apparatus 90 may include a establishing unit 901 , a first calculating unit 902 , a sampling unit 903 and a second The calculation unit 904, wherein the detailed description of each unit is as follows.
  • a establishing unit 901 configured to establish a cube model surrounding the shot scene according to the respective shooting orientations of the multiple sample images, wherein the cube model includes a plurality of small cube voxels;
  • the first calculation unit 902 is configured to calculate a plurality of cutoff distances of each small cube in the plurality of small cubes according to the plurality of sample images, wherein the distance of each small cube calculated according to the first sample image is calculated.
  • a cut-off distance includes: determining the cut-off distance according to the distance from the camera to each small cube and the distance from the camera to the surface of the object in the scene when the first sample image is captured, and the first sample image is: any one of the multiple sample images;
  • Sampling unit 903 configured to sample spatial points from the small cubes according to a plurality of cutoff distances of each small cube, wherein each sampled spatial point corresponds to a spatial coordinate;
  • the second computing unit 904 is configured to train a light field prediction model according to the sampled spatial coordinates of the spatial point, wherein the light field prediction model is used to predict the light field of the scene.
  • the voxel sampling points used for training the deep learning network are obtained based on the depth information of the image, specifically based on the depth information and the voxel to the camera.
  • a truncation distance is calculated from the distance, and then differentiated sampling is performed according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling to key areas and improve the sampling efficiency; on the other hand, the voxel sampled by this sampling method Basically, they are concentrated near the surface of the object. Therefore, the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors.
  • the device 90 further includes:
  • a prediction unit configured to predict the light field of the scene by using the light field prediction model. That is to say, after training the pre-stored light field model, the model training device will also predict the light field through the light field prediction model.
  • the sampling unit 903 is specifically configured to: for each small cube Perform fusion calculation on the plurality of truncation distances to obtain the fusion truncation distance of each small cube; and sample spatial points from each small cube according to the fusion truncation distance of each small cube.
  • the fusion cutoff distance is calculated for each small cube, and then sampling is performed according to the fusion cutoff distance.
  • the sampling unit 903 is specifically configured to: determine each small cube There is at least a first small cube whose absolute value of truncation distance is less than a preset threshold; the fusion calculation is performed on the plurality of truncation distances of the first small cube to obtain the first small cube fusion truncation distance; The fusion cutoff distance of the first small cube samples spatial points from the first small cube.
  • the fusion cut-off distance is not calculated for all small cubes, but only for the small cubes whose absolute value of cut-off distance is less than the preset threshold, because when the cut-off distance is large, the corresponding The farther the cube is from the surface of the object, the less necessary for subsequent sampling. Therefore, this application does not perform fusion calculation on the truncation distance of such small cubes, which is equivalent to excluding these small cubes from the sampling category in advance. Without reducing the effect of subsequent sampling, the amount of calculation is reduced, and the generation efficiency of the light field prediction model is improved.
  • the smaller the fusion cut-off distance is the more spatial points are sampled from the small cube. It can be understood that the smaller the fusion truncation distance is, the closer the cube is to the surface of the object. Compared with the spatial points in other small cubes on the same camera ray, the spatial points in such small cubes can better reflect the pixel information. Therefore, training the light field prediction model based more on the spatial points in such small cubes is beneficial to the subsequent prediction of more accurate images by the light field prediction model.
  • the fusion calculation includes weighted average calculation. It can be understood that the fusion cutoff distance calculated by the weighted average can more accurately reflect the distance between the small cube and the surface of the object.
  • the weight value of the cut-off distance of the second small cube calculated based on the first sample image in the weighted average calculation is the same as In the first sample image, the distance of the camera is negatively correlated, and/or, it is positively correlated with the first included angle, and the first included angle is the distance between the field of view camera ray where the second small cube is located. The included angle between the normal vectors of the nearest object surface of the second small cube, where the second small cube is any small cube in the cube model.
  • the weight value calculated based on each sample image is used to calculate the fusion cutoff distance, because the weight value is negatively correlated with the distance to the camera, and/or is positive with the first included angle Therefore, when the weight value is combined to calculate the fusion cut-off distance, the influence of different orientations on the fusion cut-off distance can be more accurately reflected.
  • the device 90 further includes:
  • an acquisition unit configured to acquire depth information from a shooting angle of view of each sample image when shooting each sample image in the plurality of sample images, wherein the depth information is used to represent the distance from the camera to the scene distance from the surface of the object.
  • the weight value of the cut-off distance calculated based on the first sample image in the weighted average calculation may also be It is called the weight value w(p) of the second small cube calculated according to the first sample image, and the weight value w(p) satisfies the following relationship:
  • ⁇ is the first included angle
  • distance(v) is the distance from the second small cube to the camera when the first sample image is captured.
  • the weight value of the second small cube calculated according to the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or, it is positively correlated with the first included angle
  • the expression of w(p) here is an optional expression of this idea, so when the weight value is combined to calculate the fusion cutoff distance, it can reflect more accurately The effect of different orientations on the fusion cutoff distance.
  • the cut-off distance d(p) of the second small cube calculated according to the first sample image satisfies the following relationship:
  • sdf(p) is the difference between the distance from the camera to the first small cube when the first sample image is captured, and the distance from the camera to the surface of the object in the scene, and u is a preset threshold.
  • d(p) is only an optional calculation formula for the cut-off distance, and other expressions may also be used in practical applications.
  • the truncation distance of the small cubes in one range is assigned as 1, and the truncation distance of the small cubes in another range is assigned as -1, which is conducive to the subsequent processing of the two types of small cubes. Thereby improving computational efficiency.
  • each unit may also correspond to the corresponding description with reference to the method embodiment shown in FIG. 4 .
  • FIG. 10 is a device 100 for generating a light field prediction model provided by an embodiment of the present application.
  • the device 100 includes a processor 1001, a memory 1002, and a communication interface 1003.
  • the interfaces 1003 are connected to each other through a bus.
  • the memory 1002 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or A portable read-only memory (compact disc read-only memory, CD-ROM), the memory 1002 is used for related instructions and data.
  • the communication interface 1003 is used to receive and transmit data.
  • the processor 1001 may be one or more central processing units (central processing units, CPUs).
  • CPUs central processing units
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the device 100 Since the device 100 needs to use multiple sample images when training the light field prediction model, the device 100 needs to acquire the multiple sample images.
  • the device 100 is provided with a camera (or referred to as an image sensor, a photographing device), and the camera can photograph and obtain the plurality of sample images.
  • a depth sensor may also be configured on the device 100 to collect the depth of the scene in the scene being shot, and the type of the depth sensor is not limited here.
  • the processor 1001 in the device 100 is configured to read the program code stored in the memory 1002, and perform the following operations: establish a cube model surrounding the shot scene according to the respective shooting orientations of the multiple sample images, wherein the cube model including a plurality of small cube voxels; then, according to the plurality of sample images, respectively calculate a plurality of cut-off distances of each small cube in the plurality of small cubes, wherein each small cube calculated according to the first sample image A cut-off distance of the is any one of the multiple sample images; after that, a spatial point is sampled from the small cube according to a plurality of cutoff distances of each small cube, wherein each sampled spatial point corresponds to a spatial coordinate; then, A light field prediction model is trained according to the sampled spatial coordinates of the spatial points, wherein the light field prediction model is used to predict the light field of the scene.
  • the voxel sampling points used for training the deep learning network are obtained based on the depth information of the image, specifically based on the depth information and the voxel to the camera.
  • a truncation distance is calculated from the distance, and then differentiated sampling is performed according to the size of the truncation distance.
  • this sampling method can quickly concentrate the sampling to key areas and improve the sampling efficiency; on the other hand, the voxel sampled by this sampling method Basically, they are concentrated near the surface of the object. Therefore, the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce the phenomenon of blur and structural errors.
  • the processor 1001 is specifically configured to: predict the light of the scene by using the light field prediction model field. That is to say, after training the pre-stored light field model, the model training device will also predict the light field through the light field prediction model.
  • the processor is specifically configured to: A fusion calculation is performed on the plurality of cutoff distances to obtain the fusion cutoff distance of each small cube; a spatial point is sampled from each small cube according to the fusion cutoff distance of each small cube.
  • the fusion cutoff distance is calculated for each small cube, and then sampling is performed according to the fusion cutoff distance.
  • the processor is specifically configured to: determine the There is at least one first small cube whose absolute value of the truncation distance is smaller than a preset threshold; the fusion calculation is performed on the plurality of truncation distances of the first small cube to obtain the first small cube fusion truncation distance; according to the The fusion cutoff distance of the first small cube samples spatial points from the first small cube.
  • the fusion cut-off distance is not calculated for all small cubes, but only for the small cubes whose absolute value of cut-off distance is less than the preset threshold, because when the cut-off distance is large, the corresponding The farther the cube is from the surface of the object, the less necessary for subsequent sampling. Therefore, this application does not perform fusion calculation on the truncation distance of such small cubes, which is equivalent to excluding these small cubes from the sampling category in advance. Without reducing the effect of subsequent sampling, the amount of calculation is reduced, and the generation efficiency of the light field prediction model is improved.
  • the smaller the fusion cut-off distance is the more spatial points are sampled from the small cube. It can be understood that the smaller the fusion truncation distance is, the closer the cube is to the surface of the object. Compared with the spatial points in other small cubes on the same camera ray, the spatial points in such small cubes can better reflect the pixel information. Therefore, training the light field prediction model based more on the spatial points in such small cubes is beneficial to the subsequent prediction of more accurate images by the light field prediction model.
  • the fusion calculation includes weighted average calculation. It can be understood that the fusion cutoff distance calculated by the weighted average can more accurately reflect the distance between the small cube and the surface of the object.
  • the weight value of the cut-off distance of the second small cube calculated based on the first sample image in the weighted average calculation is the same as In the first sample image, the distance of the camera is negatively correlated, and/or, it is positively correlated with the first included angle, and the first included angle is the distance between the field of view camera ray where the second small cube is located. The included angle between the normal vectors of the nearest object surface of the second small cube, where the second small cube is any small cube in the cube model.
  • the weight value calculated based on each sample image is used to calculate the fusion cutoff distance, because the weight value is negatively correlated with the distance to the camera, and/or is positive with the first included angle Therefore, when the weight value is combined to calculate the fusion cut-off distance, the influence of different orientations on the fusion cut-off distance can be more accurately reflected.
  • the processor before calculating the cutoff distance of each small cube in the cube model according to the multiple sample images, is further configured to: take the multiple sample images For each sample image in the scene, depth information is collected from the shooting angle of view of each sample image, where the depth information is used to represent the distance from the camera to the surface of the object in the scene.
  • the weight value of the cut-off distance calculated based on the first sample image in the weighted average calculation may also be It is called the weight value w(p) of the second small cube calculated according to the first sample image, and the weight value w(p) satisfies the following relationship:
  • ⁇ is the first included angle
  • distance(v) is the distance from the second small cube to the camera when the first sample image is captured.
  • the weight value of the second small cube calculated according to the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or, it is positively correlated with the first included angle
  • the expression of w(p) here is an optional expression of this idea, so when the weight value is combined to calculate the fusion cutoff distance, it can reflect more accurately The effect of different orientations on the fusion cutoff distance.
  • the cut-off distance d(p) of the second small cube calculated according to the first sample image satisfies the following relationship:
  • sdf(p) is the difference between the distance from the camera to the first small cube when the first sample image is captured, and the distance from the camera to the surface of the object in the scene, and u is a preset threshold.
  • d(p) is only an optional calculation formula for the cut-off distance, and other expressions may also be used in practical applications.
  • the truncation distance of the small cubes in one range is assigned as 1, and the truncation distance of the small cubes in another range is assigned as -1, which is conducive to the subsequent processing of the two types of small cubes. Thereby improving computational efficiency.
  • each operation may also correspond to the corresponding description with reference to the method embodiment shown in FIG. 4 .
  • An embodiment of the present application further provides a chip system, the chip system includes at least one processor, a memory, and an interface circuit, the memory, the transceiver, and the at least one processor are interconnected by lines, and the at least one memory Instructions are stored in the ; when the instructions are executed by the processor, the method flow shown in FIG. 4 is implemented.
  • Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on the processor, the method flow shown in FIG. 4 is implemented.
  • the embodiment of the present application further provides a computer program product, which implements the method flow shown in FIG. 4 when the computer program product runs on the processor.
  • the voxel sampling points used for training a deep learning network are obtained based on the depth information of the image, specifically based on the depth information of the image.
  • a truncation distance is calculated from the depth information and the distance from the voxel to the camera, and then differentiated sampling is performed according to the size of the truncation distance.
  • the voxels sampled by this sampling method are basically concentrated near the surface of the object, so the deep learning network trained based on this voxel can better represent the texture details of the object when performing image prediction, and can reduce blur (blur) and structural errors.
  • the process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium.
  • the program When the program is executed , which may include the processes of the foregoing method embodiments.
  • the aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un procédé pour la génération d'un modèle de prédiction de champ lumineux, et un appareil associé. Le procédé consiste à : établir un modèle cubique entourant une scène photographiée selon les orientations de photographie respectives de multiples images d'échantillon, le modèle cubique comprenant de multiples petits cubes (voxels) ; calculer respectivement de multiples distances tronquées de chacun des multiples petits cubes selon les multiples images d'échantillon (S402) ; échantillonner un point spatial à partir de chaque petit cube selon les multiples distances tronquées du petit cube, chaque point spatial échantillonné possède de manière correspondante des coordonnées spatiales ; et entraîner un modèle de prédiction de champ lumineux selon les coordonnées spatiales des points spatiaux échantillonnés, le modèle de prédiction de champ lumineux étant utilisé pour prédire un champ lumineux de la scène. Le procédé peut améliorer l'efficacité d'échantillonnage des voxels, ce qui améliore l'efficacité de génération du modèle de prédiction de champ lumineux.
PCT/CN2021/080893 2021-03-15 2021-03-15 Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé WO2022193104A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202180095331.6A CN117015966A (zh) 2021-03-15 2021-03-15 一种光场预测模型的生成方法及相关装置
PCT/CN2021/080893 WO2022193104A1 (fr) 2021-03-15 2021-03-15 Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/080893 WO2022193104A1 (fr) 2021-03-15 2021-03-15 Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé

Publications (1)

Publication Number Publication Date
WO2022193104A1 true WO2022193104A1 (fr) 2022-09-22

Family

ID=83321601

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/080893 WO2022193104A1 (fr) 2021-03-15 2021-03-15 Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé

Country Status (2)

Country Link
CN (1) CN117015966A (fr)
WO (1) WO2022193104A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101982741A (zh) * 2010-09-08 2011-03-02 北京航空航天大学 一种水下光场采样及模拟方法
CN106165387A (zh) * 2013-11-22 2016-11-23 维迪诺蒂有限公司 光场处理方法
CN107018293A (zh) * 2015-09-17 2017-08-04 汤姆逊许可公司 生成表示光场的数据的方法和装置
US20190174115A1 (en) * 2015-09-17 2019-06-06 Thomson Licensing Light field data representation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101982741A (zh) * 2010-09-08 2011-03-02 北京航空航天大学 一种水下光场采样及模拟方法
CN106165387A (zh) * 2013-11-22 2016-11-23 维迪诺蒂有限公司 光场处理方法
CN107018293A (zh) * 2015-09-17 2017-08-04 汤姆逊许可公司 生成表示光场的数据的方法和装置
US20190174115A1 (en) * 2015-09-17 2019-06-06 Thomson Licensing Light field data representation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
TANG SHENGZE, GUI TAO : "Model of 5 Light Field for Navigation in Occluded Environment", JOURNAL OF TSINGHUA UNIVERSITY (SCIENCE AND TECHNOLOGY), vol. 38, no. S1, 31 December 1998 (1998-12-31), pages 10 - 14, XP055968315, DOI: 10.16511/j.cnki.qhdxxb.1998.s1.003 *

Also Published As

Publication number Publication date
CN117015966A (zh) 2023-11-07

Similar Documents

Publication Publication Date Title
US20210110599A1 (en) Depth camera-based three-dimensional reconstruction method and apparatus, device, and storage medium
CN106780590B (zh) 一种深度图的获取方法及系统
WO2019161813A1 (fr) Procédé, appareil et système de reconstruction tridimensionnelle de scène dynamique, serveur et support
US11557083B2 (en) Photography-based 3D modeling system and method, and automatic 3D modeling apparatus and method
CN112862874B (zh) 点云数据匹配方法、装置、电子设备及计算机存储介质
CN112288853B (zh) 三维重建方法、三维重建装置、存储介质
CN110910437B (zh) 一种复杂室内场景的深度预测方法
CN106327532A (zh) 一种单幅图像的三维配准方法
CN109683699A (zh) 基于深度学习实现增强现实的方法、装置及移动终端
CN113129352B (zh) 一种稀疏光场重建方法及装置
WO2022247548A1 (fr) Procédé de positionnement, appareil, dispositif électronique et support de stockage
CN116071504B (zh) 一种面向高分辨率图像的多视图立体重建方法
CN118229901A (zh) 一种Cesium地图引擎下基于多源空间数据的三维可视化方法及系统
CN117834839A (zh) 基于移动终端的多视角3d智能成像测量系统
US20240233236A9 (en) Differentiable real-time radiance field rendering for large scale view synthesis
WO2021151380A1 (fr) Procédé de rendu d'objet virtuel sur la base d'une estimation d'éclairage, procédé de formation de réseau neuronal et produits associés
Chen et al. Densefusion: Large-scale online dense pointcloud and dsm mapping for uavs
Lyu et al. 3DOPFormer: 3D occupancy perception from multi-camera images with directional and distance enhancement
CN117726747A (zh) 补全弱纹理场景的三维重建方法、装置、存储介质和设备
CN115909255B (zh) 图像生成、图像分割方法、装置、设备、车载终端及介质
CN115578432B (zh) 图像处理方法、装置、电子设备及存储介质
WO2022193104A1 (fr) Procédé pour générer un modèle de prédiction de champ lumineux, et appareil associé
Zhang et al. UAVStereo: A Multiple Resolution Dataset for Stereo Matching in UAV Scenarios
CN115375740A (zh) 位姿确定方法和三维模型的生成方法、装置、设备、介质
Hou et al. Depth estimation and object detection for monocular semantic SLAM using deep convolutional network

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21930700

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180095331.6

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21930700

Country of ref document: EP

Kind code of ref document: A1