CN116740245A - Training method and rendering method of image rendering model - Google Patents

Training method and rendering method of image rendering model Download PDF

Info

Publication number
CN116740245A
CN116740245A CN202310601759.4A CN202310601759A CN116740245A CN 116740245 A CN116740245 A CN 116740245A CN 202310601759 A CN202310601759 A CN 202310601759A CN 116740245 A CN116740245 A CN 116740245A
Authority
CN
China
Prior art keywords
rendering
image
trained
model
initial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310601759.4A
Other languages
Chinese (zh)
Inventor
王哲
盘博文
吕江靖
贾荣飞
吕承飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Taobao China Software Co Ltd
Original Assignee
Taobao China Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Taobao China Software Co Ltd filed Critical Taobao China Software Co Ltd
Priority to CN202310601759.4A priority Critical patent/CN116740245A/en
Publication of CN116740245A publication Critical patent/CN116740245A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • G06Q30/0643—Graphical representation of items or shoppers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects
    • G06T15/506—Illumination models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Accounting & Taxation (AREA)
  • Geometry (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the application provides a training method and a rendering method of an image rendering model, wherein in the training method of the image rendering model, the rendering model to be trained is trained according to differences among a rendering image, a rendering segmentation image, an initial image and an initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained. According to the method, the rendering parameters of the rendering model to be trained are adjusted, so that differences between the rendering image and the rendering segmentation image obtained by the adjusted rendering model and the initial image and the initial segmentation image are reduced, and the object reduction degree in the rendering image is improved.

Description

Training method and rendering method of image rendering model
Technical Field
The application relates to the technical field of computers, in particular to a training method and device for an image rendering model, electronic equipment and a computer storage medium.
Background
Three-dimensional reconstruction is the opposite process to rendering, which is the three-dimensional information of a given object or scene, simulating a camera taking a two-dimensional image. The three-dimensional reconstruction is to build a mathematical model suitable for computer representation and processing on the three-dimensional object, in particular to reconstruct three-dimensional information according to single-view or multi-view images, in other words, input two-dimensional images, and infer a three-dimensional structure. For example, the three-dimensional modeling information of the object displayed by the online service platform is an object three-dimensional geometric figure obtained by reconstructing three-dimensional information from a two-dimensional image of the actual object. The three-dimensional reconstruction process of the object comprises links such as object segmentation, camera pose estimation, grid reconstruction, texture recovery and the like.
The object reduction degree in the rendered image obtained in the actual three-dimensional modeling process is low, so how to improve the object reduction degree in the rendered image obtained in the three-dimensional modeling is a problem to be solved.
Disclosure of Invention
The embodiment of the application provides a training method of an image rendering model, which is used for improving the object reduction degree in a rendered image obtained by the image rendering model. The embodiment of the application also relates to a training device of the image rendering model, electronic equipment and a computer storage medium. The embodiment of the application also relates to a rendering method, a rendering device and electronic equipment.
The embodiment of the application provides a training method of an image rendering model, which comprises the following steps: obtaining an initial image and an initial segmentation image of a target object; rendering the initial image and the initial segmentation image by utilizing a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image; training the rendering model to be trained based on differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
Optionally, the adjusting the rendering parameters of the rendering model to be trained includes: and adjusting texture parameters of the rendering model to be trained.
Optionally, the method further comprises: updating the adjusted texture parameters to the rendering model to be trained to obtain a first rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
Optionally, the adjusting the rendering parameters of the rendering model to be trained includes: and carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
Optionally, the method further comprises: updating the adjusted texture parameters and pose parameters to the rendering model to be trained to obtain a second rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
Optionally, the adjusting the rendering parameters of the rendering model to be trained includes: and carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
Optionally, the method further comprises: updating the adjusted texture parameters, pose parameters and grid parameters to the rendering model to be trained to obtain a third rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
Optionally, the adjusting the rendering parameters of the rendering model to be trained includes: and carrying out joint adjustment on the texture parameters and the illumination parameters of the rendering model to be trained.
Optionally, the method further comprises: updating the adjusted texture parameters and illumination parameters to the rendering model to be trained to obtain a fourth rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
Optionally, the method further comprises: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image; the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes: determining a loss value of the rendering model to be trained based on the texture data differences; and adjusting texture parameters of the rendering model to be trained based on the loss value of the rendering model to be trained.
Optionally, the method further comprises: obtaining texture data differences and pose data differences between the rendered image and the rendered segmentation image and the initial segmentation image; the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes: determining a loss value of the rendering model to be trained based on the texture data difference and the pose data difference; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
Optionally, the method further comprises: obtaining texture data differences and pose data differences between the rendered image and the rendered segmented image and the initial segmented image, and grid data differences between the rendered image and the initial image; the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes: determining a loss value of the rendering model to be trained based on the texture data differences, the pose data differences and the grid data differences; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
Optionally, the method further comprises: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image, and illumination data differences between the rendered image and the initial image; the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes: determining a loss value of the rendering model to be trained based on the texture data differences and the illumination data differences; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on the texture parameter and the illumination parameter of the rendering model to be trained.
Optionally, the determining the loss value of the rendering model to be trained includes: a first penalty value for object geometry differences between rendering results characterizing the rendering model to be trained and the target object is determined.
Optionally, the determining the loss value of the rendering model to be trained based on the texture data difference and the illumination data difference includes: determining a first loss value for object geometry degree difference between a rendering result used for representing the rendering model to be trained and the target object based on the texture data difference; based on the illumination data differences, a second loss value for the illumination data difference degree between the rendering result used for representing the rendering model to be trained and the target object is determined.
Optionally, the method further comprises: acquiring grid data of the target object; the grid data of the target object is acquired by the following method: acquiring a multi-view initial image of the target object and image pose data corresponding to the multi-view initial image respectively; and constructing grid data of the target object according to the multi-view initial image, the image pose data and the initial segmentation image.
Optionally, the method further comprises: obtaining illumination data of a rendering result obtained by the rendering model to be trained; the obtaining the illumination data of the rendering result obtained by the rendering model to be trained comprises the following steps: setting micro texture data, micro pose data and micro grid data of a target object in an initialized mode; and inputting the micro-texture data, the micro-pose data and the micro-grid data of the target object into a lighting model to obtain lighting data of the rendering result.
The embodiment of the application also provides a rendering method, which comprises the following steps: providing a multi-view initial image of a target object to a target rendering model to obtain an initial segmentation image and texture data of the target object, wherein the multi-view initial image corresponds to pose data and illumination data of the target object respectively; determining a rendering grid of the target object according to the multi-view initial image, the initial segmentation image of the target object, texture data and pose data respectively corresponding to the multi-view initial image; according to the texture data, performing texture mapping operation on the rendering grid of the target object; and carrying out illumination treatment on the rendering grid with the texture map according to the illumination data to obtain a rendering image.
The embodiment of the application also provides electronic equipment, which comprises a processor and a memory; the memory stores a computer program, and the processor executes the method after running the computer program.
The embodiment of the application also provides a computer storage medium, which stores a computer program, and the computer program executes the method after being executed by a processor.
Compared with the prior art, the embodiment of the application has the following advantages:
the embodiment of the application provides a training method of an image rendering model, which comprises the following steps: obtaining an initial image and an initial segmentation image of a target object; rendering the initial image and the initial segmentation image by utilizing a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image; training the rendering model to be trained based on the rendering image and the rendering segmentation image and differences between the rendering image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
According to the method, the rendering model to be trained is trained according to differences among the rendering image, the rendering segmentation image, the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained. According to the method, the rendering parameters of the rendering model to be trained are adjusted, so that differences between the rendering image and the rendering segmentation image obtained by the adjusted rendering model and the initial image and the initial segmentation image are reduced, and the object reduction degree in the rendering image is improved.
Drawings
Fig. 1 is a schematic diagram of a parameter optimization strategy of a training method of an image rendering model according to an embodiment of the present application.
Fig. 2 is a schematic scene diagram of a training method of an image rendering model according to an embodiment of the present application.
Fig. 3 is a flowchart of a training method of an image rendering model according to a first embodiment of the present application.
Fig. 4 is a flowchart of a rendering method according to a second embodiment of the present application.
Fig. 5 is a schematic diagram of a training device for an image rendering model according to a third embodiment of the present application.
Fig. 6 is a schematic diagram of a rendering device according to a fourth embodiment of the present application.
Fig. 7 is a schematic diagram of an electronic device according to a fifth embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application may be embodied in many other forms than those herein described, and those skilled in the art will readily appreciate that the present application may be similarly embodied without departing from the spirit or essential characteristics thereof, and therefore the present application is not limited to the specific embodiments disclosed below.
The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The manner of description used in the present application and in the appended claims is for example: "a", "a" and "a" etc. are not limited in number or order, but are used to distinguish the same type of information from each other.
First, the concept according to the present application will be described:
the method can be used for micro-rendering: the rendering process of the traditional graphics model is not micro-renderable, and compared with the traditional graphics model, a rendering pipeline is constructed by micro-rendering so that each link in the rendering process can be optimized.
Texture mapping: the process of mapping an image (texture) to a surface of a 3D (three-dimensional) rendered object is called texture mapping. Texture mapping provides objects with a rich detail that mimics the complex appearance of objects.
Texture reconstruction: mapping the geometry by means of an image, recovering texture mapping data of the grid in the process of reconstructing the three-dimensional model, wherein texture mapping is a method for reconstructing the texture.
Rendering parameters: rendering parameters refer to the relevant data involved in the overall rendering link, by which the final rendering result is controlled. Parameters here include model properties (e.g., mesh vertices, patches, UV coordinates, etc.), texture properties (e.g., color maps, normal maps, etc.), light source configurations (light source types, illumination maps, etc.), camera configurations (camera position, screen size, etc.), texture rendering parameters, etc.
Grid optimization: the grid is a geometrical body formed by triangular patches, and the grid is optimized to finely adjust each vertex of the triangle of the geometrical body, so that the precision of the vertex of the grid is improved.
The embodiment of the application provides a training method of an image rendering model. The embodiment of the application also provides a training device of the image rendering model, electronic equipment and a computer storage medium. The embodiment of the application also provides a rendering method, a rendering device, electronic equipment and a computer storage medium. The following examples are described in detail one by one.
In order to facilitate understanding of the method and apparatus provided by the embodiments of the present application, a background of the embodiments of the present application is described before the embodiments of the present application are described.
Three-dimensional reconstruction is the opposite process to rendering, which is the three-dimensional information of a given object or scene, simulating a camera taking a two-dimensional image. The three-dimensional reconstruction is to build a mathematical model suitable for computer representation and processing on the three-dimensional object, in particular to reconstruct three-dimensional information according to single-view or multi-view images, in other words, input two-dimensional images, and infer a three-dimensional structure. For example, the three-dimensional modeling information of the object displayed by the online service platform is an object three-dimensional geometric figure obtained by reconstructing three-dimensional information from a two-dimensional image of the actual object. The three-dimensional reconstruction process of the object comprises links such as object segmentation, camera pose estimation, grid reconstruction, texture recovery and the like.
The object reduction degree in the rendered image obtained in the actual three-dimensional modeling process is low, so how to improve the object reduction degree in the rendered image obtained in the three-dimensional modeling is a problem to be solved.
With the foregoing background description, those skilled in the art can appreciate the problems existing in the prior art, and the following details about the application scenario of the training method of the image rendering model of the present application. The training method of the image rendering model provided by the application can be applied to three-dimensional reconstruction of various online service platforms serving as target objects to obtain application scenes of three-dimensional reconstruction geometries of the target objects. For example, in an online shopping platform, in order to increase the knowledge of a user about the omnidirectional information of a commodity, a target rendering model is constructed by adopting the training method of the image rendering model provided by the application. The target rendering model obtains three-dimensional geometric bodies of the commodities by inference according to two-dimensional images of multiple visual angles of the commodities, so that a user can intuitively feel a stereoscopic visual effect diagram of the commodities in a shop of an online shopping platform, cognition and understanding of the commodities are improved, and ordering rate of the user on the commodities is improved.
The training method for the image rendering model provided by the application is used for training the rendering model to be trained to obtain the target rendering model. In the training process, the initial image and the initial segmentation image of the target object are rendered through the rendering model to be trained, and the rendering image and the rendering segmentation image of the initial image are obtained. Comparing the rendered image with the initial image, and comparing the initial segmented image with the rendered segmented image to obtain the rendered image and the rendered segmented image, and differences between the initial image and the initial segmented image. Training the rendering model to be trained according to the difference; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
In the training process of the rendering model to be trained, a loss value for representing the object difference degree between the rendering image and the target object is obtained according to the difference between the rendering image and the initial image. And adjusting rendering parameters of the rendering model to be trained based on the loss value. The rendering parameters include at least one of the following: texture parameters, pose parameters, grid parameters, illumination parameters, and material rendering parameters. Therefore, the adjusting the rendering parameter of the rendering model to be trained may be adjusting at least one rendering parameter in the rendering model to be trained, so as to improve the reduction degree of the target object in the rendering result obtained by the rendering model to be trained.
The process of adjusting the rendering parameters based on the loss values may be an iterative process, for example, in a first model training process, according to differences between the rendered image and the initial image, a first loss value for characterizing a degree of difference between the rendered image and the target object with respect to the object geometry is obtained, according to the first loss value, the rendering parameters of the rendering model to be trained are adjusted, and a rendering model with the rendering parameters adjusted is obtained.
And then, analyzing and processing the initial image by adopting a rendering model with the rendering parameters adjusted to obtain a second difference between the rendered image and the initial image, and determining a first loss value for representing the difference degree of the object geometry between the rendered image and the target object based on the second difference. And according to the first loss value, the rendering parameters of the rendering model are adjusted again.
Based on the process, the preset times of the rendering parameters of the rendering model to be trained are adjusted, and the target rendering model with the preset times of the rendering parameters is obtained. The difference between the rendering image and the rendering segmentation image obtained by analyzing the initial image and the difference between the rendering image and the rendering segmentation image are reduced by the target rendering model, so that the object reduction degree of the target object in the rendering result obtained by the rendering model after training is improved.
Please refer to fig. 1, which is a schematic diagram of a parameter optimization strategy of a training method of an image rendering model according to an embodiment of the present application.
In fig. 1, the rendering parameters of the rendering model to be trained are adjusted through a preset parameter optimization sequence, so that the difference between the rendering result and the initial image of the target object is reduced. For example, a first penalty value for characterizing a degree of difference between the rendered image and the initial image for the object geometry is reduced, and a second penalty value for characterizing a degree of difference between the rendered image and the initial image for the illumination data is reduced. The method comprises the following steps:
first, texture optimization, in other words, by adjusting texture parameters, a first loss value for an object geometry between a rendering result obtained by a rendering model after adjusting the texture parameters and an initial image of a target object is reduced. And adjusting texture parameters of preset times of the rendering model to be trained, and reducing the first loss value, so that the reduction degree of the target object in the rendering result obtained by the rendering model after adjusting the texture parameters is improved.
The process is a process of performing texture optimization on the rendering model to be trained. And the reduction degree of the target object in the rendering result obtained by the rendering model is improved by adjusting the texture parameters of the rendering model to be trained. Secondly, texture and pose optimization is also possible, in other words, the reduction degree of the target object in the rendering result is improved by jointly adjusting the texture parameters and the pose parameters of the rendering model to be trained.
Specifically, by jointly adjusting the texture parameters and the pose parameters, a first loss value for the object geometry between a rendering result obtained by a rendering model after adjusting the texture parameters and the pose parameters and an initial image of the target object is reduced. Based on the method, the reduction degree of the target object in the rendering result obtained by the rendering model after the texture parameters and the pose parameters are adjusted is improved.
The process is a process of performing texture and pose optimization on the rendering model to be trained. And the reduction degree of the target object in the rendering result obtained by the rendering model is improved by adjusting the texture parameters and the pose parameters of the rendering model to be trained. In addition, texture, pose and grid joint optimization can be adopted, in other words, the texture parameters, pose parameters and grid parameters of the rendering model to be trained are adjusted in a joint mode, so that the reduction degree of the target object in the rendering result is improved.
Specifically, the texture parameters, the pose parameters and the grid parameters are adjusted at the same time, and a first loss value for the object geometry between a rendering result obtained by a rendering model after adjusting the texture parameters, the pose parameters and the grid parameters and an initial image of the target object is reduced. Based on the method, the reduction degree of the target object in the rendering result obtained by the rendering model after the texture parameters, the pose parameters and the grid parameters are improved and adjusted.
The three processes of adjusting the model parameters of the rendering model to be trained are used for reducing the difference value between the rendering result obtained by the rendering model and the target object for the object geometry. On the basis, texture and illumination combined optimization can be realized, in other words, the texture parameters and illumination parameters are adjusted simultaneously, so that the rendering result obtained by the target rendering model obtained through training is compared with the target object, the texture, the grid and the illumination effect of the target rendering model reach the preset requirements, and the consistency of global texture illumination in the rendering result is ensured.
Please refer to fig. 2, which is a schematic view of a scene of a training method of an image rendering model according to an embodiment of the present application. In fig. 2, a process for reconstructing a three-dimensional geometry of a shoe is described by taking the shoe as an example.
Firstly, an original image of a shoe (such as an object image of the shoe in fig. 2) is collected, wherein the collected original image can be an original image of a plurality of visual angles of the shoe, and the image collecting tool positioned at the plurality of visual angles of the shoe is used for obtaining pictures corresponding to the visual angles respectively by taking the shoe as a center. Secondly, the original image of the shoe is subjected to segmentation processing, and a foreground image and a background image corresponding to the original image of the shoe are separated, so that a segmented image of the original image (such as object segmentation of the shoe in fig. 2) is obtained. Then, according to the original image of the shoe, the segmented image of the shoe, and the camera pose information of the object, the grid information of the shoe (such as the object grid of the shoe in fig. 2) is constructed.
Providing the original image of the shoe to the initial rendering model to obtain a rendering result of the shoe and a rendering segmentation image corresponding to the rendering result. And determining a first loss value for the geometric difference of the shoe between the original image of the shoe and the rendering result according to a first comparison result of the original image of the shoe and the rendering result and a second comparison result between the original segmentation image corresponding to the original image of the shoe and the rendering segmentation image of the rendering result. And adjusting the rendering parameters of the initial rendering model according to the first loss value. By adjusting the rendering parameters of the initial rendering model, the difference between the rendering result obtained by the rendering model after the rendering parameters are adjusted and the target object in terms of the geometry of the shoes is reduced, and the reduction degree of the target object in the rendering result obtained by the rendering model after the rendering parameters are adjusted is improved.
The first loss value is reduced by adjusting the rendering parameters. Wherein the rendering parameters include at least one of the following: texture parameters, pose parameters, grid parameters, illumination parameters. Thus, the present application can reduce the first penalty value by adjusting texture parameters (such as texture optimization in FIG. 2). The first penalty value may be calculated by determining the first penalty value according to a similarity between texture data of the rendering result and texture data of the target object.
The present application may also reduce the first loss value by performing a joint adjustment on the texture parameter and the pose parameter (e.g., a joint process of texture optimization and camera pose optimization in fig. 2). The first loss value is mainly calculated by weighting according to the loss value of the texture parameter and the loss value of the pose parameter to obtain the similarity between the rendering result and the original image, and the first loss value is determined.
In addition, the similarity between the rendering result and the original image can be synchronously improved by means of jointly adjusting the texture parameters, the pose parameters and the grid parameters (such as the joint process of texture optimization, camera pose optimization and grid optimization in fig. 2), so that the first loss value is reduced. The first loss value is mainly obtained by weighting according to the loss value of the texture parameter, the loss value of the pose parameter and the loss value of the grid parameter.
The three methods for adjusting the model parameters of the rendering model to be trained and improving the target object reduction degree in the rendering result mainly improve the similarity of the object geometry between the rendering result obtained by the rendering model after the model parameters are adjusted and the original image. On the basis, the texture optimization and illumination optimization as in fig. 2 are synchronously performed, in other words, on the premise that the pose parameters of the initial rendering model and the grid parameters are optimized to the target state, the texture parameters and illumination parameters of the initial rendering model are adjusted, so that the rendering result is consistent with the texture and illumination effect compared with the original image of the shoe.
The embodiment of the application provides a training method of an image rendering model, which comprises the following steps: obtaining an initial image and an initial segmentation image of a target object; rendering the initial image and the initial segmentation image by utilizing a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image; training the rendering model to be trained based on differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
According to the method, the rendering model to be trained is trained according to differences among the rendering image, the rendering segmentation image, the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained. According to the method, the rendering parameters of the rendering model to be trained are adjusted, so that differences between the rendering image and the rendering segmentation image obtained by the adjusted rendering model and the initial image and the initial segmentation image are reduced, and the object reduction degree in the rendering image is improved.
First embodiment
Fig. 3 is a flowchart of a training method of an image rendering model according to a first embodiment of the present application, and the training method of the image rendering model according to the first embodiment of the present application is described in detail below with reference to fig. 3.
As shown in fig. 3, in step S301, an initial image and an initial divided image of a target object are obtained.
This step is for obtaining an initial image and an initial divided image of the target object as a sample image and a sample divided image of the target object. Training the rendering model to be trained by adopting the initial image and the initial segmentation image, so that the similarity between the target object and the actual target object in the rendering result obtained by rendering the trained rendering model according to the initial image is improved.
Here, acquiring the initial image of the target object may include acquiring a multi-view initial image of the target object, for example, taking an original picture of the target object as an initial image of the target object by an image acquisition tool of a plurality of orientations of the target object centering on the target object.
After the initial image of the target object is obtained, segmentation processing of foreground content and background content in the picture is carried out on the initial image, and an initial segmentation image of the initial image is obtained.
As shown in fig. 3, in step S302, the initial image and the initial divided image are rendered by using a rendering model to be trained, and a rendered image and a rendered divided image of the initial image are obtained.
The method comprises the steps of rendering an initial image and an initial segmentation image through a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image. In the subsequent step, the rendering image and the initial image are compared, the rendering segmentation image and the initial segmentation image are compared, and the rendering parameters of the rendering model to be trained are adjusted according to the comparison result of the rendering image and the initial segmentation image.
As shown in fig. 3, in step S303, the rendering model to be trained is trained based on differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
The step is used for adjusting model parameters of a rendering model to be trained according to differences among the rendering image, the rendering segmentation image, the initial image and the initial segmentation image, and comprises the following steps: and adjusting rendering parameters of the rendering model to be trained. Therefore, the difference between the rendering result obtained by the rendering model after the rendering parameters are adjusted and the initial image is reduced, and the similarity between the rendering result and the target object is improved.
The method comprises the steps of adjusting rendering parameters of a rendering model to be trained, wherein the first adjusting mode comprises the following steps:
the adjusting the rendering parameters of the rendering model to be trained comprises the following steps: and adjusting texture parameters of the rendering model to be trained.
And adjusting texture parameters of a rendering model to be trained according to differences among the rendering image, the rendering segmentation image, the initial image and the initial segmentation image. Thereby improving the similarity of the rendering result obtained by the rendering model after adjusting the texture parameters and the object geometry between the rendering result and the target object.
The method comprises the steps of adjusting texture parameters of a rendering model to be trained, wherein the texture parameters are determined based on texture data differences between a rendering image and an initial image obtained by the rendering model to be trained. Thus, it further comprises: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image; the training of the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image may be achieved by:
Determining a loss value of the rendering model to be trained based on the texture data differences; and adjusting texture parameters of the rendering model to be trained based on the loss value of the rendering model to be trained.
And obtaining a rendering image of the initial image and a rendering segmentation image by utilizing the rendering model to be trained. Texture data of the initial image is acquired, and the texture data of the image is rendered. And obtaining a loss value between the initial image and the rendered image according to the texture data difference between the texture data of the initial image and the texture data of the rendered image.
Wherein the determining the loss value of the rendering model to be trained includes: a first penalty value for object geometry differences between rendering results characterizing the rendering model to be trained and the target object is determined.
Accordingly, a penalty value for the object geometry between the initial image and the rendered image is determined based on texture data differences between the texture data of the initial image and the texture data of the rendered image, and texture parameters of the rendering model to be trained are adjusted based on the penalty value for the object geometry. Based on the above, the similarity between the object geometry in the rendering result obtained by the rendering model after adjusting the texture parameters and the object geometry of the actual object is improved.
The above-mentioned adjustment process of texture parameters of the rendering model to be trained may be an iterative process. Therefore, the method provided by the embodiment of the application further comprises the following steps:
updating the adjusted texture parameters to the rendering model to be trained to obtain a first rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
For example, after the first texture parameter adjustment is performed on the rendering model to be trained, a first rendering model with the first texture parameter adjustment is obtained, and the first rendering model is adopted to render the initial image and the initial segmentation image, so as to obtain a rendering image and a rendering segmentation image of the initial image.
Then, a loss value of the first rendering model is determined based on the difference in texture data between the rendered image and the rendered segmented image and the initial segmented image, and texture parameters of the first rendering model are adjusted based on the loss value.
And adjusting texture parameters of the first rendering model for preset times according to the process to obtain a target rendering model. And carrying out a texture parameter adjustment process for the first rendering model for preset times, gradually reducing the difference of object geometry between a rendering result obtained by the first rendering model and the target object, and improving the reduction degree of the target object in the rendering result.
The above description is a first adjustment manner for improving the reduction degree of the target object in the rendering result by adjusting the texture parameters of the rendering model to be trained.
In addition, adjusting the rendering parameters of the rendering model to be trained may further include a second adjustment manner: the adjusting the rendering parameters of the rendering model to be trained comprises the following steps: and carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
After the texture parameters and the pose parameters of the rendering model to be trained are jointly adjusted, the difference of the rendering result obtained by the rendering model after the texture parameters and the pose parameters are adjusted and the initial image for the object geometry is reduced, and the similarity for the object geometry is improved.
The joint adjustment of the texture parameters and the pose parameters of the rendering model to be trained is based on the texture data difference and the pose data difference between the rendering result obtained by the rendering model to be trained and the target object, and therefore, the joint adjustment method further comprises the following steps: obtaining texture data differences and pose data differences between the rendered image and the rendered segmentation image and the initial segmentation image; the training of the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image may be achieved by:
Determining a loss value of the rendering model to be trained based on the texture data difference and the pose data difference; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
And providing the initial image and the initial segmentation image of the target object for the rendering model to be trained, and obtaining a rendering image and a rendering segmentation image of the initial image. Texture data differences and pose data differences between the rendered image and the initial image, and texture data differences and pose data differences between the rendered segmented image and the initial segmented image are obtained.
Wherein the determining the loss value of the rendering model to be trained includes: a first penalty value for object geometry differences between rendering results characterizing the rendering model to be trained and the target object is determined.
And determining a loss value of the rendering result obtained by the rendering model to be trained and the target object for the object geometry based on the texture data difference and the pose data difference, and adjusting texture parameters and pose parameters according to the loss value. Based on the method, the reduction degree of the target object in the rendering result obtained by the rendering model to be trained is improved.
The process of jointly adjusting the texture parameters and the pose parameters of the rendering model to be trained can be an iterative process. The method comprises the following steps:
after the texture parameters and the pose parameters of the rendering model to be trained are jointly adjusted, the method further comprises the following steps: updating the adjusted texture parameters and pose parameters to the rendering model to be trained to obtain a second rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
For example, after the texture parameters and the pose parameters of the rendering model to be trained are subjected to the first double-parameter joint adjustment, a second rendering model subjected to the first double-parameter joint adjustment is obtained. And rendering the initial image and the initial segmentation image by adopting a second rendering model to obtain a rendering image and a rendering segmentation image of the initial image.
Then, based on the rendered image and the rendered segmented image, a first loss value of the second rendering model for the object geometry degree of difference is determined, and based on the first loss value, the texture parameters and the pose parameters of the second rendering model are continuously adjusted.
And carrying out joint adjustment on texture parameters and pose parameters of the second rendering model for preset times according to the process to obtain the target rendering model. And carrying out a joint adjustment process of texture parameters and pose parameters for the second rendering model for preset times, gradually reducing the difference of object geometry between a rendering result obtained by the second rendering model and the target object, and improving the reduction degree of the target object in the rendering result.
The second adjustment mode for improving the reduction degree of the target object in the rendering result by jointly adjusting the texture parameters and the pose parameters of the rendering model to be trained is described above.
In addition, adjusting the rendering parameters of the rendering model to be trained may further include a third adjustment manner: the adjusting the rendering parameters of the rendering model to be trained comprises the following steps: and carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
After the texture parameters, the pose parameters and the grid parameters of the rendering model to be trained are jointly adjusted, the difference between the rendering result obtained by the rendering model after the texture parameters, the pose parameters and the grid parameters are adjusted and the initial image for the object geometry is reduced, and the similarity for the object geometry is improved.
The joint adjustment of the texture parameters, the pose parameters and the grid parameters is performed on the rendering model to be trained, and the joint adjustment is based on texture data differences, pose data differences and grid data differences between rendering results obtained by the rendering model to be trained and the target object. Thus, it further comprises: obtaining texture data differences and pose data differences between the rendered image and the rendered segmented image and the initial segmented image, and grid data differences between the rendered image and the initial image; the training of the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image may be achieved by:
determining a loss value of the rendering model to be trained based on the texture data differences, the pose data differences and the grid data differences; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
And providing the initial image and the initial segmentation image of the target object for the rendering model to be trained, and obtaining a rendering image and a rendering segmentation image of the initial image. Texture data differences, pose data differences and mesh data differences between the rendered image and the initial image, and texture data differences, pose data differences and mesh data differences between the rendered segmented image and the initial segmented image are obtained.
Wherein the determining the loss value of the rendering model to be trained includes: a first penalty value for object geometry differences between rendering results characterizing the rendering model to be trained and the target object is determined.
Based on the texture data difference, the pose data difference and the grid data difference, a first loss value for the object geometry between a rendering result obtained by the rendering model to be trained and the target object is determined, and according to the first loss value, the texture parameter, the pose parameter and the grid parameter of the rendering model to be trained are adjusted in a combined mode. Based on the method, the reduction degree of the target object in the rendering result obtained by the rendering model to be trained is improved.
The mesh data difference obtained in the above-described process is obtained from the mesh data difference between the mesh data of the rendered image and the mesh data of the target object. Thus, it further comprises: acquiring grid data of a target object; the grid data of the target object is acquired by the following method:
acquiring a multi-view initial image of the target object and image pose data corresponding to the multi-view initial image respectively; and constructing grid data of the target object according to the multi-view initial image, the image pose data and the initial segmentation image.
The process of jointly adjusting the texture parameters, the pose parameters and the grid parameters of the rendering model to be trained can be an iterative process. The method comprises the following steps:
after the texture parameters, the pose parameters and the grid parameters of the rendering model to be trained are jointly adjusted, the method further comprises the following steps: updating the adjusted texture parameters, pose parameters and grid parameters to the rendering model to be trained to obtain a third rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
For example, after the texture parameters, the pose parameters and the grid parameters of the rendering model to be trained are subjected to first three-parameter joint adjustment, a third rendering model subjected to first three-parameter joint adjustment is obtained. And rendering the initial image and the initial segmentation image by adopting a third rendering model to obtain a rendering image and a rendering segmentation image of the initial image.
Then, based on the texture data differences, pose data differences and mesh data differences between the rendered image and the rendered segmented image and the initial segmented image, a first loss value for object geometry differences of the third rendering model is determined. And continuously adjusting texture parameters, pose parameters and grid parameters of the third rendering model based on the first loss value.
And carrying out the joint adjustment of the texture parameters, the pose parameters and the grid parameters of the third rendering model for preset times according to the process, and obtaining the target rendering model. And carrying out the joint adjustment process of texture parameters, pose parameters and grid parameters for the third rendering model for preset times, gradually reducing the difference of the rendering result obtained by the third rendering model and the target object aiming at the object geometry, and improving the reduction degree of the rendering result aiming at the object geometry.
The three processes of adjusting the rendering parameters of the rendering model to be trained are to improve the similarity of the rendering result obtained by the rendering model after the parameters are adjusted and the object geometry between the rendering result and the target object. On the basis, the illumination parameters of the rendering model to be trained can be continuously adjusted, so that the difference between the rendering result obtained by the rendering model and the target object aiming at illumination data is improved.
Therefore, the application adjusts the rendering parameters of the rendering model to be trained, and also comprises a fourth adjusting mode: the adjusting the rendering parameters of the rendering model to be trained comprises the following steps: and carrying out joint adjustment on the texture parameters and the illumination parameters of the rendering model to be trained.
By carrying out joint adjustment on texture parameters and illumination parameters of the rendering model to be trained, the difference of object geometry and the difference of illumination data between the rendering result obtained by the rendering model after the texture parameters and the illumination parameters are adjusted and the initial image are reduced, and the similarity of the object geometry and the illumination data between the rendering result and the target object is improved.
The joint adjustment of the texture parameter and the illumination parameter of the rendering model to be trained may be based on adjustment when the pose parameter and the grid parameter of the rendering model to be trained reach the target state.
The joint adjustment of texture parameters and illumination parameters is performed on the rendering model to be trained, and is based on texture data differences and illumination data differences between rendering results obtained by the rendering model to be trained and the target object, so that the joint adjustment further comprises: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image, and illumination data differences between the rendered image and the initial image.
The training of the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image may be achieved by:
Determining a loss value of the rendering model to be trained based on the texture data differences and the illumination data differences; and based on the loss value of the rendering model to be trained, carrying out joint adjustment on the texture parameter and the illumination parameter of the rendering model to be trained.
And providing the initial image and the initial segmentation image of the target object for the rendering model to be trained, and obtaining a rendering image and a rendering segmentation image of the initial image. Texture data differences and illumination data differences between the rendered image and the initial image are obtained.
Wherein the determining a loss value of the rendering model to be trained based on the texture data differences and the illumination data differences comprises: determining a first loss value for object geometry degree difference between a rendering result used for representing the rendering model to be trained and the target object based on the texture data difference; based on the illumination data differences, a second loss value for the illumination data difference degree between the rendering result used for representing the rendering model to be trained and the target object is determined.
And determining a first loss value for object geometry between a rendering result obtained by the rendering model to be trained and a target object based on the texture data difference, and determining a second loss value for representing the difference degree of illumination data between the rendering result of the rendering model to be trained and the target object based on the illumination data difference. And adjusting the texture parameters according to the first loss value and adjusting the illumination parameters according to the second loss value. Based on the method, the reduction degree of the target object in the rendering result obtained by the rendering model to be trained is improved.
The process of jointly adjusting the texture parameter and the illumination parameter of the rendering model to be trained may be an iterative process. The method comprises the following steps:
after the texture parameters and the illumination parameters of the rendering model to be trained are jointly adjusted, the method further comprises the following steps: updating the adjusted texture parameters and illumination parameters to the rendering model to be trained to obtain a fourth rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
For example, after the texture parameters and the illumination parameters of the rendering model to be trained are subjected to the first double-parameter joint adjustment, a fourth rendering model after the first double-parameter joint adjustment is obtained. And rendering the initial image and the initial segmentation image by adopting a fourth rendering model to obtain a rendering image and a rendering segmentation image of the initial image.
Then, a first loss value for object geometry variability of a fourth rendering model is determined based on texture data differences between the rendered image and the rendered segmented image and the initial segmented image. A second loss value for the illumination data disparity of the fourth rendering model is determined based on the illumination data disparity between the rendered image and the initial image. And continuously adjusting texture parameters and illumination parameters of the fourth rendering model according to the first loss value and the second loss value.
And carrying out joint adjustment on texture parameters and illumination parameters of the fourth rendering model for preset times according to the process to obtain the target rendering model. And carrying out the process of jointly adjusting the texture parameters and the illumination parameters of the preset times on the fourth rendering model, gradually reducing the difference between the rendering result obtained by the fourth rendering model and the target object, and improving the reduction degree of the target object in the rendering result.
The fourth adjustment mode for improving the reduction degree of the target object in the rendering result by jointly adjusting the texture parameter and the illumination parameter of the rendering model to be trained is described above.
The obtaining of the illumination data difference between the rendered image and the initial image in the above description is based on the comparison between the illumination data of the rendered image and the illumination data of the initial image. Thus, it further comprises: and acquiring illumination data of a rendering result obtained by the rendering model to be trained.
The obtaining of the illumination data of the rendering result obtained by the rendering model to be trained can be achieved by the following modes:
setting micro texture data, micro pose data and micro grid data of a target object in an initialized mode; and inputting the micro-texture data, the micro-pose data and the micro-grid data of the target object into a lighting model to obtain lighting data of the rendering result.
The embodiment of the application provides a training method of an image rendering model, which comprises the following steps: obtaining an initial image and an initial segmentation image of a target object; rendering the initial image and the initial segmentation image by utilizing a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image; training the rendering model to be trained based on the rendering image and the rendering segmentation image and differences between the rendering image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
According to the method, the rendering model to be trained is trained according to differences among the rendering image, the rendering segmentation image, the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting at least rendering parameters of the rendering model to be trained. According to the method, at least the rendering parameters of the rendering model to be trained are adjusted, so that differences between the rendering image and the rendering segmentation image obtained by the adjusted rendering model and the initial image and the initial segmentation image are reduced, and the object reduction degree in the rendering image is improved.
Second embodiment
Fig. 4 is a flowchart of a rendering method according to a second embodiment of the present application. The rendering method according to the second embodiment of the present application is described in detail below with reference to fig. 4. The rendering method provided in the second embodiment is a target rendering model obtained through training according to the first embodiment, and a rendering result of the target object is obtained according to an initial image rendering of the target object. For the specific description process, reference may be made to the description of the above scenario embodiment and the first embodiment, and the description is omitted here.
As shown in fig. 4, in step S401, a multi-view initial image of a target object is provided to a target rendering model, and an initial segmentation image of the target object, texture data, pose data corresponding to the multi-view initial image, and illumination data of the target object are obtained.
The method comprises the step that a target rendering model is used for determining an initial segmentation image, a texture parameter, a pose parameter and an illumination parameter of a target object corresponding to an initial image of the target object according to a multi-view initial image of the target object. The data obtained above are used to construct the basis data of the three-dimensional geometry of the target object.
As shown in fig. 4, in step S402, a rendering grid of the target object is determined according to the multi-view initial image, the initial segmentation image of the target object, texture data, and pose data corresponding to the multi-view initial image, respectively.
The method is used for constructing rendering grid information of the target object, so that texture mapping processing is carried out in the rendering grid of the target object in the subsequent steps.
As shown in fig. 4, in step S403, a texture mapping operation is performed on the rendering grid of the target object according to the texture data.
The method comprises the step of performing texture mapping operation in a rendering grid of a target object according to texture parameters.
As shown in fig. 4, in step S404, according to the illumination data, illumination processing is performed on the rendering grid on which the texture map is completed, so as to obtain a rendered image.
The method comprises the steps of performing texture mapping on a rendering grid of a target object, and further performing illumination mapping processing to ensure that the texture effect and the illumination effect of an object geometry are consistent when a finally obtained first rendering image is compared with an initial image of the target object.
The rendering method in this embodiment adopts the target rendering model obtained by the training method of the image rendering model in the first embodiment, and performs rendering processing on the initial image of the target object to obtain a rendering result. The target rendering model obtained in the first embodiment is a model obtained after performing joint optimization processing of multi-level rendering parameters on the initial rendering model. The model subjected to the joint optimization treatment can achieve that the loss function value between the obtained rendering result and the initial image reaches the preset requirement, and the object reduction degree in the rendering result obtained by the target rendering model is improved.
Third embodiment
On the basis of the first embodiment, a training device for an image rendering model is provided in a third embodiment of the present application, please refer to fig. 5, which is a schematic diagram of a training device for an image rendering model provided in the third embodiment of the present application. The training apparatus of the image rendering model shown in fig. 5 includes:
a first obtaining unit 501 that obtains an initial image and an initial divided image of a target object;
a rendering unit 502, configured to render the initial image and the initial segmentation image by using a rendering model to be trained, so as to obtain a rendered image and a rendered segmentation image of the initial image;
a training unit 503, configured to train the rendering model to be trained based on differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image; wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
Fourth embodiment
On the basis of the second embodiment, a rendering device is provided in a fourth embodiment of the present application, please refer to fig. 6, which is a schematic diagram of a rendering device provided in the fourth embodiment of the present application. The rendering apparatus shown in fig. 6 includes:
A second obtaining unit 601, configured to provide a multi-view initial image of a target object to a target rendering model, to obtain an initial segmentation image and texture data of the target object, where the multi-view initial image corresponds to pose data and illumination data of the target object respectively;
a second determining unit 602, configured to determine a rendering grid of the target object according to the multi-view initial image, the initial segmentation image of the target object, texture data, and pose data corresponding to the multi-view initial image respectively;
an execution unit 603, configured to execute a texture mapping operation on a rendering grid of the target object according to the texture data;
and a third obtaining unit 604, configured to perform illumination processing on the rendering grid with the texture map according to the illumination data, so as to obtain a rendered image.
Fifth embodiment
The fifth embodiment of the present application also provides an electronic device corresponding to the methods of the first and second embodiments of the present application. Fig. 7 is a schematic diagram of an electronic device according to a fifth embodiment of the present application. The electronic device includes: at least one processor 701, at least one communication interface 702, at least one memory 703 and at least one communication bus 704; alternatively, the communication interface 702 may be an interface of a communication module, such as an interface of a GSM module; the processor 701 may be a processor CPU or a specific integrated circuit ASIC (Application Specific Integrated Circuit) or one or more integrated circuits configured to implement embodiments of the present application. The memory 703 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory. In which a memory 703 stores a program, and a processor 701 calls the program stored in the memory 703 to execute the methods of the first and second embodiments of the present application.
Sixth embodiment
The sixth embodiment of the present application also provides a computer storage medium corresponding to the methods of the first and second embodiments of the present application. The computer storage medium stores a computer program that is executed by a processor to perform the methods of the first and second embodiments.
While the application has been described in terms of preferred embodiments, it is not intended to be limiting, but rather, it will be apparent to those skilled in the art that various changes and modifications can be made herein without departing from the spirit and scope of the application as defined by the appended claims.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
1. Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable Media, as defined herein, does not include non-Transitory computer-readable Media (transmission Media), such as modulated data signals and carrier waves.
2. It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be noted that, in the embodiment of the present application, the use of user data may be involved, and in practical application, the user specific personal data may be used in the solution described herein within the scope allowed by the applicable legal regulations in the country under the condition of meeting the applicable legal regulations in the country (for example, the user explicitly agrees to the user to notify practically, etc.).
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

Claims (20)

1. A method of training an image rendering model, comprising:
obtaining an initial image and an initial segmentation image of a target object;
rendering the initial image and the initial segmentation image by utilizing a rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image;
training the rendering model to be trained based on differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image;
wherein the training the rendering model to be trained includes: and adjusting the rendering parameters of the rendering model to be trained.
2. The method of claim 1, wherein the adjusting the rendering parameters of the rendering model to be trained comprises: and adjusting texture parameters of the rendering model to be trained.
3. The method as recited in claim 2, further comprising:
updating the adjusted texture parameters to the rendering model to be trained to obtain a first rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
4. The method of claim 1, wherein the adjusting the rendering parameters of the rendering model to be trained comprises: and carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
5. The method as recited in claim 4, further comprising:
updating the adjusted texture parameters and pose parameters to the rendering model to be trained to obtain a second rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
6. The method of claim 1, wherein the adjusting the rendering parameters of the rendering model to be trained comprises: and carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
7. The method as recited in claim 6, further comprising:
updating the adjusted texture parameters, pose parameters and grid parameters to the rendering model to be trained to obtain a third rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
8. The method of claim 1, wherein the adjusting the rendering parameters of the rendering model to be trained comprises: and carrying out joint adjustment on the texture parameters and the illumination parameters of the rendering model to be trained.
9. The method as recited in claim 8, further comprising:
updating the adjusted texture parameters and illumination parameters to the rendering model to be trained to obtain a fourth rendering model, and continuing to execute the steps of rendering the initial image and the initial segmentation image by using the rendering model to be trained to obtain a rendering image and a rendering segmentation image of the initial image.
10. The method as recited in claim 2, further comprising: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image;
the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes:
determining a loss value of the rendering model to be trained based on the texture data differences;
And adjusting texture parameters of the rendering model to be trained based on the loss value of the rendering model to be trained.
11. The method as recited in claim 4, further comprising: obtaining texture data differences and pose data differences between the rendered image and the rendered segmentation image and the initial segmentation image;
the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes:
determining a loss value of the rendering model to be trained based on the texture data difference and the pose data difference;
and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters and pose parameters of the rendering model to be trained.
12. The method as recited in claim 6, further comprising: obtaining texture data differences and pose data differences between the rendered image and the rendered segmented image and the initial segmented image, and grid data differences between the rendered image and the initial image;
The training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes:
determining a loss value of the rendering model to be trained based on the texture data differences, the pose data differences and the grid data differences;
and based on the loss value of the rendering model to be trained, carrying out joint adjustment on texture parameters, pose parameters and grid parameters of the rendering model to be trained.
13. The method as recited in claim 8, further comprising: obtaining texture data differences between the rendered image and the rendered segmented image and the initial segmented image, and illumination data differences between the rendered image and the initial image;
the training the rendering model to be trained based on the differences between the rendering image and the rendering segmentation image, and the initial image and the initial segmentation image, includes:
determining a loss value of the rendering model to be trained based on the texture data differences and the illumination data differences;
And based on the loss value of the rendering model to be trained, carrying out joint adjustment on the texture parameter and the illumination parameter of the rendering model to be trained.
14. The method according to any of claims 10-12, wherein said determining a loss value of the rendering model to be trained comprises:
a first penalty value for object geometry differences between rendering results characterizing the rendering model to be trained and the target object is determined.
15. The method of claim 13, wherein the determining a loss value for the rendering model to be trained based on the texture data differences and the illumination data differences comprises:
determining a first loss value for object geometry degree difference between a rendering result used for representing the rendering model to be trained and the target object based on the texture data difference;
based on the illumination data differences, a second loss value for the illumination data difference degree between the rendering result used for representing the rendering model to be trained and the target object is determined.
16. The method according to claim 1 or 12, further comprising: acquiring grid data of the target object;
The grid data of the target object is acquired by the following method:
acquiring a multi-view initial image of the target object and image pose data corresponding to the multi-view initial image respectively;
and constructing grid data of the target object according to the multi-view initial image, the image pose data and the initial segmentation image.
17. The method as recited in claim 13, further comprising: obtaining illumination data of a rendering result obtained by the rendering model to be trained;
the obtaining the illumination data of the rendering result obtained by the rendering model to be trained comprises the following steps:
setting micro texture data, micro pose data and micro grid data of a target object in an initialized mode;
and inputting the micro-texture data, the micro-pose data and the micro-grid data of the target object into a lighting model to obtain lighting data of the rendering result.
18. A rendering method, comprising:
providing a multi-view initial image of a target object to a target rendering model to obtain an initial segmentation image and texture data of the target object, wherein the multi-view initial image corresponds to pose data and illumination data of the target object respectively;
Determining a rendering grid of the target object according to the multi-view initial image, the initial segmentation image of the target object, texture data and pose data respectively corresponding to the multi-view initial image;
according to the texture data, performing texture mapping operation on the rendering grid of the target object;
and carrying out illumination treatment on the rendering grid with the texture map according to the illumination data to obtain a rendering image.
19. An electronic device comprising a processor and a memory;
the memory having stored therein a computer program which, when executed by the processor, performs the method of any of claims 1-18.
20. A computer storage medium, characterized in that the computer storage medium stores a computer program which, when executed by a processor, performs the method of any of claims 1-18.
CN202310601759.4A 2023-05-25 2023-05-25 Training method and rendering method of image rendering model Pending CN116740245A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310601759.4A CN116740245A (en) 2023-05-25 2023-05-25 Training method and rendering method of image rendering model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310601759.4A CN116740245A (en) 2023-05-25 2023-05-25 Training method and rendering method of image rendering model

Publications (1)

Publication Number Publication Date
CN116740245A true CN116740245A (en) 2023-09-12

Family

ID=87905406

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310601759.4A Pending CN116740245A (en) 2023-05-25 2023-05-25 Training method and rendering method of image rendering model

Country Status (1)

Country Link
CN (1) CN116740245A (en)

Similar Documents

Publication Publication Date Title
CN109003325B (en) Three-dimensional reconstruction method, medium, device and computing equipment
CN107730503A (en) The image object component-level semantic segmentation method and apparatus of three-dimensional feature insertion
US11328481B2 (en) Multi-resolution voxel meshing
CN111738265B (en) Semantic segmentation method, system, medium, and electronic device for RGB-D image
Martin et al. MaterIA: Single Image High‐Resolution Material Capture in the Wild
Wei et al. Object-based illumination estimation with rendering-aware neural networks
CN114677386A (en) Special effect image processing method and device, electronic equipment and storage medium
US11501413B2 (en) Kernel reshaping-powered splatting-based efficient image space lens blur
CN113139992A (en) Multi-resolution voxel gridding
CN115375828A (en) Model shadow generation method, device, equipment and medium
US11706395B2 (en) Apparatus and method for selecting camera providing input images to synthesize virtual view images
Upadhyay et al. Enhancing diffusion models with 3d perspective geometry constraints
CN115713585B (en) Texture image reconstruction method, apparatus, computer device and storage medium
CN110378948B (en) 3D model reconstruction method and device and electronic equipment
CN116228986A (en) Indoor scene illumination estimation method based on local-global completion strategy
CN116977539A (en) Image processing method, apparatus, computer device, storage medium, and program product
CN116740245A (en) Training method and rendering method of image rendering model
CN116228850A (en) Object posture estimation method, device, electronic equipment and readable storage medium
CN117726746B (en) Three-dimensional human body reconstruction method, device, equipment, storage medium and program product
CN117422848B (en) Method and device for segmenting three-dimensional model
CN110390717A (en) 3D model reconstruction method, device and electronic equipment
CN117333627B (en) Reconstruction and complement method, system and storage medium for automatic driving scene
CN115512048A (en) Image reconstruction method and device, electronic equipment and storage medium
CN118036336A (en) Range self-adaptive high-fidelity depth data simulation method
Song et al. Revisiting the Effectiveness of 3D Object Recognition Benchmarks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination