CN116246026A

CN116246026A - Training method of three-dimensional reconstruction model, three-dimensional scene rendering method and device

Info

Publication number: CN116246026A
Application number: CN202310499294.6A
Authority: CN
Inventors: 孟庆月; 刘星; 赵晨
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2023-05-05
Filing date: 2023-05-05
Publication date: 2023-06-09
Anticipated expiration: 2043-05-05
Also published as: CN116246026B

Abstract

The disclosure provides a training method of a three-dimensional reconstruction model, a three-dimensional scene rendering method and a three-dimensional scene rendering device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, augmented reality, virtual reality, deep learning and the like, and can be applied to scenes such as meta universe, AIGC and the like. The implementation scheme is as follows: acquiring a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image; downsampling the first sample image to obtain a first input image; determining a plurality of first rays of the first input image based on the first pose; inputting information of a plurality of first rays into the three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein the size of the first predicted image is the same as that of the first sample image; determining a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image; and adjusting parameters of the three-dimensional reconstruction model based on the loss.

Description

Training method of three-dimensional reconstruction model, three-dimensional scene rendering method and device

Technical Field

The disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, augmented reality, virtual reality, deep learning and the like, and can be applied to scenes such as metauniverse, AIGC (Artificial Intelligence Generated Content) and the like. The disclosure relates to a training method and device for a three-dimensional reconstruction model, a three-dimensional scene rendering method and device, an electronic device and a computer readable storage medium.

Background

Three-dimensional Reconstruction (3D Reconstruction) refers to the establishment of a mathematical model suitable for computer representation and processing of a three-dimensional scene, is the basis for processing, operating and analyzing the properties of the three-dimensional scene in a computer environment, and is also a key technology for establishing virtual reality expressing an objective world in a computer.

In computer vision, three-dimensional reconstruction refers to the process of reconstructing three-dimensional information of a scene from single-view or multi-view images of the scene.

The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.

Disclosure of Invention

The disclosure provides a training method and device for a three-dimensional reconstruction model, a three-dimensional scene rendering method and device, electronic equipment and a computer readable storage medium.

According to an aspect of the present disclosure, there is provided a training method of a three-dimensional reconstruction model, including: acquiring a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image; downsampling the first sample image to obtain a first input image; determining a plurality of first rays of the first input image based on the first pose, wherein the plurality of first rays respectively correspond to a plurality of pixels in the first input image; inputting the information of the plurality of first rays into a three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein the size of the first predicted image is the same as the size of the first sample image; determining a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image; and adjusting parameters of the three-dimensional reconstruction model based on the loss.

According to an aspect of the present disclosure, there is provided a three-dimensional scene rendering method including: acquiring a three-dimensional reconstruction model aiming at a target scene and an observation pose of the target scene, wherein the three-dimensional reconstruction model is obtained by training based on the training method of the three-dimensional reconstruction model; and generating a rendered image of the target scene under the observation pose based on the three-dimensional reconstruction model and the observation pose.

According to an aspect of the present disclosure, there is provided a training apparatus for three-dimensional reconstruction model, including: a first acquisition module configured to acquire a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image; a first downsampling module configured to downsample the first sample image to obtain a first input image; a first determination module configured to determine a plurality of first rays of the first input image based on the first pose, wherein the plurality of first rays respectively correspond to a plurality of pixels in the first input image; a first output module configured to input information of the plurality of first rays into a three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein a size of the first predicted image is the same as a size of the first sample image; a second determination module configured to determine a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image; and an adjustment module configured to adjust parameters of the three-dimensional reconstruction model based on the loss.

According to an aspect of the present disclosure, there is provided a three-dimensional scene rendering apparatus including: the acquisition module is configured to acquire a three-dimensional reconstruction model for a target scene and an observation pose of the target scene, wherein the three-dimensional reconstruction model is trained based on a training device of the three-dimensional reconstruction model; and a generation module configured to generate a rendered image of the target scene in the viewing pose based on the three-dimensional reconstruction model and the viewing pose.

According to an aspect of the present disclosure, there is provided an electronic apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the above aspects.

According to an aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of the above aspects.

According to one or more embodiments of the present disclosure, training efficiency and rendering speed of a three-dimensional reconstruction model can be improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.

FIG. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, in accordance with an embodiment of the present disclosure;

FIG. 2 illustrates a flow chart of a training method of a three-dimensional reconstruction model in accordance with an embodiment of the present disclosure;

FIG. 3 illustrates a block diagram of a three-dimensional reconstruction model according to an embodiment of the present disclosure;

FIG. 4 illustrates a flow chart of a three-dimensional scene rendering method according to an embodiment of the disclosure;

FIG. 5 shows a block diagram of a training apparatus for three-dimensional reconstruction models in accordance with an embodiment of the present disclosure;

Fig. 6 illustrates a block diagram of a structure of a three-dimensional scene rendering device according to an embodiment of the present disclosure; and

fig. 7 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another element. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.

The terminology used in the description of the various illustrated examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.

In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order colloquial is not violated.

In the related technology, images of different view angles of a certain scene can be adopted to train a neural network model, so that the neural network learns the three-dimensional information of the scene, and the three-dimensional reconstruction of the scene is realized. The trained neural network model is then utilized to render an image that generated the scene at the new view angle. In the training process of the neural network model, all pixels of the scene image need to be converted into rays, and ray information of all pixels is input into the neural network model to perform forward computation. In order to ensure the three-dimensional reconstruction effect of the neural network model, the neural network model is usually trained by adopting a high-resolution image, so that the data volume of ray information and the calculation amount of forward calculation are large, the training efficiency of the model is low, and the rendering speed is low.

In view of the above problems, embodiments of the present disclosure provide a training method of a three-dimensional reconstruction model and a three-dimensional scene rendering method. The training method of the three-dimensional reconstruction model can improve the training efficiency and rendering speed of the three-dimensional reconstruction model while guaranteeing the three-dimensional reconstruction precision. By utilizing the three-dimensional reconstruction model of the embodiment of the invention to generate the rendering image of the target scene under the new view angle, the rendering speed and accuracy of the rendering image can be improved.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, in accordance with an embodiment of the present disclosure. Referring to fig. 1, the system 100 includes one or

more client devices

101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120.

Client devices

101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable execution of a training method of a three-dimensional reconstruction model and/or a three-dimensional scene rendering method.

In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of

client devices

101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.

In the configuration shown in fig. 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof that are executable by one or more processors. A user

operating client devices

101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be appreciated that a variety of different system configurations are possible, which may differ from system 100. Accordingly, FIG. 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

The user may use

client devices

101, 102, 103, 104, 105, and/or 106 to specify the viewing pose (i.e., perspective) of the target scene and send a rendering request (i.e., three-dimensional scene rendering request) for the target scene at that viewing pose to server 120. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that the present disclosure may support any number of client devices.

Client devices

101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, linux, or Linux-like operating systems (e.g., GOOGLE Chrome OS); or include various mobile operating systems such as MICROSOFT Windows Mobile OS, iOS, windows Phone, android. Portable handheld devices may include cellular telephones, smart phones, tablet computers, personal Digital Assistants (PDAs), and the like. Wearable devices may include head mounted displays (such as smart glasses) and other devices. The gaming system may include various handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), short Message Service (SMS) applications, and may use a variety of communication protocols.

Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a number of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. For example only, the one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a blockchain network, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, wi-Fi), and/or any combination of these and/or other networks.

The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture that involves virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices of the server). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of

client devices

101, 102, 103, 104, 105, and/or 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of

client devices

101, 102, 103, 104, 105, and/or 106.

In some implementations, the server 120 may be a server of a distributed system or a server that incorporates a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, so as to solve the defects of large management difficulty and weak service expansibility in the traditional physical host and virtual private server (VPS, virtual Private Server) service.

The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store information such as audio files and video files. Database 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. Database 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve the databases and data from the databases in response to the commands.

In some embodiments, one or more of databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key value stores, object stores, or conventional stores supported by the file system.

In embodiments of the present disclosure, a user may send a three-dimensional scene rendering request including a target scene to be rendered and a viewing pose of the target scene to the server 120 through the

client device

101, 102, 103, 104, 105, or 106. The server 120 performs the three-dimensional scene rendering method of the embodiment of the present disclosure in response to a three-dimensional scene rendering request of a user, and generates a rendered image of the target scene in a specified viewing pose based on the trained three-dimensional reconstruction model.

According to some embodiments, the three-dimensional reconstruction model may be trained by server 120, or by other servers (not shown in FIG. 1). In other words, the training method of the three-dimensional reconstruction model according to the embodiment of the present disclosure may be performed by the server 120 or may be performed by another server.

The server performing the three-dimensional scene rendering method according to the embodiment of the present disclosure and the server performing the training method of the three-dimensional reconstruction model according to the embodiment of the present disclosure may be the same server (e.g., server 120), or may be different servers (e.g., the three-dimensional scene rendering method is performed by server 120, and the training method of the three-dimensional reconstruction model is performed by a server other than server 120).

The system 100 of fig. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.

FIG. 2 illustrates a flow chart of a training method 200 for a three-dimensional reconstruction model in accordance with an embodiment of the present disclosure. As described above, the subject of execution of method 200 may be a server, such as server 120 in fig. 1 or other servers other than server 120. In other embodiments, the subject of execution of method 200 may also be a client device, such as client devices 101-106 in FIG. 1.

As shown in fig. 2, the method 200 includes steps S210-S260.

In step S210, a first sample image of a target scene and a first pose of an image acquisition device at the time of acquisition of the first sample image are acquired.

In step S220, the first sample image is downsampled to obtain a first input image.

In step S230, a plurality of first rays of the first input image are determined based on the first pose. The plurality of first rays respectively correspond to a plurality of pixels in the first input image.

In step S240, information of the plurality of first rays is input into the three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model. The size of the first predicted image is the same as the size of the first sample image.

In step S250, a loss of the three-dimensional reconstruction model is determined based at least on the difference of the first prediction image and the first sample image.

In step S260, parameters of the three-dimensional reconstruction model are adjusted based on the above-described loss.

According to an embodiment of the present disclosure, a first input image of a small size is obtained by downsampling a first sample image. Ray information (i.e., information of a plurality of first rays) of a small-sized first input image is input to a three-dimensional reconstruction model to obtain a primary-sized rendered image (i.e., a first predicted image). Compared with the scheme that the ray information of the first sample image is directly input into the three-dimensional reconstruction model, the method greatly reduces the data volume of the ray information and the calculated volume in the forward calculation process, thereby improving the training efficiency and the rendering speed of the three-dimensional reconstruction model and improving the usability of the three-dimensional reconstruction model.

Further, according to the embodiments of the present disclosure, determining the loss of the three-dimensional reconstruction model based on the difference of the original-size first sample image and the first prediction image and adjusting the parameters of the model, it is possible to ensure the three-dimensional reconstruction accuracy of the three-dimensional reconstruction model, that is, to ensure the accuracy of the rendered image.

In embodiments of the present disclosure, the target scene may be any three-dimensional scene to be reconstructed, such as a city street, a field landscape, a metauniverse avatar, and the like.

In an embodiment of the present disclosure, the first sample image may be an image of the target scene acquired by the image acquisition device from an arbitrary view angle. The image capturing device may be any device having image capturing capabilities, including but not limited to cameras, video cameras, cell phones, tablet computers, etc. The first pose is the pose (i.e., perspective) of the image acquisition device when acquiring the first sample image.

According to some embodiments, a first sample image of the target scene and a first pose of the image acquisition device to which the first sample image corresponds may be derived from a motion recovery Structure (SFM) algorithm (Structure From Motion). For example, sample images of a target scene at a plurality of different view angles can be acquired, and then the pose of the image acquisition device corresponding to each sample image is calculated by adopting an SFM algorithm. It will be appreciated that the pose of the image acquisition device may allow for a small amount of error. The first sample image may be a sample image of any view of the target scene, and correspondingly, the first pose is a pose of the image acquisition device when acquiring the sample image.

The pose of the image acquisition device (including the first pose and the second pose described below) is used to indicate the position and pose of the image acquisition device. The position of the image acquisition device may be represented, for example, by three-dimensional coordinates in the form of (x, y, z). The pose of the image acquisition device may be represented, for example, by a pose angle. The attitude angles further include pitch angle (pitch or θ), yaw angle (yaw or ψ), and roll angle (roll or ϕ). In practice, the attitude of the image pickup apparatus may be expressed using only the pitch angle and the roll angle, and the yaw angle may be uniformly set to 0.

In step S220, the first sample image is downsampled (or referred to as downsampled) to obtain a first input image. The downsampling operation can reduce the size of the image. Since the downsampling operation is performed on the first sample image, the size of the resulting first input image is smaller than the size of the first sample image.

The size of the first input image is determined from the downsampled ratio p. Let the size of the first sample image be W (the number of horizontal pixels) H (the number of vertical pixels), the size of the first input image be (W/p) H/p. The ratio of downsampling may be set as desired. For example, the downsampling ratio may be set to 1/4, 1/8, or the like.

In step S230, a plurality of first rays of the first input image may be determined based on the first pose of the image acquisition device. The plurality of first rays respectively correspond to a plurality of pixels in the sample image, and each first ray comprises a plurality of sampling points.

According to some embodiments, each first ray of the plurality of first rays is a ray directed by the image acquisition device having the first pose to a corresponding pixel in the first input image. Specifically, based on the first pose and the internal parameters of the image capturing apparatus (including the focal length, the physical size of the pixels, the number of pixels whose image center differs from the origin of the image, and the like), and the two-dimensional coordinates of the respective pixels of the first input image in the first input image, the positions in space of the respective pixels in the first input image, that is, the three-dimensional coordinates, may be determined. Further, the first ray corresponding to the pixel can be obtained by connecting the position of the image acquisition device (i.e., the three-dimensional coordinate of the image acquisition device in the first pose) with the position of the pixel (i.e., the three-dimensional coordinate of the pixel).

The points on the first ray are sampled, and a plurality of sampling points can be obtained. For example, starting from the origin of the first ray (i.e., the position of the image acquisition device in the first pose), sampling is performed at a certain interval length (i.e., one point is sampled at regular intervals), resulting in a plurality of sampling points. The number of sampling points may be set as desired, for example, to 64, 238, 256, etc. It can be appreciated that the more the number of sampling points is set, the more accurate the three-dimensional reconstruction result of the target scene is, but the lower the computational efficiency is.

The information of the first ray may be a vector obtained by encoding position information (e.g., three-dimensional coordinates) of a plurality of sampling points on the first ray.

In step S240, the information of the plurality of first rays is input into the three-dimensional reconstruction model, and a rendered image, that is, a first predicted image, of the target scene output by the three-dimensional reconstruction model in the first pose may be obtained. The size of the first predicted image is the same as the size of the first sample image. That is, the three-dimensional reconstruction model takes the ray information of the first input image of a small size as an input, and renders and outputs the first predicted image of a large size.

The first predicted image includes predicted color values (e.g., RGB values) for a plurality of pixels. It will be appreciated that pixels in the first predicted image correspond one-to-one with pixels in the first sample image. The color values of the pixels in the first sample image may be used as true values of the predicted color values of the corresponding pixels in the first predicted image.

According to some embodiments, the three-dimensional reconstruction model may be implemented as a multi-layer perceptron (MultiLayer Perceptron, MLP). In other embodiments, the three-dimensional reconstruction model may also be implemented as a convolutional neural network (Convolutional Neural Network, CNN), a deep neural network (Deep Neural Network, DNN), or the like.

Fig. 3 shows a block diagram of a three-dimensional reconstruction model 300 in accordance with an embodiment of the present disclosure. As shown in fig. 3, the three-dimensional reconstruction model 300 includes a convolution layer 310, an activation layer 320, an intermediate processing module 330, a deconvolution layer 340, a deconvolution layer 350, and a deconvolution layer 360.

The intermediate processing module 330 further includes a convolution layer 331, a batch normalization (Batch Normalization, BN) layer 332, an activation layer 333, a convolution layer 334, and a batch normalization layer 335.

The intermediate processing module 330 may set T. T may be any positive integer, such as 20, 30, etc. In the case that there are a plurality of first process modules 330 (i.e., T > 1), the input of the first intermediate process module 330 is connected to the output of the active layer 320, and the inputs of the other intermediate process modules 330 are connected to the output of the last intermediate process module 330. And the output of the last intermediate processing module 330 is connected to the input of the deconvolution layer 340.

The convolution layers 310, 331 and 334 may employ, for example, a convolution kernel of 1*1. The activation layers 320, 333 may, for example, employ a ReLU activation function. Each of the deconvolution layers 340-360 is used to achieve up-sampling of the image at a scale of 2, i.e., for each deconvolution layer, the width and height of the image output by that deconvolution layer is twice the width and height, respectively, of the image input to that deconvolution layer. Let the size of the image input to any deconvolution layer be w×h, the size of the image output is 2w×2h.

The first sample image has a size w×h. The size of the first input image may be, for example, (W/8) × (H/8). Each pixel in the first input image may be converted into a first ray based on the first pose of the image acquisition device. And sampling points on the first ray to obtain a plurality of sampling points. The position information of a plurality of sampling points on the first ray is encoded to obtain the encoding vector of the first ray, namely, the information of the first ray. The dimension of the encoded vector may be, for example, C, i.e., C elements are included in the encoded vector.

Information of a plurality of first rays of a first input image is input to the convolution layer 310. The information of the plurality of first rays may be represented, for example, as a tensor of n×c (W/8) ×h/8), where N is the number of first input images (i.e., the number of first sample images), C is the dimension of the encoding vector of the first rays, and (W/8) ×h/8 is the dimension of the first input images.

The forward computation of the convolution layer 310, the activation layer 320, the intermediate processing module 330, the deconvolution layer 340, the deconvolution layer 350 and the deconvolution layer 360 results in a rendered image with a size w×h, i.e., a first predicted image, output by the deconvolution layer 360. The first predicted image may be represented as, for example, a tensor of n×3×w×h, where N is the number of first predicted images (i.e., the number of first sample images), 3 is the dimension of the color vector of the pixel (i.e., three channels of RGB), and w×h is the size of the first predicted image.

According to some embodiments, the loss of the three-dimensional reconstruction model may be determined based only on a first difference of the first prediction image and the first sample image. This can improve the calculation efficiency.

The first difference of the first prediction image and the first sample image may be a color difference of the first prediction image and the first sample image, and accordingly, the loss of the three-dimensional reconstruction model includes only a color loss of the first prediction image and the first sample image. According to some embodiments, the color Loss may be expressed as an average absolute error (Mean Absolute Error, MAE), also known as L1 Loss, of color values of pixels of the first prediction image and corresponding locations of the first sample image. According to other embodiments, the color Loss may also be expressed as a mean square error (Mean Square Error, MSE), also known as L2 Loss, of color values of pixels of the first predicted image and corresponding locations of the first sample image. It should be appreciated that other functions (i.e., loss functions) may be employed to calculate color Loss in addition to the L1 Loss and L2 Loss described above. The present disclosure does not limit the loss function of color loss.

According to other embodiments, the loss of the three-dimensional reconstruction model may also be determined in combination with a second sample image having repeated pixels with the first sample image. Therefore, constraints among images can be added in the model training process, consistency among images of the three-dimensional reconstruction result is guaranteed, and reconstruction accuracy is improved. According to some embodiments, the method 200 further comprises: and acquiring a second sample image of the target scene, a second prediction image corresponding to the second sample image and a pixel matching relation between the first sample image and the second sample image. Wherein the first pixel in the matched first sample image and the second pixel in the second sample image correspond to the same spatial point. Accordingly, in step S250, a loss of the three-dimensional reconstruction model may be determined based on the first difference of the first prediction image and the first sample image, the second difference of the second prediction image and the second sample image, and the pixel matching relationship.

According to the above-described embodiments, the differences (i.e., the first difference and the second difference) of the predicted image and the real image can ensure the accuracy of the color of the single image rendered by the model. The pixel matching relationship can ensure consistency of colors between different images. The loss of the three-dimensional reconstruction model is determined by combining the first difference, the second difference and the pixel matching relation, so that the three-dimensional reconstruction precision of the three-dimensional reconstruction model can be improved.

According to some embodiments, the second predicted image corresponding to the second sample image may be acquired according to the following steps S210 '-S240'.

In step S210', a second pose of the image acquisition device at the time of acquisition of a second sample image is acquired.

In step S220', the second sample image is downsampled to obtain a second input image.

In step S230', a plurality of second rays of the second input image are determined based on the second pose, wherein the plurality of second rays respectively correspond to a plurality of pixels in the second input image.

In step S240', information of the plurality of second rays is input into the three-dimensional reconstruction model to obtain a second predicted image output by the three-dimensional reconstruction model, wherein a size of the second predicted image is the same as a size of the second sample image.

The steps S210 to S240' correspond to the steps S210 to S240, respectively. Details of the implementation of steps S210'-S240' may be referred to the relevant descriptions of steps S210-S240, and will not be repeated here.

According to some embodiments, the difference between the first pose of the first sample image and the second pose of the second sample image can be made smaller (for example, smaller than a first threshold value), so that the first sample image and the second sample image can be ensured to have enough repeated pixels, thereby facilitating the addition of inter-image constraints (i.e., inter-frame constraints) in the model training process, further ensuring the inter-frame consistency of the three-dimensional reconstruction result, and improving the reconstruction precision.

According to some embodiments, the first sample image and the second sample image may be obtained through the following steps S271 to S273.

In step S271, a plurality of sample images of a target scene and a plurality of poses of an image capturing apparatus respectively corresponding to the plurality of sample images are acquired.

In step S272, a plurality of sample image pairs are determined from the plurality of sample images based on the plurality of poses, wherein a difference in poses of two images in any of the plurality of sample image pairs is less than a first threshold. The difference in pose of the two images can be represented by the distance of the pose vectors of the two images. The pose vector may be a six-dimensional vector composed of three-dimensional coordinates and three-dimensional attitude angles of the image capturing apparatus.

In step S273, two images in the sample image pair are determined as the first sample image and the second sample image, respectively.

According to the above-described embodiment, the difference in the pose of the two images in the sample image pair is small (i.e., smaller than the first threshold). The two images in the sample image pair are respectively determined to be the first sample image and the second sample image, so that the first sample image and the second sample image can be ensured to contain more repeated pixels, and the inter-frame constraint is convenient to add, so that the inter-frame consistency of the three-dimensional reconstruction result is ensured, and the reconstruction precision of the three-dimensional reconstruction model is improved.

According to some embodiments, the image capturing device may be used to capture a video of the target scene, and a plurality of consecutive video frames in the video may be used as the plurality of sample images in step S271. That is, the plurality of sample images are a plurality of video frames in a video of the target scene captured by the image capturing apparatus. Video frames in the video have continuity, and the shooting visual angles (namely, the pose of the image acquisition device) of two video frames with a short distance are not greatly different. Accordingly, in step S272, two video frames having a difference in video frame numbers smaller than a second threshold (e.g., 10) may be combined as a sample image pair. The difference in the numbers of the two video frames can be represented by the absolute value of the difference in the numbers of the two video frames.

According to the embodiment, in the case of shooting video on a target scene, video frames with a relatively close distance can be formed into a sample image pair, so that the generation efficiency of the sample image pair is improved.

After the second sample image corresponding to the first sample image is determined through the above-described steps S271 to S273, the pixel matching relationship of the first sample image and the second sample image may be further determined.

The pixel matching relationship is used to indicate whether a first pixel in the first sample image and a second pixel in the second sample image correspond to the same spatial point. If the first pixel p1 in the first sample image and the second pixel p2 in the second sample image correspond to the same spatial point, the first pixel p1 and the second pixel p2 are matched. If the first pixel p1 in the first sample image and the second pixel p2 in the second sample image correspond to different spatial points, the first pixel p1 and the second pixel p2 are not matched.

Note that, when the first pixel p1 and the second pixel p2 are matched, since the imaging angles of the first sample image and the second sample image are different, the pixel coordinates of the two are generally different.

According to some embodiments, the pixel matching relationship of the first sample image and the second sample image may be determined by analyzing the optical flow changes of the first sample image to the second sample image.

According to some embodiments, optical flow changes of the first sample image to the second sample image may be determined based on the trained optical flow estimation network; and determining a pixel matching relationship of the first sample image and the second sample image based on the optical flow change.

According to the embodiment, the optical flow estimation network is utilized to estimate the optical flow change of the two sample images, and the pixel matching relationship of the two sample images is determined based on the optical flow change, so that the pixel matching relationship can be determined quickly, and the calculation efficiency is improved.

The optical flow estimation network may be, for example, a neural network, such as FlowNet or the like. The optical flow estimation network takes the first sample image and the second sample image as inputs, and outputs optical flow changes from the first sample image to the second sample image. Optical flow change refers to the displacement of a first pixel in a first sample image to a second pixel in a second sample image that matches it (i.e., corresponds to the same spatial point). The optical flow change may be expressed as a tensor of 2 x W x H, where 2 represents the displacement in both the lateral and longitudinal directions and W x H is the size of the first sample image and the second sample image.

Since the optical flow change indicates the displacement of a first pixel to a second pixel that matches the first pixel, the pixel coordinates of the first pixel are added to the displacement to obtain the pixel coordinates of the second pixel that matches the first pixel, thereby determining the second pixel that matches the first pixel.

After obtaining the pixel matching relationship of the first sample image and the second prediction image corresponding to the second sample image, in step S250, a loss of the three-dimensional reconstruction model may be determined based on the first difference of the first prediction image and the first sample image, the second difference of the second prediction image and the second sample image, and the pixel matching relationship.

According to some embodiments, the loss of the three-dimensional reconstruction model may be determined based on a first difference of the first prediction image and the first sample image, a second difference of the second prediction image and the second sample image, and a third difference of the color value of the first target pixel and the color value of the second target pixel. The first target pixel is any pixel in the first predicted image, and the second target pixel is a pixel matched with the first target pixel in the second predicted image.

According to the above embodiment, the model loss is determined based on the difference (i.e., the third difference) of the color values of the matched pixels, and the third difference is irrelevant to the pose of the image acquisition device, and does not cause accumulation of pose errors, thereby ensuring the accuracy of model training.

It will be appreciated that the first predicted image (predicted value) corresponds one-to-one to the pixels of the first sample image (true value) and the second predicted image (predicted value) corresponds one-to-one to the pixels of the second sample image (true value). Accordingly, the pixel matching relationship of the first sample image and the second sample image may be multiplexed with the first prediction image and the second prediction image, that is, the pixel matching relationship of the first prediction image and the second prediction image is the same as the pixel matching relationship of the first sample image and the second sample image.

The second difference of the second prediction image and the second sample image may be a color difference of the second prediction image and the second sample image. According to some embodiments, the color difference may be expressed as an average absolute error (L1 Loss) or a mean square error (L2 Loss) of color values of the corresponding position pixels of the second prediction image and the second sample image. It should be appreciated that other functions may be employed to calculate the color difference in addition to the L1 Loss and L2 Loss described above. The present disclosure does not limit the calculation function of the color difference.

The third difference between the color value of the first target pixel and the color value of the second target pixel may be, for example, an average absolute error (L1 Loss) or a mean square error (L2 Loss) of the two. It should be appreciated that other functions may be employed to calculate the color difference in addition to the L1 Loss and L2 Loss described above. The present disclosure does not limit the calculation function of the color difference.

According to some embodiments, the loss of the three-dimensional reconstruction model may be a weighted sum of the first difference, the second difference, and the third difference described above.

According to some embodiments, a re-projection error may be calculated based on the pixel matching relationship, and a loss of the three-dimensional reconstruction model may be determined based on the re-projection error. Specifically, the loss of the three-dimensional reconstruction model may be determined according to the following steps S281 to S283.

In step S281, the first target pixel in the first predicted image is re-projected into the second predicted image based on the first pose and the second pose, so as to obtain a re-projected pixel in the second predicted image corresponding to the first target pixel.

In step S282, a second target pixel in the second predicted image that matches the first target pixel is determined based on the pixel matching relationship.

In step S283, a loss of the three-dimensional reconstruction model is determined based on the first difference of the first predicted image and the first sample image, the second difference of the second predicted image and the second sample image, and the fourth difference of the coordinates of the re-projection pixel and the coordinates of the second target pixel.

According to the above embodiment, the model loss is determined based on the re-projection error (i.e., the fourth difference) of the matched pixels. It should be noted that, since pose information of the image capturing apparatus is used in calculating the re-projection error, accumulation of the pose error may be caused. In the case of a large pose error, the accuracy and convergence speed of the model may be affected to some extent.

According to some embodiments, the fourth difference of the coordinates of the re-projection pixel and the coordinates of the second target pixel may be, for example, the euclidean distance of the coordinates of both.

According to some embodiments, the loss of the three-dimensional reconstruction model may be a weighted sum of the first difference, the second difference, and the fourth difference described above.

After determining the loss of the three-dimensional reconstruction model through step S250, in step S260, parameters of the three-dimensional reconstruction model may be adjusted using an algorithm such as back propagation, based on the determined loss.

It will be appreciated that steps S210-S260 may be repeated a number of times, thereby performing a number of iterations, gradually optimizing the parameters of the three-dimensional reconstruction model. And stopping the training process when the preset termination condition is reached, and obtaining the trained three-dimensional reconstruction model. The termination condition may be, for example, that the penalty is less than a penalty threshold, that the number of iterations reaches a number of times threshold, that the penalty converges, etc.

Based on the trained three-dimensional reconstruction model, embodiments of the present disclosure also provide a three-dimensional scene rendering method. Fig. 4 shows a flow chart of a three-dimensional scene rendering method 400 according to an embodiment of the disclosure. The method 400 may be performed by a server or by a client device.

As shown in fig. 4, the method 400 includes steps S410-S420.

In step S410, a three-dimensional reconstruction model for a target scene and an observation pose of the target scene are acquired, wherein the three-dimensional reconstruction model is trained based on the training method of the three-dimensional reconstruction model in the embodiment of the present disclosure.

In step S420, a rendered image of the target scene in the viewing pose is generated based on the three-dimensional reconstruction model and the viewing pose.

In accordance with embodiments of the present disclosure, a trained three-dimensional reconstruction model is employed to generate a rendered image of a target scene at a specified perspective (i.e., viewing pose). According to the three-dimensional reconstruction model disclosed by the embodiment of the invention, the rendering speed and accuracy of a rendered image can be improved.

According to some embodiments, a three-dimensional scene rendering request may be received for a user, the three-dimensional scene rendering request including a target scene to be rendered and a viewing pose of the target scene specified by the user.

The observation pose includes, for example, a position (expressed in three-dimensional coordinates) and a posture angle (including pitch angle, yaw angle, and roll angle) of an observation target scene. And inputting the observation pose into the trained three-dimensional reconstruction model, and obtaining a rendered image of the target scene output by the three-dimensional reconstruction model under the observation pose.

According to an embodiment of the present disclosure, there is also provided a training apparatus for three-dimensional reconstruction model.

Fig. 5 shows a block diagram of a training apparatus 500 for three-dimensional reconstruction model according to an embodiment of the present disclosure. As shown in fig. 5, the apparatus 500 includes a first acquisition module 510, a first downsampling module 520, a first determination module 530, a first output module 540, a second determination module 550, and an adjustment module 560.

The first acquisition module 510 is configured to acquire a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image.

The first downsampling module 520 is configured to downsample the first sample image to obtain a first input image.

The first determination module 530 is configured to determine a plurality of first rays of the first input image based on the first pose, wherein the plurality of first rays respectively correspond to a plurality of pixels in the first input image.

The first output module 540 is configured to input information of the plurality of first rays into a three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein a size of the first predicted image is the same as a size of the first sample image.

The second determination module 550 is configured to determine a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image.

An adjustment module 560 is configured to adjust parameters of the three-dimensional reconstruction model based on the loss.

According to an embodiment of the present disclosure, a first input image of a small size is obtained by downsampling a first sample image. Ray information (i.e., information of a plurality of first rays) of a small-sized first input image is input to a three-dimensional reconstruction model to obtain a primary-sized rendered image (i.e., a first predicted image). Compared with the scheme that the ray information of the first sample image is directly input into the three-dimensional reconstruction model, the method greatly reduces the data volume of the ray information and the calculated volume in the forward calculation process, and therefore the training efficiency and the rendering speed of the model are improved.

According to some embodiments, the apparatus 500 further comprises: a second acquisition module configured to acquire a second sample image of a target scene, a second prediction image corresponding to the second sample image, and a pixel matching relationship between the first sample image and the second sample image, wherein a first pixel in the matched first sample image and a second pixel in the second sample image correspond to the same spatial point; wherein the second determination module is further configured to: the loss is determined based on a first difference of the first predicted image and the first sample image, a second difference of the second predicted image and the second sample image, and the pixel matching relationship.

According to some embodiments, the second determining module comprises: a first determining unit configured to determine the loss based on the first difference, the second difference, and a third difference of a color value of a first target pixel and a color value of a second target pixel, wherein the first target pixel is any pixel in the first predicted image, and the second target pixel is a pixel in the second predicted image that matches the first target pixel.

According to some embodiments, the second determining module comprises: a reprojection unit configured to reproject a first target pixel in the first predicted image into a second predicted image based on the first pose and a second pose of the image acquisition device when acquiring the second sample image, to obtain a reprojected pixel in the second predicted image; a matching unit configured to determine a second target pixel in the second predicted image that matches the first target pixel based on the pixel matching relationship; and a second determining unit configured to determine the loss based on the first difference, the second difference, and a fourth difference of coordinates of the re-projection pixel and coordinates of the second target pixel.

According to some embodiments, the apparatus 500 further comprises: an estimation module configured to determine an optical flow change of the first sample image to the second sample image based on a trained optical flow estimation network; and a third determination module configured to determine the pixel matching relationship based on the optical flow variation.

According to some embodiments, the apparatus 500 further comprises: a third acquisition module configured to acquire a plurality of sample images of a target scene and a plurality of poses of an image acquisition device respectively corresponding to the plurality of sample images; a fourth determination module configured to determine a plurality of sample image pairs from the plurality of sample images based on the plurality of poses, wherein a difference in poses of two images in any of the plurality of sample image pairs is less than a first threshold; and a fifth determining module configured to determine two images of the sample image pair as the first sample image and the second sample image, respectively.

According to some embodiments, the plurality of sample images are a plurality of video frames in a video of the target scene captured by the image capture device, and wherein the fourth determination module is further configured to: two video frames having a difference in video frame numbers less than a second threshold are determined as the sample image pair.

According to some embodiments, the second acquisition module comprises: a fourth acquisition module configured to acquire a second pose of the image acquisition device when acquiring the second sample image; a second downsampling module configured to downsample the second sample image to obtain a second input image; a sixth determining module configured to determine a plurality of second rays of the second input image based on the second pose, wherein the plurality of second rays respectively correspond to a plurality of pixels in the second input image; and a second output module configured to input information of the plurality of second rays into the three-dimensional reconstruction model to obtain a second predicted image output by the three-dimensional reconstruction model, wherein a size of the second predicted image is the same as a size of the second sample image.

According to some embodiments, each first ray of the plurality of first rays is a ray directed by an image acquisition device having the first pose to a corresponding pixel in the first input image.

It should be appreciated that the various modules or units of the apparatus 500 shown in fig. 5 may correspond to the various steps in the method 200 described with reference to fig. 2. Thus, the operations, features and advantages described above with respect to method 200 apply equally to apparatus 500 and the modules and units comprised thereof. For brevity, certain operations, features and advantages are not described in detail herein.

According to an embodiment of the present disclosure, there is also provided a three-dimensional scene rendering apparatus.

Fig. 6 shows a block diagram of a structure of a three-dimensional scene rendering device 600 according to an embodiment of the present disclosure. As depicted in fig. 6, the apparatus 600 includes an acquisition module 610 and a generation module 620.

The acquisition module 610 is configured to acquire a three-dimensional reconstruction model for a target scene and an observed pose of the target scene, wherein the three-dimensional reconstruction model is trained based on the apparatus of any of claims 10-17.

The generation module 620 is configured to generate a rendered image of the target scene in the viewing pose based on the three-dimensional reconstruction model and the viewing pose.

It should be appreciated that the various modules or units of the apparatus 600 shown in fig. 6 may correspond to the various steps in the method 400 described with reference to fig. 4. Thus, the operations, features and advantages described above with respect to method 400 apply equally to apparatus 600 and the modules and units comprised thereof. For brevity, certain operations, features and advantages are not described in detail herein.

Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules and/or at least some of the functions of the multiple modules may be combined into a single module.

It should also be appreciated that various techniques may be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to fig. 5, 6 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the modules 510-620 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip including one or more components of a processor (e.g., a central processing unit (Central Processing Unit, CPU), microcontroller, microprocessor, digital signal processor (Digital Signal Processor, DSP), etc.), memory, one or more communication interfaces, and/or other circuitry, and may optionally execute received program code and/or include embedded firmware to perform functions.

There is also provided, in accordance with an embodiment of the present disclosure, an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor to enable the at least one processor to perform a training method and/or a three-dimensional scene rendering method of a three-dimensional reconstruction model of an embodiment of the present disclosure.

According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the training method of the three-dimensional reconstruction model and/or the three-dimensional scene rendering method of the embodiments of the present disclosure.

According to an embodiment of the present disclosure, there is also provided a computer program product comprising computer program instructions which, when executed by a processor, implement the training method of the three-dimensional reconstruction model and/or the three-dimensional scene rendering method of the embodiments of the present disclosure.

Referring to fig. 7, a block diagram of an electronic device 700 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 may also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the electronic device 700 are connected to the I/O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, the input unit 706 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and/or a remote control. The output unit 707 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 708 may include, but is not limited to, magnetic disks, optical disks. The communication unit 709 allows the electronic device 700 to exchange information/data with other devices through computer networks, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth devices, 802.11 devices, wi-Fi devices, wiMAX devices, cellular communication devices, and/or the like.

The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as method 200 and/or method 400. For example, in some embodiments, method 200 and/or method 400 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 700 via the ROM 702 and/or the communication unit 709. When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of method 200 and method 400 described above may be performed. Alternatively, in other embodiments, computing unit 701 may be configured to perform method 200 and/or method 400 by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), the internet, and blockchain networks.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely illustrative embodiments or examples and that the scope of the present disclosure is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.

Claims

1. A training method of a three-dimensional reconstruction model, comprising:

Acquiring a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image;

downsampling the first sample image to obtain a first input image;

determining a plurality of first rays of the first input image based on the first pose, wherein the plurality of first rays respectively correspond to a plurality of pixels in the first input image;

inputting the information of the plurality of first rays into a three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein the size of the first predicted image is the same as the size of the first sample image;

determining a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image; and

and adjusting parameters of the three-dimensional reconstruction model based on the loss.

2. The method of claim 1, further comprising:

acquiring a second sample image of the target scene, a second prediction image corresponding to the second sample image and a pixel matching relation between the first sample image and the second sample image, wherein a first pixel in the matched first sample image and a second pixel in the second sample image correspond to the same space point;

Wherein said determining a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image comprises:

the loss is determined based on a first difference of the first predicted image and the first sample image, a second difference of the second predicted image and the second sample image, and the pixel matching relationship.

3. The method of claim 2, wherein the determining the penalty based on the first difference of the first predicted image and the first sample image, the second difference of the second predicted image and the second sample image, and the pixel matching relationship comprises:

and determining the loss based on the first difference, the second difference and a third difference between a color value of a first target pixel and a color value of a second target pixel, wherein the first target pixel is any pixel in the first predicted image, and the second target pixel is a pixel matched with the first target pixel in the second predicted image.

4. The method of claim 2, wherein the determining the penalty based on the first difference of the first predicted image and the first sample image, the second difference of the second predicted image and the second sample image, and the pixel matching relationship comprises:

Based on the first pose and a second pose of the image acquisition device when the second sample image is acquired, re-projecting a first target pixel in the first predicted image into a second predicted image to obtain a re-projected pixel in the second predicted image;

determining a second target pixel in the second predicted image that matches the first target pixel based on the pixel matching relationship; and

the loss is determined based on the first difference, the second difference, and a fourth difference of coordinates of the re-projected pixel and coordinates of the second target pixel.

5. The method of any of claims 2-4, further comprising:

determining an optical flow change of the first sample image to the second sample image based on a trained optical flow estimation network; and

the pixel matching relationship is determined based on the optical flow variation.

6. The method of any of claims 2-4, further comprising:

acquiring a plurality of sample images of the target scene and a plurality of poses of an image acquisition device respectively corresponding to the plurality of sample images;

determining a plurality of sample image pairs from the plurality of sample images based on the plurality of poses, wherein a difference in poses of two images in any of the plurality of sample image pairs is less than a first threshold; and

Two images in the sample image pair are determined as the first sample image and the second sample image, respectively.

7. The method of claim 6, wherein the plurality of sample images are a plurality of video frames in a video of the target scene captured by the image capture device, and wherein the determining, based on the plurality of poses, a plurality of sample image pairs from the plurality of sample images comprises:

two video frames having a difference in video frame numbers less than a second threshold are determined as the sample image pair.

8. The method of claim 2, wherein acquiring a second predicted image corresponding to the second sample image comprises:

acquiring a second pose of the image acquisition device when acquiring the second sample image;

downsampling the second sample image to obtain a second input image;

determining a plurality of second rays of the second input image based on the second pose, wherein the plurality of second rays respectively correspond to a plurality of pixels in the second input image; and

and inputting the information of the plurality of second rays into the three-dimensional reconstruction model to obtain the second predicted image output by the three-dimensional reconstruction model, wherein the size of the second predicted image is the same as the size of the second sample image.

9. The method of claim 1, wherein each first ray of the plurality of first rays is a ray directed by an image acquisition device having the first pose to a corresponding pixel in the first input image.

10. A three-dimensional scene rendering method, comprising:

acquiring a three-dimensional reconstruction model for a target scene and an observation pose of the target scene, wherein the three-dimensional reconstruction model is trained based on the method of any one of claims 1-9; and

and generating a rendered image of the target scene under the observation pose based on the three-dimensional reconstruction model and the observation pose.

11. A training apparatus for a three-dimensional reconstruction model, comprising:

a first acquisition module configured to acquire a first sample image of a target scene and a first pose of an image acquisition device when acquiring the first sample image;

a first downsampling module configured to downsample the first sample image to obtain a first input image;

a first determination module configured to determine a plurality of first rays of the first input image based on the first pose, wherein the plurality of first rays respectively correspond to a plurality of pixels in the first input image;

A first output module configured to input information of the plurality of first rays into a three-dimensional reconstruction model to obtain a first predicted image output by the three-dimensional reconstruction model, wherein a size of the first predicted image is the same as a size of the first sample image;

a second determination module configured to determine a loss of the three-dimensional reconstruction model based at least on a difference of the first predicted image and the first sample image; and

an adjustment module configured to adjust parameters of the three-dimensional reconstruction model based on the loss.

12. The apparatus of claim 11, further comprising:

a second acquisition module configured to acquire a second sample image of the target scene, a second prediction image corresponding to the second sample image, and a pixel matching relationship between the first sample image and the second sample image, wherein a first pixel in the matched first sample image and a second pixel in the second sample image correspond to the same spatial point;

wherein the second determination module is further configured to:

13. The apparatus of claim 12, wherein the second determination module comprises:

a first determining unit configured to determine the loss based on the first difference, the second difference, and a third difference of a color value of a first target pixel and a color value of a second target pixel, wherein the first target pixel is any pixel in the first predicted image, and the second target pixel is a pixel in the second predicted image that matches the first target pixel.

14. The apparatus of claim 12, wherein the second determination module comprises:

a reprojection unit configured to reproject a first target pixel in the first predicted image into a second predicted image based on the first pose and a second pose of the image acquisition device when acquiring the second sample image, to obtain a reprojected pixel in the second predicted image;

a matching unit configured to determine a second target pixel in the second predicted image that matches the first target pixel based on the pixel matching relationship; and

a second determination unit configured to determine the loss based on the first difference, the second difference, and a fourth difference of coordinates of the re-projection pixel and coordinates of the second target pixel.

15. The apparatus of any of claims 12-14, further comprising:

an estimation module configured to determine an optical flow change of the first sample image to the second sample image based on a trained optical flow estimation network; and

a third determination module configured to determine the pixel matching relationship based on the optical flow variation.

16. The apparatus of any of claims 12-14, further comprising:

a third acquisition module configured to acquire a plurality of sample images of the target scene and a plurality of poses of an image acquisition device respectively corresponding to the plurality of sample images;

a fourth determination module configured to determine a plurality of sample image pairs from the plurality of sample images based on the plurality of poses, wherein a difference in poses of two images in any of the plurality of sample image pairs is less than a first threshold; and

a fifth determination module configured to determine two images in the sample image pair as the first sample image and the second sample image, respectively.

17. The apparatus of claim 16, wherein the plurality of sample images are a plurality of video frames in a video of the target scene captured by the image capture device, and wherein the fourth determination module is further configured to:

18. The apparatus of claim 12, wherein the second acquisition module comprises:

a fourth acquisition module configured to acquire a second pose of the image acquisition device when acquiring the second sample image;

a second downsampling module configured to downsample the second sample image to obtain a second input image;

a sixth determining module configured to determine a plurality of second rays of the second input image based on the second pose, wherein the plurality of second rays respectively correspond to a plurality of pixels in the second input image; and

and a second output module configured to input information of the plurality of second rays into the three-dimensional reconstruction model to obtain a second predicted image output by the three-dimensional reconstruction model, wherein the size of the second predicted image is the same as the size of the second sample image.

19. The apparatus of claim 11, wherein each first ray of the plurality of first rays is a ray directed by an image acquisition device having the first pose to a corresponding pixel in the first input image.

20. A three-dimensional scene rendering device, comprising:

an acquisition module configured to acquire a three-dimensional reconstruction model for a target scene and an observation pose of the target scene, wherein the three-dimensional reconstruction model is trained based on the apparatus of any one of claims 11-19; and

a generation module configured to generate a rendered image of the target scene in the viewing pose based on the three-dimensional reconstruction model and the viewing pose.

21. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-10.