EP4595001A1

EP4595001A1 - Neural segmentation fields for representing three-dimensional scenes

Info

Publication number: EP4595001A1
Application number: EP23777460.9A
Authority: EP
Inventors: Peihan Tu; Tsung-Wei Huang; Guan-Ming Su
Original assignee: Dolby Laboratories Licensing Corp
Current assignee: Dolby Laboratories Licensing Corp
Priority date: 2022-09-27
Filing date: 2023-08-31
Publication date: 2025-08-06
Also published as: JP7853519B2; WO2024073214A1; JP2025531498A; CN120266156A

Abstract

Methods and apparatus for rendering segmentation maps of a 3D scene using machine learning. According to an example embodiment, a method of training a neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene comprises: computing color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; generating a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the neural network based on said color texture and said volume density; and adjusting configuration parameters of network nodes of the neural network based on a loss function configured to receive, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further configured to receive, as a second input thereof, the predicted segmentation map.

Description

NEURAL SEGMENTATION FIELDS FOR REPRESENTING THREE-DIMENSIONAL SCENES 1. Cross-Reference to Related Applications [0001] This application claims the benefit of priority from U.S. Provisional Application Serial No.63/410,344 filed on 27 September 2022, which is incorporated by reference herein in its entirety. 2. Field of the Disclosure [0002] Various example embodiments relate to rendering a three-dimensional (3D) scene into a two-dimensional (2D) space. 3. Background [0003] The process of linking each pixel in an image to a class label is referred to as semantic segmentation. The label could be, for example, “person,” “flower,” “building,” etc. Semantic segmentation can be thought of as image classification at the pixel level. Therefore, in semantic segmentation, every pixel of the image is typically associated with a corresponding class label. Semantic segmentation differs from object detection because, unlike object detection, semantic segmentation works at the pixel level to determine relatively accurate contours of objects within an image. [0004] Unlike semantic segmentation, which treats multiple objects within a class as one entity, instance segmentation identifies individual objects within classes. Instance segmentation is sometimes considered to be a refined version of semantic segmentation. Various computer vision applications use either semantic segmentation or a combination of semantic and instance segmentation. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS [0005] Disclosed herein are various embodiments of methods and apparatus for rendering segmentation maps of a 3D scene using machine learning. According to an example embodiment, a smaller neural network (NN) is trained to render segmentation maps corresponding to arbitrarily 1 selected views of the 3D scene, with the training being performed using a larger neural network pretrained to represent the 3D scene in color texture and volume density. In other words, the small neural network is configured to work on top of the pretrained larger neural network, thereby providing the capability to obtain segmentation information faster and with relatively low additional load on the computational resources. Also disclosed are several embodiments of an entropy-based loss function and regularization terms therefor which are constructed to facilitate the training process for the small neural network, e.g., by taking advantage of the auxiliary information available in conjunction with or from the previous training of the larger neural network. [0006] According to an example embodiment, provided is an image-processing method, comprising training, via a processor, a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene, the training including: computing, with the processor, color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; generating, with the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjusting, with the processor, configuration parameters of NN nodes of the first neural network based on a loss function configured to receive, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further configured to receive, as a second input thereof, the predicted segmentation map. [0007] According to another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising the above or below image-processing method. [0008] According to yet another example embodiment, provided is an image-processing apparatus, comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to train a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene; and wherein to train the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; generate a predicted segmentation map corresponding to the 2 selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjust configuration parameters of NN nodes of the first neural network based on a loss function receiving, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further receiving, as a second input thereof, the predicted segmentation map. [0009] According to yet another example embodiment, provided is an image-processing method comprising testing, via a processor, a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene, the testing including: computing, with the processor, color texture and volume density corresponding to a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generating, with the processor, a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained, with the processor, using the pretrained 3D representation. [0010] According to yet another example embodiment, provided is an image-processing apparatus, comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to test a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene; wherein to test the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generate a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained using the pretrained 3D representation. BRIEF DESCRIPTION OF THE DRAWINGS [0011] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which: 3 [0012] FIGs.1A-1B are block diagrams illustrating example process flows for training and testing, respectively, of a neural segmentation field according to various embodiments. [0013] FIG.2 is a block diagram illustrating a multilayer perceptron (MLP) that can be used to implement a neural segmentation field according to an embodiment. [0014] FIG.3 is a flowchart illustrating a process of training the MLP of FIG.2 according to various embodiments. [0015] FIG.4 is a flowchart illustrating a process of testing the MLP of FIG.2 according to various embodiments. [0016] FIG.5 is a flowchart illustrating computations of a loss function used in the process of FIG.3 according to some examples. [0017] FIG.6 is a flowchart illustrating computations of an example regularization term for the loss function used in the process of FIG.3 according to some examples. [0018] FIG.7 is a flowchart illustrating computations of another example regularization term for the loss function used in the process of FIG.3 according to some examples. [0019] FIG.8 is a block diagram illustrating a computing device according to an embodiment. DETAILED DESCRIPTION [0020] This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of various aspects of the present disclosure, and does not limit the scope of the disclosure in any way. [0021] In the following description, numerous details are set forth, such as optical device configurations, timings, operations, and the like, in order to provide an understanding of one or more 4 aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely exemplary and not intended to limit the scope of this application. [0022] Moreover, while the present disclosure focuses mainly on examples in which the various circuits are used in digital projection systems, it will be understood that these are merely examples. It will further be understood that the disclosed systems and methods can be used in any device in which there is a need to project light; for example, cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, and the like. [0023] A neural network (NN) is a typically nonlinear trainable circuit comprising a plurality of processing elements (PEs), also referred to as “neurons,” “artificial neurons,” or “NN nodes.” In some implementations, a neural network is an application-specific circuit, wherein different PEs are implemented as respective configurable sub-circuits connected by physical links forming the corresponding physical network. In some other implementations, a neural network may be computer emulated, in which cases one or more electronic processors are programmed to perform signal processing analogous to that of the corresponding application-specific circuit. [0024] Each PE of a neural network typically has connections with one or more other PEs. The plurality of connections between the (physical or computer-emulated) PEs defines the neural network’s topology. In some topologies, the PEs may be aggregated into layers. Different layers may have different types of PEs configured to perform different respective kinds of transformations on their inputs. Signals may travel from the first PE layer (typically referred to as the input layer) to the last PE layer (typically referred to as the output layer). In some topologies, the neural network may have one or more intermediate PE layers (typically referred to as the hidden layers) located between the input and output PE layers. An example PE may scale, sum, and bias the incoming signals and use an activation function to produce an output signal that is a static nonlinear function of the biased sum. The resulting PE output may become either one of the neural network’s outputs or be sent to one or more other PEs through the corresponding connection(s). The respective weights and/or biases applied by individual PEs can be changed (e.g., optimized) during the training (learning) mode of operation and are typically fixed (i.e., constant) during the testing (working) mode of operation. Various embodiments disclosed herein below may employ or rely on one or more neural networks. 5 Neural Radiance Fields [0025] A Neural Radiance Field (NeRF) represents a 3D scene implicitly, e.g., using a neural network that takes a 3D position and viewing direction as inputs and generates the corresponding predicted color texture and volume density as outputs. The corresponding neural network can be trained using a set of 2D images with known camera poses and pertinent intrinsic information. After having been trained, the neural network can render color texture and volume density of arbitrary views of the 3D scene by querying the corresponding 3D positions and viewing directions for the various pixels in the views. [0026] In various examples, a NeRF is configured to implicitly represent a scene using the continuous 3D scene density ^^ and color ^^ ൌ ^ ^^, ^^, ^^^ as a function of continuous five-dimensional (5D) input vectors of spatial coordinates ^^ ൌ ^ ^^, ^^, ^^^ and viewing directions ^^ ൌ ^ ^^, ^^^, where ^^, ^^, ^^ are the Cartesian coordinates, and ^^, ^^ are the angles used in the spherical coordinate system. The continuous function that converts a 5D input ^ ^^, ^^^ to a four-dimensional (4D) output ^ ^^, ^^, ^^, ^^^ is approximated using an neural network, such as a multilayer perceptron (MLP). [0027] In some NeRF configurations, the density ^^ is a function of only ^^, whereas the radiance ^^ is a function of both position ^^ and viewing direction ^^. The color of a pixel in a 2D rendering of the scene is computed via volume rendering. Given a set of images of a 3D scene with known camera parameters, the neural network’s PE parameters can be optimized via gradient descent by minimizing photometric differences between the rendered and ground-truth images. A brief description of example corresponding MLP, positional encoding, and volume rendering are given below in the remainder of this section. [0028] An MLP is a fully connected feedforward neural network that has an input layer, one or more hidden layers, and an output layer. The “fully connected” attribute means that there is a respective weighted connection between each PE from the previous layer to each PE of the adjacent subsequent layer. An entry of the corresponding weight matrix ^^ is ^^_^^, which includes weights for the connections between the a-th PE of the subsequent layer and the b-th PE of the previous layer. For a layer having a plurality of PEs, the output ^^ ∈ ^^^{^^} thereof can be expressed as a function ^^ of the weighted sum of input values ^^ ∈ ^^^{^^}, where ^^_^ is the number of PEs in the 6 subsequent layer, and ^^_^ is the number of PEs in the previous layer. The function ^^ is the above- mentioned activation function. In mathematical terms, ^^ ൌ ^^^ ^^ ^^ ^ ^^^ (1) where the vector B is the bias vector. A nonlimiting example of the activation function f is the ReLU (Rectified Linear Unit) function expressed as follows: ^^{^^ ^^^ ൌ ^ ^^ for ^^ ^ 0} 0_{for ^^ ^ 0} ⁽²⁾ Other suitable activation functions can be used as well. [0029] Since deep neural networks may intrinsically be biased towards preferentially learning lower frequency functions, a NeRF may be configured to additionally map the inputs to a higher dimensional space using a series of trigonometric functions γ and then pass the higher dimensional inputs to the neural network for better fitting output data with high-frequency components. In various examples, the series γ is defined as follows: ^^^ ^^^ ൌ ^sin^2^{^} ^^ ^^^, cos^2^{^} ^^ ^^^, … , sin^2^{^ି^} ^^ ^^^, cos^2^{^ି^} ^^ ^^^^ (3) where L is the number of to each coordinate in ^^ ൌ ^ ^^, ^^, ^^^ and to each component in the viewing direction ^^ ൌ ^ ^^, ^^^. Before positional encoding, the coordinates and viewing directions are typically normalized to lie in the interval [−1, 1]. Note that Eq. (3) describes an example implementation of positional encoding. In some other implementations, positional encoding is applied to some or all of the coordinates ^ ^^, ^^, ^^^ but not to the viewing direction (e.g., see FIG.3). [0030] An example NeRF represents a 3D scene as the volume density and directional emitted radiance at any point of the 3D space and renders the color of any ray passing through the scene using volume rendering by numerical integration. Let ^^^{^} ^^^{^} ൌ ^^ ^ ^^ ^^_^, ^^_^ ^ ^^ ^ ^^_^ be the ray emitted from the camera through a given pixel, traversing between near and far bounds ( ^^_^ and ^^_^), where ^^ is the origin, and ^^_^ is the unit vector of the direction of the ray. For a given ^^, ^^^{^} ^^^{^} is the 3D coordinate of the point on the ray whose distance to the camera origin is ^^. Then, for selected ^^ random quadrature points ^ ^^_^ ^| ^^ ൌ 1, … , ^^^{^} between ^^_^ and ^^_^, the approximated expected color can be computed using the following equation: ^_^ ^{^} _^^ ^{^} _ൌ ^{∑^} ^_{ୀ^ ^} ^{^} _{^^ ^^^^ ^^^ ^^} ^{^} _^^^ ^{^} _{^^^^ ^^^ ^^^^ (4)} where ^^{^}^^ ^^_^^ ൌ exp൫െ∑^{^ି^} ^_ᇲୀ^ ^^^ ^^_^ᇲ^ ^^_^ᇲ ൯; ^^^ ^^^ ൌ 1 െ exp^െ ^^^; and ^^_^ ൌ ^^_^ା^ െ ^^_^ is the distance ^_^ ^{^} _^^ ^{^} _{to represent} [0031] In addition to the color texture, some applications and/or users require and/or expect semantic segmentation of the 3D scene, which the above-described NeRF does not provide. Beneficially, various embodiments disclosed herein address at least this problem in the state of the art by providing a neural segmentation field implemented as an additional smaller neural network trained and configured to implicitly represent semantic segmentation of the same 3D scene using the initial pretrained neural network, e.g., the NeRF. According to an example embodiment, the training of the smaller neural network takes advantage of the fact that the semantics, color texture, and geometry of a 3D scene are typically correlated to a significant degree. As such, given the initial pretrained neural network’s 3D scene representation and a set of 2D segmentations with known camera poses and intrinsic parameters, the neural segmentation field can be learned in a relatively fast and efficient manner and then the corresponding smaller neural network can be used to render 2D segmentation maps of the 3D scene corresponding to arbitrarily selected viewpoints. At least some embodiments of the training process implement enhancements, e.g., in the form of one or more regularization terms, that can improve the segmentation quality for at least some 3D scenes. In some specific instances, an example embodiment of the disclosed neural segmentation field beneficially achieves a relatively high segmentation quality (e.g., mIoU > 0.9) with a relatively small model size (e.g., < 0.4 MB) for a relatively large number (e.g., > 25) of semantic classes. Various embodiments of the neural segmentation field can be applied, e.g., to object editing (such as color manipulation or enhancement) and/or to 3D-scene encoding and compression. [0032] Herein, the acronym “mIoU” stands for mean intersection-over-union. The mIoU metric is typically computed from the ground truth mask and the predicted mask and is conventionally used to evaluate the performance of various object-detection and semantic-segmentation models. Qualitatively, this metric can be understood as measuring the overlap between the ground truth mask and the predicted mask. Thus, the higher the mIoU value, the better the model’s performance. An mIoU value greater than 0.9 typically signals an excellent performance. Neural Segmentation Fields 8 [0033] FIGs.1A-1B are block diagrams illustrating example process flows (100, 101) for training and testing, respectively, of a neural segmentation field (110) according to various embodiments. A training stage (108) of the training process flow (100) produces the neural segmentation field (110), which is an implicit segmentation representation that represents segmentation in a 3D scene as a neural network, based on a 3D-scene representation (102), a set of 2D segmentation maps (104), and corresponding camera parameters (106). In some instances, the 3D-scene representation (102) is a NeRF. In some other instances, the 3D-scene representation (102) is another suitable 3D scene representation pretrained to represent the 3D scene. For example, such a 3D scene representation can be a non-NN-based solution, e.g., Spherical Harmonics or another suitable implementation thereof. In various examples, semantic segmentation and instance segmentation can be represented by the neural segmentation field (110). A testing stage (114) of the testing process flow (101) produces 2D segmentation maps (116) corresponding to testing (e.g., new or arbitrary) viewpoints using the 3D-scene representation (102), the neural segmentation field (110) produced in the training process flow (100), and camera parameters (112) for the testing viewpoints. Specific nonlimiting examples of the training stage (108) and the testing stage (114) are described in more detail below, e.g., in reference to FIGs.3 and 4. [0034] FIG.2 is a block diagram illustrating an MLP (200) that can be used to implement a neural segmentation field according to an embodiment. The MPL (200) has four layers (2101-2104). The first layer (210₁) is the input layer. The next two layers (210₂, 210₃) are the hidden layers. The fourth layer (210₄) is the output layer. In general, MLP (200) can have N hidden layers, where N is a positive integer. Thus, the specific example of the MLP (200) illustrated in FIG.2 corresponds to N=2. In some specific examples, the number N is in the range from 1 to 4. [0035] Each of the layers (210₁-210₄) has M PEs (202), where M is a positive integer. In some specific examples, the number M is in the range from 16 to 256. Each of the PEs (20221-2022M) of the second layer (2102) is directly connected to receive respective inputs from each of the PEs (202₁₁-202_1M) of the input layer (210₁). Each of the PEs (202₃₁-202_3M) of the third layer (210₃) is directly connected to receive respective inputs from each of the PEs (20221-2022M) of the second layer (2102). Each of the PEs (20241-2024M) of the output layer (2104) is directly connected to receive respective inputs from each of the PEs (202₃₁-202_3M) of the third layer (210₃). Similar PE/layer connections are implemented for additional embodiments corresponding to other values of 9 N. In some examples, each of the PEs (202) operates using the ReLU activation function (see Eq. (2)). In other examples, other suitable activation functions can also be used. [0036] In one specific example, the inputs to the MLP (200) include the positional encoded 3D coordinates ^γ^ ^^^, γ^ ^^^, γ^ ^^^^ (also see Eq. (3)) as well as a 4D input including the radiance ^^ ൌ ^ ^^, ^^, ^^^ and the density ^^. The output of the MLP (200) is a plurality of semantic logits ^^ ∈ ^^^{^ೞ}, where ^^_^ is the number of segmentation classes. [0037] FIG.3 shows a flowchart (300) illustrating a process of training the MLP (NN 1, 200) according to various embodiments. Such training is performed using camera parameters (302) of a set of training views. Each of the training views is associated (342) with a corresponding ground- truth label map (340), which serves as one of the inputs to a loss function (330). Iterative adjustments (328) of the PE parameters used in the MLP (200) are performed to optimize (e.g., approximately minimize) the loss function (330) over the set of training views. Once optimization criteria are met, the training process is terminated, and the corresponding “optimal” PE parameters are fixed and subsequently used in the testing (working) mode of operation, which is described in more detail below in reference to FIG.4. [0038] For a selected view of the set of training views, the camera parameters (302) are used to obtain a 5D input (304, 306) for a pretrained neural network (NN 2, 310). In some specific examples, the neural network (310) implements a NeRF, e.g., as described above. The 5D input (304, 306) includes the corresponding 3D spatial coordinates ^ ^^, ^^, ^^^ (306) and the 2D viewing direction ^ ^^, ^^^ (304). In response to the 5D input (304, 306), the pretrained neural network (310) generates a 4D output ^ ^^, ^^, ^^, ^^^ (314). Positional encoding (312) is applied to the 3D spatial coordinates ^ ^^, ^^, ^^^ (306) to generate a corresponding series ^^^ ^^^ (318) (e.g., see Eq. (3)). The 4D output ^ ^^, ^^, ^^, ^^^ (314) of the neural network (310) together with the series ^^^ ^^^ (318) provide an input for the MLP (200), which is being trained. In response to this input, the MLP (200) generates a plurality of semantic logits ^^ ∈ ^^^{^ೞ} (320). [0039] Volume rendering (322), which can be implemented, e.g., in a manner consistent with Eq. (4), is used to compute, from the logits ^^ (320), a corresponding 2D semantic segmentation map ^^^ ^^^ ∈ ^^^{ுൈ^ൈ^ೞ} (324) of width ^^ and height ^^. An example mathematical equation for the computation of the map (324) is as follows: 10 _^^ ^{^} _^^ ^{^} _ൌ ^{∑^} ^_{ୀ^ ^} ^{^} _{^^ ^^^^ ^^^ ^^} ^{^} _^^^ ^{^} _{, ^^^^ ^^^ ^^^^ (5)} Eq. (5) uses the same notation as . The loss function is then computed based on the semantic segmentation , map (340), and one or more optional parameters (332). In one specific example, the parameters (332) include an RGB image corresponding to the selected view of the set of training views. Various embodiments of the loss function (330) are described in more detail below in reference to Eqs. (7)-(16) and FIGs.5-7. [0040] FIG.4 shows a flowchart (400) illustrating a process of testing the MLP (NN 1, 200) according to various embodiments. As already indicated above, the term “testing” refers to the working mode of operation using which semantic segmentation maps corresponding to arbitrary views of the 3D scene can be generated. Prior to entering the working mode of operation, the MLP (200) is trained as described above in reference to FIG.3. In the testing process (400), the PE parameters of the MLP (200) are constant, e.g., fixed at the optimal values determined using the training process (300). The pretrained neural network (310) remains in the same configuration in the testing process (400) as in the training process (300). [0041] For a selected (e.g., arbitrary) new view, the corresponding camera parameters (402) are used to obtain the corresponding 5D input (304, 306) for the neural network (310). In response to the 5D input (304, 306), the neural network (310) generates the 4D output ^ ^^, ^^, ^^, ^^^ (314). Positional encoding (312) is applied to the 3D spatial coordinates ^ ^^, ^^, ^^^ (306) to generate the corresponding series ^^^{^} ^^^{^} (318). The 4D output ^ ^^, ^^, ^^, ^^^ (314) of the neural network (310) together with the series ^^^{^} ^^^{^} (318) provide an input for the trained MLP (200). In response this input, the trained MLP (200) generates the plurality of semantic logits ^^ ∈ ^^^{^ೞ} (320). Subsequent processing of the MLP’s output includes applying a softmax function (426) to the semantic logits (320) to obtain the probabilities ^^ of rays/pixels belonging to the class ^^. For example, for the pixel ^{^}ℎ, ^^^{^}, the probability of being in the class ^^ can be computed as follows: ^^ ^{^^୮ ^^^,^,^} ^_,௪,^ ൌ (6) where S_h,w,c is the probabilities for the pixels representing the new view make up a probability map (428). The probability map (428) is further processed to generate a label map (430), wherein different objects within the new view are delineated and labeled (e.g., using color coding) in accordance with the respective classes c thereof. 11 Example Loss Functions [0042] In one specific example, the loss function (330) is defined as the cross entropy between the predicted semantic segmentation map (324) and the ground-truth segmentation map (340), e.g., as follows: ^^{^} ^{^^ ∑^ೞି^ ^௧} ^{^ୀ^ ௪,^ ^,௪,^ ∑^ೞି^ ୮ ^^^,^,^ ^௧} ^{^,௪ ൌ െ log ^^^, ∙ ^^ ൌ െ ^ୀ^ log∑^ೞషభ ^^^,௪,^ (7)} ^{^ᇲసబ ^^୮^ ^^^,^,^ᇲ^} (₈₎ where which is the target label; ^^_^ is the total number of classes c, and _^,௪,^ is the ground-truth label for _^,௪,^ ∈ ^0, 1^ at pixel (h, w) of the map (340) for the class ^^. The iterative adjustments (328) are implemented using an Adam optimizer with a learning rate of 5 ൈ 10^ିସ for 20000 iterations. [0043] In another specific example, the loss function (330) is defined as the weighted cross entropy between the predicted semantic segmentation map (324) and the ground-truth segmentation map (340), e.g., as follows: ௪^{^^ ^ ି^ ^ ି^ ^^୮ ^^} ^{^^ ^^௧^ௗ} ^{^,௪ ൌ െ ^^ ∑ ೞ} ^{^ୀ^ l ^௧} ^{,^ ^,௪,^ ^,௪ ∑ ೞ ^,^,^ ^௧} ^{^,௪ og ^^^,௪ ∙ ^^ ൌ െ ^^ ^ୀ^ log ^^୮^ ^^ ^^^,௪,^ (9)} where ^^ be defined as follows: ^_^ ^{^} _^^ ^{^} _{ൌ exp^ ^^^^^^ ^^^^ (11)} where ^^_^^^^ ^^^ ∈ ^0, 1^ is the magnitude of an image edge in the rendered 2D RGB image (332). In contrast to the cross-entropy loss defined by Eqs. (7)-(8), in which every pixel is weighted equally, in the weighted cross-entropy loss defined by Eqs. (9)-(11), the pixels on object boundaries are weighted more heavily than the pixels located relatively deep inside an object. The edges of the RGB image (332) are used to indicate object boundaries in the maps (324, 340) because the object boundaries typically occur on or very near to the RGB edges. [0044] FIG.5 shows a flowchart (500) illustrating computations of the weighted cross entropy for the loss function (330) according to some examples. Inputs to the computations (500) include the predicted semantic segmentation map (324), the ground-truth segmentation map (340), and the 12 RGB image (332). Edge detection (504) is performed for the RGB image (332) to detect object edges therein. The weighting function w of Eq. (11) is then computed (502) such that the object- edge pixels are given higher weights than object-body pixels. In some examples, a deep-learning- based edge detector can be used to implement the computation blocks (502, 504). The loss function (330) is then computed in accordance with Eqs. (9)-(10). [0045] In some other specific examples, the loss function (330) includes one or more weighted regularization terms added to the cross entropy of Eq. (8) or to the weighted cross entropy of Eq. (10). Two illustrative examples of such regularization terms are described below in reference to FIGs.6-7. In various other embodiments, other suitable regularization terms can be used in addition to or instead of those example regularization terms. Eq. (12) provides an example of the loss function (330) employing such regularization terms. ^^ ^^ ^^ ^^ ^^_ ^^ ^^ ^^ ^^ ൌ ^^ ^^ ^^ ^^ℎ ^^ ^^ ^^_ ^^ ^^ ^^ ^^ ^^_ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^ ^^_^ ൈ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^_ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^ ^^_ଶ ൈ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ Herein, the pixelwise regularization and pairwise regularization are example regularization terms; ^^_^ and ^^_ଶ are constants whose values can be empirically selected. In some examples, ^^_^ and ^^_ଶ can be 0.01 and 0.001, respectively. [0046] One example regularization term implements pixelwise edge regularization. The corresponding unary regularization term ^^^{^} ^^^{^} at pixel ^^ is defined as: ^^^ ^^^ ൌ ^{ெೞ^^^} ெ_^^್^^^ (13) where ^^_^ is the segmentation edge map; and ^^_^^^ is the edge detected, e.g., using the above- described edge detector. The map ^^_^ is computed as gradient magnitude over the segmentation probability map, which is expressed as follows: ଶ ^_{^ ^^} ^{^} _{^^ ^ ^} where ^^_௫^. ^ and ^^_௬^. ^ probability map of the c-th class. Intuitively, if the segmentation edge is aligned with the RGB image edge, then the regularization term ^^^ ^^^ of Eq. (13) is relatively small. Otherwise, the regularization term ^^^ ^^^ of 13 Eq. (13) is relatively large. As a result, the regularization term ^^^ ^^^ of Eq. (13) helps to achieve more accurate object delineation in the predicted semantic segmentation maps. [0047] FIG.6 shows a flowchart (600) illustrating computations of a pixelwise regularization term (610) for the loss function (330) according to some examples. Inputs to the computations (600) include the predicted semantic segmentation map (324) and the RGB image (332). Edge detection (504) is performed for the RGB image (332) to detect object edges therein. An edge map (604) is generated based on the object edges detected in the block (504). A segmentation edge map (602) is generated based on the predicted semantic segmentation map (324). The pixelwise edge regularization term (610) is then computed in accordance with Eq. (13) based on the edge maps (602, 604). [0048] Another example regularization term implements pairwise edge regularization. The corresponding pairwise regularization term ^^^ ^^, ^^^ for pixels ^^ and q is defined as: ^^^ ^^, ^^^ ൌ ^^^ ^^_^^^^ ^^, ^^^, ^^_^^ ^^, ^^^^ (16) where ^^_^^^, ^^_^ are the (332) and on the predicted segmentation map (324), respectively; ^^ is a function configured to penalize large segmentation distances for the pixel pairs whose RGB distance is small. ^^_^^^ can be provided by the pretrained neural network (310). The pairwise edge regularization term is configured to use ^^_^^^ for improving ^^_^ and thus ^^. [0049] In one specific example, ^^_^^^ is the length of the shortest path ^^ between two pixels ^^ and ^^ on a weighted graph ^^_^^^, which is a 4-neighbor graph with pixels as nodes and the RGB image-edge magnitude as the edge weight. The corresponding mathematical expression is as follows: ^^_^^^^ ^^, ^^^ ൌ ∑_௭∈^^^,^^ ^^_^^^^ ^^^ (18) where ^^^ ^^, ^^^ is the shortest path between ^^ and ^^ on ^^_^^^ computed using the Dijkstra’s algorithm; and ^^_^^^^ ^^, ^^^ is the length of ^^^ ^^, ^^^. If there are image edges between two pixels on ^^_^^^, then ^^_^^^ is large. Otherwise, ^^_^^^ is small. ^^_^^ ^^, ^^^ is the L2 distance between two _{probability vectors ^^, i.e., ^^^} ^{^} _{^^, ^^} ^{^} _{ൌ ^^2^ ^^} ^{^} _^^ ^{^} _{, ^^^ ^^^^.} 14 [0050] FIG.7 shows a flowchart (700) illustrating computations of a pairwise edge regularization term (710) for the loss function (330) according to some examples. Inputs to the computations (700) include the predicted semantic segmentation map (324) and the RGB image (332). Edge detection (504) is performed for the RGB image (332) to detect object edges therein. An edge map (604) is generated based on the object edges detected in the block (504). A pairwise distance map (702) is generated in accordance with Eq. (18) based on the edge map (604). A pairwise distance map (704) is generated based the predicted semantic segmentation map (324) and using the above definition of ^^_^^ ^^, ^^^. The pairwise edge regularization term (710) is then computed in accordance with Eq. (16) based on the distance maps (702, 704). In some specific examples, the pairwise edge regularization term (710) may be calculated over a local region surrounding the pixel in question, the local area having the size of, e.g., 8 ^8 or 16 ^16 pixels. Example Hardware [0051] FIG.8 is a block diagram illustrating a computing device (800) according to an embodiment. The device (800) can be used, e.g., to implement the process flows (100, 101). The device (800) comprises input/output (I/O) devices (810), an image-processing engine (IPE, 820), and a memory (830). The I/O devices (810) may be used to enable the device (800) to receive at least the parameters of the 3D representation (102) and to output at least the parameters of the neural segmentation field (110) and the segmentation maps (116). The I/O devices (810) may also be used to connect the device (800) to a display. [0052] The memory (830) may have buffers to receive image data corresponding to a 3D scene to be rendered. The image data may be, e.g., in the form of one or more image files. Once the image data are received, the memory (830) may provide parts of the data to the IPE (820) for processing therein. The IPE (820) includes a processor (822) and a memory (824). The memory (824) may store therein program code, which when executed by the processor (822) enables the IPE (820) to perform image processing, including but not limited to the image processing in accordance with some or all of the above-described flowcharts (100, 101, 300, 400, 500, 600, 700). The program code may include, inter alia, the program code used to emulate the various neural networks, e.g., the NN 1 (200) and the NN 2 (310) described above. Once the IPE (820) generates various above described maps by executing the corresponding portions of the code, the IPE (820) may perform rendering processing thereof and provide the corresponding viewable image(s) for 15 being viewed on the display. The viewable image can be, e.g., in the form of a suitable image file outputted through the I/O devices (810). [0053] According to an example embodiment disclosed above, e.g., in the summary section and/or in reference to any one or any combination of some or all of FIGs.1-8, provided is an image- processing method, comprising training, via a processor, a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene, the training including: computing, with the processor, color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a second neural network pretrained to represent the 3D scene; generating, with the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjusting, with the processor, configuration parameters of processing elements of the first neural network based on a loss function configured to receive, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further configured to receive, as a second input thereof, the predicted segmentation map. [0054] In some embodiments of the above method, the first neural network is a multilayer perceptron; and wherein the second neural network is another multilayer perceptron that implements a neural radiance field. [0055] In some embodiments of any of the above methods, the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to said color texture and said volume density. [0056] In some embodiments of any of the above methods, training further includes: calculating a cross entropy between the predicted segmentation map and the ground-truth segmentation map; and obtaining a loss-function value using the cross entropy. [0057] In some embodiments of any of the above methods, the loss function is configured to receive, as a third input thereof, a color image of the 3D scene corresponding to the selected training view. 16 [0058] In some embodiments of any of the above methods, the training further includes: detecting object edges in the color image; assigning different weights to object-edge pixels and to object-body pixels; calculating a cross entropy between the predicted segmentation map and the ground-truth segmentation map using the different weights; and obtaining a loss-function value using the cross entropy. [0059] In some embodiments of any of the above methods, the loss function includes one or more regularization terms favoring alignment of corresponding edges in the predicted segmentation map and the ground-truth segmentation map. [0060] In some embodiments of any of the above methods, the generating is performed in response to positional encoding of a 3D spatial coordinate corresponding to the selected training view. [0061] In some embodiments of any of the above methods, the image-processing method further comprises creating, with the processor, a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creating being performed using the first neural network after the training. [0062] In some embodiments of any of the above methods, the image-processing method further comprises: applying, with the processor, a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; and converting, with the processor, the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene. [0063] According to another example embodiment disclosed above, e.g., in the summary section and/or in reference to any one or any combination of some or all of FIGs.1-8, provided is a non- transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising any of the above methods. [0064] According to yet another example embodiment disclosed above, e.g., in the summary section and/or in reference to any one or any combination of some or all of FIGs.1-8, provided is an image-processing apparatus, comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to train a first neural network to render 17 segmentation maps corresponding to arbitrarily selected views of a 3D scene; and wherein to train the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a second neural network pretrained to represent the 3D scene; generate a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjust configuration parameters of processing elements of the first neural network based on a loss function receiving, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further receiving, as a second input thereof, the predicted segmentation map. [0065] In some embodiments of the above apparatus, the first neural network is a multilayer perceptron; and wherein the second neural network is another multilayer perceptron that implements a neural radiance field. [0066] In some embodiments of any of the above apparatus, the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to said color texture and said volume density. [0067] In some embodiments of any of the above apparatus, to train the first neural network, the apparatus is further configured to: calculate a cross entropy between the predicted segmentation map and the ground-truth segmentation map; and obtain a loss-function value using the cross entropy. [0068] In some embodiments of any of the above apparatus, the loss function is configured to receive, as a third input thereof, a color image of the 3D scene corresponding to the selected training view. [0069] In some embodiments of any of the above apparatus, to train the first neural network, the apparatus is further configured to: detect object edges in the color image; assign different weights to object-edge pixels and to object-body pixels; calculate a cross entropy between the predicted segmentation map and the ground-truth segmentation map using the different weights; and obtain a loss-function value using the cross entropy. 18 [0070] In some embodiments of any of the above apparatus, the loss function includes one or more regularization terms favoring alignment of corresponding edges in the predicted segmentation map and the ground-truth segmentation map. [0071] In some embodiments of any of the above apparatus, the at least one memory and the program code are further configured to, with the at least one processor, cause the apparatus to create a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creating being performed using the first neural network after the training. [0072] In some embodiments of any of the above apparatus, the at least one memory and the program code are further configured to, with the at least one processor, cause the apparatus to: apply a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; and convert the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene. [0073] According to yet another example embodiment disclosed above, e.g., in the summary section and/or in reference to any one or any combination of some or all of FIGs.1-8, provided is an image-processing method, comprising testing, via a processor, a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene, the testing including: computing, with the processor, color texture and volume density corresponding to a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generating, with the processor, a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained, with the processor, using the pretrained 3D representation. [0074] According to yet another example embodiment disclosed above, e.g., in the summary section and/or in reference to any one or any combination of some or all of FIGs.1-8, provided is an image-processing apparatus, comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to test a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene; wherein to test the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to 19 a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generate a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained using the pretrained 3D representation. [0075] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims. [0076] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation. [0077] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary. [0078] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed 20 Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter. [0079] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims. [0080] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit. [0081] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing various embodiments described herein. Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and/or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing various embodiments described herein. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits. [0082] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range. 21 [0083] The use of figure numbers and/or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures. [0084] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence. [0085] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or additional embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.” [0086] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner. [0087] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if” may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].” [0088] Also for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of 22 one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements. [0089] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and/or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context. [0090] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware. The term circuitry also covers, for 23 example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device. [0091] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown. [0092] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and/or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. 24

Claims

CLAIMS What is claimed is: 1. An image-processing method, comprising training, via a processor, a first neural network (NN) to render segmentation maps corresponding to arbitrarily selected views of a three- dimensional (3D) scene, the training including: computing, with the processor, color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; generating, with the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjusting, with the processor, configuration parameters of NN nodes of the first neural network based on a loss function configured to receive, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further configured to receive, as a second input thereof, the predicted segmentation map.

2. The image-processing method of claim 1, wherein the first neural network is a multilayer perceptron; and wherein the 3D representation is a second neural network implementing a neural radiance field.

3. The image-processing method of claim 1 or claim 2, wherein the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to said color texture and said volume density.

4. The image-processing method of any one of claims 1 to 3, wherein the training further includes: calculating a cross entropy between the predicted segmentation map and the ground-truth segmentation map; and obtaining a loss-function value using the cross entropy. 25

5. The image-processing method of any one of claims 1 to 4, wherein the loss function is configured to receive, as a third input thereof, a color image of the 3D scene corresponding to the selected training view.

6. The image-processing method of claim 5, wherein the training further includes: detecting object edges in the color image; assigning different weights to object-edge pixels and to object-body pixels; calculating a cross entropy between the predicted segmentation map and the ground-truth segmentation map using the different weights; and obtaining a loss-function value using the cross entropy.

7. The image-processing method of any one of claims 1 to 6, wherein the loss function includes one or more regularization terms favoring alignment of corresponding edges in the predicted segmentation map and the ground-truth segmentation map.

8. The image-processing method of any one of claims 1 to 7, wherein the generating is performed in response to positional encoding of a 3D spatial coordinate corresponding to the selected training view.

9. The image-processing method of any one of claims 1 to 8, further comprising: creating, with the processor, a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creating being performed using the first neural network after the training; applying, with the processor, a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; and converting, with the processor, the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene.

10. A non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising the method of any one of claims 1 to 9. 26

11. An image-processing apparatus, comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to train a first neural network (NN) to render segmentation maps corresponding to arbitrarily selected views of a three-dimensional (3D) scene; and wherein to train the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to a selected training view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; generate a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and adjust configuration parameters of NN nodes of the first neural network based on a loss function receiving, as a first input thereof, a ground-truth segmentation map corresponding to the selected training view and further receiving, as a second input thereof, the predicted segmentation map.

12. The image-processing apparatus of claim 11, wherein the first neural network is a multilayer perceptron; and wherein the 3D representation is a second neural network implementing a neural radiance field.

13. The image-processing apparatus of claim 11 or claim 12, wherein the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to said color texture and said volume density.

14. The image-processing apparatus of any one of claims 11 to 13, wherein to train the first neural network, the apparatus is further configured to: calculate a cross entropy between the predicted segmentation map and the ground-truth segmentation map; and 27 obtain a loss-function value using the cross entropy.

15. The image-processing apparatus of any one of claims 11 to 14, wherein the loss function is configured to receive, as a third input thereof, a color image of the 3D scene corresponding to the selected training view.

16. The image-processing apparatus of claim 15, wherein to train the first neural network, the apparatus is further configured to: detect object edges in the color image; assign different weights to object-edge pixels and to object-body pixels; calculate a cross entropy between the predicted segmentation map and the ground-truth segmentation map using the different weights; and obtain a loss-function value using the cross entropy.

17. The image-processing apparatus of any one of claims 11 to 16, wherein the loss function includes one or more regularization terms favoring alignment of corresponding edges in the predicted segmentation map and the ground-truth segmentation map.

18. The image-processing apparatus of any one of claims 11 to 17, wherein the at least one memory and the program code are further configured to, with the at least one processor, cause the apparatus to: create a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creating being performed using the first neural network after the training; apply a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; and convert the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene.

19. An image-processing method, comprising testing, via a processor, a first neural network to render segmentation maps corresponding to arbitrarily selected views of a three-dimensional (3D) scene, the testing including: 28 computing, with the processor, color texture and volume density corresponding to a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generating, with the processor, a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained, with the processor, using the pretrained 3D representation.

20. An image-processing apparatus, comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to test a first neural network to render segmentation maps corresponding to arbitrarily selected views of a three-dimensional (3D) scene; wherein to test the first neural network, the apparatus is configured to: compute color texture and volume density corresponding to a selected view of the 3D scene, the computing being performed using a 3D representation pretrained to represent the 3D scene; and generate a segmentation map corresponding to the selected view of the 3D scene, the generating being performed using the first neural network based on said color texture and said volume density; and wherein the first neural network has been trained using the pretrained 3D representation. 29