EP4612903A1 - Residual encoding and decoding using a neural field - Google Patents
Residual encoding and decoding using a neural fieldInfo
- Publication number
- EP4612903A1 EP4612903A1 EP23814007.3A EP23814007A EP4612903A1 EP 4612903 A1 EP4612903 A1 EP 4612903A1 EP 23814007 A EP23814007 A EP 23814007A EP 4612903 A1 EP4612903 A1 EP 4612903A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- parameters
- neural
- image
- residual
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
- H04N19/31—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the temporal domain
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
Definitions
- visual computing is used to synthesize 3-dimensional (3D) shapes and 2-dimensional (2D) images, render novel views of various scenes, and animate articulating human bodies.
- 3D 3-dimensional
- 2D 2-dimensional
- visual computing is used to reconstruct 3D appearance, shape, pose, and deformation.
- human-computer interaction visual computing is used to enable interactive investigation of space- time data.
- robotics applications visual computing is used to support planning and execution of robotic actions.
- a framework for encoding/decoding a sequence of residual images with optional reshaping in a dual-layer neural-network system wherein a base layer provides a baseline representation, and an enhancement layer provides enhancement information using a trained neural field.
- the optional reshaping can be used to reduce the neural- network size.
- the disclosed framework can beneficially provide better quality and/or coding efficiency compared to those achieved when coding is applied to the original image directly.
- the disclosed framework can be used to generate novel frames not encountered in the training data, thereby supporting a wide range of applications with different visual experiences.
- a method for residual encoding using a neural field comprising: accessing, via a processor, a plurality of residual images or reshaped images, each of the residual images representing a difference between a respective reference image and a base image, different respective reference images corresponding to various respective values of one or more first parameters; training, via the processor, a neural-field network to represent the plurality of residual images or reshaped images, the neural-field network being responsive to a variable input of the one or more first parameters and being characterized by a set of second parameters, the training producing fixed values of the second parameters; and communicating the base image and the fixed values of the second parameters to an electronic decoder.
- a method for residual decoding using a neural field comprising: accessing, via a processor, a base image and a set of fixed parameter values of a neural-field network trained to represent a plurality of residual images, the neural-field network being responsive to a variable input of one or more first parameters and being characterized by a set of second parameters, the fixed parameter values being values of the second parameters; testing, via the processor, the neural-field network using a selected value of the variable input to generate an output residual image; and computing, via the processor, a reconstructed image by combining the output residual image and the base image.
- an apparatus for residual encoding comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: access a plurality of residual images or reshaped images, each of the residual images representing a difference between a respective reference image and a base image, different respective reference images corresponding to various respective values of one or more first parameters; train a neural-field network to represent the plurality of residual images, the neural-field network being responsive to a variable input of the one or more first parameters and being characterized by a set of second parameters, the training producing fixed values of the second parameters; and communicate the base image and the fixed values of the second parameters to an electronic decoder.
- an apparatus for residual decoding comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: access a base image and a set of fixed parameter values of a neural-field network trained to represent a plurality of residual images, the neural-field network being responsive to a variable input of one or more first parameters and being characterized by a set of second parameters, the fixed parameter values being values of the second parameters; test the neural-field network using a selected value of the variable input to generate an output residual image; and compute a reconstructed image by summing the output residual image and the base image.
- FIG.1 is a block diagram illustrating a multilayer perceptron (MLP) that can be used to implement a neural field according to an embodiment.
- FIG.2 is a block diagram illustrating an encoder using the MLP of FIG.1 according to an embodiment.
- FIG.3 is a block diagram illustrating a decoder corresponding to the encoder of FIG.2 according to an embodiment.
- FIG.4 is a block diagram illustrating a part of motion-blur image processing using the encoder of FIG.2 according to an embodiment.
- FIG.5 is a block diagram illustrating a computing device according to an embodiment.
- DETAILED DESCRIPTION This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like.
- Neural Field Under the neural-field framework, field quantities are produced by sampling coordinates and feeding the sampled coordinates into a neural network.
- Neural Radiance Field is an implicit 3D scene representation that takes the spatial location (x, y, z) and the viewing direction ( ⁇ , ⁇ ) as inputs and generates the corresponding predicted color texture and volume density as outputs.
- the corresponding neural network can be trained, e.g., using a set of 2D images with known camera poses and pertinent intrinsic information. After having been trained, the neural network can be used to render arbitrary views of the 3D scene by (i) querying the corresponding 3D positions and viewing directions for the various pixels in the views and (ii) performing volume rendering to construct a projected 2D image.
- FIG.1 is a block diagram illustrating a multilayer perceptron (MLP, 100) that can be used to implement a neural field according to an embodiment.
- the MLP (100) has three layers (1101-1103). The first layer (1101) is an input layer. The next layer (1102) is a hidden layer. The third layer (110 3 ) is an output layer.
- the MLP (100) is a fully connected feedforward neural network.
- the “fully connected” attribute means that there is a respective weighted connection between each neural-network (NN) node (also referred to as “processing element,” “neuron,” or “artificial neuron”) from the previous layer to each NN node of the adjacent subsequent layer.
- NN node also referred to as “processing element,” “neuron,” or “artificial neuron”
- An example NN node may scale, sum, and bias the incoming signals and use an activation function to produce an output signal that is a static nonlinear function of the biased sum.
- the node’s output may become either one of the neural network’s outputs or be sent to one or more other NN nodes through the corresponding connection(s).
- the respective weights and/or biases applied by individual NN nodes can be changed (e.g., optimized) during the training (learning) mode of operation and are typically fixed (i.e., constant) during the testing (working) mode of operation.
- Various embodiments disclosed herein below may employ or rely on one or more neural networks, such as the MLP (100).
- An entry ⁇ ⁇ of the weight matrix ⁇ of the MLP (100) is the weight for the connection between the a-th NN node of the subsequent layer and the b-th NN node of the previous layer.
- the output ⁇ ⁇ ⁇ ⁇ thereof can be expressed as a function ⁇ of the weighted sum of input values ⁇ ⁇ ⁇ ⁇ , where ⁇ ⁇ is the number of NN nodes in the subsequent layer, and ⁇ ⁇ is the number of NN nodes in the previous layer.
- the function ⁇ is the above- mentioned activation function.
- ⁇ ⁇ ( ⁇ + ⁇ ) (1) where the vector B is the bias vector.
- a nonlimiting example of the activation function f is the ReLU (Rectified Linear Unit) function expressed as follows: In other examples, other suitable activation functions can be used as well.
- one or more of the final MLP outputs are configured to be in the range [0,1].
- the output layer e.g., the output layer (1103)
- the output layer is configured to apply a sigmoid function ⁇ ( ⁇ ) to map a wider range of values onto the range [0,1].
- the layers (1101, 1102, 1103) of the MLP (100) have two, three and one NN nodes (102), respectively.
- the number of the NN nodes (102) in an MLP layer can be in the range from 1 to 256.
- Different hidden layers may have different respective numbers of the NN nodes (102) or the same number of the NN nodes (102).
- the input layer has 41 NN nodes (102), and each of five hidden layers has 256 NN nodes (102).
- Residual Encoding [0025]
- FIG. 2 is a block diagram illustrating an electronic encoder (200) according to an embodiment.
- a corresponding electronic decoder (300) is described below in reference to FIG. 3.
- the encoder (200) and the decoder (300) can beneficially produce better image quality of reconstructed images than at least some conventional methods due to the use of residual images, which can be represented by floating point values for more precise calculations compared to calculations relying on integer values.
- the encoder (200) and the decoder (300) can be used to accurately interpolate the neural-field residual from nearby reference residuals along a user- selected dimension of the pertinent image sequence.
- the decoder (300) can be operated to accurately predict a novel residual and then generate a corresponding novel image by adding the novel residual to the base image.
- the encoder (200) receives, as an input, a sequence (202) of target (e.g., reference) images.
- each image of the sequence (202) is normalized to have values thereof in the range between 0 and 1.
- the sequence (202) is a sequence of video frames ( ⁇ 5 , ..., ⁇ ⁇ 8 ⁇ ), where ⁇ 5 denotes the first frame in time, and ⁇ ⁇ 8 ⁇ denotes the last frame in time.
- the sequence (202) is not necessary a sequence in the time domain.
- the indices t representing the order of the target images in the sequence (202) correspond to a user-defined domain or dimension other than time.
- the user-defined domain is the dynamic range of the content, wherein ⁇ 5 represents a 100 nits SDR version of an image, and ⁇ ⁇ 8 ⁇ represents a 4000 nits HDR version of the same image.
- ⁇ 5 represents a 4000 nits HDR version of an image
- ⁇ ⁇ 8 ⁇ represents a 100 nits SDR version of the same image.
- the acronyms SDR and HDR stand for Standard Dynamic Range and High Dynamic Range, respectively.
- One image of the sequence (202) is selected to serve as a base image (204), which is denoted as ⁇ ⁇ .
- the encoder (200) operates to encode the base image (204) in a base layer bitstream (294), which is transmitted out, e.g., to the decoder (300).
- the encoder (200) further operates to compute a sequence (208) of residual images by subtracting (206) the base image (204) from each image of the sequence (202).
- a corresponding forward-reshaped sequence (212) is generated by applying forward reshaping (210) to the sequence (208).
- Pertinent parameters of the forward reshaping (210) are transmitted out in a metadata bitstream (296).
- the forward-reshaped sequence (212) is used to train, in a neural field encoding block (214), a suitable neural network, e.g., the MLP (100), to represent the sequence (212).
- the parameters of the trained neural network undergo a compression (216) and are transmitted out in a neural network bitstream (298).
- Example operations performed by the decoder (200) according to various embodiments are described in more detail below.
- the elements (204, 206, 208, 294) are absent or not used.
- pixel values of the residual images > ? have a sign, i.e., can be positive or negative.
- a rescaling operation is performed on the sequence (208), e.g., when the MLP (100) has a sigmoid layer configured to operate in accordance with Eq. (3).
- a neural network such as the MLP (100) may be biased towards preferentially learning lower frequency functions.
- bias can be reduced or eliminated using the above-described positional encoding (e.g., see Eq. (4)).
- One result of positional encoding is an increase in the number of inputs to the input layer of the neural network, such as the input layer (1101).
- the forward reshaping (210) is used to mitigate such an increase. More specifically, the forward reshaping (210) operates to change the signal-frequency distribution such that the weight of higher- frequency components is reduced, thereby enabling a corresponding reduction of the number 2L of trigonometric components of ⁇ (also see Eq. (4)).
- the forward reshaping (210) is optional and can be omitted.
- a region of interest is selected in at least some examples. Only the selected ROI is then fed to the neural network for the training in the neural network encoding block (214). Accordingly, the corresponding neural network bitstream (298) in such examples only contains the coefficients and biases for the ROI.
- the ROI is reconstructed and added to the base image according to the ROI location communicated to the decoder (300) via the metadata bitstream (296).
- the boundary of the ROI is determined as a tight (e.g., a tightest) bounding box enclosing the ROI such that all pixels outside the ROI are nonactive, e.g., have a value of 0.5 in the rescaled range.
- ⁇ F, GF 0.5, GF > ⁇ HI W ⁇ , ⁇ U ⁇ (10)
- a small margin of ⁇ is added to enlarge the ROI for a cleaner transition in the decoded image from the active area encoded by the neural-field ROI to the non-active area from the base image (204).
- the two frame indices, ⁇ HI ⁇ ]5 and ⁇ HI ⁇ ] ⁇ determined in the above-indicated manner define the FOI and are transmitted in the metadata bitstream (296).
- the corresponding spatial axis is divided into M bins, where M is an integer greater than one. In some examples, M can be equal to the number of pixels in the corresponding dimension.
- the forward reshaping (210) operates to compute average local standard deviation (SD) along the corresponding (x or y) dimension for each bin (or overlapped bin).
- SD local standard deviation
- the x and y SDs so computed are denoted as _ ⁇ ( ⁇ ) and _ a (G), respectively.
- SD calculations rely on a sliding window, whose size is denoted as ⁇ .
- _ ⁇ ( ⁇ ) ⁇ c ⁇ ⁇ c ⁇ 8 ⁇ i j5 _ ⁇ ( ⁇ , h) (20)
- _ ⁇ ( ⁇ ) is nonuniform, i.e., changes as a function of x.
- the forward reshaping (210) is configured to apply a mapping function, that maps the uniform grid ⁇ onto a non-uniform grid ⁇ k ⁇ such that the local SD in the new grid ⁇ k ⁇ is more uniform than in the uniform grid ⁇ .
- CDF Cumulative Distribution Function
- the above-described operations are used to implement the forward reshaping (210) in the encoder (200) to achieve the following result: given a uniform grid ⁇ , the mapping function ⁇ ( ⁇ ) is applied to generate a corresponding nonuniform grid ⁇ k ⁇ .
- the nonuniform grid ⁇ k ⁇ is used as an input to the above-described positional encoding.
- the metadata of the mapping function ⁇ ( ⁇ ) are transmitted to the decoder (300) in the metadata bitstream (296).
- the mapping-function metadata are extracted from the metadata bitstream (296), which enables the decoder (300) to reconstruct the mapping function ⁇ ( ⁇ ).
- the decoder (300) can determine the corresponding ⁇ k of the nonuniform grid ⁇ k ⁇ .
- the decoder (300) uses the ⁇ k determined in this manner to query the MLP (100) to obtain the corresponding Red, Green, Blue (RGB) pixel value.
- the decoder (300) then assigns this RGB value to the pixel ⁇ of the uniform grid ⁇ .
- similar operations are performed to implement, in the forward reshaping block (210), reshaping for the y-axis dimension.
- the corresponding mapping function ⁇ a () is constructed to satisfy
- the metadata of the reshaping function ⁇ a () are transmitted to the decoder (300) in the metadata bitstream (296).
- User-Defined-Axis Reshaping [0039]
- the above-described spatial-reshaping algorithm is adapted to apply reshaping to a user-defined dimension.
- the encoder (200) is configured to measure the SD, _ ? ( ⁇ ), along the time dimension t and apply substantially the same calculations to determine a reshaping function ⁇ ? () in the time domain, with the function ⁇ ? () satisfying Eq. (30): where ct is a constant.
- the metadata of the reshaping function ⁇ ? () are transmitted to the decoder (300) in the metadata bitstream (296).
- Codeword-Domain Reshaping [0040]
- the above-described spatial-reshaping algorithm is adapted to apply reshaping in the codeword domain.
- the encoder (200) is configured to compute a histogram in a codeword domain, _ ⁇ (%).
- a corresponding reshaping function ⁇ ⁇ () is then constructed using the above-described method such that: ⁇ ⁇ l_ ⁇ ( % ) m ⁇ o ⁇ (31) where c c is a constant.
- the forward reshaping (210) includes two or more forward reshaping operations performed based on the above-described mapping functions ⁇ a (), ⁇ ? (), ⁇ ⁇ ()). Such multiple reshaping operations can be performed, e.g., sequentially.
- the order in which these reshaping operations are applied to the sequence (208) is communicated to the decoder (300) in the metadata bitstream (296). Based on the communicated order, the decoder (300) operates to apply the corresponding backward reshaping operations in the reverse order.
- Neural Field Encoding As already indicated above, positional encoding (see Eq. (4)) is used to generate the inputs to the MLP (100) in the neural field encoding block (214). To control the MLP overhead, it may be desirable to constrain the total number 2L of trigonometric components of ⁇ for performance optimization purposes in at least some embodiments.
- the corresponding optimization problem can be stated as follows: how to choose the values for the positional encoding to optimize the quality of the images computed by the decoder (300)?
- the above optimization problem is a combinatory problem, which is NP hard. As such, for a relatively large L, finding an optimal solution may take a relatively long (e.g., impractical) time.
- the neural field encoding block (214) is configured, in various examples, to use a greedy sub-optimal iterative algorithm. At the beginning of the algorithm, the frequency candidate list, ⁇ , is initialized with ⁇ ⁇ J different frequencies, where ⁇ ⁇ J > ⁇ .
- the selected frequency list, ⁇ is an empty set at the initialization. Then, in each iteration of the iterative algorithm, for each frequency in the set ⁇ , the algorithm operates to include the frequency into the frequency list and measure the corresponding distortion. For all frequencies in the set ⁇ , the algorithm picks the one, 4 ⁇ , which corresponds to an approximately smallest distortion. The value 4 ⁇ is placed into the selected frequency list ⁇ and is removed from the frequency candidate list ⁇ . This selection procedure is repeated until L frequencies are selected.
- the following pseudocode illustrates an example of the iterative frequency selection algorithm.
- the function D() is a function used to compute the distortion.
- the final frequency list ⁇ is transmitted to the decoder (300) in the metadata bitstream (296).
- the neural network compression (216) operates to compress the various neural-network parameters to generate the neural-network bitstream (298).
- different suitable compression methods can be used.
- the neural network compression (216) operates in accordance with the standard for Neural Network Compression and Representation (NNCR), or part 17 of the ISO/IEC 15938 standard, which is incorporated herein by reference in its entirety.
- the NNCR standard is promulgated by the ISO/IEC Moving Picture Experts Group (MPEG) and specifically targets efficient compression and transmission of neural networks, such as the MLP (100).
- MPEG Moving Picture Experts Group
- FIG. 3 is a block diagram illustrating the electronic decoder (300) according to an embodiment.
- the corresponding electronic encoder (200) has been described above.
- the decoder (300) generally performs inverse coding operations in the reverse order.
- the inputs to the decoder (300) are provided by the above-described bitstreams (294, 296, 298).
- the decoder (300) applies neural network decompression (316) to the neural-network bitstream (298) to obtain the various neural-network parameters.
- the obtained neural-network parameters are then used to reconstruct the MLP (100) in the trained configuration previously computed at the encoder (200).
- the decoder (300) uses the reconstructed MLP (100) and a pertinent portion of the metadata bitstream (296) in a neural field decoding block (314). This metadata portion is used for spatial reshaping such that the reshaped coordinate is then used to query the reconstructed MLP (100).
- a user-specified input (315) provides one or more parameters, such as time, luminance range, etc., based on which the reconstructed MLP (100) is operated in the testing mode to render a corresponding reshaped residual (312).
- the decoder (300) further operates to apply backward reshaping (310) to the reshaped residual (312) to generate a corresponding residual image (308).
- the backward reshaping (310) is configured to perform reshaping operations that are inverse to the reshaping operation of the forward reshaping (210).
- Summation (306) is used to combine the base image (204) received via the base-layer bitstream (294) and the residual image (308) to generate a corresponding output image (302).
- the image (302) can be one of the target (reference) images of the sequence (202) or a novel view generated using the reconstructed MLP (100).
- FIG.4 is a block diagram illustrating a part of motion-blur image processing using the encoder (200) according to an embodiment. More specifically, FIG.4 illustrates preprocessing (400) for generating the sequence (202) for the encoder (200).
- the preprocessing (400) uses a sequence of still images (402 0 , 402 1 , ..., 402 K ⁇ 1 ) to compute the sequence (202) of the target motion- blur (MB) images (2021, 2022, ..., 202K ⁇ 1).
- an MB image at the time instant ⁇ U is generated by taking an average of the still images (402 0 , 402 1 , ..., 402 K ⁇ 1 ) from time 0 to time ⁇ U as follows: where I ⁇ ? ⁇ ? is the corresponding MB image; and I : is the still image (402k).
- the MB image (2021) is the same as the first still image (4020), i.e., I ⁇ ?
- the first still image (402 0 ) is also designated as the base image (204).
- the encoder (200) further operates to compute the sequence (208) of residual images by subtracting (206) the base image (204) from each of the target MB images (202 1 , 202 2 , ..., 202 K ⁇ 1 ).
- the sequence (208) of the residual images so computed is then processed by the encoder (200) as described above in reference to FIG.2.
- the corresponding MLP (100) can be operated to predict a pixel value ( ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ ) based on the pixel coordinate ( ⁇ , G) and further based on the time ⁇ U.
- the time ⁇ U is used as the user-specified variable input (315) at the decoder (300).
- the value of ⁇ U can be integer or non- integer (e.g., fractional).
- the decoder (300) operates the MLP (100) to generate the reconstructed image (302) corresponding to one of the reference MB images (2021, 2022, ..., 202K ⁇ 1).
- the decoder (300) operates the MLP (100) to generate the reconstructed image (302) representing a corresponding novel MB image.
- the parameter ⁇ U represents the dynamic range.
- the sequence (202) comprises target images of the same scene with the dynamic ranges of 100, 162, 260, 413, 652, 1026, 1612, 2536, and 4000 nits, respectively.
- a descending order of dynamic ranges can similarly be used in the sequence (202).
- the sequence (208) of the residual images is computed using this sequence (202) and the base image (204) as described above in reference to FIG.2.
- the base image (204) is the target image having the dynamic range of 100 nits or the target image having the dynamic range of 4000 nits.
- the corresponding MLP (100) can be operated at the decoder (300) to compute images having different dynamic ranges based on the user- specified variable input (315).
- the user-specified input (315) can be selected to have any value of ⁇ U between 100 nits and 4000 nits.
- the decoder (300) operates the MLP (100) to generate the reconstructed image (302) corresponding to one of the reference images (202) used at the encoder (200).
- the decoder (300) operates the MLP (100) to generate the reconstructed image (302) having a corresponding novel dynamic range in accordance with the specified value of ⁇ U.
- Varying Age Face [0049]
- the parameter ⁇ U represents the age of the person whose face is in the target images (202).
- the sequence (202) comprises 23 images of the same person between the ages of 6 years old and 60 years old.
- the sequence (208) of the residual images is computed using this sequence (202) and the base image (204) as described above in reference to FIG. 2.
- the base image (204) is the target image corresponding to the age of 6 years old.
- the base image (204) is the target image corresponding to the age of 60 years old.
- the corresponding MLP (100) can be operated at the decoder (300) to compute images corresponding to different ages based on the user-specified variable input (315).
- the decoder (300) operates the MLP (100) to generate the reconstructed image (302) corresponding to one of the reference images (202) used at the encoder (200).
- FIG.5 is a block diagram illustrating a computing device (500) according to an embodiment.
- the device (500) can be used, e.g., to implement the encoder (200) or the decoder (300).
- the computing device (500) comprises input/output (I/O) devices (510), an image- processing engine (520), and a memory (530).
- the I/O devices (510) may be used to enable the device (500) to receive various input signals (502) and to output various output signals (504).
- the input signals (502) include the sequence (202), whereas the output signals (504) include the bitstreams (294, 296, 298).
- the input signals (502) include the bitstreams (294, 296, 298), whereas the output signals (504) include the reconstructed image(s) (302).
- the memory (530) may have buffers to receive image data and/or other pertinent data.
- the image data may be, e.g., in the form of one or more image files. Once the data are received, the memory (530) may provide parts of the data to the image-processing engine (520) for processing therein.
- the image-processing engine (520) includes a processor (522) and a memory (524).
- the memory (524) may store therein program code, which when executed by the processor (522) enables the image-processing engine (520) to perform image processing, including but not limited to the image processing in accordance with some or all of the above-described process flows (200, 300, 400).
- the program code may include, inter alia, the program code used to emulate the various neural networks, e.g., the MLP (100) described above.
- the image-processing engine (520) may perform rendering processing thereof and provide the corresponding viewable image(s) for being viewed on the display.
- the viewable image can be, e.g., in the form of a suitable image file outputted through the I/O devices (510).
- an apparatus for residual encoding comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: access a plurality of residual images or reshaped images, each of the residual images representing a difference between a respective reference image and a base image, different respective reference images corresponding to various respective values of one or more first parameters; train a neural-field network to represent the plurality of residual images or reshaped images, the neural-field network being responsive to a variable input of the one or more first parameters and being characterized by a set of second parameters, the training producing fixed values of the second parameters; and communicate the base image and
- an apparatus for residual decoding comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: access a base image and a set of fixed parameter values of a neural-field network trained to represent a plurality of residual images, the neural-field network being responsive to a variable input of one or more first parameters and being characterized by a set of second parameters, the fixed parameter values being values of the second parameters; test the neural-field network using a selected value of the variable input to generate an output residual image; and compute a reconstructed image by summing the output residual image and the base image.
- a method for residual encoding using a neural field comprising: accessing, via a processor, a plurality of residual images or reshaped images, each of the residual images representing a difference between a respective reference image and a base image, different respective reference images corresponding to various respective values of one or more first parameters; training, via the processor, a neural-field network to represent the plurality of residual images or reshaped images, the neural-field network being responsive to a variable input of the one or more first parameters and being characterized by a set of second parameters, the training producing fixed values of the second parameters; and communicating the base image and the fixed values of the second parameters to an electronic decoder.
- the one or more first parameters are selected from the group consisting of time, dynamic range, and age.
- the neural-field network comprises a multilayer perceptron responding to an input of a coordinate or other field-type variable; and wherein the set of second parameters comprises weights for connections between various neural- network nodes of the multilayer perceptron and biases applied to signals by individual neural- network nodes of the multilayer perceptron.
- the method further comprises: mapping, via the processor, a first number of inputs to a second number of inputs using a series of trigonometric functions, the second number being greater than the first number, the mapping being performed using a greedy algorithm; and applying, via the processor, the second number of inputs to the neural-field network.
- the method further comprises performing, via the processor, reshaping of the plurality of residual images to generate the first number of inputs, the reshaping being configured to reduce relative weight of higher-frequency signal components or to limit a size of the neural-field network.
- the method further comprises selecting, via the processor, a region of interest in the residual images, the region of interest being smaller than a full image frame.
- the reshaping includes at least one of spatial-domain reshaping, time-domain reshaping, and codeword-domain reshaping.
- the method further comprises communicating one or more parameters of the reshaping to the electronic decoder.
- communicating the fixed values of the second parameters to the electronic decoder comprises: generating a compressed data stream by applying, via the processor, compression to the fixed values of the second parameters; and transmitting the compressed data stream to the electronic decoder.
- the neural-field network is operable when the variable input of the one or more first parameters specifies a value that is different from any one of the various respective values of the one or more first parameters.
- a method for residual decoding using a neural field comprising: accessing, via a processor, a base image and a set of fixed parameter values of a neural-field network trained to represent a plurality of residual images, the neural-field network being responsive to a variable input of one or more first parameters and being characterized by a set of second parameters, the fixed parameter values being values of the second parameters; testing, via the processor, the neural-field network using a selected value of the variable input to generate an output residual image; and computing, via the processor, a reconstructed image by combining the output residual image and the base image.
- the one or more first parameters are selected from the group consisting of time, dynamic range, and age.
- the neural-field network comprises a multilayer perceptron; and wherein the set of second parameters comprises weights for connections between various neural-network nodes of the multilayer perceptron and biases applied to signals by individual neural-network nodes of the multilayer perceptron.
- the method further comprises: receiving one or more reshaping parameters from an electronic encoder for reshaping configured to increase relative weight of higher-frequency signal components; and applying the reshaping, via the processor, to an output of the neural-field network to generate the output residual image.
- the method further comprises selecting, via the processor, a region of interest for the output residual image, the region of interest being smaller than a full image frame.
- the reshaping includes at least one of spatial- domain reshaping, time-domain reshaping, and codeword-domain reshaping.
- accessing the set of fixed parameter values comprises: receiving a compressed data stream from an electronic encoder; and obtaining the fixed values of the second parameters by applying, via the processor, decompression to the compressed data stream.
- the neural-field network is operable when the variable input of the first parameter specifies a value that is different from any value of the first parameter using which the neural-field network has been trained.
- Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing various embodiments described herein.
- Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and/or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing various embodiments described herein.
- the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”
- the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.
- processors may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
- the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.
- processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and/or custom, may also be included.
- DSP digital signal processor
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- ROM read only memory
- RAM random access memory
- nonvolatile storage nonvolatile storage.
- Other hardware conventional and/or custom, may also be included.
- any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
- circuit may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”
- This definition of circuitry applies to all uses of this term in this application, including in any claims.
- circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware.
- circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202263419099P | 2022-10-25 | 2022-10-25 | |
| PCT/US2023/077678 WO2024091969A1 (en) | 2022-10-25 | 2023-10-24 | Residual encoding and decoding using a neural field |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4612903A1 true EP4612903A1 (en) | 2025-09-10 |
Family
ID=88975644
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP23814007.3A Pending EP4612903A1 (en) | 2022-10-25 | 2023-10-24 | Residual encoding and decoding using a neural field |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP4612903A1 (en) |
| CN (1) | CN120239969A (en) |
| WO (1) | WO2024091969A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP7150668B2 (en) * | 2019-06-21 | 2022-10-11 | 株式会社日立製作所 | HIGH-LEVEL SYNTHESIS DEVICE AND HIGH-LEVEL SYNTHESIS METHOD |
| US20250078397A1 (en) * | 2023-09-01 | 2025-03-06 | Google Llc | Prior for high-resolution image synthesis |
-
2023
- 2023-10-24 WO PCT/US2023/077678 patent/WO2024091969A1/en not_active Ceased
- 2023-10-24 EP EP23814007.3A patent/EP4612903A1/en active Pending
- 2023-10-24 CN CN202380080647.7A patent/CN120239969A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024091969A1 (en) | 2024-05-02 |
| CN120239969A (en) | 2025-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Akhtar et al. | PU-Dense: Sparse tensor-based point cloud geometry upsampling | |
| CN107730503B (en) | Image object component level semantic segmentation method and device embedded with three-dimensional features | |
| CN109961507B (en) | A face image generation method, device, device and storage medium | |
| JP2023549821A (en) | Deformable neural radiance field | |
| EP2850835B1 (en) | Estimation, encoding and decoding of motion information in multidimensional signals through motion zones, and of auxiliary information through auxiliary zones | |
| KR20210131358A (en) | Image processing to determine object thickness | |
| EP4612903A1 (en) | Residual encoding and decoding using a neural field | |
| JP2002506585A (en) | Method for sprite generation for object-based coding systems using masks and rounded averages | |
| Xie et al. | Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering | |
| WO2022208440A1 (en) | Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture | |
| KR20220031945A (en) | Device and method of training a generative neural network | |
| CN117083638A (en) | Accelerated Neural Radiation Fields for View Synthesis | |
| US20260073611A1 (en) | Generating a rendered image of a three-dimensional object | |
| CN117115047A (en) | An image enhancement method, device, equipment and storage medium | |
| Amarù et al. | Dalib: a curated repository of libraries for data augmentation in Computer Vision | |
| Zhu et al. | Dynamic scene reconstruction: Recent advance in real-time rendering and streaming | |
| CN121053335A (en) | 3D Reconstruction Method and Electronic Equipment Based on Pulse Camera | |
| Hou | Permuted sparse representation for 3D point clouds | |
| Ruan et al. | Implicit neural compression of point clouds | |
| Yang et al. | UVS-CNNs: Constructing general convolutional neural networks on quasi-uniform spherical images | |
| Nan et al. | Multi-modal Dynamic Point Cloud Geometric Compression Based on Bidirectional Recurrent Scene Flow | |
| Mamou et al. | Multi-chart geometry video: A compact representation for 3D animations | |
| CN113170094B (en) | Video decoding method, computer device, computer readable medium, and video decoder | |
| Zhang et al. | ParaNet: Deep regular representation for 3D point clouds | |
| CN120876693B (en) | A Method and System for Generating 3D Scene Styles Based on Weighted Style Neural Radiation Fields |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20250509 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| P01 | Opt-out of the competence of the unified patent court (upc) registered |
Free format text: CASE NUMBER: UPC_APP_0012842_4612903/2025 Effective date: 20251111 |
|
| DAV | Request for validation of the european patent (deleted) | ||
| DAX | Request for extension of the european patent (deleted) |