WO2025210218A1 - Method and data processing system for lossy image or video encoding, transmission and decoding - Google Patents

Method and data processing system for lossy image or video encoding, transmission and decoding

Info

Publication number
WO2025210218A1
WO2025210218A1 PCT/EP2025/059249 EP2025059249W WO2025210218A1 WO 2025210218 A1 WO2025210218 A1 WO 2025210218A1 EP 2025059249 W EP2025059249 W EP 2025059249W WO 2025210218 A1 WO2025210218 A1 WO 2025210218A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
sequence
images
neural network
input images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/EP2025/059249
Other languages
French (fr)
Inventor
Bilal ABBASI
Christopher FINLAY
Arsalan ZAFAR
Lucas BENNETT
Evan CATON
Aaron BERK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Deep Render Ltd
Original Assignee
Deep Render Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from GBGB2415182.1A external-priority patent/GB202415182D0/en
Application filed by Deep Render Ltd filed Critical Deep Render Ltd
Publication of WO2025210218A1 publication Critical patent/WO2025210218A1/en
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H04N19/23—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning

Definitions

  • This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding.
  • the compression of image and video content can be lossless or lossy compression.
  • lossless compression the image or video is compressed such that all of the original information in the content can be recovered on decompression.
  • lossless compression there is a limit to the reduction in data quantity that can be achieved.
  • lossy compression some information is lost from the image or video during the compression process.
  • Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system.
  • JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and/or video files.
  • known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression.
  • inter-frame redundancy One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames.
  • I-frames or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, I-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video.
  • P-frames, or predictive frames utilize temporal redundancy in video sequences to achieve greater compression.
  • a P-frame represents the difference between itself and the closest preceding I- or P-frame.
  • the process known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality.
  • B-frames or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process.
  • B-frames By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames.
  • Artificial intelligence (AI) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content.
  • AI based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted.
  • An example of an AI based image compression process comprising a hyper-network is described in Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXiv preprint arXiv:1802.01436 (2016), which is hereby incorporated by reference.
  • An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression.
  • a method of training one or more neural networks comprising the steps of: receiving a first sequence of input images at a first computer system; producing a masked sequence of input images by masking a portion of one or more of the input images; with a first neural network, for a pair of input images of the masked sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; updating the parameters of the first neural network and the second neural network based
  • the masking comprises selecting a portion of a first image of the first sequence, and replacing pixels in the other images of the first sequence using the selected portion.
  • an unselected portion of the first image of the first sequence remains unmasked in the first sequence of input images.
  • a position and size of the portion for first sequence is different to the position and size of the position for one or more of the the further sequences.
  • the position and size of the portions for the first sequence and the further sequences are distributed according to a predetermined distribution.
  • the distribution is a uniform distribution.
  • the distribution is a Gaussian distribution with a mean at a center coordinate of the input images.
  • the distribution is based on a type of the input images.
  • the type comprises at least one of: webcam video frames, static scene frames, video game frames, high motion video frames, low motion video frames.
  • the function is based on a difference between the first sequence of output images and the masked sequence of input images.
  • a method of performing lossy image or video encoding, transmission and decoding comprising the steps of: r eceiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images;
  • a method of performing lossy image or video encoding and transmission comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods.
  • a method of performing lossy image or video decoding comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to any of the above described methods.
  • a data processing apparatus configured to perform any of the above methods.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the above methods.
  • a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of the above methods.
  • a method of performing lossy image or video decoding comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to any of the above methods.
  • a data processing apparatus configured to perform any of the above methods.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
  • a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods.
  • a method of training one or more neural networks the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; modifying the pixels of the first image and the second image at respective first and second coordinates to introduce a pixel representation of an object into the first image and the second image; with a first neural network, producing a latent representation of a difference between the first image and the second image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; u pdating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks.
  • a method of performing lossy image or video encoding, transmission and decoding comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a method of performing lossy image or video encoding and transmission comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods
  • a method of performing lossy image or video decoding comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a data processing apparatus configured to perform any of the above methods.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
  • a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods.
  • a method of training one or more neural networks comprising the steps of: receiving a first image and a plurality of second images at a first computer system; selecting one of the plurality of second images as a reference image; with a first neural network, producing a latent representation of a difference between the first image and the reference image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; u pdating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks.
  • said selecting comprises sampling an image from the plurality of second images.
  • said selecting comprises selecting one of the plurality of second images from the image sequence based on one or more predetermined rules.
  • the first image and the plurality of second images comprise consecutive images of an image sequence.
  • the selected one of the plurality of second images is a positioned in the sequence before the first image.
  • the selected one of the plurality of second images is a positioned in the sequence after the first image .
  • the method comprises biasing said selecting towards a beginning of the image sequence.
  • the method comprises biasing said selecting towards an end of the image sequence.
  • said biasing is based on a property of the first image and/or the plurality of second images.
  • said property is a scene type of the first image and/or the plurality of second images.
  • the one or more sequences of images comprise a predetermined balance of image sequences having optical flows above a threshold and image sequences having optical flows below the threshold, and wherein said selecting is based on the threshold.
  • a method of performing lossy image or video encoding, transmission and decoding comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a method of performing lossy image or video encoding and transmission comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods.
  • a method of performing lossy image or video decoding comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a data processing apparatus configured to perform any of the above methods.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
  • producing the combined optical flow information comprises producing a parame- terised combination of the first optical flow information and the second optical flow information using one or more combination parameters.
  • the method comprises updating the combination parameters based on one or more terms of the evaluated function.
  • the parameterised combination comprises a weighted sum of the first optical flow information and the second optical flow information and the combination parameters comprise one or more weights of the weighted sum.
  • the parameterised combination comprises a pixel-wise parameterised combination of the first optical flow information and the second optical flow information.
  • the function is further based on a rate term and a distortion term.
  • the difference between the combined optical flow information and the third optical flow information comprises a mean squared error.
  • the method comprises estimating the first optical flow information using a first optical flow algorithm, and estimating the second optical flow information using a second optical flow algorithm.
  • the first optical flow algorithm and the second optical flow algorithm comprise an ensemble of optical flow algorithms.
  • the method comprises estimating further optical flow information using a further optical flow algorithm, and producing the combined optical flow information from the first optical flow information, the second optical flow information and the further optical flow information.
  • the method comprises estimating the difference between the combined optical flow information and the third optical flow information using a discriminator function.
  • a method of performing lossy image or video encoding, transmission and decoding comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a method of performing lossy image or video encoding and transmission comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods.
  • a method of performing lossy image or video decoding comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods.
  • a data processing apparatus configured to perform any of the above methods.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
  • Figure 1 illustrates an example of an image or video compression, transmission and decom- pression pipeline.
  • Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network.
  • Figure 3 illustrates an example of a video compression, transmission and decompression pipeline.
  • Figure 4 illustrates an example of a video compression, transmission and decompression system.
  • Figure 5a illustrates an example plot of a distortion score of a frame sequence against frame number.
  • Figure 5b illustrates an example plot of a distortion score of of a frame sequence against frame number.
  • Figure 6 illustratively shows steps of a method of modifying training data of a video compression, transmission and decompression pipeline.
  • Figure 7a illustrates an example plot of a distortion score of a frame sequence against frame number.
  • Figure 7b illustrates an example plot of a distortion score of a frame sequence against frame number.
  • Figure 8 illustrates a high motion frame sequence.
  • Figure 9 illustrates a training data augmentation method for use with a video compression, transmission, and decompression method.
  • Figure 10 illustrates a training data augmentation method for use with a video compression, transmission, and decompression method.
  • Figure 11 illustrates method for use with training a video compression, transmission, and decompression method.
  • Figure 12 illustrates method for use with training a video compression, transmission, and decompression method.
  • DETAILED DESCRIPTION OF THE DRAWINGS Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information.
  • Image and video information is an example of information that may be compressed.
  • the file size required to store the information, particularly during a compression process when referring to the compressed file may be referred to as the rate.
  • compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process.
  • Image and video files containing image and video data are common targets for compression.
  • the input image may be represented as ⁇ .
  • the data representing the image may be stored in a tensor of dimensions ⁇ ⁇ ⁇ ⁇ ⁇ , where ⁇ represents the height of the image, ⁇ represents the width of the image and ⁇ represents the number of channels of the image.
  • Each ⁇ ⁇ ⁇ data point of the image represents a pixel value of the image at the corresponding location.
  • Each channel ⁇ of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device.
  • an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively.
  • the image information is stored in the RGB colour space, which may also be referred to as a model or a format.
  • Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.
  • the channels of an image file are not limited to storing colour information and other information may be represented in the channels.
  • a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video.
  • Each image making up a video may be referred to as a frame of the video.
  • the output image may differ from the input image and may be represented by ⁇ .
  • the difference between the input image and the output image may be referred to as distortion or a difference in image quality.
  • the distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way.
  • An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art.
  • the distortion function may comprise a trained neural network.
  • the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner.
  • a relation between these quantities for a given compression technique may be defined by a rate-distortion equation.
  • AI based compression processes may involve the use of neural networks.
  • a neural network is an operation that can be performed on an input to produce an output.
  • a neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network.
  • Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer.
  • Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer.
  • a node may receive an input from one or more nodes of the previous layer.
  • the one or more operations may include a convolution, a weight, a bias and an activation function.
  • Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer.
  • Each of the one or more operations is defined by one or more parameters that are associated with each operation.
  • the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer.
  • each of the values in the weight matrix is a parameter of the neural network.
  • the convolution may be defined by a convolution matrix, also known as a kernel.
  • one or more of the values in the convolution matrix may be a parameter of the neural network.
  • the activation function may also be defined by values which may be parameters of the neural network.
  • the parameters of the network may be varied during training of the network. Other features of the neural network may be predetermined and therefore not varied during training of the network.
  • the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network.
  • a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known.
  • the initial parameters of the neural network are randomized and the first training input is provided to the network.
  • the output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced.
  • an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function.
  • This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features.
  • training or “learning” as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model’s predictions and the actual data, represented by the above-described rate-distortion loss function.
  • the training process may comprise multiple epochs.
  • An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm.
  • the model’s parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data.
  • the training data may be divided into smaller subsets known as batches.
  • the size of a batch referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model’s parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources.
  • a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution.
  • the learnable parameters are updated by a specified amount each time, determined by the learning rate.
  • the learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process.
  • a smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution.
  • a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge.
  • a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches.
  • a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system.
  • a straight-through estimator (STE) quantisation model or one other quantisation models may be used.
  • different quantisation models may be used for during evaluation of different term of the loss function.
  • noise quantisation may be used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term.
  • end-to-end learning of the quantisation process achieves a similar effect.
  • learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes. Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off.
  • the system described above may be distributed across multiple locations and/or devices.
  • the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server.
  • the decoder 120 may be located on a separate device which may be referred to as a recipient device.
  • the system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline.
  • the AI based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process.
  • the hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder ⁇ h ⁇ and a trained neural network 125 acting as a hyper-decoder ⁇ h ⁇ .
  • An example of such a is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above.
  • the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation.
  • the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation.
  • the mean standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation.
  • the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150.
  • Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the AI based compression process 100.
  • At least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed.
  • the residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent.
  • the residual values may also be normalised.
  • a training set of input images may be used as described above.
  • the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step.
  • the training process may further include a generative adversarial network (GAN).
  • GAN generative adversarial network
  • an additional neutral network acting as a discriminator is included in the system.
  • the discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake.
  • the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input.
  • a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake.
  • the output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process.
  • the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process.
  • Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously.
  • the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6. Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination.
  • the frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process.
  • frames which are decoded using no information from other frames may be referred to as I-frames.
  • Frames which are decoded using information from past frames may be referred to as P-frames.
  • Frames which are decoded using information from past frames and future frames may be referred to as B-frames.
  • Frames may not be encoded and/or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time.
  • the images represented by each frame of a video may be related.
  • a number of frames in a video may show the same scene.
  • a number of different parts of the scene may be shown in more than one of the frames.
  • objects or people in a scene may be shown in more than one of the frames.
  • the background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame.
  • the transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation.
  • the flow may be represented by a vector.
  • One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map.
  • An example AI based video compression, transmission, and decompression process 200 is shown in Figure 3.
  • the process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may be also be considered as a single, end-to-end pipeline.
  • I-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2.
  • an input image ⁇ 0 is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204.
  • the bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image ⁇ 0 which in this case is an I-frame.
  • the decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image ⁇ 0 to be available for later use by components of both the encoding and decoding sides of the pipeline.
  • P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image ⁇ that is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images ⁇ 1 from an earlier frame t-1.
  • the previously reconstructed ⁇ 1 is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides.
  • previously reconstructed images may be used for generating a flow maps containing information indicative of inter-frame movement of pixels between frames.
  • both the image being compressed ⁇ and the previously reconstructed image from an earlier frame ⁇ 1 are passed into a flow module part 206 of the pipeline.
  • the flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs ⁇ 1 and ⁇ , which is indicative of inter-frame movement of pixels or pixel groups between ⁇ 1 and ⁇ .
  • the latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.
  • the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map ⁇ .
  • the reconstructed flow map ⁇ is applied to the previously reconstructed image ⁇ 1 to generate a warped image ⁇ 1, ⁇ .
  • any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used.
  • the warped image ⁇ 1, ⁇ is a prediction of how the previously reconstructed image ⁇ 1 might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of ⁇ and ⁇ 1.
  • the reconstructed flow map ⁇ and corresponding warped image ⁇ 1, ⁇ may be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides.
  • both the image being compressed ⁇ and the ⁇ 1, ⁇ are passed into a residual module part 210 of the pipeline.
  • the residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage ⁇ and the warped image ⁇ 1, ⁇ .
  • the latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted.
  • the bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map ⁇ from the decoded latent representation.
  • a residual map may first be pre-calculated between ⁇ and the ⁇ 1, ⁇ and the pre-calculated residual map may be passed into an autoencoder for compression only.
  • the residual map ⁇ is applied (e.g. combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image ⁇ which is a reconstruction of image ⁇ and accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image ⁇ can then be used to process the next frame. That is, it can be used to compress, transmit and decompress ⁇ +1, and so on until an entire video stream or chunk of a video stream has been processed.
  • the residual autoencoder may be trained to reconstruct the frame ⁇ directly from the entropy decoded bitstream by removing the connection between ⁇ 1, ⁇ and the output of the residual block 210, thereby eliminating any direct combination step with the warped previously decoded image to speed up inference.
  • the flow information is intuitively understood to be indirectly captured within the residual information, which the residual decoder is able to learn to use to directly reconstruct the output image ⁇ .
  • the residual autoencoder may be trained to reconstruct the frame ⁇ directly from the entropy decoded bitstream in combination with some representation of flow injected into one or more layers of the residual decoder.
  • the bitstream may contain (i) a quantised, entropy encoded latent representation of the I-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image.
  • any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2.
  • the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable.
  • the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6680-6689).
  • the above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g.
  • FIG. 4 shows an example of an AI image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400.
  • the system 400 comprises a first device 401 and a second device 402.
  • the first and second devices 401, 402 may be user devices such as smartphones, tablets, AR/VR headsets or other portable devices.
  • the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively.
  • the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU.
  • the first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 404.
  • the video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and/or B-frames.
  • the blocks of frames are then compressed by an AI-compression module 407 comprising the encode side of the AI-based video compression pipeline of Figure 3.
  • the output of the AI-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications.
  • the second device 402 receives the communicated bitstream 408b which is passed to an AI-decompression module 410 comprising the decode side of the AI-based video compression pipeline of Figure 3.
  • the output of the AI-decompression module 402 is the reconstructed I-frames, P-frames and/or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402.
  • a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402.
  • the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider AI-compression literature.
  • execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods.
  • Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time.
  • efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations.
  • Figure 5a is a plot 500 of reconstruction accuracy (quantified in mean squared error, MSE, between the input frame and the output frame) against frame number of a sequence of one I- frame (frame index 1) and 149 P-frames (frame index 2 to 150) produced by an AI-compression pipeline trained on training data where frozen-patch long GOP training was not applied. It can be seen from the gradually increasing MSE as the frames progress that the image reconstruction accuracy is decreasing, indicating that unrolling artefacts start to dominate the further away the last I-frame was.
  • MSE mean squared error
  • Figure 6 illustratively shows steps 600 of a method of modifying training data using the above-described frozen-patch long GOP training.
  • a GOP 601 is provided. Whilst only a small number of frames are shown for brevity, in practice a full GOP of some predetermined length is provided. Each of the frames is part of a sequence of a scene in which a character is moving his arms up and down and then walking off.
  • a patch 602 of the first frame of the GOP, for example an I-frame 603, is selected by pixel coordinates and the pixels of the first frame outside of those coordinates are copied as a mask 604.
  • the mask 604, made up of the non-patch first frame pixels is are applied 605 across all the other frames of the GOP 601, leaving only those patches at the non-mask pixel coordinates of the other frames untouched.
  • the mask frame remains unchanged, containing the original I-frame pixels that include the characters arm, body and part of the head. Effectively, the I-frame is frozen and applied to the whole GOP except for the pixels within the selected patch.
  • the modified GOP is then stored as a sequence of frames of training data, and the same process is repeated for another GOP, and so on to produce the full training data set.
  • patch sizes and positions may be chosen randomly according to some distribution, for example a uniform distribution, or Gaussian distribution or some other distribution. It has been found that a uniform distribution of different patch sizes and positions produces a training data set that, when trained on, is most robust to different GOP lengths (both long and short) during inference.
  • patch sizes and positions may be selected based on some property of the data the method is being applied to (e.g. selecting a patch size of around 128x128 pixels or 256x256 pixels or more pixels and positioned around the center pixels of webcam video data around where the person in the webcam video data is positioned).
  • Envisaged data types where the patch size and position may be predetermined include but are not limited to webcam video data, static scene data, video game data, high motion video data, low motion CCTV data.
  • the patch size and position is typically based around typical motion or non-motion patterns and where detail is to be preserved as much as possible, such as at or around edges, at or around typical movement paths and patterns, and so on.
  • the distribution of patch sizes and positions can be used to control the performance of the model trained on it for different needs. For example, if a given model is only intended to be used on long GOP sequences where the center of the frame is expected to contain detail and movement (e.g.
  • the distribution of patch sizes and positions can be focused around the center of the frames when producing the frozen-patch long GOP training data (e.g. a Gaussian distribution with a mean at the center pixel of the frames).
  • the frozen-patch long GOP training approach can advantageously be used as a way to pursue the networks of an AI-based compression pipeline for a given purpose, such as compressing and reconstructing webcam video conference frames.
  • the frozen-patch approach can also be applied to short GOP sequences (e.g. fewer than 10 frames).
  • Neural Network Training frozen-patch GOPs Require: Training dataset D, learning rate ⁇ , regularization parameter ⁇ , number of epochs ⁇ Select patches and frozen pixels of GOPs of D Generate frozen-patch training data D ⁇ using selected patches and frozen pixels
  • a further technique that can be used, either together with the above-described frozen-patch GOP training, or in a standalone manner is long GOP training with truncated back propagation in time, as will be described below in concept 2.
  • TBPTT Truncated Backpropagation Through Time
  • TBPTT techniques comprise the following steps: 1. Segment the Input Sequence: Divide the input sequence into overlapping smaller chunks of a fixed length, ⁇ . 2. Forward Pass: For each chunk, perform a forward pass through the neural network for ⁇ time steps, computing the hidden states and outputs. 3. Backward Pass: Backpropagate the error through only the last ⁇ time steps, instead of the entire sequence. 4. Update Parameters: Update the network parameters based on the gradients computed in the backward pass. 5.
  • Step 1 Divide the sequence into overlapping chunks: [ ⁇ 1, ⁇ 2, ⁇ 3, ⁇ 4], [ ⁇ 2, ⁇ 3, ⁇ 4, ⁇ 5], [ ⁇ 3, ⁇ 4, ⁇ 5, ⁇ 6], ...,[ ⁇ 9, ⁇ 10, ⁇ 11, ⁇ 12].
  • the training dataset, D is divided into batches. These batches are processed one at a time. The division into batches helps manage memory efficiently and can speed up the training process.
  • Each batch from the dataset is further divided into non-overlapping chunks ⁇ that together cover a number of time steps ⁇ , each containing a plurality of image pairs ⁇ and ⁇ 1, for example images from a first point in time and a second point in time across the total number of time steps ⁇ , whereby when compressing and reconstructing one of those images during the forward pass, the temporal redundancy between the two images is used (e.g. by applying a flow-residual approach as described above in the introductory section).
  • This update may be performed using an optimizer, O, which adjusts the parameters, ⁇ , based on the accumulated gradients, the learning rate, ⁇ , and optionally other factors, for example regularization term.
  • O optimizer
  • validation may be performed using a validation dataset.
  • the window size need not be static across a batch. Instead it can vary across a sequence of frames of a GOP.
  • the forward pass is performed for ⁇ h ⁇ 1, the distortion scores (e.g. MSE) are calculated for each of the frames of ⁇ h ⁇ 1. If the distortion scores are indicative of the appearance of artefacts (such as unrolling artefacts) in the reconstructed frames, the chunk size for the next chunk, ⁇ h ⁇ 2, can be reduced or kept the same.
  • the chunk size can be increased for the next chunk.
  • Checking distortion scores for signs of artefacts may be performed by applying a threshold to the distortion score. For example, if the distortion is above the threshold it is indicative that the reconstructed image is too different to the ground truth image and thus that artefacts are present. Conversely, if they are at or below the threshold, the reconstructed image is sufficiently similar to the ground truth image and no or only minor artefacts are present. More complex methods of detecting artefacts may additionally or alternatively be applied. For example, by comparing image statistics of the reconstructed image with the ground truth image, by applying a classifier, and so on.
  • ⁇ h ⁇ 2 [ ⁇ 4, ..., ⁇ 9]
  • the forward pass is then performed for ⁇ h ⁇ 2, the distortion scores are calculated for each of the frames of ⁇ h ⁇ 2 and checked for signs of artefacts. If there are no signs of artefacts, the chunk size for ⁇ h ⁇ 3 may be increased again. For the sake of illustration, we increase it to e.g. 27.
  • predetermined points or points determined dynamically we take some reference frame ⁇ , for example an I-frame or a frame at the start of a chunk, we measure the distortion loss (e.g. MSE) at that point, and then when we get to frame ⁇ + ⁇ we measure the distortion loss again and compare it to the previously measured distortion for the reference frame ⁇ . If the new distortion is some amount greater than the previously measured distortion (e.g. some percentage, for example 1-25%, such as 5%, 10%, or 20%), then it is indicative that the distortion is going up too fast and accordingly that we should cut the chunk short there and proceed to backpropagate, rather than waiting until the end of all the frames in that chunk.
  • some reference frame ⁇ for example an I-frame or a frame at the start of a chunk
  • the distortion loss e.g. MSE
  • this approach effectively immediately cuts short any chunks when we see the distortion increasing too much, and adapts the next chunk size accordingly based on where the current chunk was cut short, rather than waiting for the end of the chunk to adapt the size of the next chunk. That is, by measuring a relative distortion of frames of the chunk compared to a reference frame, it is possible to track changes in the distortion at an arbitrary level of granularity (e.g. frame to frame, every 5 frames, every 10 frames, and so on), thus allowing for the dynamic updating of chunk sizes during training.
  • the above approaches approaches also open up training (and finetuning) on substantially longer GOP size data than has been possible up to now and on long GOP data of all types (e.g.
  • the adaptive window facilitates the automatic finding optimal chunk sizes for truncated back propogation through time, unique for each GOP and data type - something which is burdensome using handcrafted chunk sizes or static chunk sizes across an entire GOP.
  • the presently described truncated backpropagation through time approaches may be used with concept 1, whereby the frozen patch approach may be applied to a GOP before that GOP is chunked into non-overlapping chunks as described above.
  • both of the above approaches may be jointly or individually combined with the concept described below.
  • the networks of the AI-compression pipeline to learn the easier task of short GOP compression first without the issues of long GOP training, resulting in the overall loss moving roughly in the direction of a global minimum. Then, after base training is complete, the finetuning on long GOP training data allows the already-performant networks to learn the more difficult task of long GOP compression, which moves the loss even closer to the global minimum for long GOP compression.
  • the term short GOP may be understood to be GOP lengths of below substanially 10 frames, while 10 frames or above may be understood as a long GOP.
  • the network For each epoch, up to a predetermined number of epochs ⁇ , the network processes batches of data from D ⁇ . In each batch, the network performs a forward pass to compute predictions based on its current parameters. It then calculates a loss, ⁇ . A backward pass is performed to compute gradients of the loss with respect to the parameters. The parameters are updated using an optimization algorithm, O ⁇ , which applies these adjustments in a direction expected to reduce the loss, using the learning rate ⁇ as a step size. After each epoch, the network can be optionally evaluated on a validation set to monitor its performance on unseen data.
  • the networks of the compression pipeline ⁇ are able to compress short GOP sequences well but will produce errors and artefacts, such as unrolling, on long GOP sequences.
  • the network undergoes fine-tuning on long GOP data.
  • This phase adjusts the network to perform well on a specific long GOP compression task using long GOP training data, D ⁇ ⁇ .
  • the fine-tuning process also proceeds in epochs, for a number of epochs ⁇ ⁇ ⁇ .
  • the network processes batches from D ⁇ ⁇ in a manner similar to the base training phase, performing forward passes, computing a fine-tuning loss ⁇ ⁇ ⁇ , and then performing backward passes to adjust the parameters.
  • FIG. 7a illustrates a plot 700 of distortion (MSE) score of compressed and reconstructed frames compared to corresponding ground truth frames of a sequence that has been compressed and reconstructed by an AI-based compression pipeline that has been base-trained on short GOP data.
  • MSE distortion
  • the figure shows a long GOP data series 701 made up of a single sequence of the same 150 frames as a single GOP. It is apparent that the distortion increases as the sequence progresses on the long GOP series 701 after the initial I frame.
  • I-frames cannot take advantage of temporal redundancies so use up more bits than P-frames to compress resulting in the initial MSE spike at frame 0 (i.e. the I frame). It can be seen that from the initial MSE spike at frame 0 of around 40, the MSE hits 40 again around frame 50 and continues to go extend higher and higher reaching MSE scores of around 55-60 at frame 150.
  • Figure 7b illustrates the same plot 700 but now after finetuning on long GOP data.
  • the long GOP data series 701 in Figure 7b has the same initial MSE spike to around 40 for the I frame before dropping as the P-frames progress. However, this time the MSE scores only hit 40 again by frame 150 (contrasting to Figure 7a where the MSE scores already hit 40 by 50 frame).
  • Figure 7b illustrates that finetuning on long GOP data after base training on short GOP data produces networks of an AI-compression pipeline that achieve better bit rates while maintaining low distortion across longer GOP data during inference.
  • the finetuning of the networks of the pipeline associated with Figure 7b may be performed using GOPs of whatever GOP length the finetuned networks are intended to used on.
  • the inference toy example of Figure 7b is performed at GOP length 150, accordingly it is envisaged that the finetune data comprises GOP lengths of 150.
  • a balanced training data sets i.g. a data set that is not specifically biased towards a higher proportion of high motion scenes
  • modifying some or all of the scenes to include a synthetically introduced box or patch that moves across the scene for example gradually at some predetermined speed in some predetermined direction or path, or for example appearing and disappearing at different places in the frames
  • a synthetically introduced box or patch that moves across the scene, for example gradually at some predetermined speed in some predetermined direction or path, or for example appearing and disappearing at different places in the frames
  • box is used for ease of understanding but it will be appreciated that any shape of any size may be used.
  • the underlying training data set has few high motion image sequences (for example, optical flow values across the data set are in aggregate below some threshold), then it may be advantageous to increase the movement speed of the "flying" boxes, and vice versa.
  • the neural networks of an AI-based compression pipeline are then trained on such a "flying" box enhanced training data set, the inventors have found that the trained network has improved rate distortion scores on high motion scenes (compared to training on a non-enhanced data set) without significantly negatively impacting rate distortion scores on low motion scenes. This can be contrasted with simply biasing a training data set with more natively high motion scenes which the inventors have found in many cases catastrophically reduces performance on low motion scenes.
  • the "flying" box enhancement may be introduced into some or all of the image sequences of a training data set. For example, it is envisaged that it may be introduced only on scenes where some motion requirement is considered to be met. For example, where motion is below a threshold motion amount based on optical flow or some other pixel difference metric. Or it may be introduced based on some other requirement (e.g. predetermined scene type such as screen capture content, action content, or some other content), or indeed it may be introduced on in all scenes in the training data set.
  • some motion requirement e.g. predetermined scene type such as screen capture content, action content, or some other content
  • Figure 9 illustrates an image sequence 900 that may be used in a training data set showing a static nature scene into which a "flying" box has been introduced. It will be appreciated that Figures 8 and 9, for illustrative purposes, show only three frames of the respective sequences but that in practice each image sequence may GOP sizes of any number and the "flying" boxes (or indeed other shapes) may be introduced onto some or all of the frames of a given GOP.
  • An illustrative method of adding "flying" boxes, in this case black pixel rectangles, to a sequence of images is set out in the pseudocode below:
  • Algorithm 5 Enhance training image sequence with flying box Inputs: s equence, rectangle_width, rectangle_height, start_x, start_y, speed_x, speed_y
  • a further optional enhancement of the above methods is the introduction of a non-uniform texture or pattern into the pixels of the flying box.
  • the inventors have unexpectedly found that introducing image features that AI-based compression pipelines in many cases have difficulty with into the pixels of the flying boxes results in the same effect of improving the rate distortion scores of the pipeline on frames with those features importantly without negatively impacting rate distortion scores associated with other types of scene features.
  • Figure 10 illustrates an example image sequence 1000 onto which a flying box 1001 containing alpha numeric characters has been introduced onto a scene, in this case a high motion scene of a rocket during lift off as it moves across the scene.
  • a flying box 1001 containing alpha numeric characters has been introduced onto a scene, in this case a high motion scene of a rocket during lift off as it moves across the scene.
  • the pixel distribution of the majority of the scene remains consistent with the natively captured footage of the scene while super imposing the "flying" box with text onto the scene introduces an easy-to-learn high motion feature, as well as an easy to learn alpha-numeric feature.
  • Training an AI-based compression pipeline on such data results in a set of networks that have improved rate distortion scores on high motion scenes, low motion scenes, and scenes that include alphanumeric text compared to networks trained on the same data set but without such enhancements.
  • the alpha-numeric character example has been found to be particularly effective at improving the rate distortion performance of AI-based compression pipelines on screen capture image sequences such as screen sharing by participants in a video call, and such as video game streamers who may quickly switch between browser screens and have various overlays on their streams, many of which contain text, without significantly reducing the rate distortion performance of the pipeline on other types of content such as movies and so on.
  • the inventors have found that applying the "flying" box methodology to training data enhancement may be seen as a method of blending specific training data types (e.g. certain pixel distributions associated with certain scene and content types, and or motions associated therewith) into an underlying, balanced training dataset without negatively effecting the balance of that underlying training dataset.
  • the networks trained thereon retain the good across-the-board performance of the balanced training data while boosting the performance on the specific data types blended into it using the "flying" boxes.
  • this effect is particularly advantageous because it allows a single model (i.e. the weights and biases and so on of the neural networks of the pipeline) to be performant across a much higher number of different use cases, thereby reducing the memory requirements of the pipeline when it is deployed as a codec on edge devices.
  • This is a significantly different approach to the paradigm of using multiple different models (i.e. sets of weights) and selecting that model which is finetuned specifically for good performance on a given image sequence type.
  • the texture of the "flying" box may be sourced from other image sequences in the underlying training data set. That is, rather than synthetically creating the pixel values of the "flying" box, they may instead be sampled, for example randomly, from image patches of the training data set into which the "flying" boxes are being introduced. That is, for a given image sequence A, a patch of pixels from one or more frames of image sequence B may be sampled and used as the texture of the "flying" box added to image sequence A.
  • This approach increases the variance of the pixel distribution of the "flying" box, which can help to prevent overfitting that may occur when solid colour block "flying" boxes are used.
  • the motion of the "flying" box across the frames of the sequence may be different between sequences. That is, it is envisaged that "flying" boxes together having a distribution of different motion amounts, directions and speeds may be introduced into the sequences to increase the variance of the pixel distributions attributed to the "flying" boxes. As above, this can help to prevent overfitting that may occur when all "flying" boxes across all sequences have the same or similar motion.
  • the network architecture of ⁇ is defined, for example as shown in Figure 3.
  • the network parameters ⁇ are randomly initialised and then the training loop is started.
  • flying boxes are introduced as described above, and the forward pass is computed for the network being trained ⁇ .
  • the total ⁇ is calculated by combining a distortion term ⁇ and a rate term ⁇ , and any other loss terms (not shown).
  • the backwards pass is then performed to compute gradients based on the loss, and the parameters ⁇ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser.
  • a validation loss can be calculated and the training loop is repeated until the predetermined number of steps or epochs ⁇ have been calculated, or some other criteria have been reached.
  • the optical flow is not necessarily always estimated between ⁇ and ⁇ 1, but instead, for each current frame ⁇ , the optical flow may instead be estimated by comparing ⁇ with ⁇ where ⁇ may be sampled from ⁇ ⁇ [1, 2, 3, ...].
  • the effect of this approach is that the networks of the pipeline learn better how to predict larger optical flows as the pixel differences between a frame at ⁇ and for example ⁇ ⁇ 5 are typically greater than those between a frame at ⁇ and ⁇ ⁇ 1.
  • the value of ⁇ may be randomly sampled from Z using any suitable method, for example using a random or pseudo-random number generator, where such sampling may be drawn from a parameterised distribution to allow for some control over the value of ⁇ , and/or ⁇ may be sampled according to some predetermined heuristics (e.g. one or more predetermined rules). For example, certain scene types may result in lower loss values when ⁇ is biased to lower values compared to higher values, and so on and accordingly the sampling of ⁇ may take this into account by being biased towards lower ⁇ values (for example by parameterising a distribution of values from which ⁇ may be selected using a mean, variance and/or other parameters), or vice versa.
  • a predetermined heuristics e.g. one or more predetermined rules
  • Example scene types include high motion action scenes of movies, low motion nature scenes from documentaries, and so on.
  • the result is that there is a distribution of different frame positions from which the reference frame is selected, and thus a greater distribution of motion amounts in the training data while otherwise leaving the pixel distributions of the underlying data unchanged.
  • the rate and distortion losses after training converge to lower values than that pipeline is able to achieve without using the methods of concept 5. More generally, the higher the frame gap (i.e. where ⁇ is large), the higher the pixel differences between ⁇ and ⁇ are and thus the higher the motion that the network has to try to learn during training to minimise the rate distortion for.
  • this approach has the effect of synthetically introducing motion into the training data set without otherwise changing the underlying pixel distributions of the training data. That is, it facilitates the learning of high motion without otherwise biasing an already balanced training data set with an increase of additional natively high motion image sequences, which in some cases has the effect of reducing performance on low motion scenes.
  • the result is accordingly a network that has improved performance (better rate distortion scores) on low and high motion scenes simultaneously.
  • the sampling of ⁇ e.g. its biasing towards higher or lower ⁇ values
  • Figure 11 illustratively shows this approach with reference to a sequence of frames 1100 with frame positions ⁇ , ⁇ ⁇ 1, ⁇ ⁇ 2,..., ⁇ ⁇ ⁇ .
  • Frame ⁇ is the current frame of a sequence being processed while ⁇ is the frame which is being used as a reference frame against which optical flow information is being determined (e.g. in a P-frame module of Figure 3).
  • a current frame of the image sequence being fed into the pipeline e.g.
  • ⁇ may be randomly generated from a subset of Z, for example ⁇ ⁇ [1, 2, 3, 4, 5, ..., 60]. Whilst not shown in Figure 11, it is also envisaged that, when using B-frames, ⁇ may take on negative values so that the reference frame is selected from e.g. ⁇ + 1, ⁇ + 2, ... ⁇ + ⁇ . Once the processing of the current frame ⁇ is complete, ⁇ +1 becomes the current frame and the process is repeated for ⁇ +1 and so on until the entire sequence of frames (which may make up a GOP) has been processed.
  • An example training implementation is illustrated in the pseudocode provided below.
  • the network architecture of ⁇ is defined, for example as shown in Figure 3.
  • the network parameters ⁇ are randomly initialised and then the training loop is started. For each batch in the training data X, an ⁇ value is generated and used to select the reference frame ⁇ , and the forward pass is computed for the network being trained ⁇ .
  • the total ⁇ is calculated by combining a distortion term ⁇ and a rate term ⁇ , and any other loss terms (not shown).
  • the backwards pass is then performed to compute gradients based on the loss, and the parameters ⁇ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser.
  • the loss function may incorporate one or more additional terms, that are based on a difference between some ground truth optical flow information ⁇ , and the optical flow information ⁇ that a given forward pass of the networks produce during a training step.
  • each of the loss terms may be regularised relative to each other using a regularisation parameter such as ⁇ .
  • ⁇ ground truth optical flow information
  • ground truth optical flow information ⁇ is not readily available and must be estimated first from the input image ⁇ and a reference frame ⁇ 1.
  • there are many different methods of estimating optical flow each with their own strengths and weaknesses, and each producing different optical flow maps.
  • Horn-Schunck algorithm this algorithm takes a global approach by computing a dense flow field over the entire image. It minimizes an energy function that includes both a data fidelity term and a smoothness constraint, resulting in a smooth flow field.
  • the algorithm estimates motion for every pixel, providing a comprehensive motion map. While it is mathematically rigorous and offers global consistency, it may over-smooth motion boundaries, blurring fine details. It also requires careful tuning of the smoothness parameter and is more computationally intensive than local methods like Lucas-Kanade.
  • Farnebburg’s algorithm this algorithm models the neighborhood of each pixel using quadratic polynomials, allowing it to calculate flow in terms of displacement fields.
  • TV-L1 optical flow algorithm this algorithm introduces robustness to noise and illumination changes by using a total variation regularization term and an L1 norm for the data fidelity term. This variational approach preserves sharp motion boundaries and is less sensitive to outliers, making it effective in handling abrupt motion changes and varying lighting conditions.
  • Brox’s optical flow algorithm incorporates additional constancy assumptions, such as gradient constancy, to improve performance in challenging scenarios. It is better at capturing significant displacements and is more invariant to illumination changes due to its use of gradient information.
  • SimpleFlow algorithm this is a non-parametric algorithm that relies on color matching and spatial consistency rather than image gradients. It can handle large, non-linear motions and is less sensitive to textureless regions. However, it can be slow due to exhaustive search mechanisms and may miss subtle motions, making it less suitable for applications requiring fine detail.
  • FlowNet this is a convolutional neural network architecture designed for flow estimation. It is capable of modeling intricate motions learned from data but may not generalize well to data significantly different from its training data.
  • PWC-Net this is based on FlowNet but is an improvement over FlowNet in that it incorporates a feature pyramid and warping layers, effectively capturing large motions with better efficiency and accuracy than FlowNet.
  • SIFT Flow algorithm this algorithm aligns images through dense matching of Scale-Invariant Feature Transform (SIFT) descriptors. This method is robust to scale, rotation, and illumination changes, making it useful for aligning images with similar scenes but different content.
  • Frequency domain methods these types of methods estimate optical flow by analyzing phase information using Fourier transforms. These methods can effectively capture periodic motions and are resistant to certain types of noise. However, they may not localize motion accurately in spatial terms and struggle with non-linear or non-periodic movements.
  • ⁇ is estimated using a Horn-Schunck algorithm implementation which performs well on global motion
  • the networks of the flow module 206 after training will also perform better on global motion but not on local motion.
  • Concept 6 is directed to harnessing these correlations by providing a learnable or criterion- based ensemble flow estimation to use as ⁇ that comprises flow information (e.g. a flow map) produced by one or more of a plurality of different flow algorithms based on whichever one or combination minimises the loss function or a specific loss term, or based on some other minimisation criteria.
  • flow information e.g. a flow map
  • the resulting "ground truth" flow map may be stitched together on a pixel-wise basis from pixels of whichever flow algorithm produced an optimal flow value for each pixel.
  • ⁇ is a learnable or criterion-based parameter or parameters, for example in tensor form, indicative of how much each of the different flow estimations produced by the respective optical flow algorithms in F contribute to the values that make up the ensemble flow map ⁇ .
  • This approach also allows different optical flow algorithms to contribute different amounts to different parts of the flow map.
  • the networks may update the weights when applying the optimiser to increase the contribution of this optical flow algorithm to the optical flow information for such patches by updating the values of ⁇ accordingly.
  • the networks learn by updating ⁇ which optical flow algorithms are best applied to different pixels of different inputs to minimise the rate distortion loss.
  • ⁇ may be criterion-based where, for each training step of the AI-based compression pipeline, the ⁇ is stitched together from flow pixels selected based on whichever algorithm produced a flow value for that pixel which met some criterion (e.g. lowest difference between ⁇ and ⁇ , and so on).
  • a first optical flow algorithm produces a flow value for that pixel which is very different from ⁇ whereas a second optical flow algorithm produces a flow value for that pixel which results in a smaller difference.
  • the flow value selected for pixel (0,0) will be that produced by the second optical flow algorithm rather than the first.
  • many more optical flow algorithms may be used and the resulting final flow map may be stitched together from flow values taken from many of such optical flow algorithms.
  • ⁇ is a learnable or criterion-based parameter(s) that defines the contribution of each of the optical flow algorithms ⁇ in F to the ensemble flow ⁇ which acts as a "ground truth” or "teacher” optical flow against which the estimated flow ⁇ may be compared when calculating the flow-specific loss term ⁇ during a training step of the networks of the flow module of an AI-compression pipeline.
  • each of the loss terms may be regularised relative to each other using a regularisation parameter such as ⁇ .
  • a regularisation parameter such as ⁇ .
  • any combination may be used, for example any parameterised combination based on one or more combination parameters may be used such as a parameterised linear or non-linear combination, a parameterised power law combination, a parameterised polynomial combination, a parameterised logarithmic combination and so on.
  • This approach is also advantageous from an implementation perspective because it makes it straightforward to include any number (for example 2, 3, 4, 5 or more) and any type of optical flow algorithms (for example any of the optical flow algorithms listed above and/or any others not listed above) in F without burdensome, manual experimentation around which works best because the learnable parameter ⁇ during training automatically learns how much of each optical flow algorithm should contribute to each pixel of a flow map for a given input ⁇ and ⁇ 1 to achieve a given training objective (e.g. minimised rate distortion loss) for a given training data set.
  • this approach allows the networks to decide on their own which behaviours of which optical flow algorithms the networks of the flow module should emulate to achieve the desired minimisation of the rate distortion loss.
  • a criterion-based approach may require additional work in terms of manual hand-crafting of the criterion (e.g. selecting the pixel flow values that are least different to ⁇ ), the criterion-based approach is advantageous is that it does not typically training stability when training the rest of the pipeline - something which may occur when using a learnable ⁇ .
  • An illustrative training implementation is set out in the pseudocode below:
  • the network architecture of ⁇ is defined, for example as shown in Figure 3.
  • the network parameters ⁇ are randomly initialised and then the training loop is started.
  • the forward pass is computed for the network being trained ⁇ .
  • ⁇ is a learnable parameter specifying how much of each flow algorithm contributes to the values of the ensemble flow map and will accordingly be updated by the optimiser when it is applied.
  • the total ⁇ is then calculated by combining a distortion term ⁇ and a rate term ⁇ , and the flow loss term ⁇ ( ⁇ , ⁇ ) which is indiciative of a difference between the actual flow ⁇ estimated by the networks in the current training step and the ensemble flow ⁇ .
  • the backwards pass is then performed to compute gradients based on the loss, and the parameters ⁇ and ⁇ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser.
  • a validation loss can be calculated and the training loop is repeated until the predetermined number of steps or epochs ⁇ have been calculated, or some other criteria have been reached.
  • the learning rate, batch size, and or number of epochs may be optimised during training, for example using a learning rate scheduler or some other hyperparameter optimisation method. More generally, the hyperparameters may be optimised experimentally. As training progresses, ⁇ will converge to a set of values defining the optimal composition of flow algorithms, either in aggregate or on a pixel-wise basis, that achieve a minimised rate distortion loss. In this way, an optimal flow "teacher" is learned which results in the networks of the flow module 206 better emulating the most desired behaviours from the flow algorithms in F while avoiding the least desired behaviours. It is envisaged that the optimiser may be applied at different times to both of ⁇ and ⁇ , and to only one of them.
  • FIG. 12 illustratively part 1200 of the above method.
  • a plurality of optical flow estimation algorithms ⁇ 1201, 1202, 1203 are provided and used to calculate an output optical flow between a first frame ⁇ and a second frame ⁇ 1. Their respective output flow maps are weighted or otherwise emphasised or de-emphasised by learned parameters ⁇ and combined into an ensemble flow map or other representation of optical flow information ⁇ . This provides a reference "ground truth" flow for use in a flow loss term ⁇ of the loss function.
  • the optical flow part 206 of the P-frame module comprising at least an encoder neural network 1204 and decoder neural network 1205 also performs a forward pass to produce its own estimate of optical flow ⁇ from the first frame ⁇ and the second frame ⁇ 1.
  • the optical flow ⁇ produced by the (initially untrained) flow part 206 of the P-frame module may not be particularly accurate and/or may not be entropy encodable into a particularly small bitstream size - this is something that is learned during training.
  • the flow loss term ⁇ is estimated from the the optical flow ⁇ from the forward pass of the untrained networks of the flow part 206 and from the ensemble flow ⁇ .
  • the flow loss term ⁇ may be a simple difference metric such as a mean sqaured error, or other difference metric and/or it may comprise a discriminator loss estimate, whereby the generator is the flow part 206 performing its forward pass and the discriminator may comprise a neural network trained to distinguish ⁇ from ⁇ .
  • the rate term ⁇ and the distortion term ⁇ may then be calculated in the usual way as described earlier herein to produce the overall loss ⁇ , on which the updates to the weights of the neural networks of the AI-based compression pipeline may be based.
  • the criterion-based approach is also envisaged as a more easily implemented approach given the training instability that a learned ⁇ can introduce.
  • each flow algorithm computes a flow and then the choice of which teacher to use is made (pixel-wise) by applying a criterion. For example, whichever teacher minimises one or more loss terms of the loss function or the loss function overall is the one that is selected. In this way, the resulting optical flow information is still "combined" in the sense that the flow algorithm choice is still a pixel-wise choice resulting in an overall optical flow map which is the aggregation of all the individual flow pixels from a chosen algorithm for that pixel.
  • the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.
  • the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
  • the computer storage medium is not, however, a propagated signal.
  • data processing apparatus encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
  • a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD-ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
  • LAN local area network
  • WAN wide area network

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; producing a masked sequence of input images by masking a portion of one or more of the input images; with a first neural network, for a pair of input images of the masked sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps for a plurality of further sequences of input images to produce first and second trained neural networks.

Description

Method and data processing system for lossy image or video encoding, transmission and decoding BACKGROUND This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding. There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted. To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and/or video files. In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take up fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations. A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations. In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known inter-frame redundancy. One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality. I-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, I-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video. P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality. B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (AI) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, AI based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted. An example of an AI based image compression process comprising a hyper-network is described in Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXiv preprint arXiv:1802.01436 (2018), which is hereby incorporated by reference. An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. A further example of an AI based video compression approach is shown in Mentzer, F., Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI (pp. 562-578), which is hereby incorporated by reference. SUMMARY According to an aspect of the present disclosure, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; producing a masked sequence of input images by masking a portion of one or more of the input images; with a first neural network, for a pair of input images of the masked sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for a plurality of further sequences of input images to produce first and second trained neural networks. Optionally, the masking comprises selecting a portion of a first image of the first sequence, and replacing pixels in the other images of the first sequence using the selected portion. Optionally, an unselected portion of the first image of the first sequence remains unmasked in the first sequence of input images. Optionally, a position and size of the portion for first sequence is different to the position and size of the position for one or more of the the further sequences. Optionally, the position and size of the portions for the first sequence and the further sequences are distributed according to a predetermined distribution. Optionally, the distribution is a uniform distribution. Optionally, the distribution is a Gaussian distribution with a mean at a center coordinate of the input images. Optionally, the distribution is based on a type of the input images. Optionally, the type comprises at least one of: webcam video frames, static scene frames, video game frames, high motion video frames, low motion video frames. Optionally, the function is based on a difference between the first sequence of output images and the masked sequence of input images. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to any of the above described methods. According to an aspect of the present disclosure, there is provided a data processing apparatus configured to perform any of the above methods. According to an aspect of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the above methods. According to an aspect of the present disclosure, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of the above methods. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; repeating the above steps for one or more further sequences of input images; accumulating gradients associated with said evaluating of the function for the first sequence and for the one or more further sequences; updating the parameters of the first neural network and the second neural network based on the accumulated gradients; and repeating the above steps for a plurality of said sequences of input images to produce first and second trained neural networks. Optionally, the first sequence of input images and the one or more further sequences of input images comprise different images. Optionally, a final hidden state of the first and second neural networks after evaluating the function for the first sequence is used as a starting hidden state of the first and second neural networks when repeating the steps for one of the one or more further sequences. Optionally, the first sequence comprises 10 or more input images. Optionally, the first sequence comprises 30 or more input images. Optionally, the first sequence comprises 60 or more input images. Optionally, a number of input images in the first sequence of input images is different to a number of frames in at least one of the one or more further sequences of input images. Optionally, the number of input images in the one or more further sequences of input images is based on a difference between the first output image and the one of the images of the pair. Optionally, the number of input images in the one or more further sequences is based on a difference between the first sequence of output images and the first sequence of input images. Optionally, the number of input images in the one or more further sequences is based on an output of a classifier configured to detect an artefact in the first output image. Optionally, the first sequence and the further sequences comprise a batch, and repeating said steps for a plurality of batches. Optionally, said updating the parameters comprises applying an optimiser to weights of the first and second neural networks based on said accumulated gradients. Optionally, the accumulated gradients are associated with a greater number of input images than a number of input images in the first sequence or a number input images in any one of the one or more further sequences. Optionally, the method comprises repeating said steps a first number of times using sequences having a first number of input images, and repeating said steps a second number of times using one or more further sequences having a second, greater number of input images. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a data processing apparatus configured to perform any of the above methods. According to an aspect of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the above methods. According to an aspect of the present disclosure, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of the above methods. According to an aspect of the present disclosure, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for a plurality of further sequences of input images to produce first and second trained neural networks, wherein at least some further sequences comprise a greater number of input images than the first sequence of input images. Optionally, the first sequence of input images comprises fewer than 10 input images. Optionally, the at least some further sequences each comprise 10 or more input images. Optionally, the at least some further sequences each comprise 30 or more input images. Optionally, the at least some further sequences each comprise 60 or more input images. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to any of the above methods. According to an aspect of the present disclosure, there is provided a data processing apparatus configured to perform any of the above methods. According to an aspect of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods. According to an aspect of the present disclosure, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods. According to an aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; modifying the pixels of the first image and the second image at respective first and second coordinates to introduce a pixel representation of an object into the first image and the second image; with a first neural network, producing a latent representation of a difference between the first image and the second image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks. Optionally, the first and second coordinates are based on a predetermined motion path of the object between the first image and the second image. Optionally, the one or more sequences of images comprise a predetermined balance of image sequences having optical flows above a threshold and image sequences having optical flows below the threshold. Optionally, said modifying the pixels is performed on one or more of said image sequences having optical flows below the threshold. Optionally, pixels of the object define one or more alphanumeric characters. Optionally, pixels of the object define a non-uniform texture. Optionally, pixels of the object define a solid colour block. Optionally, pixels of the object are based on an image patch from an image sequence of the sequences of image different to the image sequence comprising the first image and the second image. Optionally, pixels of the object define a box shape. Optionally, the pixels of the object define a circular shape. Optionally, the pixels of the object define a polygonal shape. Optionally, the first and second coordinates for a first image sequence of the one or more sequences of images are different to the first and second coordinates for a second image sequence of the one or more sequences of images, and wherein said modifying is based on the threshold. According to a further aspect, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods According to a further aspect, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a data processing apparatus configured to perform any of the above methods. According to a further aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods. According to a further aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a plurality of second images at a first computer system; selecting one of the plurality of second images as a reference image; with a first neural network, producing a latent representation of a difference between the first image and the reference image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks. Optionally, said selecting comprises sampling an image from the plurality of second images. Optionally, said selecting comprises selecting one of the plurality of second images from the image sequence based on one or more predetermined rules. Optionally, the first image and the plurality of second images comprise consecutive images of an image sequence. Optionally, the selected one of the plurality of second images is a positioned in the sequence before the first image. Optionally, the selected one of the plurality of second images is a positioned in the sequence after the first image . Optionally, the method comprises biasing said selecting towards a beginning of the image sequence. Optionally, the method comprises biasing said selecting towards an end of the image sequence. Optionally, said biasing is based on a property of the first image and/or the plurality of second images. Optionally, said property is a scene type of the first image and/or the plurality of second images. Optionally, the one or more sequences of images comprise a predetermined balance of image sequences having optical flows above a threshold and image sequences having optical flows below the threshold, and wherein said selecting is based on the threshold. According to a further aspect, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods. According to a further aspect, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a data processing apparatus configured to perform any of the above methods. According to a further aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods. According to a further aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; estimating first optical flow information and second optical flow information, each being indicative of a difference between the first image and the second image; producing combined optical flow information from the first optical flow information and second optical flow information; with a first neural network, producing third optical flow information indicative of a difference between the first image and the second image; with a second neural network, decoding the third optical flow information to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the combined optical flow information and the third optical flow information; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks. Optionally, producing the combined optical flow information comprises producing a parame- terised combination of the first optical flow information and the second optical flow information using one or more combination parameters. Optionally, the method comprises updating the combination parameters based on one or more terms of the evaluated function. Optionally, the parameterised combination comprises a weighted sum of the first optical flow information and the second optical flow information and the combination parameters comprise one or more weights of the weighted sum. Optionally, the parameterised combination comprises a pixel-wise parameterised combination of the first optical flow information and the second optical flow information. Optionally, the function is further based on a rate term and a distortion term. Optionally, the difference between the combined optical flow information and the third optical flow information comprises a mean squared error. Optionally, the method comprises estimating the first optical flow information using a first optical flow algorithm, and estimating the second optical flow information using a second optical flow algorithm. Optionally, the first optical flow algorithm and the second optical flow algorithm comprise an ensemble of optical flow algorithms. Optionally, the method comprises estimating further optical flow information using a further optical flow algorithm, and producing the combined optical flow information from the first optical flow information, the second optical flow information and the further optical flow information. Optionally, the method comprises estimating the difference between the combined optical flow information and the third optical flow information using a discriminator function. According to a further aspect, there is provided a method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to any of the above methods. According to a further aspect, there is provided a method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to any of the above methods. According to a further aspect, there is provided a data processing apparatus configured to perform any of the above methods. According to a further aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods. According to a further aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS Aspects of the invention will now be described by way of examples, with reference to the following figures in which: Figure 1 illustrates an example of an image or video compression, transmission and decom- pression pipeline. Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network. Figure 3 illustrates an example of a video compression, transmission and decompression pipeline. Figure 4 illustrates an example of a video compression, transmission and decompression system. Figure 5a illustrates an example plot of a distortion score of a frame sequence against frame number. Figure 5b illustrates an example plot of a distortion score of of a frame sequence against frame number. Figure 6 illustratively shows steps of a method of modifying training data of a video compression, transmission and decompression pipeline. Figure 7a illustrates an example plot of a distortion score of a frame sequence against frame number. Figure 7b illustrates an example plot of a distortion score of a frame sequence against frame number. Figure 8 illustrates a high motion frame sequence. Figure 9 illustrates a training data augmentation method for use with a video compression, transmission, and decompression method. Figure 10 illustrates a training data augmentation method for use with a video compression, transmission, and decompression method. Figure 11 illustrates method for use with training a video compression, transmission, and decompression method. Figure 12 illustrates method for use with training a video compression, transmission, and decompression method. DETAILED DESCRIPTION OF THE DRAWINGS Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The file size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression. In a compression process involving an image, the input image may be represented as ^^. The data representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^ represents the height of the image, ^^ represents the width of the image and ^^ represents the number of channels of the image. Each ^^ × ^^ data point of the image represents a pixel value of the image at the corresponding location. Each channel ^^ of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video. The output image may differ from the input image and may be represented by ^^. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network. Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a rate-distortion equation. AI based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network. Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer. Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network. Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network. To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients ^^^^/^^^^ of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network. In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features. It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model’s predictions and the actual data, represented by the above-described rate-distortion loss function. The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model’s parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data. Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model’s parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution. The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge. The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model’s performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1/L2 regularization, can also be used to mitigate overfitting. It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an “untrained” neural network is considered be “trained” is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein. More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately. In the case of AI based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^, where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network. In the case of AI based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu/ xypan/research/snr/Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (for example at www.image-net.org/download). An example training set of input images is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example at http://challenge.compression.cc/tasks/). An example of an AI based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the AI based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function ^^^^ acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation ^^, resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function. In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network. In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function ^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the AI based compression process 100. The encoder-decoder system may be referred to as an autoencoder. Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation is distributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian or Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the rate-distortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in WO2021/220008A1, which is incorporated in its entirety by reference. Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term. In a similar manner to how learning parameters top produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes. Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off. The system described above may be distributed across multiple locations and/or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server. The decoder 120 may be located on a separate device which may be referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline. The AI based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ ^^ and a trained neural network 125 acting as a hyper-decoder ^^ℎ ^^. An example of such a is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above. The neural network 115 acting as a hyper-decoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. The quantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140 characterised by ^^ discussed above. In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150. Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the AI based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised. To perform training of the AI based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an AI based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake. When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6. Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120. Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. AI based compression process 100 described above may be applied multiple times to perform compression, transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video. The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and/or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time. The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map. An example AI based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may be also be considered as a single, end-to-end pipeline. As described above, I-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image ^^0 is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, for example, an I-frame and a plurality of subsequent P-frames and/or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image ^^0 which in this case is an I-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image ^^0 to be available for later use by components of both the encoding and decoding sides of the pipeline. In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image ^^^^ that is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images ^^^^−1 from an earlier frame t-1. As described above, the previously reconstructed ^^^^−1 is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides. At the encode side, previously reconstructed images may be used for generating a flow maps containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed ^^^^ and the previously reconstructed image from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-frame movement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map ^^ . The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generate a warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image ^^^^−1 might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1. As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^ may be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides. In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map ^^ from the decoded latent representation. Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and the pre-calculated residual map may be passed into an autoencoder for compression only. This hand-crafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function. Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image ^^^^ which is a reconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then be used to process the next frame. That is, it can be used to compress, transmit and decompress ^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed. Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directly from the entropy decoded bitstream by removing the connection between ^^^^−1,^^ and the output of the residual block 210, thereby eliminating any direct combination step with the warped previously decoded image to speed up inference. In this case, the flow information is intuitively understood to be indirectly captured within the residual information, which the residual decoder is able to learn to use to directly reconstruct the output image ^^^^ . Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directly from the entropy decoded bitstream in combination with some representation of flow injected into one or more layers of the residual decoder. In this case, the flow information is intuitively understood to be indirectly captured within the injected information, which the residual decoder is able to learn to use while decoding the latent representation of flow information to directly reconstruct the output image ^^^^ . Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the I-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable. Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6680-6689). The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame. Figure 4 shows an example of an AI image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR/VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively. That is, compute for performing both encoding and decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing AI-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU. The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and/or B-frames. The blocks of frames are then compressed by an AI-compression module 407 comprising the encode side of the AI-based video compression pipeline of Figure 3. The output of the AI-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications. The second device 402 receives the communicated bitstream 408b which is passed to an AI-decompression module 410 comprising the decode side of the AI-based video compression pipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructed I-frames, P-frames and/or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402. It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider AI-compression literature. For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations. A number of concepts related to the AI compression processes and/or their implementation in a hardware system discussed above will now be described. Although each concept is described separately, one or more of the concepts described below may be applied in an AI based compression process as described above. Concept 1: Frozen-patch long GOP training A compression artefact type that appears specifically in AI-based image and video compression, but not in traditional compression, is known as "unrolling". Unrolling is an artefact that typically appears in image sequences where there is little to no movement between frames and manifests itself in the reconstructed images as halo-like rings around edges and objects in the images. These artefacts typically arise where small, barely noticeable changes in pixel values between frames such as tiny lighting changes, block artefacts from traditional compression (if the input to the AI compression pipeline came from a traditionally compressed source), noise, and so on become amplified over time by the AI based compression pipeline. This results in the networks of the compression pipeline treating what is effectively image noise as real movement and then exaggerating this as the sequence of frames progresses. At the end of the sequence of frames, the artefacts can be quite severe and detrimental to the image reconstruction quality. In long group of picture (GOP) sequences, the effect can be particularly pronounced as errors propagate much further over the course of long P- and/or B-frame sequences before the next I-frame provides "fresh" information without artefacts for the next set of P- and/or B-frames to be generated from. Unrolling artefacts occur across mostly all types of image sequences but are particularly apparent when AI based flow-residual compression models are used to encode and decode image sequences of internet video calls (whereby a foreground user who moves very little is surrounded by a largely static background which also does not move much), screen sharing image sequences, security camera feeds and others. As described above, if there is no movement in the scene after a period of time, objects and edges start to exhibit halo-like ringing around their edges. The artefacts also result in the compression pipeline assigning a disproportionate number of bits to handle the hallucinated movement of the unrolling artefacts even though the corresponding parts of the actual images aren’t changing or are hardly changing between frames so ought not to have many bits assigned to them. One advantage of an AI-based compression pipeline over traditional compression is that neural networks are significantly more expressive than the hand-crafted techniques of traditional compression and can adapt very well to different training data. At a very general level, if an AI-based compression pipeline is not good at compressing certain types of sequences, biasing the training data the networks of the pipeline have been trained on by adding more examples of the problematic sequences is a generally successful approach to improving that pipeline’s ability to handle the problematic sequences. This can be achieved for example by doing a base model training run on a balanced data set and then finetuning the (trained) base model on the problematic sequences, or it can be achieved by simply biasing the base model training data from the beginning. However, even when taking these approach, there are some types of sequences where biasing the training data does not result in the expected improvement. For Long GOP sequences are an example of such a type of sequence. The term "long GOP" as used herein means a sequence of frames of more than substantially 10 frames. For example, a sequence of one I-frame and nine P- and/or B frames, or some other combination thereof. The advantages of the present disclosure become particularly apparent in respect of long GOP sequences of at least 100 frames, for example 120 or more frames (for example one I-frame and 119 P- and/or B-frames). It turns out that even when the networks of the architecture are trained on training data biased with more long GOP sequences, there is only a very small performance improvement (in terms of bit rate and reconstruction accuracy e.g. as measured by PSNR), and the above-described unrolling artefacts are still present. Accordingly, the present disclosure proposes a number of different approaches to help improve performance on long GOP sequences. The first of these approaches is referred to herein as frozen-patch long GOP training. Frozen-patch long GOP training modifies one or more GOPs in the training data by selecting only a patch of the frames of the GOP to to show to the network, and masking the rest of the frame using a frozen frame from the GOP. For example, we take the first I-frame of a GOP, select a patch (e.g. by pixel coordinates) I-frame, and modify the rest of the frames in the GOP by masking everything outside of those pixel coordinates in each frame by copying over the pixels from the I-frame that were not in the patch. The selection of the patch may be random, or based on some other criteria such as an image statistics property associated with the frame, and the patch size can be different across the training data (e.g. some with a patch size of 0 where we mask the entire frame sequence with a duplicated I-frame. some with a patch size corresponding to the entire image resolution where we mask none of the pixels and keep the frame sequence unmodified, and some patch sizes in between these two extremes). In this way, the networks during training are exposed to static portions (the masked portion that is duplicated from the I-frame), as well as movement containing portions (the selected patch in which the scene is played in the usual way across the sequence of frames). Whilst this modification of the training data results in what, at first glance, looks like artefact- ridden, blocky training data, it turns out, counter-intuitively, that an AI-based compression pipeline trained on this type of training data is substantially better at handling longer GOP sequences in terms of bit rate (e.g. bits per pixel) and reconstruction accuracy (e.g. PSNR, MSE or some other quantified difference). It is also noted that this improvement is seen across all types of long GOP sequence scene types including video conference frame sequences, video game sequences, nature scene sequences, and so on. This above-described approach also helps to mitigate unrolling artefacts. This is in part understood to be due to the networks being given the (noiseless) frozen parts of the GOP in each frame and so learning that any small deviations that can result in unrolling artefacts should not be propagated across the sequence. Figure 5a is a plot 500 of reconstruction accuracy (quantified in mean squared error, MSE, between the input frame and the output frame) against frame number of a sequence of one I- frame (frame index 1) and 149 P-frames (frame index 2 to 150) produced by an AI-compression pipeline trained on training data where frozen-patch long GOP training was not applied. It can be seen from the gradually increasing MSE as the frames progress that the image reconstruction accuracy is decreasing, indicating that unrolling artefacts start to dominate the further away the last I-frame was. Frames beyond number 150 are not shown as the MSE just continues to increase until the artefacts are so significant the image becomes unrecognisable. Figure 5b is a similar plot 501 but now the frames are produced by an AI-compression pipeline that has been trained using the above-described frozen-patch long GOP training. It is immediately apparent that the MSE does not substantially grow as the frames progress indicating that artefacts are not propagating through time across the sequence of frames. Indeed, data up to frame number 300 is plotted in the plot of Figure 5b and there is no substantial increase in MSE compared to the early frame numbers. This can be contrasted with the plot 500 in Figure 5a which shows substantial MSE increases even after only 150 frames. Figure 6 illustratively shows steps 600 of a method of modifying training data using the above-described frozen-patch long GOP training. First, a GOP 601 is provided. Whilst only a small number of frames are shown for brevity, in practice a full GOP of some predetermined length is provided. Each of the frames is part of a sequence of a scene in which a character is moving his arms up and down and then walking off. A patch 602 of the first frame of the GOP, for example an I-frame 603, is selected by pixel coordinates and the pixels of the first frame outside of those coordinates are copied as a mask 604. The mask 604, made up of the non-patch first frame pixels is are applied 605 across all the other frames of the GOP 601, leaving only those patches at the non-mask pixel coordinates of the other frames untouched. In the example of Figure 6, as the character walks off, this will only be apparent from the pixels within the patch, but the mask frame remains unchanged, containing the original I-frame pixels that include the characters arm, body and part of the head. Effectively, the I-frame is frozen and applied to the whole GOP except for the pixels within the selected patch. The modified GOP is then stored as a sequence of frames of training data, and the same process is repeated for another GOP, and so on to produce the full training data set. It is envisaged that the patch sizes and positions may be chosen randomly according to some distribution, for example a uniform distribution, or Gaussian distribution or some other distribution. It has been found that a uniform distribution of different patch sizes and positions produces a training data set that, when trained on, is most robust to different GOP lengths (both long and short) during inference. Alternatively, patch sizes and positions may be selected based on some property of the data the method is being applied to (e.g. selecting a patch size of around 128x128 pixels or 256x256 pixels or more pixels and positioned around the center pixels of webcam video data around where the person in the webcam video data is positioned). Envisaged data types where the patch size and position may be predetermined include but are not limited to webcam video data, static scene data, video game data, high motion video data, low motion CCTV data. In each case, the patch size and position is typically based around typical motion or non-motion patterns and where detail is to be preserved as much as possible, such as at or around edges, at or around typical movement paths and patterns, and so on. Further, it is envisaged that the distribution of patch sizes and positions can be used to control the performance of the model trained on it for different needs. For example, if a given model is only intended to be used on long GOP sequences where the center of the frame is expected to contain detail and movement (e.g. webcam video conference scenes), the distribution of patch sizes and positions can be focused around the center of the frames when producing the frozen-patch long GOP training data (e.g. a Gaussian distribution with a mean at the center pixel of the frames). At a high level, this means the frozen-patch long GOP training approach can advantageously be used as a way to specialise the networks of an AI-based compression pipeline for a given purpose, such as compressing and reconstructing webcam video conference frames. For completeness, it is noted that the frozen-patch approach can also be applied to short GOP sequences (e.g. fewer than 10 frames). In this case, whilst short GOP sequences are already typically less problematic, the technique can still be used to control the performance of the model at by, for example, focusing the patch distribution around different positions within the frame to teach the networks to improve performance around the positions of the patches in the frame. Below is provided pseudocode for training one or more networks of an AI-based compression pipeline using the above-described frozen-patch GOP training approach.
Algorithm 1 Neural Network Training frozen-patch GOPs Require: Training dataset D, learning rate ^^, regularization parameter ^^, number of epochs ^^ Select patches and frozen pixels of GOPs of D Generate frozen-patch training data D^^^^^^ using selected patches and frozen pixels Initialize network parameters ^^ randomly for ^^^^^^^^ℎ = 1 to ^^ do for each batch (^^, ^^) in D^^^^^^ do Forward pass to compute predictions ^^ = ^^^^ (^^) Compute ^^^^^^^^ Backward pass to compute gradients ∇^^ ^^^^^^^^ Update parameters with optimizer O: ^^ ← O(^^, ∇^^ ^^^^^^^^, ^^) end for Optionally evaluate on validation set end for A further technique that can be used, either together with the above-described frozen-patch GOP training, or in a standalone manner is long GOP training with truncated back propagation in time, as will be described below in concept 2. Concept 2: Long GOP training with truncated back propagation through time One problem with training on long GOP sequences is that training can be unstable. This can, in part, be attributed to the unrolling artefacts (and other artefacts) manifesting themselves in the reconstructed frames of the long GOPs. This in turn increases the distortion loss when training on long GOP sequences and can prevent the loss from converging well. In the case where unrolling artefacts are particularly prominent this can cause the loss to increase and behave unpredictably. The inventors have realised that one way to address this problem is by using TBPTT. Truncated Backpropagation Through Time (TBPTT) is a technique used to train neural networks by limiting the number of time steps over which gradients are propagated backward through the network. This approach helps to reduce computational costs and mitigate the vanishing or exploding gradient problem, enhancing the stability and efficiency of training neural networks. At a very general level, known TBPTT techniques comprise the following steps: 1. Segment the Input Sequence: Divide the input sequence into overlapping smaller chunks of a fixed length, ^^ . 2. Forward Pass: For each chunk, perform a forward pass through the neural network for ^^ time steps, computing the hidden states and outputs. 3. Backward Pass: Backpropagate the error through only the last ^^ time steps, instead of the entire sequence. 4. Update Parameters: Update the network parameters based on the gradients computed in the backward pass. 5. Move to Next Chunk: Repeat the process for the next chunk of the sequence, using the final hidden state of the previous chunk as the initial state for the next chunk. By way of worked example, consider a sequence of length 12 and a TBPTT chunk size of 4 for a sequence modeling task using an neural network. • Sequence: [^^1, ^^2, ..., ^^12] • Chunk Size: 4 The following steps use a known TBPTT technique with overlapping chunks whereby the overlap in this example is 1: Step 1: Divide the sequence into overlapping chunks: [^^1, ^^2, ^^3, ^^4], [^^2, ^^3, ^^4, ^^5], [^^3, ^^4, ^^5, ^^6], ...,[^^9, ^^10, ^^11, ^^12]. Steps 2: For a first chunk: 1. Perform a forward pass for 4 time steps. 2. Compute the loss at the end of the chunk. 3. Backpropagate the error through these 4 time steps. 4. Apply optimiser to update weights. Step 3: Move to the next chunk and repeat the process, using the final hidden state of the previous chunk as the initial state for the next chunk. This approach can facilitate the efficient learning from sequences by reducing the computational cost and preventing issues with vanishing or exploding gradients. However, one challenge in the field of image and video compression is that sequences are long. For example in a video sequence running at 30fps, large parts of an image may be unchanged for 1, 2, 3 minutes or more at a time, which means temporal information and correlations can stretch across sequences of thousands of frames or more. For example, if a background of a scene remains unchanged for 3 minutes, that information is redundant and in an ideal world would be compressed away and not transmitted in a bit stream. Increasing the GOP length allows for more of this temporal information to be harnessed to facilitate better compression rates by removal of redundant information. For the networks to learn how to use temporal information across long GOP sizes, one approach is to increase GOP sizes in training data. However, this results in a very significant compute overhead and training time increase that is expensive and/or not practical, even when using truncated backpropagation through time with a sliding window of overlapping chunks. For example, consider a GOP frame sequence of 60 frames long [^^0, ..., ^^59], with a batch size of 1 (i.e. the single 60 frame sequence is a single batch) and chunk size 10. Applying an overlapping, sliding window of truncated back propagation through time, we have to perform gradient calculations for all of the 51 following chunks of the single batch: [^^0, ..., ^^9], [^^1, ..., ^^10], [^^2, ..., ^^11], [^^50, ..., ^^59] This can be contrasted with short GOP lengths, where for example, a GOP of 4 frames long would only have a single chunk. Accordingly, it can be seen that increasing GOP size during training and using truncated back propagation through time on its own can result in orders of magnitude increase in compute overhead and training time. The present disclosure proposes a solution to this problem by counter-intuitively removing the overlap of the chunks while performing truncated back propagation through time. For example, considering the same GOP length of 60 frames and batch size 1. By removing the overlap, we now only have to perform gradient calculations for the 6 following chunks of the single batch: [^^0, ..., ^^9], [^^10, ..., ^^19], [^^20, ..., ^^29], [^^50, ..., ^^59] This approach is counter-intuitive because one advantage typically associated with truncated back propagation through time is that the overlap in the sliding window helps to capture temporal relationships across all time steps. The present inventors have realised that, in the context of image and video compression, removing this overlap does not result in any significant drops of performance that might intuitively be expected by less temporal overlap. Accordingly, training performed using this approach is 5-10x faster than using traditional truncated back propagation through time, while at the same time being just as stable and resulting in comparable final loss levels. As a result, the present disclosure facilitates the use of long GOPs during training, which contributes to overall reduction in artefacts such as unrolling, as is described above. More generally, the above approach may be illustrated with the following pseudocode:
Algorithm 2 Truncated back propagation through time without overlap Require: Training dataset D, learning rate ^^, regularization parameter ^^, number of epochs ^^ Initialize network parameters ^^ randomly for ^^^^^^^^ℎ = 1 to ^^ do for each batch in D^^^^^^ do Chunk batch into non-overlapping chunks (^^) of image pairs ^^^^ and ^^^^−1 for each chunk ^^ in ^^ do Forward pass to compute predictions ^^^^,^^ = ^^^^ (^^^^,^^ , ^^^^,^^−1) Compute ^^^^^^^^^^ Backward pass to compute gradients ∇^^ ^^^^^^^^^^ Accumulate gradients in ∇^^ ^^^^^^^^^^ end for With accumulated gradients, update parameters with optimizer O: ^^ ← O(^^, ∇^^ ^^^^^^^^^^ , ^^) end for Optionally evaluate on validation set end for That is, initially, the network parameters ^^ are set to random values. Within each epoch, the training dataset, D, is divided into batches. These batches are processed one at a time. The division into batches helps manage memory efficiently and can speed up the training process. Each batch from the dataset is further divided into non-overlapping chunks ^^ that together cover a number of time steps ^^, each containing a plurality of image pairs ^^^^ and ^^^^−1, for example images from a first point in time and a second point in time across the total number of time steps ^^, whereby when compressing and reconstructing one of those images during the forward pass, the temporal redundancy between the two images is used (e.g. by applying a flow-residual approach as described above in the introductory section). For each chunk, ^^, a forward pass is performed to reconstruct the images of the sequence, ^^^^, based on the current parameters, ^^, and the input images of the sequence, e.g. using all the pairs ^^^^,^^ and ^^^^,^^−1 in the chunk ^^. The loss, ^^^^^^^^^^, for the chunk is computed. This loss measures between the predicted values and the actual target values in the chunk and may be, for example an MSE loss or any other loss. A backward pass is then executed to calculate the gradients of chunk ^^ of the loss with respect to the network parameters, ∇^^^^^^^^^^^^. The calculated gradients are accumulated into an overall accumulated gradient ∇^^^^^^^^^^^^ for the batch. Only once all chunks ^^ in the batch have been processed and the gradients accumulated are the network parameters are updated. This update may be performed using an optimizer, O, which adjusts the parameters, ^^, based on the accumulated gradients, the learning rate, ^^, and optionally other factors, for example regularization term. Finally, optionally, at the end of each epoch, validation may be performed using a validation dataset. For completeness, the final hidden state of the first chunk once completed is stored and used as the initial hidden state from which the forward pass of the second chunk is started. The same process is repeated for the second chunk, the third chunk, and the fourth chunk to complete the batch. In a further generalisation of the above approach, the window size need not be static across a batch. Instead it can vary across a sequence of frames of a GOP. Consider for example, the same GOP length of 60 frames and batch size 1 e.g. [^^0, ..., ^^59]. Instead of setting static window or chunk length as described above, the chunk sizes can be dynamically adjusted as the forward passes progress. For example, consider a 60 frame GOP with a starting chunk size of 4. That is: ^^ℎ^^^^^^1 = [^^0, ..., ^^3] The forward pass is performed for ^^ℎ^^^^^^1, the distortion scores (e.g. MSE) are calculated for each of the frames of ^^ℎ^^^^^^1. If the distortion scores are indicative of the appearance of artefacts (such as unrolling artefacts) in the reconstructed frames, the chunk size for the next chunk, ^^ℎ^^^^^^2, can be reduced or kept the same. Conversely, if the distortion scores are not indicative of the appearance of artefacts in the reconstructed frames, the chunk size can be increased for the next chunk. Checking distortion scores for signs of artefacts may be performed by applying a threshold to the distortion score. For example, if the distortion is above the threshold it is indicative that the reconstructed image is too different to the ground truth image and thus that artefacts are present. Conversely, if they are at or below the threshold, the reconstructed image is sufficiently similar to the ground truth image and no or only minor artefacts are present. More complex methods of detecting artefacts may additionally or alternatively be applied. For example, by comparing image statistics of the reconstructed image with the ground truth image, by applying a classifier, and so on. In the above example a chunk size of 4 is relatively small so it is unlikely that there will be unrolling artefacts. Accordingly, the chunk size of ^^ℎ^^^^^^2 is increased to e.g. 6. That is: ^^ℎ^^^^^^2 = [^^4, ..., ^^9] The forward pass is then performed for ^^ℎ^^^^^^2, the distortion scores are calculated for each of the frames of ^^ℎ^^^^^^2 and checked for signs of artefacts. If there are no signs of artefacts, the chunk size for ^^ℎ^^^^^^3 may be increased again. For the sake of illustration, we increase it to e.g. 27. That is: ^^ℎ^^^^^^3 = [^^10, ..., ^^36] As a chunk size of 27 is relatively large, unrolling artefacts are likely to appear and this will be apparent from the increase in distortion scores in the reconstructed frames as the forward pass progresses through the sequence of 30 frames. In this case, the chunk size for the next chunk ^^ℎ^^^^^^4 can be decreased, for example to 23. That is: ^^ℎ^^^^^^4 = [^^37, ..., ^^59] A chunk size of 23 is also relatively large, but smaller than 27, so it is expected that the distortion scores will be less than for the chunk size of 27. If the decrease is enough, the distortion scores may fall below the threshold or meet whatever criteria of the image statistics and/or classifier being used to detect distortions. Thus, for the single GOP of 60 frames, an optimum chunk size has been identified at which temporal data from longer GOPs is optimally captured at just the threshold chunk length at which artefacts such as unrolling artefacts are unlikely appear. This adaptive chunk size approach accordingly facilitates the use of truncated backpropagation through time in a way that is optimal for AI-based image and video compression. The above example is generalised in the following pseudocode:
Algorithm 3 Truncated back propagation through time with adaptive chunk size Require: Training dataset D, learning rate ^^, regularization parameter ^^, number of epochs ^^ Initialize network parameters ^^ randomly for ^^^^^^^^ℎ = 1 to ^^ do for each batch in D^^^^^^ do Chunk batch into non-overlapping chunks (^^) of size ^^ of image pairs ^^^^ and ^^^^−1 for each chunk ^^ in ^^ do Forward pass to compute predictions ^^^^,^^ = ^^^^ (^^^^,^^ , ^^^^,^^−1) Compute ^^^^^^^^^^ Backward pass to compute gradients ∇^^ ^^^^^^^^^^ Accumulate gradients in ∇^^ ^^^^^^^^^^ Identify presence of artefacts in ^^^^,^^ and update ^^ Update ^^ end for With accumulated gradients, update parameters with optimizer O: ^^ ← O(^^, ∇^^ ^^^^^^^^^^ , ^^) end for Optionally evaluate on validation set end for In an additional or alternative approach to adaptive chunk sizes is to introduce a relative distortion check at some arbitrary point or points in the sequence (e.g. predetermined points or points determined dynamically). That is, we take some reference frame ^^^^ , for example an I-frame or a frame at the start of a chunk, we measure the distortion loss (e.g. MSE) at that point, and then when we get to frame ^^^^+^^ we measure the distortion loss again and compare it to the previously measured distortion for the reference frame ^^^^ . If the new distortion is some amount greater than the previously measured distortion (e.g. some percentage, for example 1-25%, such as 5%, 10%, or 20%), then it is indicative that the distortion is going up too fast and accordingly that we should cut the chunk short there and proceed to backpropagate, rather than waiting until the end of all the frames in that chunk. At a general level, this approach effectively immediately cuts short any chunks when we see the distortion increasing too much, and adapts the next chunk size accordingly based on where the current chunk was cut short, rather than waiting for the end of the chunk to adapt the size of the next chunk. That is, by measuring a relative distortion of frames of the chunk compared to a reference frame, it is possible to track changes in the distortion at an arbitrary level of granularity (e.g. frame to frame, every 5 frames, every 10 frames, and so on), thus allowing for the dynamic updating of chunk sizes during training. The above approaches approaches also open up training (and finetuning) on substantially longer GOP size data than has been possible up to now and on long GOP data of all types (e.g. webcam call scenes, high motion scenes, low motion scenes etc) without a priori knowledge of that data. This is because the adaptive window facilitates the automatic finding optimal chunk sizes for truncated back propogation through time, unique for each GOP and data type - something which is burdensome using handcrafted chunk sizes or static chunk sizes across an entire GOP. As alluded to above, the presently described truncated backpropagation through time approaches may be used with concept 1, whereby the frozen patch approach may be applied to a GOP before that GOP is chunked into non-overlapping chunks as described above. Finally, both of the above approaches may be jointly or individually combined with the concept described below. Concept 3: Long GOP finetuning A further problem of training on long GOP training data is that the training signal strength and training stability can decrease as GOP length increases. This can be attributed, at least in part, due to errors and artefacts propagating temporally across the sequence as each subsequent frame relies on information from earlier frames that can include those errors and artefacts, resulting in worse MSE losses and unpredictable loss curve behaviours. The present inventors have realised that this problem can be partially mitigated by performing base training on short GOP lengths which are associated with stronger training signal strength and stability and, once base training is completed (e.g. once loss is below a predetermined threshold or has plateaued or after a predetermined number of training steps), finetuning on longer GOP length training data. This approach allows the networks of the AI-compression pipeline to learn the easier task of short GOP compression first without the issues of long GOP training, resulting in the overall loss moving roughly in the direction of a global minimum. Then, after base training is complete, the finetuning on long GOP training data allows the already-performant networks to learn the more difficult task of long GOP compression, which moves the loss even closer to the global minimum for long GOP compression. As above, the term short GOP may be understood to be GOP lengths of below substanially 10 frames, while 10 frames or above may be understood as a long GOP. The above approach may be illustrated using the following pseudocode: Algorithm 4 Neural Network Training with Long GOP Fine-tuning Require: Base training dataset D^^^^^^^^, fine-tuning dataset D ^^ ^^^^^^, learning rate ^^, regularization parameter ^^, number of epochs ^^^^^^^^^^, number of fine-tuning epochs ^^ ^^ ^^^^^^ Initialize network parameters ^^ randomly ⊲ Base Training Loop for ^^^^^^^^ℎ = 1 to ^^^^^^^^^^ do for each batch (^^, ^^) in D^^^^^^^^ do Forward pass to compute predictions ^^ = ^^^^ (^^) Compute base loss ^^^^^^^^^^^^^^^^ Backward pass to compute gradients ∇^^ ^^^^^^^^^^^^^^^^ Update parameters with optimizer O^^^^^^^^: ^^ ← O^^^^^^^^ (^^, ∇^^ ^^^^^^^^^^^^^^^^, ^^) end for Optionally evaluate on a base validation set end for ⊲ Fine-tuning Loop for ^^^^^^^^ℎ = 1 to ^^ ^^ ^^^^^^ do for each batch (^^, ^^) in D ^^ ^^^^^^ do Forward pass to compute predictions ^^ = ^^^^ (^^) Compute fine-tuning loss ^^^^^^^^ ^^ ^^^^^^ Backward pass to compute gradients ∇^^ ^^^^^^^^ ^^ ^^^^^^ Update parameters with optimizer O ^^ ^^^^^^: ^^ ← O ^^ ^^^^^^ (^^, ∇^^ ^^^^^^^^ ^^ ^^^^^^, ^^) end for Optionally evaluate on a fine-tuning validation set end for That is, for the base training phase, the network’s parameters ^^ are initialised randomly and the base, short GOP dataset, D^^^^^^^^ is provided. For each epoch, up to a predetermined number of epochs ^^^^^^^^^^, the network processes batches of data from D^^^^^^^^. In each batch, the network performs a forward pass to compute predictions based on its current parameters. It then calculates a loss, ^^^^^^^^^^^^^^^^. A backward pass is performed to compute gradients of the loss with respect to the parameters. The parameters are updated using an optimization algorithm, O^^^^^^^^, which applies these adjustments in a direction expected to reduce the loss, using the learning rate ^^ as a step size. After each epoch, the network can be optionally evaluated on a validation set to monitor its performance on unseen data. After base training on short GOP data, the networks of the compression pipeline ^^^^ are able to compress short GOP sequences well but will produce errors and artefacts, such as unrolling, on long GOP sequences. To mitigate this, the network undergoes fine-tuning on long GOP data. This phase adjusts the network to perform well on a specific long GOP compression task using long GOP training data, D ^^ ^^^^^^, The fine-tuning process also proceeds in epochs, for a number of epochs ^^ ^^ ^^^^^^. The network processes batches from D ^^ ^^^^^^ in a manner similar to the base training phase, performing forward passes, computing a fine-tuning loss ^^^^^^^^ ^^ ^^^^^^, and then performing backward passes to adjust the parameters. The parameters are updated, either using the same optimiser as the base training or optionally using a different optimizer, O ^^ ^^^^^^. Finally, similar to the base training phase, the network can be evaluated on a separate validation set specific to the fine-tuning dataset. The number of training steps may be based on epochs or may instead be set as a predetermined number of steps or based on some stopping criteria such as loss curve plateau or some other stop criteria. Counter-intuitively, the present inventors have found that finetuning on long GOP data does not result in a significant drop in performance on short GOP sequences during validation even though the performance on long GOP sequences significantly increases (e.g. in terms of rate and distortion scores in e.g. bits per pixel and/or MSE scores). It is understood that this phenomena is likely due to the tasks of short GOP and long GOP compression being similar in nature. Thus, finetuning does not result in the base training model suffering from catastrophic forgetting as might be the case if the finetuning task was significantly different to the base training task. Figure 7a illustrates a plot 700 of distortion (MSE) score of compressed and reconstructed frames compared to corresponding ground truth frames of a sequence that has been compressed and reconstructed by an AI-based compression pipeline that has been base-trained on short GOP data. The figure shows a long GOP data series 701 made up of a single sequence of the same 150 frames as a single GOP. It is apparent that the distortion increases as the sequence progresses on the long GOP series 701 after the initial I frame. It is also noted that I-frames cannot take advantage of temporal redundancies so use up more bits than P-frames to compress resulting in the initial MSE spike at frame 0 (i.e. the I frame). It can be seen that from the initial MSE spike at frame 0 of around 40, the MSE hits 40 again around frame 50 and continues to go extend higher and higher reaching MSE scores of around 55-60 at frame 150. Figure 7b illustrates the same plot 700 but now after finetuning on long GOP data. The long GOP data series 701 in Figure 7b has the same initial MSE spike to around 40 for the I frame before dropping as the P-frames progress. However, this time the MSE scores only hit 40 again by frame 150 (contrasting to Figure 7a where the MSE scores already hit 40 by 50 frame). This indicates an improvement of temporal stability in sequences of roughly 3x the length. Accordingly, Figure 7b illustrates that finetuning on long GOP data after base training on short GOP data produces networks of an AI-compression pipeline that achieve better bit rates while maintaining low distortion across longer GOP data during inference. For completeness, the finetuning of the networks of the pipeline associated with Figure 7b may be performed using GOPs of whatever GOP length the finetuned networks are intended to used on. For example, the inference toy example of Figure 7b is performed at GOP length 150, accordingly it is envisaged that the finetune data comprises GOP lengths of 150. Extending this principle generally, including a distribution of different GOP lengths in the finetune data can be used to produce a pipeline that is able to perform well across said variety of GOP lengths. More specifically, it has been found that performing the base training on the easy task of short GOPs produces a base model that is particularly suited to subsequent finetuning on longer GOPs and results in a pipeline that outperforms a model where the base training data comprises a full, complete distribution of GOP sizes. Concept 4: Flying-box data augmentation As described above in connection with concept 1, AI-based compression pipelines frequently struggle to handle high motion scenes well. That is, they struggle to achieve compression rates competitive with traditional codecs and assign a disproportionately large number of bits to the compression of high motion scenes, and/or they struggle to achieve reconstruct the frames of the scene with high accuracy, resulting in poor distortion scores. Concept 4 is directed to a data augmentation technique that has been found to significantly improve the ability of AI-based compression pipelines to learn how to handle high motion scenes, resulting in better compression rates and distortion scores. It is noted for completeness that improving the ability to learn may be understood as meaning that, for a given AI-based compression pipeline, the rate and distortion losses after training converge to lower values than that pipeline is able to achieve without using the methods of concept 4. Figure 8 illustrates a sequence of frames 800 of an image sequence that may be part of a training data set for training an AI-based compression pipeline, for example the pipeline of Figure 3. The sequence of frames 800 contains high motion features: a figure 801 moves from one side of the scene to the other side of the scene within two frames. This kind of high motion feature is difficult for AI-based compression pipelines to learn to handle well. Whilst it is in principle possible to bias a training data set with a higher proportion of this kind of high motion scene data, the inventors have found that an AI-based compression pipeline that is the result of training on a high-motion biased training data set will improve its rate distortion scores on such high-motion scenes but at the expense of very poor performance on static scenes that were previously not a challenge. The inventors have instead realised that taking a balanced training data sets (i.g. a data set that is not specifically biased towards a higher proportion of high motion scenes) and modifying some or all of the scenes to include a synthetically introduced box or patch that moves across the scene, for example gradually at some predetermined speed in some predetermined direction or path, or for example appearing and disappearing at different places in the frames, has the effect of improving high motion scene rate distortion scores when the data set is used for training, without negatively impacting the low motion scene rate distortion scores in the same way that biasing the balanced training data set towards a higher proportion of natively high motion scenes. The term box is used for ease of understanding but it will be appreciated that any shape of any size may be used. In general terms, it is understood that the synthetic enhancement of the training data that introduces "flying" boxes into the scenes but leaves the scenes otherwise unmodified, facilitates the learning of high motion features (i.e. the flying box is an "easy-to-learn", high motion feature) in scenes which are otherwise low motion. For example, an almost perfectly static nature panorama scene is very low motion. A "flying" box that starts in one corner of the scene and moves to another corner at some predetermined speed as the frame sequence progresses is a very simple, but high motion feature effectively super imposed onto the static nature scene. The resulting scene thus simultaneously comprises easy-to-learn low motion features (the pixel distributions of an almost unchanging static nature scene), and easy-to-learn high motion features (the pixel distributions of a simple, synthetic "flying" box). This synthetically enhanced scene can be contrasted with native, high motion training data scenes (e.g. an action scene from a movie with explosions and debris) where the high motion features are significantly more complex and difficult to learn, and do not frequently appear at same time as the kind of pixel distributions that are present in low motion scenes. That is, the distributions of pixels (and any associated optical flow information) in natively high motion scene are distinct from distributions of pixels and optical flow information in natively low motion scenes. Biasing a training data set towards the natively high motion scenes results in the distributions of the low motion scenes becoming under-represented, resulting in reduced performance on such scenes. This problem is at least partially solved through the introduction of a synthetic "flying" box across the training data set because, unlike biasing the data set with natively high motion scenes, the "flying" box does not significantly alter the underlying distribution of the low motion scenes onto which the "flying" box is super-imposed, thus ensuring that these types of distributions do not become under-represented in the training data. Thus, the properties of the "flying" boxes (e.g. their starting positions, movement speed and so on) may be based be based on the balance of motion in the underlying training data set. For example, if the underlying training data set has few high motion image sequences (for example, optical flow values across the data set are in aggregate below some threshold), then it may be advantageous to increase the movement speed of the "flying" boxes, and vice versa. When the neural networks of an AI-based compression pipeline are then trained on such a "flying" box enhanced training data set, the inventors have found that the trained network has improved rate distortion scores on high motion scenes (compared to training on a non-enhanced data set) without significantly negatively impacting rate distortion scores on low motion scenes. This can be contrasted with simply biasing a training data set with more natively high motion scenes which the inventors have found in many cases catastrophically reduces performance on low motion scenes. Thus training data augmentation using "flying" boxes allows the networks to better learn how to simultaneously handle both high and low motion scenes. It will be appreciated that the "flying" box enhancement may be introduced into some or all of the image sequences of a training data set. For example, it is envisaged that it may be introduced only on scenes where some motion requirement is considered to be met. For example, where motion is below a threshold motion amount based on optical flow or some other pixel difference metric. Or it may be introduced based on some other requirement (e.g. predetermined scene type such as screen capture content, action content, or some other content), or indeed it may be introduced on in all scenes in the training data set. Figure 9 illustrates an image sequence 900 that may be used in a training data set showing a static nature scene into which a "flying" box has been introduced. It will be appreciated that Figures 8 and 9, for illustrative purposes, show only three frames of the respective sequences but that in practice each image sequence may GOP sizes of any number and the "flying" boxes (or indeed other shapes) may be introduced onto some or all of the frames of a given GOP. An illustrative method of adding "flying" boxes, in this case black pixel rectangles, to a sequence of images is set out in the pseudocode below:
Algorithm 5 Enhance training image sequence with flying box Inputs: sequence, rectangle_width, rectangle_height, start_x, start_y, speed_x, speed_y Outputs: output_sequence Variables: current_x, current_y, new_frame current_x← start_x current_y← start_y for each frame in sequence do new_frame ← copy(frame) for y from current_y to (current_y + rectangle_height) do for x from current_x to (current_x + rectangle_width) do if x is within frame width and y is within frame height then set_pixel(new_frame, x, y, color.BLACK) end if end for end for Add new_frame to output_sequence current_x ← current_x + speed_x current_y ← current_y + speed_y end for Return: output_sequence That is, an input image sequence is provided, and the flying box width, height, start x and y pixel coordinate position and movement speed in each direction is specified. Then, for each frame in the input image sequence, the frame is copied and the pixels at the position where the flying box is set to be are set to black, or some other colour, to introduce the flying box into the frame. The position of where the flying box will be in the next frame is then determined by considering the current positions and speeds, and then the process is repeated for all the frames of the sequence. A further optional enhancement of the above methods is the introduction of a non-uniform texture or pattern into the pixels of the flying box. The inventors have unexpectedly found that introducing image features that AI-based compression pipelines in many cases have difficulty with into the pixels of the flying boxes results in the same effect of improving the rate distortion scores of the pipeline on frames with those features importantly without negatively impacting rate distortion scores associated with other types of scene features. One specific example of this effect is the introduction of text (e.g. alpha numeric or other characters, typed or hand written) into the flying boxes. In a similar manner as described above, it is understood that synthetically introducing such features onto scenes where such features would not typically be expected to be found has the effect of allowing the networks to learn such to effectively compress and reconstruct such features without negatively impacting the networks’ abilities to effectively compress and reconstruct the features of the rest of the scene onto which the flying box is introduced. That is, in general terms the training data set substantially retains whatever balance of pixel distributions it originally had while the the introduction of the textured or patterned flying box becomes an easy-to-learn feature superimposed on those pixel distributions. Figure 10 illustrates an example image sequence 1000 onto which a flying box 1001 containing alpha numeric characters has been introduced onto a scene, in this case a high motion scene of a rocket during lift off as it moves across the scene. As explained above, the pixel distribution of the majority of the scene remains consistent with the natively captured footage of the scene while super imposing the "flying" box with text onto the scene introduces an easy-to-learn high motion feature, as well as an easy to learn alpha-numeric feature. Training an AI-based compression pipeline on such data results in a set of networks that have improved rate distortion scores on high motion scenes, low motion scenes, and scenes that include alphanumeric text compared to networks trained on the same data set but without such enhancements. The alpha-numeric character example has been found to be particularly effective at improving the rate distortion performance of AI-based compression pipelines on screen capture image sequences such as screen sharing by participants in a video call, and such as video game streamers who may quickly switch between browser screens and have various overlays on their streams, many of which contain text, without significantly reducing the rate distortion performance of the pipeline on other types of content such as movies and so on. In very general terms, the inventors have found that applying the "flying" box methodology to training data enhancement may be seen as a method of blending specific training data types (e.g. certain pixel distributions associated with certain scene and content types, and or motions associated therewith) into an underlying, balanced training dataset without negatively effecting the balance of that underlying training dataset. Thus the networks trained thereon, retain the good across-the-board performance of the balanced training data while boosting the performance on the specific data types blended into it using the "flying" boxes. In the specific case of video compression, this effect is particularly advantageous because it allows a single model (i.e. the weights and biases and so on of the neural networks of the pipeline) to be performant across a much higher number of different use cases, thereby reducing the memory requirements of the pipeline when it is deployed as a codec on edge devices. This is a significantly different approach to the paradigm of using multiple different models (i.e. sets of weights) and selecting that model which is finetuned specifically for good performance on a given image sequence type. In a further optional enhancement, it is envisaged that the texture of the "flying" box may be sourced from other image sequences in the underlying training data set. That is, rather than synthetically creating the pixel values of the "flying" box, they may instead be sampled, for example randomly, from image patches of the training data set into which the "flying" boxes are being introduced. That is, for a given image sequence A, a patch of pixels from one or more frames of image sequence B may be sampled and used as the texture of the "flying" box added to image sequence A. This approach increases the variance of the pixel distribution of the "flying" box, which can help to prevent overfitting that may occur when solid colour block "flying" boxes are used. In a further optional enhancement, it is envisaged that the motion of the "flying" box across the frames of the sequence may be different between sequences. That is, it is envisaged that "flying" boxes together having a distribution of different motion amounts, directions and speeds may be introduced into the sequences to increase the variance of the pixel distributions attributed to the "flying" boxes. As above, this can help to prevent overfitting that may occur when all "flying" boxes across all sequences have the same or similar motion. In general terms, training the neural networks of an AI-based compression pipeline may be defined by the following pseudocode: Algorithm 6 Training AI-based compression pipeline using flying box data augmentation Inputs: Training dataset X, learning rate ^^, regularization parameter ^^, number of epochs ^^ , network architecture ^^^^ Initialize network parameters ^^ for epoch = 1 to ^^ do for each batch in X do Add flying boxes to (^^^^−1, ^^^^ ) and perform forward pass to compute predictions ^^^^ = ^^^^ (^^^^−1, ^^^^ ) Compute loss Loss = ^^ + ^^^^ Backward pass to compute gradients ∇^^Loss Update parameters with optimizer O: ^^ ← O(^^, ∇^^Loss, ^^) end for Optionally evaluate on validation set end for That is, a training data set X, a learning rate ^^, regularisation parameter ^^, and a number of training steps or epochs ^^ is selected. The network architecture of ^^^^ is defined, for example as shown in Figure 3. The network parameters ^^ are randomly initialised and then the training loop is started. For each batch in the training data X, flying boxes are introduced as described above, and the forward pass is computed for the network being trained ^^^^ . The total ^^^^^^^^ is calculated by combining a distortion term ^^ and a rate term ^^, and any other loss terms (not shown). The backwards pass is then performed to compute gradients based on the loss, and the parameters ^^ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser. Optionally, a validation loss can be calculated and the training loop is repeated until the predetermined number of steps or epochs ^^ have been calculated, or some other criteria have been reached. The learning rate, batch size, and or number of epochs may be optimised during training, for example using a learning rate scheduler or some other hyperparameter optimisation method. More generally, the hyperparameters may be optimised experimentally. It will be appreciated that the flying boxes may be added "on the fly" or pre-computed for a given sequence and added to an input training dataset X before starting training so that the training dataset X already contains the flying boxes when training commences, in which case the "add flying boxes" step in the pseudocode above does not need to be performed again. It will be appreciated that the presently decscribed concept may be used in a standalone manner, and/or together with any of the other concepts described herein. Concept 5: Temporal frame sub-sampling As described above, AI-based compression pipelines in many cases struggle to effectively compress high motion scenes and training data augmentations, such as those of concept 4, may be used to improve the ability of the networks of the pipeline to learn to better compress such scenes. An alternative or additional approach is to increase motion variance in the image sequences of the training data during training by varying how far back (or forward) in terms of frame position in the image sequence the reference frame is taken from. That is, how far separated from the current frame the reference frame for estimating the motion against is. For example, when estimating optical flow, the optical flow is not necessarily always estimated between ^^^^ and ^^^^−1, but instead, for each current frame ^^^^ , the optical flow may instead be estimated by comparing ^^^^ with ^^^^−^^ where ^^ may be sampled from ^^ ∈ [1, 2, 3, ...]. The effect of this approach is that the networks of the pipeline learn better how to predict larger optical flows as the pixel differences between a frame at ^^ and for example ^^ − 5 are typically greater than those between a frame at ^^ and ^^ − 1. The value of ^^ may be randomly sampled from Z using any suitable method, for example using a random or pseudo-random number generator, where such sampling may be drawn from a parameterised distribution to allow for some control over the value of ^^, and/or ^^ may be sampled according to some predetermined heuristics (e.g. one or more predetermined rules). For example, certain scene types may result in lower loss values when ^^ is biased to lower values compared to higher values, and so on and accordingly the sampling of ^^ may take this into account by being biased towards lower ^^ values (for example by parameterising a distribution of values from which ^^ may be selected using a mean, variance and/or other parameters), or vice versa. Example scene types include high motion action scenes of movies, low motion nature scenes from documentaries, and so on. In any event, the result is that there is a distribution of different frame positions from which the reference frame is selected, and thus a greater distribution of motion amounts in the training data while otherwise leaving the pixel distributions of the underlying data unchanged. When adopting this approach, the inventors have found that, for a given AI-based compression pipeline, the rate and distortion losses after training converge to lower values than that pipeline is able to achieve without using the methods of concept 5. More generally, the higher the frame gap (i.e. where ^^ is large), the higher the pixel differences between ^^^^ and ^^^^−^^ are and thus the higher the motion that the network has to try to learn during training to minimise the rate distortion for. As a result, this approach has the effect of synthetically introducing motion into the training data set without otherwise changing the underlying pixel distributions of the training data. That is, it facilitates the learning of high motion without otherwise biasing an already balanced training data set with an increase of additional natively high motion image sequences, which in some cases has the effect of reducing performance on low motion scenes. The result is accordingly a network that has improved performance (better rate distortion scores) on low and high motion scenes simultaneously. It is also envisaged that, the sampling of ^^ (e.g. its biasing towards higher or lower ^^ values) may be based on the balance of motion in the underlying training data set. For example, if the underlying training data set has few high motion image sequences (for example, optical flow values across the data set are in aggregate below some threshold), then it may be advantageous to bias the sampling of ^^ towards higher values, or vice versa. Figure 11 illustratively shows this approach with reference to a sequence of frames 1100 with frame positions ^^, ^^ − 1, ^^ − 2,...,^^ − ^^. Frame ^^^^ is the current frame of a sequence being processed while ^^^^−^^ is the frame which is being used as a reference frame against which optical flow information is being determined (e.g. in a P-frame module of Figure 3). For a current frame of the image sequence being fed into the pipeline e.g. ^^^^ , a random integer ^^ is generated and ^^ is set to the value of ^^ and used to select the reference frame ^^^^−^^ against which ^^^^ will be compared. For example, if ^^ = 1, then the reference frame input into the P-frame module of the pipeline will be ^^^^−1 whereby, if the operation of the networks of the optical flow part of the pipeline are defined as a function ^^, then: ^^ ^^^^^^ = ^^(^^^^ , ^^^^−1) Or, for example, if ^^ = 4, then: ^^ ^^^^^^ = ^^(^^^^ , ^^^^−4) and so on up to the general case of: ^^ ^^^^^^ = ^^(^^^^ , ^^^^−^^) where ^^ ∈ Z. In some implementations ^^ may be randomly generated from a subset of Z, for example ^^ ∈ [1, 2, 3, 4, 5, ..., 60]. Whilst not shown in Figure 11, it is also envisaged that, when using B-frames, ^^ may take on negative values so that the reference frame is selected from e.g. ^^ + 1, ^^ + 2, ... ^^ + ^^. Once the processing of the current frame ^^^^ is complete, ^^^^+1 becomes the current frame and the process is repeated for ^^^^+1 and so on until the entire sequence of frames (which may make up a GOP) has been processed. An example training implementation is illustrated in the pseudocode provided below. Algorithm 7 Training AI-based compression pipeline using temporal frame sub-sampling Inputs: Training dataset X, learning rate ^^, regularization parameter ^^, number of epochs ^^ , network architecture ^^^^ Initialize network parameters ^^ for epoch = 1 to ^^ do for each batch in X do Generate ^^ ∈ Z and perform forward pass to compute predictions ^^^^ = ^^^^ (^^^^−^^ , ^^^^ ) Compute loss Loss = ^^ + ^^^^ Backward pass to compute gradients ∇^^Loss Update parameters with optimizer O: ^^ ← O(^^, ∇^^Loss, ^^) end for Optionally evaluate on validation set end for That is, a training data set X, a learning rate ^^, regularisation parameter ^^, and a number of training steps or epochs ^^ is selected. The network architecture of ^^^^ is defined, for example as shown in Figure 3. The network parameters ^^ are randomly initialised and then the training loop is started. For each batch in the training data X, an ^^ value is generated and used to select the reference frame ^^^^−^^, and the forward pass is computed for the network being trained ^^^^ . The total ^^^^^^^^ is calculated by combining a distortion term ^^ and a rate term ^^, and any other loss terms (not shown). The backwards pass is then performed to compute gradients based on the loss, and the parameters ^^ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser. Optionally, a validation loss can be calculated and the training loop is repeated until the predetermined number of steps or epochs ^^ have been calculated, or some other criteria have been reached. The learning rate, batch size, and or number of epochs may be optimised during training, for example using a learning rate scheduler or some other hyperparameter optimisation method. More generally, the hyperparameters may be optimised experimentally. It will be appreciated that the presently decscribed concept may be used in a standalone manner, and/or together with any of the other concepts described herein. Concept 6: Flow teacher ensemble As has been described generally above, successfully training the networks of an AI-based compression pipeline, such as that of Figure 3, to perform well simultaneously on high and low motion image sequences can be challenging. One area of focus of such training is the training of the flow part of the P-frame module of such a pipeline. By way of illustrative example, this may refer to the training of one or more of the networks of the flow module 206 in Figure 3. As described generally above, the distortion term(s) of a rate distortion loss function Loss = ^^ +^^^^ are typically based on a difference between a ground truth input frame, for example ^^^^ and the reconstructed version of that frame ^^^^ , that is: Loss = ^^ (^^^^ , ^^^^) + ^^^^ When considering the training of the networks of the flow module 206 specifically, the loss function may incorporate one or more additional terms, that are based on a difference between some ground truth optical flow information ^^^^^^ , and the optical flow information ^^ that a given forward pass of the networks produce during a training step. For example: Loss = ^^ (^^^^ , ^^^^) + ^^( ^^^^^^ (^^^^ , ^^^^−1), ^^ (^^^^ , ^^^^−1)) + ^^^^ Whilst not shown for ^^ and ^^, each of the loss terms may be regularised relative to each other using a regularisation parameter such as ^^. Unlike in the case of the basic distortion loss term where the ground truth may simply be the input image ^^^^ , ground truth optical flow information ^^^^^^ is not readily available and must be estimated first from the input image ^^^^ and a reference frame ^^^^−1. However, there are many different methods of estimating optical flow, each with their own strengths and weaknesses, and each producing different optical flow maps. There is accordingly not necessarily any way to determine a single ground truth of optical flow information in the same way that there is always a ground truth input image ^^^^ that can be accessed. There are instead only estimates of ground truth optical flow information that are highly dependent on the estimation method or algorithm used. Some exemplary optical flow estimation techniques and their strengths and weaknesses are set out below. Lucas-Kanade algorithm - this algorithm relies on the assumption that the optical flow is constant within a small neighborhood of each pixel. The method solves the optical flow equations by performing a least-squares fit of the local image velocities, making it computationally efficient and relatively simple to implement. It performs well in regions with high-texture content and small movements. However, it struggles with large displacements between images and is sensitive to image noise due to its reliance on intensity gradients. Additionally, it does not handle occlusions well, where pixels may appear or disappear between frames. Horn-Schunck algorithm - this algorithm takes a global approach by computing a dense flow field over the entire image. It minimizes an energy function that includes both a data fidelity term and a smoothness constraint, resulting in a smooth flow field. The algorithm estimates motion for every pixel, providing a comprehensive motion map. While it is mathematically rigorous and offers global consistency, it may over-smooth motion boundaries, blurring fine details. It also requires careful tuning of the smoothness parameter and is more computationally intensive than local methods like Lucas-Kanade. Farnebäck’s algorithm - this algorithm models the neighborhood of each pixel using quadratic polynomials, allowing it to calculate flow in terms of displacement fields. It provides dense flow fields and is relatively fast. However, the polynomial approximation may not capture complex motions accurately, and the method requires careful selection of window sizes and other parameters to function optimally. TV-L1 optical flow algorithm - this algorithm introduces robustness to noise and illumination changes by using a total variation regularization term and an L1 norm for the data fidelity term. This variational approach preserves sharp motion boundaries and is less sensitive to outliers, making it effective in handling abrupt motion changes and varying lighting conditions. Brox’s optical flow algorithm - incorporates additional constancy assumptions, such as gradient constancy, to improve performance in challenging scenarios. It is better at capturing significant displacements and is more invariant to illumination changes due to its use of gradient information. While it offers high-quality flow estimation in complex scenes, it is sensitive to the choice of regularization parameters. SimpleFlow algorithm - this is a non-parametric algorithm that relies on color matching and spatial consistency rather than image gradients. It can handle large, non-linear motions and is less sensitive to textureless regions. However, it can be slow due to exhaustive search mechanisms and may miss subtle motions, making it less suitable for applications requiring fine detail. FlowNet - this is a convolutional neural network architecture designed for flow estimation. It is capable of modeling intricate motions learned from data but may not generalize well to data significantly different from its training data. PWC-Net - this is based on FlowNet but is an improvement over FlowNet in that it incorporates a feature pyramid and warping layers, effectively capturing large motions with better efficiency and accuracy than FlowNet. SIFT Flow algorithm - this algorithm aligns images through dense matching of Scale-Invariant Feature Transform (SIFT) descriptors. This method is robust to scale, rotation, and illumination changes, making it useful for aligning images with similar scenes but different content. Frequency domain methods - these types of methods estimate optical flow by analyzing phase information using Fourier transforms. These methods can effectively capture periodic motions and are resistant to certain types of noise. However, they may not localize motion accurately in spatial terms and struggle with non-linear or non-periodic movements. This list is not intended to be exhaustive and is provided primarily to illustrate that the there are many different ways to estimate optical flow, each of which may produce different optical flow information (e.g. a flow map). The inventors have realised that the performance on different scene types of the networks of AI-based compression pipelines that estimate flow, such as the flow module 206 in Figure 3, is correlated with the performance on those scene types of the algorithm used to estimate the ground truth flow ^^^^^^ in the loss function. For example, if ^^^^^^ is estimated using a Lucas-Kanae algorithm implementation which performs well on small local motion but not on large global motion then the networks of the flow module 206 after training also perform well on small local motion but not on large global motion. Conversely, if ^^^^^^ is estimated using a Horn-Schunck algorithm implementation which performs well on global motion, then the networks of the flow module 206 after training will also perform better on global motion but not on local motion. Concept 6 is directed to harnessing these correlations by providing a learnable or criterion- based ensemble flow estimation to use as ^^^^^^ that comprises flow information (e.g. a flow map) produced by one or more of a plurality of different flow algorithms based on whichever one or combination minimises the loss function or a specific loss term, or based on some other minimisation criteria. For example, the resulting "ground truth" flow map may be stitched together on a pixel-wise basis from pixels of whichever flow algorithm produced an optimal flow value for each pixel. Where optimal here refers to meeting some criteria such as a smallest flow pixel difference between the ^^ and ^^^^^^ , or minimises the overall loss, or one or more other specific loss terms, and so on. For example, consider a set of optical flow algorithms F = {^^1, ^^2, . .. , ^^^^} where n is the number of flow estimation algorithms included in the set. During a forward pass of the networks of the flow module, an ensemble flow may be calculated by producing a respective individual flow maps using each of the optical flow algorithms in F and combining these, for example using a weighted sum, or some any other combination By way of illustrative example, consider estimation of optical flow information ^^ such as a flow map for a given input frame ^^^^ and ^^^^−1. We can define an ensemble ^^^^^^^^ of optical flow information estimations with: ^^ (^^ , ^^ ∑^^ ^^^^^^ ^^ ^^−1) = ^^=1 ^^^^^^^^ (^^^^ , ^^^^−1) where ^^^^ is a learnable or criterion-based parameter or parameters, for example in tensor form, indicative of how much each of the different flow estimations produced by the respective optical flow algorithms in F contribute to the values that make up the ensemble flow map ^^^^^^^^. This approach also allows different optical flow algorithms to contribute different amounts to different parts of the flow map. For example, if one optical flow algorithm produces optical flow information for certain pixels or patches of pixels that results in a very low loss during a training step, the networks may update the weights when applying the optimiser to increase the contribution of this optical flow algorithm to the optical flow information for such patches by updating the values of ^^^^ accordingly. As training progresses, the networks learn by updating ^^^^ which optical flow algorithms are best applied to different pixels of different inputs to minimise the rate distortion loss. This approach effectively results in a learned teacher flow ensemble that forces the networks of the AI-compression pipeline to learn to emulate the good-performance abilities of the various optical flow algorithms of the ensemble for different scenarios on a pixel-wise basis, while simultaneously learning to not emulate the bad-performance abilities of those algorithms (because that would result in a higher loss and thus training pushes the values of ^^^^ towards minimising any contribution of bad flow values to the ensemble flow). Alternatively, ^^^^ may be criterion-based where, for each training step of the AI-based compression pipeline, the ^^^^^^ is stitched together from flow pixels selected based on whichever algorithm produced a flow value for that pixel which met some criterion (e.g. lowest difference between ^^ and ^^^^^^ , and so on). For example, consider a toy example where we consider what flow value to use for pixel (0,0) of a flow map ^^^^^^ , a first optical flow algorithm produces a flow value for that pixel which is very different from ^^ whereas a second optical flow algorithm produces a flow value for that pixel which results in a smaller difference. In this case, the flow value selected for pixel (0,0) will be that produced by the second optical flow algorithm rather than the first. This is of course only a toy example and in practice, many more optical flow algorithms may be used and the resulting final flow map may be stitched together from flow values taken from many of such optical flow algorithms. We can thus define our new loss function as: Loss = ^^ (^^^^ , ^^^^) + ^^( ^^^^^^^^ (^^^^ , ^^^^−1), ^^ (^^^^ , ^^^^−1)) + ^^^^ and: Loss = ^^ (^^^^ , ^^^^) + ^^(∑^^ ^^=1 ^^^^^^^^ (^^^^ , ^^^^−1), ^^ (^^^^ , ^^^^−1)) + ^^^^ where ^^^^ is a learnable or criterion-based parameter(s) that defines the contribution of each of the optical flow algorithms ^^^^ in F to the ensemble flow ^^^^^^^^ which acts as a "ground truth" or "teacher" optical flow against which the estimated flow ^^ may be compared when calculating the flow-specific loss term ^^ during a training step of the networks of the flow module of an AI-compression pipeline. As above, whilst not shown, each of the loss terms may be regularised relative to each other using a regularisation parameter such as ^^. Whilst the example of a weighted sum is used here, it is envisaged that any combination may be used, for example any parameterised combination based on one or more combination parameters may be used such as a parameterised linear or non-linear combination, a parameterised power law combination, a parameterised polynomial combination, a parameterised logarithmic combination and so on. This approach is also advantageous from an implementation perspective because it makes it straightforward to include any number (for example 2, 3, 4, 5 or more) and any type of optical flow algorithms (for example any of the optical flow algorithms listed above and/or any others not listed above) in F without burdensome, manual experimentation around which works best because the learnable parameter ^^^^ during training automatically learns how much of each optical flow algorithm should contribute to each pixel of a flow map for a given input ^^^^ and ^^^^−1 to achieve a given training objective (e.g. minimised rate distortion loss) for a given training data set. In very general term, this approach allows the networks to decide on their own which behaviours of which optical flow algorithms the networks of the flow module should emulate to achieve the desired minimisation of the rate distortion loss. It should be noted in contrast that a criterion-based approach may require additional work in terms of manual hand-crafting of the criterion (e.g. selecting the pixel flow values that are least different to ^^ ), the criterion-based approach is advantageous is that it does not typically training stability when training the rest of the pipeline - something which may occur when using a learnable ^^^^. An illustrative training implementation is set out in the pseudocode below:
Algorithm 8 Training AI-based compression pipeline using ensemble flow teacher Inputs: Training dataset X, learning rate ^^, regularization parameter ^^, number of epochs ^^ , network architecture ^^^^ , flow algorithm ensemble F Initialize network parameters ^^ for epoch = 1 to ^^ do for each batch in X do Calculate ^^ ∑ ^^^^^^ (^^^^ , ^^^^−1) = ^^ ^^=1 ^^^^^^^^ (^^^^ , ^^^^−1) and perform forward pass to compute predictions ^^^^ = ^^^^ (^^^^ , ^^^^−1) and ^^ (^^^^ , ^^^^−1) Compute loss Loss = ^^ (^^^^ , ^^^^−1) + ^^( ^^^^^^^^ , ^^ ) + ^^^^ Backward pass to compute gradients ∇^^Loss Update parameters with optimizer O: ^^, ^^ ← O(^^, ∇^^Loss, ^^) end for Optionally evaluate on validation set end for That is, a training data set X, a learning rate ^^, regularisation parameter ^^, a number of training steps or epochs ^^ , and a set of optical flow estimation algorithms (the flow algorithm ensemble) F is selected. The network architecture of ^^^^ is defined, for example as shown in Figure 3. The network parameters ^^ are randomly initialised and then the training loop is started. For each batch in the training data X the forward pass is computed for the network being trained ^^^^ . Additionally, the flow ensemble produces an ensemble flow estimation ^^^^^^^^ (^^^^ , ^^^^−1) = ∑^^ ^^=1 ^^^^^^^^ (^^^^ , ^^^^−1) which will later be used as a "ground truth" flow when estimating the loss function. Note that ^^ is a learnable parameter specifying how much of each flow algorithm contributes to the values of the ensemble flow map and will accordingly be updated by the optimiser when it is applied. The total ^^^^^^^^ is then calculated by combining a distortion term ^^ and a rate term ^^, and the flow loss term ^^( ^^^^^^^^, ^^ ) which is indiciative of a difference between the actual flow ^^ estimated by the networks in the current training step and the ensemble flow ^^^^^^^^. The backwards pass is then performed to compute gradients based on the loss, and the parameters ^^ and ^^ are optimised using the optimiser, such as stochastic gradient descent SGD, or some other known optimiser. Optionally, a validation loss can be calculated and the training loop is repeated until the predetermined number of steps or epochs ^^ have been calculated, or some other criteria have been reached. The learning rate, batch size, and or number of epochs may be optimised during training, for example using a learning rate scheduler or some other hyperparameter optimisation method. More generally, the hyperparameters may be optimised experimentally. As training progresses, ^^ will converge to a set of values defining the optimal composition of flow algorithms, either in aggregate or on a pixel-wise basis, that achieve a minimised rate distortion loss. In this way, an optimal flow "teacher" is learned which results in the networks of the flow module 206 better emulating the most desired behaviours from the flow algorithms in F while avoiding the least desired behaviours. It is envisaged that the optimiser may be applied at different times to both of ^^ and ^^, and to only one of them. For example, during early stages of a training schedule (e.g. for a predetermined number of steps), ^^ may be frozen in its initialisation state, for example, all algorithms contribute equally to the ensemble flow, or only one does. This may increase the initial stability of the training of ^^. Then, after a next predetermined number of training steps, ^^ may be unfrozen, allowing the composition of the flow ensemble to vary, and thus allowing ^^ and ^^ to be trained together to achieve the training objective of minimising the loss function. Other training schedules with freezing and unfreezing of ^^ and ^^ after predetermined number of steps are also envisaged. The specific details of the training schedule may be determined empirically during ablation runs. Alternatively, training can be simplified by using a criterion-based selection of the pixels used to produce ^^^^^^^^, in which case there is no need to update ^^ with the optimiser as this is instead a hard coded criterion-based parameter. Figure 12 illustratively part 1200 of the above method. A plurality of optical flow estimation algorithms ^^^^ 1201, 1202, 1203 are provided and used to calculate an output optical flow between a first frame ^^^^ and a second frame ^^^^−1. Their respective output flow maps are weighted or otherwise emphasised or de-emphasised by learned parameters ^^^^ and combined into an ensemble flow map or other representation of optical flow information ^^^^^^^^. This provides a reference "ground truth" flow for use in a flow loss term ^^ of the loss function. The optical flow part 206 of the P-frame module, comprising at least an encoder neural network 1204 and decoder neural network 1205 also performs a forward pass to produce its own estimate of optical flow ^^ from the first frame ^^^^ and the second frame ^^^^−1. As this occurs during training, the optical flow ^^ produced by the (initially untrained) flow part 206 of the P-frame module may not be particularly accurate and/or may not be entropy encodable into a particularly small bitstream size - this is something that is learned during training. The flow loss term ^^ is estimated from the the optical flow ^^ from the forward pass of the untrained networks of the flow part 206 and from the ensemble flow ^^^^^^^^. As described above, the flow loss term ^^ may be a simple difference metric such as a mean sqaured error, or other difference metric and/or it may comprise a discriminator loss estimate, whereby the generator is the flow part 206 performing its forward pass and the discriminator may comprise a neural network trained to distinguish ^^ from ^^^^^^^^. The rate term ^^ and the distortion term ^^ may then be calculated in the usual way as described earlier herein to produce the overall loss ^^, on which the updates to the weights of the neural networks of the AI-based compression pipeline may be based. Whilst the above described example describes a learned combination of flow algorithms, the criterion-based approach is also envisaged as a more easily implemented approach given the training instability that a learned ^^ can introduce. That is, rather than a learned parameter that determines how much of each flow algorithm contributes to the flow maps, instead each flow algorithm computes a flow and then the choice of which teacher to use is made (pixel-wise) by applying a criterion. For example, whichever teacher minimises one or more loss terms of the loss function or the loss function overall is the one that is selected. In this way, the resulting optical flow information is still "combined" in the sense that the flow algorithm choice is still a pixel-wise choice resulting in an overall optical flow map which is the aggregation of all the individual flow pixels from a chosen algorithm for that pixel. In very general terms the final flow map may be said to be stitched together from flow pixels taken from whichever algorithm resulted in most closely matching a given criterion (e.g. minimising a specific loss term such as a rate term or distortion term or some other criterion). Thus, the final flow comprises a weighted combination where one weight is 1 and the others are 0 without any learned parameters. This implementation is substantially simpler and avoids any training instability that a learned combination can in some cases introduce when training the pipeline. It will be appreciated that the presently described concept may be used in a standalone manner, and/or together with any of the other concepts described herein. The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal. The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. The subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. While this specification contains many specific implementation details, these should be construed as descriptions of features that may be specific to particular examples of particular inventions. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable subcombination. Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Finally, for completeness, whilst a short GOP is described herein as a GOP of length under 10 frames and a long GOP is a GOP of 10 frames or greater. The advantages of each concept becomes significantly more apparent and visible as GOP length increases. For example, at 30 frames, 60 frames, 90 frames, 120 frames and so on, training time and stability is significantly worse without using the presently described concepts. Further, artefacts and errors produced by networks trained on long GOPs without using the presently described concepts are significantly more apparent at GOP lengths of 30, 60, 90, 120, and greater numbers of frames. Accordingly, whilst the present disclosure is presented in the context of the boundary between long and short GOP terminology is 10 frames, it is also envisaged that the boundary between long and short GOP terminology may be considered at 30, 60, 90, 120 frames or other frames of these orders of magnitude or greater.

Claims

1. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; producing a masked sequence of input images by masking a portion of one or more of the input images; with a first neural network, for a pair of input images of the masked sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for a plurality of further sequences of input images to produce first and second trained neural networks.
2. The method of claim 1, wherein masking comprises selecting a portion of a first image of the first sequence, and replacing pixels in the other images of the first sequence using the selected portion. 3. The method of claim 2, wherein an unselected portion of the first image of the first sequence remains unmasked in the first sequence of input images. 4. The method of claim 3, wherein a position and size of the portion for first sequence is different to the position and size of the position for one or more of the the further sequences. 5. The method of claim 4, wherein the position and size of the portions for the first sequence and the further sequences are distributed according to a predetermined distribution. 6. The method of claim 5, wherein the distribution is a uniform distribution. 7. The method of claim 5, wherein the distribution is a Gaussian distribution with a mean at a center coordinate of the input images. 8. The method of claim 5, wherein the distribution is based on a type of the input images. 9. The method of claim 8, wherein the type comprises at least one of: webcam video frames, static scene frames, video game frames, high motion video frames, low motion video frames. 10. The method of any of claims 1 to 9, wherein the function is based on a difference between the first sequence of output images and the masked sequence of input images.
11. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to the method of any of claims 1 to 10. 12. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 1 to 10.
13. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to the method of any of claims 1 to 10. 14. A data processing apparatus configured to perform the method of any of claims 1 to 13. 15. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 1 to 13. 16. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 1 to 13. 17. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; repeating the above steps for one or more further sequences of input images; accumulating gradients associated with said evaluating of the function for the first sequence and for the one or more further sequences; updating the parameters of the first neural network and the second neural network based on the accumulated gradients; and repeating the above steps for a plurality of said sequences of input images to produce first and second trained neural networks. 18. The method of claim 17, wherein the first sequence of input images and the one or more further sequences of input images comprise different images. 19. The method of claim 17 or 18, wherein a final hidden state of the first and second neural networks after evaluating the function for the first sequence is used as a starting hidden state of the first and second neural networks when repeating the steps for one of the one or more further sequences. 20. The method of any of claims 17 to 19, wherein the first sequence comprises 10 or more input images.
21. The method of any of claims 17 to 20, wherein the first sequence comprises 30 or more input images. 22. The method of any of claims 17 to 21, wherein the first sequence comprises 60 or more input images. 23. The method of any of claims 17 to 22, wherein a number of input images in the first sequence of input images is different to a number of frames in at least one of the one or more further sequences of input images. 24. The method of claim 23, wherein the number of input images in the one or more further sequences of input images is based on a difference between the first output image and the one of the images of the pair. 25. The method of claim 23, wherein the number of input images in the one or more further sequences is based on a difference between the first sequence of output images and the first sequence of input images. 26. The method of claim 23, wherein the number of input images in the one or more further sequences is based on an output of a classifier configured to detect an artefact in the first output image. 27. The method of any of claims 17 to 26, wherein the first sequence and the further sequences comprise a batch, and repeating said steps for a plurality of batches.
28. The method of any of claims 17 to 27, wherein said updating the parameters comprises applying an optimiser to weights of the first and second neural networks based on said accumulated gradients. 29. The method of any of claims 17 to 28, wherein the accumulated gradients are associated with a greater number of input images than a number of input images in the first sequence or a number input images in any one of the one or more further sequences. 30. The method of claim 17 to 29, comprising repeating said steps a first number of times using sequences having a first number of input images, and repeating said steps a second number of times using one or more further sequences having a second, greater number of input images. 31. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to the method of any of claims 17 to 30. 32. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 17 to 30. 33. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to the method of any of claims 17 to 30. 34. A data processing apparatus configured to perform the method of any of claims 17 to 33.
35. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 17 to 33. 36. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 17 to 33. 37. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; with a second neural network, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; evaluating a function based on a difference between the first sequence of output images and the first sequence of input images; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for a plurality of further sequences of input images to produce first and second trained neural networks, wherein at least some further sequences comprise a greater number of input images than the first sequence of input images.
38. The method of claim 37, wherein the first sequence of input images comprises fewer than 10 input images. 39. The method of claim 37 or 38, wherein the at least some further sequences each comprise 10 or more input images. 40. The method of claim 37 or 38, wherein the at least some further sequences each comprise 30 or more input images. 41. The method of claim 37 or 38, wherein the at least some further sequences each comprise 60 or more input images. 42. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce a first output image, wherein the first output image is an approximation of one of the images of the pair; repeating the above steps to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the first neural network and the second neural network are produced according to the method of any of claims 37 to 41. 43. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first sequence of input images at a first computer system; with a first neural network, for a pair of input images of the first sequence, producing a latent representation of a difference between the images of the pair; transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 37 to 41. 44. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at a second computer system, decoding a latent represen- tation to produce a first output image, wherein the first output image is an approximation of one image of an image pair of a first sequence of input images; repeating the above step to produce a first sequence of output images, the first sequence of output images being an approximation of the first sequence of input images; wherein the second neural network is produced according to the method of any of claims 37 to 41. 45. A data processing apparatus configured to perform the method of any of claims 37 to 44.
46. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 37 to 44. 47. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 37 to 44. 48. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; modifying the pixels of the first image and the second image at respective first and second coordinates to introduce a pixel representation of an object into the first image and the second image; with a first neural network, producing a latent representation of a difference between the first image and the second image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks.
49. The method of claim 48, wherein the first and second coordinates are based on a predetermined motion path of the object between the first image and the second image. 50. The method of any of claims 48 to 49, wherein the one or more sequences of images comprise a predetermined balance of image sequences having optical flows above a threshold and image sequences having optical flows below the threshold. 51. The method of claim 50, wherein said modifying the pixels is performed on one or more of said image sequences having optical flows below the threshold. 52. The method of any of claims 48 or 51, wherein pixels of the object define one or more alphanumeric characters. 53. The method of any of claims 48 to 51, wherein pixels of the object define a non-uniform texture. 54. The method of any of claims 48 to 51, wherein pixels of the object define a solid colour block. 55. The method of any of claims 48 to 51, wherein pixels of the object are based on an image patch from an image sequence of the sequences of image different to the image sequence comprising the first image and the second image. 56. The method of any of claims 48 to 55, wherein pixels of the object define a box shape.
57. The method of any of claims 48 to 55, wherein the pixels of the object define a circular shape. 58. The method of any of claims 48 to 55, wherein the pixels of the object define a polygonal shape. 59. The method of any of claims 48 to 58, wherein the first and second coordinates for a first image sequence of the one or more sequences of images are different to the first and second coordinates for a second image sequence of the one or more sequences of images, and wherein said modifying is based on the threshold. 60. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 48 to 59.
61. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 48 to 59. 62. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 48 to 59. 63. A data processing apparatus configured to perform the method of any of claims 48 to 62. 64. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 48 to 62. 65. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 48 to 62.
66. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a plurality of second images at a first computer system; selecting one of the plurality of second images as a reference image; with a first neural network, producing a latent representation of a difference between the first image and the reference image; with a second neural network, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the output image image and the first image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks. 67. The method of claim 66, wherein said selecting comprises sampling an image from the plurality of second images. 68. The method of claim 66, wherein said selecting comprises selecting one of the plurality of second images from the image sequence based on one or more predetermined rules. 69. The method of any of claims 66 to 68, wherein the first image and the plurality of second images comprise consecutive images of an image sequence.
70. The method of claim 69, wherein the selected one of the plurality of second images is a positioned in the sequence before the first image. 71. The method of claim 69, wherein the selected one of the plurality of second images is a positioned in the sequence after the first image . 72. The method of any of claims 66 to 71, comprising biasing said selecting towards a beginning of the image sequence. 73. The method of any of claims 66 to 71, comprising biasing said selecting towards an end of the image sequence. 74. The method of claim 72 or 73, wherein said biasing is based on a property of the first image and/or the plurality of second images. 75. The method of claim 74, wherein said property is a scene type of the first image and/or the plurality of second images. 76. The method of any of claims 66 to 75, wherein the one or more sequences of images comprise a predetermined balance of image sequences having optical flows above a threshold and image sequences having optical flows below the threshold, and wherein said selecting is based on the threshold. 77. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 66 to 76. 78. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 66 to 76. 79. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 66 to 76. 80. A data processing apparatus configured to perform the method of any of claims 66 to 79. 81. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 66 to 79. 82. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 66 to 79. 83. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; estimating first optical flow information and second optical flow information, each being indicative of a difference between the first image and the second image; producing combined optical flow information from the first optical flow information and second optical flow information; with a first neural network, producing third optical flow information indicative of a difference between the first image and the second image; with a second neural network, decoding the third optical flow information to produce an output image, wherein the output image is an approximation of the first image; evaluating a function based on a difference between the combined optical flow information and the third optical flow information; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps for one or more sequences of images to produce first and second trained neural networks. 84. The method of claim 83, wherein producing the combined optical flow information comprises producing a parameterised combination of the first optical flow information and the second optical flow information using one or more combination parameters. 85. The method of claim 84, comprising updating the combination parameters based on one or more terms of the evaluated function. 86. The method of claim 85, wherein the parameterised combination comprises a weighted sum of the first optical flow information and the second optical flow information and the combination parameters comprise one or more weights of the weighted sum. 87. The method of any of claims 84 to 86, wherein the parameterised combination comprises a pixel-wise parameterised combination of the first optical flow information and the second optical flow information. 88. The method of any of claims 83 to 87, wherein the function is further based on a rate term and a distortion term.
89. The method of any of claims 83 to 88, wherein the difference between the combined optical flow information and the third optical flow information comprises a mean squared error. 90. The method of any of claims 83 to 89, comprising estimating the first optical flow information using a first optical flow algorithm, and estimating the second optical flow information using a second optical flow algorithm. 91. The method of claim 90, wherein the first optical flow algorithm and the second optical flow algorithm comprise an ensemble of optical flow algorithms. 92. The method of claim 90 or 91, comprising estimating further optical flow information using a further optical flow algorithm, and producing the combined optical flow information from the first optical flow information, the second optical flow information and the further optical flow information. 93. The method of any of claims 83 to 92, comprising estimating the difference between the combined optical flow information and the third optical flow information using a discriminator function. 94. A method of performing lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; transmitting the latent representation to a second computer system; with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of the first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 83 to 93. 95. A method of performing lossy image or video encoding and transmission, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of a difference between the first image and the second image; and transmitting the latent representation to a second computer system; wherein the first neural network is produced according to the method of any of claims 83 to 93. 96. A method of performing lossy image or video decoding, the method comprising the steps of: with a second neural network, at the second computer system, decoding the latent representation to produce an output image, wherein the output image is an approximation of a first image; wherein the first neural network and the second neural network are produced according to the method of any of claims 83 to 93.
97. A data processing apparatus configured to perform the method of any of claims 83 to 96. 98. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 83 to 96. 99. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer carry out the method of any of claims 83 to 96.
PCT/EP2025/059249 2024-04-05 2025-04-04 Method and data processing system for lossy image or video encoding, transmission and decoding Pending WO2025210218A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
GB202404867 2024-04-05
GB2404867.0 2024-04-05
GB2415182.1 2024-10-16
GBGB2415182.1A GB202415182D0 (en) 2024-10-16 2024-10-16 Method and data processing system for lossy image or video encoding, transmission and decoding

Publications (1)

Publication Number Publication Date
WO2025210218A1 true WO2025210218A1 (en) 2025-10-09

Family

ID=95519128

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2025/059249 Pending WO2025210218A1 (en) 2024-04-05 2025-04-04 Method and data processing system for lossy image or video encoding, transmission and decoding

Country Status (1)

Country Link
WO (1) WO2025210218A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021220008A1 (en) 2020-04-29 2021-11-04 Deep Render Ltd Image compression and decoding, video compression and decoding: methods and systems

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021220008A1 (en) 2020-04-29 2021-11-04 Deep Render Ltd Image compression and decoding, video compression and decoding: methods and systems

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
AGUSTSSON, EMINNEN, D.JOHNSTON, NBALLE, J.HWANG, S. JTODERICI, G.: "Scale-space flow for end-to-end optimized video compression", IN PROCEEDINGS OF THE IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2020, pages 8503 - 8512
BALLÉ, JOHANNES ET AL.: "Variational image compression with a scale hyperprior", ARXIV PREPRINT ARXIV: 1802.01436, 2018
HABIBIAN AMIRHOSSEIN ET AL: "Video Compression With Rate-Distortion Autoencoders", 2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, 27 October 2019 (2019-10-27), pages 7032 - 7041, XP033723097, [retrieved on 20200224], DOI: 10.1109/ICCV.2019.00713 *
LI XIN ET AL: "Self-Supervised Tracking via Target-Aware Data Synthesis", IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, IEEE, USA, vol. 35, no. 7, 2 January 2023 (2023-01-02), pages 9186 - 9197, XP011975176, ISSN: 2162-237X, [retrieved on 20230102], DOI: 10.1109/TNNLS.2022.3231537 *
MENTZER, FAGUSTSSON, EBALLE, J.MINNEN, DJOHNSTON, N.TODERICI, G: "Neural video compression using gans for detail synthesis and propagation", IN COMPUTER VISION-ECCV 2022: 17TH EUROPEAN CONFERENCE, TEL AVIV, ISRAEL, 23 October 2022 (2022-10-23), pages 562 - 578
POURREZA, R.COHEN, T.: "Extending neural p-frame codecs for b-frame coding", IN PROCEEDINGS OF THE IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2021, pages 6680 - 6689
YILMAZ M AKIN ET AL: "End-to-End Rate-Distortion Optimization for Bi-Directional Learned Video Compression", 2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), IEEE, 25 October 2020 (2020-10-25), pages 1311 - 1315, XP033868936, [retrieved on 20200909], DOI: 10.1109/ICIP40778.2020.9190881 *

Similar Documents

Publication Publication Date Title
TWI850806B (en) Attention-based context modeling for image and video compression
US12108050B2 (en) Method, an apparatus and a computer program product for video encoding and video decoding
US12113985B2 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
US12483715B2 (en) Method, an apparatus and a computer program product for video encoding and video decoding
US11570465B2 (en) Machine-learned in-loop predictor for video compression
US12394100B2 (en) Video coding using camera motion compensation and object motion compensation
US11936866B2 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
US20130279598A1 (en) Method and Apparatus For Video Compression of Stationary Scenes
KR20230070492A (en) Decoding with signaling of feature map data
EP4666579A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
Xiang et al. Generative adversarial networks based error concealment for low resolution video
WO2025082896A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding using image comparisons and machine learning
WO2025073564A1 (en) Method and data processing system for lossy image or video compression
JP2025532413A (en) Neural network with variable number of channels and method of operation thereof
EP4655942A1 (en) Method and apparatus for video frame synthesis
WO2025172429A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
WO2025168485A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
Ajal et al. Scalable and Resolution Data Analysis of Image and Video Compression using DL-CNNS Neural Network
WO2025162929A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
WO2025242587A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
WO2025061586A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
WO2025196024A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
WO2026082860A2 (en) Method and data processing system for lossy image or video encoding, transmission and decoding
Cao et al. Saliency-Based Neural Representation for Videos
WO2025088034A1 (en) Method and data processing system for lossy image or video encoding, transmission and decoding

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25721157

Country of ref document: EP

Kind code of ref document: A1