JP2023063166A - Compression apparatus, decompression apparatus, and method of generating model - Google Patents

Abstract

To improve accuracy and efficiency in compression of high-dimensional data.SOLUTION: A compression apparatus includes a memory storing parameters of a first model, a second model, and a third model, and a processor configured to form the first model, the second model, and the third model on the basis of the parameters stored in the memory. The processor divides a first image and a second image one unit time before into tiles, the tiles of the first and second images into the first model, acquires information on discretized latent variables of the tiles of the first and second images, inputs the information on the discretized latent variables of the tiles of the second image into the third model, acquires a probability distribution of the discretized latent variables with respect to the latent variables of the tiles of the first image, encodes the discretized latent variables of the tiles of the first image on the basis of the probability distribution, and stores encoding information in the memory in association with time information of the first image.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to compressors, decompressors and methods of generating models.

Moving image compression technology is an important technology in the age of low communication speeds, and has been developed to improve the accuracy and compression rate of image reconstruction. Even today, when communication speeds are becoming more stable and faster, the development of semiconductor technology that enables the output of high-resolution images such as 4K and 8K monitors has made video compression an important technology. Improvements in the accuracy and compression ratio of . The same can be said for not only communication speed but also storage performance.

As a technique for compressing a moving image, there is a technique for conditionally entropy-encoding an image for each frame of the moving image. There is also a technique for reproducing the color of a pixel at coordinates in the original image using latent variables encoded from the image. Therefore, it is still necessary to avoid the above-mentioned problems of reproduction of high-resolution images and videos in communication and storage, data volume, and the like, and further improvement of moving image compression technology is desired.

J. Liu, et. al., “Conditional Entropy Coding for Efficient Video Compression,” arXiv:2008.09180, 20 August 2020, https://arxiv.org/abs/2008.09180v1 I. Mehta, et. al., “Modulated Periodic Activations for Generalizable Local Functional Representations,” arXiv:2104.03960, 8 April 2021, https://arxiv.org/abs/2104.03960v1

This disclosure provides data compression techniques that improve the accuracy and efficiency of compression of high-dimensional data.

According to one embodiment, the compressor comprises one or more memories storing parameters of the trained first, second and third models, and parameters stored in the one or more memories. and one or more processors capable of forming said first model, said second model or said third model on the basis of. The one or more processors divide a first image and a second image one unit time before the first image into tiles, and divide the tiles of the first image and the tiles of the second image into the first model. input to obtain information of the discretized latent variables of the tiles of the first image and of the tiles of the second image; 3. inputting into the model, obtaining a probability distribution of the discretized latent variables for the latent variables of the tiles of the first image; obtaining the discretized latent variables of the tiles of the first image based on the probability distribution; A variable is encoded, a code is obtained, and information of the code is stored in the one or more memories in association with information regarding the time of the first image.

According to one embodiment, the decompressor comprises one or more memories storing parameters of the third model and the fourth model trained by the trained training device and stored in the one or more memories and one or more processors operable to form the third model or the fourth model based on the parameters. The one or more processors receive compressed data containing encoded information about the discretized latent variables of the tiles of the first image, previously obtained one unit time before the first image. Input the discretized latent variables of the second image into the third model, obtain a probability distribution of the discretized latent variables of the tiles of the first image, and use the probability distributions to process the compressed data decoding the information in which the discretized latent variables of the tiles of the first image contained in are encoded, inputting the decoded latent variables into the fourth model, and inputting the decoded latent variables of the tiles of the first image Get the color value.

According to one embodiment, it comprises one or more memories and one or more processors. The one or more processors divide a first image and a second image one unit time before the first image into tiles, and convert the discretized latent variables of the tiles into estimating, from the position in the image of the tile of the first image and the latent variable, estimating the color value of the pixel corresponding to the position using a fourth model, from the first image and the second image, estimating a hyperprior code for the probability distribution of the latent variable corresponding to the first image using a second model; A probability distribution of a latent variable of a tile image is estimated using a third model, the color value estimated using the fourth model, the correct color value estimated using the second model, and the color value estimated using the fourth model. the first model and the first 2 models, said third model and said fourth model are trained.

Each model trained by this training device may be used in the compression device and decompression device described above.

The block diagram which shows an example of the compression apparatus which concerns on one Embodiment. 4 is a flowchart showing processing of a compression device according to one embodiment; FIG. 4 is a diagram schematically showing how tiles are divided according to one embodiment; FIG. 4 is a diagram schematically showing an example of an index representing latent variables according to one embodiment; FIG. 4 is a diagram showing the range of variables for probability distributions in encoding according to an embodiment; 1 is a block diagram showing an example of a restoration device according to one embodiment; FIG. 4 is a flowchart showing processing of the restoration device according to one embodiment. 1 is a block diagram showing an example of a training device according to one embodiment; FIG. 4 is a flow chart showing processing of the training device according to one embodiment. 1 is a diagram showing a hardware implementation example of an information processing apparatus according to an embodiment; FIG.

Embodiments of the present invention will be described below with reference to the drawings. The drawings and description of the embodiments are given by way of example and are not intended to limit the invention. In an embodiment of the present disclosure, an information processing device relating to compression of moving image data, decompression of compressed data, and training of a model used for compression and decompression will be described.

(data compression)
FIG. 1 is a diagram showing an example of the configuration of a data compression device according to one embodiment.
The compression device 1 includes an input unit 100, a storage unit 102, an output unit 104, a division unit 106, a feature amount acquisition unit 108, a hyperprior acquisition unit 110, a probability distribution acquisition unit 112, and an encoding unit 114. And prepare. The compression device 1 is a device that uses several trained models to generate and output compressed video data. The operation of the trained model is described in detail in the description of the training device, and the input/output data is described in the description of the compression device 1. FIG.

As will be described later in detail, the compression device 1 uses the current frame image, which is the object of data compression, for each frame image, or uses the current frame image and a frame image one time or more before. to compress the data. In the following, the frame image I _t alone at the current time t, or the current frame image I _t and the frame image at the time t - 1 one unit time earlier (hereinafter referred to as the previous frame image I _{t -1)} . ), the frame image one time ago may be read as a frame image two hours ago or more, or may be read as a plurality of frame images one time or more ago. .

The input unit 100 has an input interface. The compression device 1 receives input of data, for example, video data to be compressed, through the input unit 100 .

The storage unit 102 stores necessary data such as data such as model parameters used for the operation of the compression device 1, data to be subjected to data compression, and intermediate input/output data. In the following description, data processed by each component may be stored in storage unit 102 at appropriate timing. The stored data is read and used by the utilizing component at appropriate timing.

The output unit 104 outputs the compressed moving image data. Note that the compression device 1 may store the compressed moving image data in the storage unit 102. FIG.

The dividing unit 106 divides the frame images I in the moving image data into predetermined sizes. For example, the dividing unit 106 divides the frame image I into tiles T(m, n) of a predetermined size. (m, n) indicates the position of the tile. For example, the dividing unit 106 divides the frame image I into M×N tiles of a predetermined size T(0, 0), T(1, 0), . , T(M - 1, N - 1). When compressing the current frame image I _t , the dividing unit 106 divides the current frame image I _t or the current frame image _It and the previous frame image I _t-1 into tiles. _For example, the current frame image I _t is divided into tiles T _{t (} 0, 0), _{. 1} (0, 0), ..., T _t-1 (M - 1, N - 1).

The feature quantity acquiring unit 108 acquires the latent variables (feature quantities) of the tile T(m, n) divided by the dividing unit 106 from the tile T(m, n). The feature acquisition unit 108 inputs the tile T(m, n) to the trained first model M1, and calculates the latent variable z(m , n).

The latent variable z(m, n) is, for example, a vector indicating the feature amount of the tile T(m, n), and the storage capacity indicates the color information of all pixels of the tile T(m, n) of a predetermined size. is also a vector represented by a small storage capacity. This latent variable z(m, n) may be a vector e _k defined discretized. For example, the first model M1 to which the tile T(m, n) is input, has a predetermined number of vectors e _k (k = 1, . (probability that the latent variable of the tile is vector e _k or selection probability that vector e _k is selected as the latent variable of the tile) π _k is output.

For example, the feature quantity acquisition unit 108 acquires the probability π _k of the vector e _k (k ∈ {1, K}) via the first model M1, samples the vector e _i having the maximum probability π _i , and Let it be a latent variable. The vector e _k is, for example, a D-dimensional vector. The output layer of the first model M1 has, for example, K output nodes, and each vector e _k (k ∈ {1, K}) is the latent variable z(m, n) of the tile T(m, n). Output the probability of being selected as

_When the latent variable is obtained stochastically using the embedding vector discretized in this way, the vector e _k is obtained from the dimension D and the number It was previously obtained based on K.

Note that the latent variable may be a tensor having any dimension instead of a vector.

As another example, the latent variable may be a vector defined by continuous values output from the output layer of the first model M1 instead of being given in advance. In this case, the first model M1 outputs the latent variables z _t (m, n) as continuous values.

The hyper prior acquisition unit 110 acquires a hyper prior distribution (hyper prior) of the prior distribution, with the probability distribution for the previous frame image I _{t−1 as the} prior distribution and the posterior probability for the current frame image I _t as the posterior distribution. The hyperprior acquisition unit 110 inputs the previous frame image I _t-1 and the current frame image I _t to the trained second model M2 to acquire the hyperprior code u _t . The second model M2 may be a neural network model that outputs a hyperpliance code u _t when two temporally consecutive frame images I _t and I _t-1 are input.

The hyperplier code u _t is a vector indicating the conditioning that encodes the latent variable z _t of the tile at time t, the variable characterizing the probability distribution. This hyperplier code u _t is the vector used to properly model the conditioned probability of the tile latent variable z _t at time t conditioned on the tile latent variable z _t−1 at time t − 1. .

It may be assumed that the probability distribution P(u _t ) of the hyperplier code u _t is modeled as the product of the probability distributions P(u _tj ) of each element of the vector. where j represents the dimension of u _t . The element probability distribution P(u _tj ) may, for example, be assumed to be normally distributed. As a non-limiting example, the configuration of the second model M2 that outputs the hyperpliance code u _t includes an image encoder that encodes each of the frame images I _t and I _t-1 into latent variables, and an image encoder that encodes the frame images I _t and I _t-1 and a hyperprior encoder that estimates the hyperprior code u _t from the two latent variables of . A hyperprior encoder may comprise a residual block and a neural network that includes convolutional layers. Note that the residual block may be, for example, a block of one sequence that concatenates a 3×3 convolutional layer, a Leaky ReLU, and a 3×3 convolutional layer.

Probability distribution acquisition unit 112 uses latent variable z _{t -1 of each tile T t-1} acquired by feature amount acquisition unit 108 and hyper-prior code u _t acquired by hyper-prior acquisition unit 110 to determine the current time Estimate the probability distribution P(z _t | z _t-1 , u _t ) of the latent variable z _t in tile T _t . The probability distribution acquisition unit 112 can output the probability distribution P as discrete values or continuous values.

When outputting discrete values, the probability distribution acquisition unit 112 converts the tile T _t-1 (m, n) in the previous frame and the hyperprior u _t that characterizes the distribution at the current time from the distribution at the previous time into the trained 3 Input to the model M3, the probability distribution P(e _k | z _t-1 (m, n), u _t ) of the latent variable z _t (m, n) of a certain tile T _t (m, n) at the current time , k ∈ {1, K}.

The third model M3 has, for example, K output nodes in the output layer, and the probability distribution P( _ek | z _t-1 (m, n), u _t ) for each vector e _k is expressed as Output from In other words, the output of the softmax function of the vector e _k , that is, the output probability of the vector e _k is output from the nodes of the output layer of the third model M3. Probability distribution acquisition section 112 obtains probability distribution _P (e _k | z _{t -1} , u _t ), k ∈ {1, K}.

For example, probability distribution acquisition section 112 obtains _P ( _ek | z _{t -1} (0, 0), u _t ). Alternatively, the probability distribution P(e _k | z _t-1 (m0, n0), u _t ). As yet another example, if sufficient time can be secured for encoding and decoding, probability distributions may be obtained from multiple tiles, and the probability distribution P may be obtained as a statistical value such as the average of the probability distributions. .

When outputting as continuous values, the probability distribution acquisition unit 112 acquires the parameters of the Gaussian mixture distribution (weight w, mean u, variance σ) via the third model M3, for example, and obtains the probability distribution P(z _t | z _t-1 , u _t ) are calculated based on these parameters.

The encoding unit 114 acquires a binary string encoded code for the frame image I _t based on the latent variable z output by the feature amount acquisition unit 108 and the probability distribution P acquired by the probability distribution acquisition unit 112. . The encoding unit 114 calculates the latent _variable z(m _, n) Convert (m ∈ {0, M − 1}, n ∈ {0, N − 1}) into entropy code (code) to obtain entropy code C _t .

The coding method used may be any entropy coding method such as Huffman coding, arithmetic coding, range coders, or the like. From the viewpoint of compression rate, it is desirable to use arithmetic codes, range coders, and the like, which have higher compression efficiency, than Huffman codes, which reliably convert one type of description to more than 1 bit on average.

For example, when using arithmetic coding, the encoding unit 114 can encode latent variables (that is, latent variable sequences) obtained from each of a plurality of tiles, and may encode every predetermined number of tiles. For example, when using a range coder, the encoding unit 114 may reset the range every predetermined number of tiles, or may reset the range when the range becomes smaller than a predetermined value. These encoding units or parameters related to entropy encoding such as range changes may be changed at any timing as long as they are similarly defined in the compression device 1 and the decompression device described later.

The compression _device 1 entropy-encodes the latent variable _{z t (} m, n) ( m ∈ {0, M − 1}, n ∈ {0, N − 1}), the hyperpliance code u _t , and the latent variable z _t of the tile Tt at the time t specified in the keyframe as compressed data Generate.

A key frame is a reference frame. For example, the first frame may be set as a key frame, and a key frame may be set every predetermined number of frames. Also, a scene detection unit that detects a change in the scene of the moving image may be further provided, and a key frame may be set at the timing when the change in the scene is detected.

As another example of encoding, a probability distribution P is obtained for each tile, and based on this probability distribution P, the encoding unit 114 determines the range of variables in a range coder or the like for each tile, and finally may output the entropy code _Ct . For example, the probability distribution acquisition unit 112 acquires the probability distribution P of the latent variables of the tile at time t from the latent variables of the tile at the same position at time t - 1, and executes entropy encoding according to this probability distribution P. do.

More specifically, probability distribution acquisition section 112 obtains _probability distribution P( _ek | z _{t -1} (0, 0), u _t ), k ∈ { 1, K} is obtained from the hyperplier code u _t and the latent variables of the tile T _t-1 (0, 0) one time ago using the third model M3. Similarly, the probability distribution acquisition unit 112 obtains each of the tiles T _t (1, 0), ..., T _t (M - 1, 0), ..., T _t (M - 1, N - 1) , the probability distribution P(e _k | z _t-1 (m, n), u _t ) of vector e _k is estimated, and encoding section 114 uses the estimated probability distribution P for each tile T. encode the latent variable z for each tile T.

By encoding in this way, it is possible to perform encoding according to the probability distribution of the latent variables at the time to be processed that are conditioned by the latent variables one unit time earlier for each tile. If there are many tiles for which changes in latent variables can be predicted, it is possible to further increase the compression rate.

As still another example, the frame image I _t is divided into a predetermined number of tiles, a representative tile is determined for each divided tile group, and a hyperprior code is used for this representative tile to generate the vector e _k . A probability distribution may be obtained. For example, the probability distribution may be obtained by defining one representative tile for each tile group obtained by dividing the frame image I _t vertically or horizontally. Further, the tile group may be divided into four tile groups, vertically and horizontally, or another tile group may be divided.

Methods for obtaining these probability distributions are defined as those common to the compression device 1 and the decompression device. When compressing data in the compression device 1, the compression efficiency in encoding can be improved as the granularity of tiles, which is the reference for calculating the probability distribution, is finer, but the calculation cost in the decompression device increases. For this reason, the user may select a compression mode in consideration of the calculation cost in decompression, and the compression device 1 may change the method of calculating the probability distribution of the vector e _k according to the mode specified by the user.

In addition to these data, the compression device 1 may output parameters required for decoding together. For example, the value of the vector e _k , the parameters of each neural network model, and the like may be appropriately compressed and added to the moving image data. These may be those that the user can select as compression modes for compressed data. For example, the compression device 1 may compress data using preset parameters, vectors, and the like. A plurality of preset modes may be used as described above.

As another example, the compression device 1 may have a mode in which parameters, vectors, and the like are optimized by simple training during data compression to increase the accuracy of moving image restoration. The user may switch between these modes as desired, and based on this mode, the compressor 1 may implement processing appropriately. In the latter case, as described above, parameters, vectors, and the like are assigned to the compressed data of the moving image as variables, and the decompressing device, which will be described later, expands these parameters and the like to execute the decompressing process of the moving image. may

FIG. 2 is a flow chart showing the processing of the compression device 1. As shown in FIG.
First, the compression device 1 receives input of data to be compressed (moving image data will be described here) via the input unit 100 (S100). The moving image data may be stored in the storage unit 102. FIG. Note that the format of the moving image data is not particularly limited. For example, the moving image data may be in a format in which an image for each frame can be acquired appropriately. The compression device 1 extracts the image I into a frame at each time and compresses the moving image.

The dividing unit 106 divides the frame image I _t at time t to be processed and the frame image I _t-1 one unit time earlier into tiles T of a predetermined size (S102).

FIG. 3 is a diagram schematically showing how a frame image I _t is divided into tiles T _t by the dividing unit 106 according to one embodiment.
As shown in this figure, the frame image I _t is divided into M×N tiles T _t for each predetermined size. Although indexes are assigned, this index assignment is an example and is not limited to this assignment. The dividing unit 106 also divides the frame image I _t-1 at time t −1 into tiles T _t-1 in the same manner as at time t. Note that the dividing unit 106 may divide the frame image so that the tiles do not overlap each other, or may divide the frame image so that the tiles partially overlap each other.

Instead of explicitly dividing and storing each area in the storage unit 102 as shown in the drawing, the division may be configured such that the array index for the divided area is stored as information of each tile. An index may be calculated to obtain the elements of the array to be input to the first model M1 when obtaining the variables.

If there is a fraction in the boundary area of the image, the dividing unit 106 may, for example, pad the area outside the image area with an appropriate value to form tiles of a predetermined size. For example, if the predetermined size is 32 × 32 pixels, and a video of 1920 × 1080 pixels is divided into tiles, the last tile in the vertical direction is composed of 32 × 24 pixel image data and zero-padded 32 × Data of 8 pixels may be combined to form a tile. The processing when there is a fraction is not limited to this, and any appropriate method can be adopted.

Conversely, the size of the tiles may be determined so that the number of tiles is the predetermined number. For example, an image with a size of full HD (1920×1080) may be divided into 128×108 tiles. In this case, the tile size is 15×10 pixels.

If the tiles are allowed to overlap pixels, in each of the above cases, a margin for the overlapping pixels may be provided, and the tiles may share the pixels within this margin.

The dividing unit 106 may thus divide the frame image into tiles of a predetermined size, or may divide the frame image into a predetermined number of tiles. The predetermined size or predetermined number may be determined by compression rate or restoration accuracy. This determination may be made in advance by the information processing apparatus, or may be made by the user at the timing of data compression. Such additional information may be stored, for example, as header information of the compressed file, and appropriate processing may be implemented by referring to this header information upon decompression.

Returning to FIG. 2, next, the feature quantity acquisition unit 108 acquires latent variables of tiles T _t and T _t- 1 at time t and time t - 1 (S104). The feature amount acquisition unit 108 acquires latent variables for each tile by, for example, inputting color information of pixels for each tile into the first model M1. As described above, the latent variable may be a continuous value, or may be output as a coordinate value indicating one point in a discretely given latent space. The feature quantity acquiring unit 108 acquires, for example, the selection probability π _k of the vector e _k output from the first model M1, and acquires the vector e with the maximum π as a latent variable.

Note that when the frame to be processed is a key frame, the processing of S102 and S104 for the frame image at time t - 1 one unit time earlier may be omitted.

FIG. 4 is a diagram schematically showing an example of indices of latent variables according to one embodiment when the latent variables are represented by discrete tensors (vectors e _k ).
The feature quantity acquisition unit 108 may output the index of the vector e _k output by the Softmax function for each tile, as shown in FIG. For example, the latent variable z _t (0, 0) of tile T _t (0, 0) is e ₇₀₃₇ and the latent variable z _t (1, 0) of T _t (1, 0) is e ₆₈₉₈ . . This latent variable is a quantity that characterizes the color value for each tile. For example, K=8192, and in this case there are 8192 types of embedding vectors that can be selected as latent variables.

Returning to FIG. 2, next, the compression device 1 determines whether or not the frame being processed is a key frame (S106).

If it is a key frame (S106: YES), information corresponding to the frame number, a flag indicating that it is a key frame, and the latent variable z of tile T are stored in storage unit 102 (S108), and Move on to processing. The latent variable z may be stored in the storage unit 102 in a non-reversible-compressed state, or may be stored in the storage unit 102 in a reversible-compressed state. The information corresponding to the frame number may be, for example, a sequential number uniquely assigned from the first frame to the last frame of the moving image, or information according to this.

As described above, a key frame may be set for each predetermined number of frames, or may be set by processing such as scene detection. Also, the first frame of the moving image may be determined as a key frame and processed.

If it is not a keyframe (S106: NO), the processing from S110 is executed.

First, the hyper-prior acquisition unit 110 inputs the frame images I _t and I _t-1 to the second model M2 and acquires the hyper-prior code u _t between time t and time t - 1 (S110). .

Next, the probability distribution acquisition unit 112 inputs the hyperpliance code u _t and the latent variable z _t of the tile T _t- 1 at the time t - 1 to the third model M3, and estimates the probability distribution of the vector e _k . (S112).

Next, the encoding unit 114 encodes the latent variable _zt of the tile _Tt acquired in S104 according to the range based on the probability distribution acquired by the probability distribution acquisition unit 112 to acquire the entropy code _Ct . (S114). Whether one entropy code _Ct is assigned to a plurality of latent variables (that is, a latent variable string) or one entropy code _Ct to one latent variable depends on the type of entropy encoding.

FIG. 5 is a diagram showing ranges of variables for probability distributions in encoding according to an embodiment.
Assume that the probabilities acquired by the probability distribution acquisition unit 112 are estimated as shown in the middle column for the vectors shown on the left end. In this case, encoding section 114 defines the range of numerical values corresponding to variables to be encoded according to this probability distribution as shown in the right column. Based on this range, the encoding unit 114 acquires an entropy code C by encoding one or more indices of e selected as latent variables for each tile acquired by the feature amount acquisition unit 108 .

This range follows the probability distribution inferred from the relationship between the previous frame image I _t-1 and the current frame image I _t . The tile image at time t in each tile is inferred by the prior probability hyperparameter, which can be inferred by the hyperprior code u _t . For example, data characterizing (conditioning) what has changed from the previous frame to the current frame is included in the hyperpliance code u _t .

From this, it is possible to infer how each tile is likely to change from the tile in the previous frame image by using this hyperpliance code _ut . Therefore, by using the probability distribution of the vector e _k inferred using this hyperplier code u _t , it becomes possible to set the range used for appropriate coding, and the latent variable can be converted into an entropy code such as a range coder. It is possible to increase the compression efficiency when using

The output of the third model M3 is based on, for example, the Softmax function, and the use of the Softmax function may generate a vector with a probability distribution of approximately 0. In such cases, at least one bit in the range may be set to be assigned to each vector. By setting in this way, loss of data can be avoided.

Returning to FIG. 2, next, the compression device 1 stores the compressed data of the frame image _It in the non-key frame in the storage unit 102 (S116). In a frame that is not a key frame, the compression device 1 stores at least information corresponding to the frame number, the hyperpliance code u _t and the entropy code C _t of the encoded latent variable. If necessary, other information such as tile position information used when estimating the probability distribution is also stored. In addition, a flag indicating that the frame is not a key frame may be stored.

The compression device 1 determines whether or not processing has been completed for all frames (S118), and if there are still frames left (S118: NO), continues the processing from S102, and processes all frames. If the processing has ended (S118: YES), the processing ends.

As described above, the compression device 1 expresses the latent variable for each tile of the frame image as a discretized embedded vector in the key frame, and uses it as compressed data. The latent variable is entropy-encoded using the probability distribution of the embedding vector and expressed as compressed data.

(data recovery)
Next, restoration of moving image data compressed by the compression device 1 will be described. In principle, the restoration device described below restores moving image data by performing the same processing as that of each component in the compression device 1 or the reverse processing. That is, the data and the like to be input to the model are basically the same data as in the compression device 1 .

For example, if the user can specify a plurality of modes in the compression device 1, the restoration process is executed by the same process as the data input to the model in the mode used for compression. In a more specific example, the method of calculating the probability distribution of embedding vectors in the compression device 1 and the method of selecting tiles in the method of calculating the probability distribution of embedding vectors in the restoration device described below use the same method. Information about branching of these processes may be data attached to a header of compressed data or the like.

FIG. 6 is a diagram showing an example of the configuration of a data restoration device according to one embodiment.
The restoration device 2 includes an input unit 200, a storage unit 202, an output unit 204, a probability distribution acquisition unit 206, a decoding unit 208, and a color information acquisition unit 210. The decompression device 2 is a device that decompresses and outputs compressed video data using several trained models. The operation of the trained model will be described in detail in the description of the training device, and the input/output data will be described in the description of the reconstruction device 2.

The input unit 200 has an input interface. The decompression device 2 receives input of data, for example, compressed video data to be decompressed, through the input unit 200 .

The storage unit 202 stores necessary data such as data such as model parameters used in the operation of the restoration device 2, data to be restored, and data input/output in the middle. In the following description, data processed by each component may be stored in the storage unit 202 at appropriate timing. The stored data is read and used by the utilizing component at appropriate timing.

The output unit 204 outputs the restored moving image data. For example, the output unit 204 may have a display as an output interface, and may convert the video data restored by the restoration device 2 into a displayable format and display it. Also, the output unit 204 may output audio data in accordance with the display of the moving image, and in this case, may be provided with an output interface such as a speaker.

The probability distribution acquisition unit 206 inputs the latent variable z of the tile T and the hyperpliance code u to the trained third model M3, and estimates the probability distribution P of the latent variable z of the tile T. The probability distribution acquisition unit 206 can output the probability distribution P as discrete values or continuous values. The third model M3 in the decompression device 2 is a model common to the third model M3 in the compression device 1 used for compressing the moving image data. Since the operation is the same as that of the probability distribution acquisition unit 112 of the compression device 1, detailed description will be omitted.

The decoding unit 208 decodes the encoded latent variable entropy code C based on the probability distribution P of the vector e _k acquired by the probability distribution acquisition unit 206 . The decoding unit 208 obtains the latent variable z of one or more tiles T by decoding the entropy code C. FIG.

The color information acquisition unit 210 acquires the color value F of the pixels included in the tile T based on the latent variable z for each tile T. FIG. The color information acquisition unit 210 inputs the latent variable z to the trained fourth model M4 to infer the color value of the pixel (x, y) in the tile T. The color value F may be, for example, a quantity having an integer value of [0, 255] for each of the three primary colors. In addition, it may be a value represented by a continuous value of [0, 1], and at the time of output, these values may be appropriately converted in the output unit 204 and displayed.

Several types of formation of the fourth model M4 are conceivable. The fourth model M4 may be formed, for example, as a model that, given the size of the tile and the latent variable z of the tile, restores and outputs the color values of all the pixels in the tile T. More specifically, the fourth model M4 receives the first MLP that outputs the modulation information α when the latent variable z is input, and the color value F(p, α( and a second MLP that outputs z)). Alternatively, for example, a model including an MLP that receives a vector obtained by concatenating a latent variable z and a position p and outputs a color value F(p, z) may be used.

When the first MLP and the second MLP are provided, the first MLP includes an input layer to which the latent variable z of the number of dimensions D is input, _NL hidden layers h', and an output layer. The second MLP comprises an input layer to which position p is input, _NL hidden layers h, and an output layer to output color value c. For example, ReLUctantly may be used as the activation function of the first MLP, and the input layer of the first MLP outputs _h'0 to the first hidden layer, and the i + 1th hidden layer is , output h' _i+1 to the next layer. In addition, the i+1-th hidden layer uses the output h′ _i+1 as the modulation information α _i+1 for modulating the activity of the activation function of the second MLP to the i-th hidden layer of the second MLP. + Output to the 1st layer.

The input layer of the 2nd MLP outputs the input position p as h ₀ to the 1st layer of the hidden layer, and the i + 1th layer of the hidden layer of the 2nd MLP outputs α _i+ input from the 1st MLP Modulate the activity of the activation function by ₁ and output h _i+1 to the next layer. The output layer receives the output h from the last hidden layer and calculates the color value F(p, z) of the pixel at coordinates (x, y) in the tile image with latent variable z. A periodic nonlinear function such as a sine function is used for the activation function of the second MLP.

ここで、wは、各層を接続する重み行列、bは、各層に入力されるバイアス、○の中に・を記載した記号(odot)は、アダマール積(要素同士の積)、Tは、転置を示す。なお、上記において、ReLU、sin等の関数又は演算子は、成分ごとに文脈により、行列、ベクトルの成分ごとに演算するものと読む。以下においても同様の記載があれば同様に適宜スカラーに対する演算又はベクトル若しくは行列等の要素それぞれに対する演算であると理解されたい。 Note that the outputs h' ₀ , h' _i+1 , α _i+1 related to the first MLP, and the output h i ₊₁ related to the second MLP may be defined as follows.

Here, w is the weight matrix that connects each layer, b is the bias input to each layer, the symbol (odot) written in the circle is the Hadamard product (the product of the elements), and T is the transposition. indicates In the above description, functions or operators such as ReLU and sin are read as operating for each component of a matrix or vector depending on the context for each component. If there is a similar description below, it should be understood that it is an operation on a scalar or an operation on each element of a vector, matrix, or the like, as appropriate.

As another example, the fourth model M4 may be in the form of inputting the position in the tile of the pixel whose color value is to be restored together with the latent variable _zt . In this case, by specifying a pixel position p in tile T _t (eg, coordinates in tile T), the color value F(p, z) at this pixel can be restored. By restoring the color values F(p,z) for all pixels in the tile _Tt , the image of the tile _Tt can be restored.

When the pixel position p is input as described above, the fourth model M4 may accept the size of the tile in the input layer. It may be configured to accept input. In either case, it is also possible to output an image with a higher resolution than when compression is performed. For example, if the tile size is accepted by the input layer, by inputting a decimal value as the position p, the color value F(p, z) between the pixels of the compressed tile T is It is possible to infer

When inputting a continuous value from 0 to 1, it is possible to infer the color value F(p, z) at an arbitrary position p by inputting an arbitrary value. For example, if the tile size is 16 × 16, to get a tile image with the same size as the original, set x, y ∈ {0, 1/15, 2/15, …, 1} You can restore the color value F(p, z) by inputting p. In contrast, using the same latent variables, we can find the number of pixels in a tile of size 32 × 32 by entering x, y ∈ {0, 1/31, 2/31, …, 1} as position p. It is possible to infer the color value F(p, z).

When inferring the restored image of the tile T(m, n), the color information acquisition unit 210 obtains not only the latent variable z(m, n) but also the latent variables z(m - 1, n - 1), z(m, n - 1), z(m + 1, n - 1), z(m - 1, n), z(m + 1, n), z(m - 1, n + 1), z(m, n + 1), z(m + 1, n + 1) may be input to the fourth model M4. For tiles on the edge of the image, the above latent variables may be appropriately reduced to change the mixing ratio. For example, the latent variable z(m, n) may be updated based on the following formula and input to the fourth model M4. Note that the coefficients and weighting method are not limited to the following formulas.

As another example, the mixing ratio of the latent variables of the tile in 8-connection may be changed based on the position p(=(x, y)) within the tile. As yet another example, a table of combinations of latent variables is prepared in the restoration device 2, and latent variables to be transformed for combinations of latent variables of tiles to be restored or combinations close to latent variables of tiles to be restored are determined. and recover the color value F(p,z) based on the obtained latent variables.

The reconstruction device 2 reconstructs the frame image I _t based on the information of the color value F(p, z) acquired by the color information acquisition unit 210, and outputs it from the output unit 204 (for example, displays). Further, the restoration device 2 may convert the restored color values into a format suitable for the output interface and output (display) them in a signal processing unit (not shown). As described above, when displaying, the audio information may be output together, or the vibration information may be output via the vibration module.

FIG. 7 is a flow chart showing the processing of the restoration device 2. As shown in FIG.
First, the decompression device 2 receives input of compressed data to be decompressed via the input unit 200 (S200). Compressed data may be stored in the storage unit 202 . The decompression device 2 extracts data for each frame from the compressed data and executes processing.

The restoration device 2 determines whether the frame at time t to be processed is a key frame (S202). The decompression device 2 refers to the key frame flag attached to the frame in the compressed data and determines whether or not the frame to be processed is a key frame.

If the frame at time t is a key frame (S202: YES), the latent variable _zt of the tile _Tt of the frame image It is extracted from the compressed data, and using this latent variable _zt , _the color information acquisition unit 210 obtains the color value of each pixel of the tile T _t to restore the frame image I _t (S208). Note that if the latent variable _zt is reversibly compressed, it is decoded and used.

If the frame at time t to be processed is not a key frame (S202: NO), the latent variable _zt corresponding to the frame _image It is not directly included in the compressed data. By decoding C, we obtain the latent variable _zt .

For this process, the probability distribution acquisition unit 206 acquires the probability distribution of the vector e _k (S204). The probability distribution acquisition unit 206 extracts the hyperpliance code u _t corresponding to the frame at time t from the compressed data, and acquires the latent variable z t-1 used to restore the frame image I _t-1 at time t _-1. do. For this latent variable z _t-1 , for example, a value obtained at the timing of restoring the frame image I _{t-1 at time t - 1} may be used. The probability distribution acquisition unit 206 inputs the hyperpliance code u _t and the latent variable z _t−1 to the third model M3, and acquires the probability distribution of the vector e _k with respect to the latent variable z _t at time t. Since this obtained probability distribution is used for decoding entropy code C in the next step S206, it is the same as the one used in entropy coding when obtaining entropy code C to be decoded.

Next, decoding section 208 decodes entropy code _Ct based on the probability distribution of vector e _k (S206). Through this decoding process, the latent variable _zt of each tile _Tt at time t can be obtained. If one entropy code C is obtained from multiple tiles worth of latent variables, the decoding process decodes this entropy code into multiple tiles worth of latent variables.

Then, the color information acquiring unit 210 uses the acquired latent variable _zt to restore the frame image I _t in the non-key frame (S208).

That is, the restoration device 2 restores the frame image I _t using the latent variable z _t stored in the compressed data in the key frame. Except for key frames, compressed data is calculated based on the probability distribution of the vector e _k in the current frame obtained from the latent variable z _t-1 obtained in the immediately preceding frame processing and the hyperprior code u _t stored in the compressed data. The latent variable _zt is decoded from the entropy code _Ct stored in , and the frame image _It is restored using this latent variable _zt .

Thus, the decompression device 2 can decompress the moving image data compressed by the compression device 1 appropriately.

In the above description, the restoration device 2 restores sequentially from the first frame to the last frame of the moving image, but the operation of the restoration device 2 is not limited to this. For example, the restoration device 2 may restore from the middle part of the moving image to the end, or any continuous frame of the moving image according to the user's designation or the like. A configuration may be adopted in which the user can input the start time and end time of the restoration range of the moving image to the restoration device 2 . In this case, the frame images may be sequentially restored from the first detected key frame including the start frame by going back to the past frames from the frame designated as the start of restoration.

Moreover, the configuration may be such that not only temporal designation but also spatial designation is possible. In this case, the restoration device 2 may restore the moving image within the range of information of the tiles included in the restoration target range and the surrounding tiles that are necessary for restoration.

(model training)
Next, a training device for training a model used in the compression device 1 and the decompression device 2 will be described.

FIG. 8 is a diagram showing an example of the configuration of a model training device according to one embodiment.
The training device 3 includes an input unit 300, a storage unit 302, an output unit 304, a division unit 306, an encoding unit 308, a decoding unit 310, a feature acquisition unit 312, a hyperprior acquisition unit 314, A probability distribution acquisition unit 316, a color information acquisition unit 318, and an update unit 320 are provided. The training device 3 trains at least one of the first model M1, the second model M2, the third model M3, or the fourth model M4 used in the compression device 1 or the decompression device 2 by machine learning techniques. It is a device.

The first model M1, the second model M2, the third model M3, and the fourth model M4 may each be formed as an appropriate neural network model. For example, these models may be formed by a CNN (Convolutional Neural Network) containing at least one convolutional layer, may be formed by an MLP (Multi-Layer Perceptron), or other suitable models. may be formed by a neural network model.

The input unit 300 has an input interface. The training device 3 receives input of data via the input unit 300, for example, video data, which is teacher data necessary for training a model.

The storage unit 302 stores necessary data such as various parameters used for the operation of the training device 3, parameters of the model to be trained, and intermediate data in training (eg, encoded data, restored data, latent variables, probability distributions). do. In the following description, data processed by each component may be stored in the storage unit 302 at appropriate timing. The stored data is read and used by the utilizing component at appropriate timing.

The output unit 304 outputs parameters of the model optimized by training. This output may be a concept including storing in the storage unit 302 . In other words, the output unit 304 may replace the storage of parameters in the storage unit 302 with output processing to the outside.

A dividing unit 306 divides the frame image I in the moving image data into predetermined sizes. The dividing unit 306 performs the same processing as the dividing unit 106 in the compression device 1, so the details are omitted.

The encoding unit 308 acquires an encoded code for the frame image I _t based on the latent variable z output by the feature amount acquisition unit 312 and the probability distribution P output by the probability distribution acquisition unit 316 . Since this encoding unit 308 executes the same processing as the encoding unit 114 in the compression device 1, details thereof will be omitted.

The decoding unit 310 decodes the encoded latent variable entropy code C _t based on the probability P of the vector e _k at the time t acquired by the probability distribution acquisition unit 316 . The decoding unit 310 performs the same processing as the decoding unit 208 in the decompression device 2, so the details are omitted.

The feature amount acquisition unit 312 acquires latent variables (feature amounts) of the tile T from the tile T divided by the division unit 306 . The feature acquisition unit 108 inputs the tile T to the first model M1 to be trained, and acquires the latent variable z of the tile T.

The feature quantity acquisition unit 312 may have a configuration different from that of the feature quantity acquisition unit 108 in the compression device 1 . The feature quantity obtaining unit 312 inputs the tile T divided by the dividing unit 306 into the first model M1, and obtains the selection probability of the vector e _k . As a non-limiting example, the feature quantity acquisition unit 312 acquires latent variables using the Gumbel Softmax technique based on the probabilities output by the first model M1. For example, the feature quantity acquisition unit 312 acquires the latent variable z for the tile T based on the following formula.

Uniform(0, 1) is a uniform random number in [0, 1], i.e. u is the value obtained by the uniform random number in [0, 1], g is the noise value given by the uniform random number, g _k is the noise value obtained according to equation (3) for each k, one_hot() is the function generating the one-hot vector, π _k is the probability of e _k output from the first model M1 . As shown in Equation (7), noise is given to π to generate a one-hot vector based on this probability, and a vector e corresponding to this one-hot vector is obtained and used as a latent variable.

ここで、τは、温度を表すパラメータであり、訓練における順伝播、すなわち、特徴量取得部312の処理においては、十分小さい値、例えば、τ = 0.01等と設定してもよい。τを十分に小さい値とすることで、第1モデルM1の出力にSoftmax関数を通した出力と略一致させることができる。すなわち、圧縮装置1における潜在変数zの選択と略一致させることができる。 The feature quantity acquisition unit 312 may use the following formula instead of formula (7).

Here, τ is a parameter representing temperature, and may be set to a sufficiently small value, for example, τ = 0.01, in forward propagation in training, that is, in the processing of the feature quantity acquisition unit 312 . By setting τ to a sufficiently small value, the output of the first model M1 can be approximately matched with the output obtained by passing the Softmax function. That is, it can be substantially matched with the selection of the latent variable z in the compression device 1. FIG.

By using the form of formula (8), in the training process, it is possible to execute backpropagation without going through argmax( ), that is, to acquire the gradient information of the first model M1. By using the Gumbel Softmax distribution, as described above, it is possible to estimate the gradient by sampling discrete variables other than the discrete variable with the maximum probability, and gradually reduce the noise to reduce the probability using the Softmax function. and obtain a result consistent with the case of choosing the discrete variable that takes the maximum probability.

The hyper prior acquisition unit 314 acquires a hyper-prior distribution of the prior distribution, with the probability distribution for the frame image I _t-1 at time t −1 as the prior distribution and the posterior probability for the frame image I _t at time t as the posterior distribution. do. The processing of the hyper-prior acquisition unit 314 is the same as that of the hyper-prior acquisition unit 110 of the compression device 1, so detailed description thereof will be omitted.

The probability distribution _acquisition unit 316 obtains the latent variable z t of the tile T t at the time t from the latent variable _z of the tile T acquired by the feature acquisition unit 312 and the hyper prior code u acquired by the hyper prior acquisition unit 314. Estimate the probability distribution P. The processing of the probability distribution acquisition unit 316 is the same processing as that of the probability distribution acquisition unit 112 of the compression device 1, so detailed description is omitted.

The color information acquisition unit 318 samples a plurality of pixels in each tile T, and calculates the color value of the sampled pixel based on the coordinate value (position p) of the sampled pixel and the latent variable z of the tile. get an F. All pixels in a tile may be sampled. The processing of the color information acquisition unit 318 is substantially the same as that of the color information acquisition unit 210 of the restoration device 2, so detailed description is omitted.

The update unit 320 calculates an error by comparing the color value output by the color information acquisition unit 318 and the color value of the input video data, and back-propagates the error to obtain the parameters of each model. to update. Furthermore, in order to train for increasing the compression rate, the compression rate, ie, an index for generation of the entropy code C, may be considered as an error. For example, for the frame image I _t at time t, the objective function is defined as follows.

ここで、Dist()は、距離関数を表し、例えば、L1ノルム(マンハッタン距離)、L2ノルム(ユークリッド距離)であってもよいし、他の距離関数であってもよい。Nは、色情報取得部318においてサンプリングされた画素の数であり、c(p_t)は、サンプリングされた画素の元の画像における画素の色(正解値)である。 Here, L _t is an evaluation corresponding to the reconstruction error of the frame image I _t and may be expressed by the following equation.

Here, Dist( ) represents a distance function, and may be, for example, L1 norm (Manhattan distance), L2 norm (Euclidean distance), or other distance functions. N is the number of pixels sampled in the color information acquisition unit 318, and c(p _t ) is the pixel color (correct value) in the original image of the sampled pixels.

第1項は、確率分布取得部316が取得した確率分布から求めることが可能であり、第2項のP()は、例えば、正規分布等の所与の確率分布が用いられる。なお、λは、復元精度と圧縮率とに基づいて、適切に設定されればよい。例えば、精度よりも圧縮率を高めるモデルを訓練する場合には、λをより大きくし、圧縮率よりも精度を高めるモデルを訓練する場合には、λをより小さく設定して訓練を実行してもよい。 Also, R _t is an evaluation corresponding to the compression ratio of the frame image I _t and may be expressed by the following formula.

The first term can be obtained from the probability distribution acquired by the probability distribution acquisition unit 316, and P( ) of the second term uses, for example, a given probability distribution such as a normal distribution. Note that λ may be appropriately set based on the restoration accuracy and compression rate. For example, if you want to train a model that gives more compression than accuracy, set λ higher, and if you want to train a model that gives more accuracy than compression, set λ smaller. good too.

As a non-limiting example, the updating unit 320 calculates the objective function by Equation (11), and updates the parameters of each model so as to reduce the objective function. Error backpropagation can be used for this update. In the backpropagation of the first model M1, gradient information can be obtained appropriately by using Gumbel Softmax instead of using argmax as described above.

FIG. 9 is a flow chart showing an example of processing of the training device 3. As shown in FIG.
The training device 3 first acquires training data (video data) via the input unit 300 (S300). This moving image data may be a moving image including at least two consecutive frame images. A moving image including many frame images may be used for highly accurate training.

The training device 3 extracts two frames at consecutive times t - 1 and t and starts processing. First, the frame image I _t at time t is compressed for each tile T _t (S302), in the same manner as the processing described for the compression device 1.

The color information acquisition unit 318 samples a plurality of pixels, acquires the color value F using the fourth model M4, and restores part or all of the frame image _It (S304).

The updating unit 320 updates the parameters of each model by backpropagation so that the objective function of Equation (11) becomes smaller (S308). As described above, in the process of acquiring the discretized latent variables for each tile, the process is executed using a function capable of calculating the gradient.

The training device 3 determines whether or not the termination condition is satisfied (S308), and repeats the training until an appropriate termination condition is satisfied. That is, if the end condition is not satisfied (S308: NO), the training process (S302-S306) is repeated, and if the end condition is satisfied (S310), the parameters are output (S310) and the process ends.

The training process continues until an appropriate termination condition, such as until each parameter update converges, until the accuracy falls below a predetermined value, or until a predetermined number of epochs are completed. The training process may, for example, be performed in a batch process and update the parameters of each model based on the gradient averages computed in the batch process.

As described above, according to the present embodiment, the processing method of the training device 3 makes it possible to optimize the models used in the compression device 1 and the decompression device 2. FIG. By using these models, the compression device 1 can achieve compression of moving image data that achieves both high accuracy and high compression rate. Similarly, by using these models, the decompression device 2 can decompress the video data generated by the compression device 1 appropriately.

As an example, if the number of embedding vectors e _k is 4096 (12 bits), it can be expected that this value can be compressed to about 1 to 6 bits by entropy coding. As a result, the compression can be reduced to about 1/12 to 1/2 compared to the case where embedding vectors are used for each tile. Furthermore, although it depends on the size of the tile, it is possible to compress the color information for each primary color for the number of pixels of the size of the tile using a discretized latent variable. can do. Since the tile information itself can be used as a latent variable (dimensionally compressed tile information), further compression efficiency can be achieved.

The order of processing in each of the information processing devices described above (the compression device 1, the decompression device 2, or the training device 3) may be changed as long as there is no contradiction in data dependency. Similarly, a plurality of processes of the information processing device may be executed in parallel (for example, by SIMD instructions or MIMD instructions) as long as there is no contradiction in data dependency. For example, in compression or decoding, the information processing device sequentially compresses or decodes key frames, but can also compress or decode multiple key frames in parallel.

Note that the frame image at time t - 1 is the frame image to be processed immediately before, so if there is sufficient storage capacity, the frame image generated in the previous loop may be used. For example, a ring buffer or the like may be configured to store information (tiles, latent variables, etc.) one time ago. This is a rate-limiting configuration issue, so if recalculation is faster or cheaper than keeping an extra frame's worth of images in memory, then the can be recalculated to

In the above description, the first model M1 to the fourth model M4 are used, but the present invention is not limited to this, and compression processing may be performed using some models. For example, using only the first model M1, the frame image I _t for each time may be converted into the latent variable z for each tile T, and this may be used as compressed data. Although the compression efficiency is not as good as in the above-described embodiment, it is possible to compress and decompress moving images with simple processing. For this reason, it may be advantageous in moving image restoration or the like that requires real-time performance.

Also, for example, the second model M2 and the third model M4 may be formed as one model. In this way, the combination of models can be selected as appropriate within the range of appropriate processing.

The vector e _k , the first model M1, the second model M2, the third model M3, and the fourth model M4 are parameters related to compression such as the compression rate (quantization rate), the accuracy of the restored moving image, or the restoration time. Multiple ones optimized based on at least one of them may be prepared. For example, an optimized embedding vector and model may be trained for each parameter (mode) of compression, and compression and decompression based on user-selected parameters. By preparing a plurality of data in this manner, compression and decompression can be performed based on parameters selected by the user.

The compression device 1, decompression device 2, and training device 3 may each be configured as an independent information processing device, or at least two of them may be configured as the same information processing device.

All of the above trained models may be concepts including, for example, models that have been trained as described and further distilled by general techniques.

A part or all of each device (information processing device: compression device 1, decompression device 2 or training device 3; hereinafter the same) in the above-described embodiment may be configured by hardware, or may be implemented by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) or the like may be configured by information processing of software (program) executed. In the case of software information processing, software that realizes at least part of the functions of each device in the above-described embodiments can be stored in CD-ROM (Compact Disc-Read Only Memory), USB (Universal Serial Bus) memory. The information processing of the software may be executed by storing it in a non-temporary storage medium (non-temporary computer-readable medium) such as the above and reading it into a computer. Alternatively, the software may be downloaded via a communication network. Furthermore, by implementing all or part of the software processing in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), the information processing by the software may be executed by hardware. .

A storage medium for storing software may be a detachable storage medium such as an optical disk, or a fixed storage medium such as a hard disk or memory. Also, the storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or may be provided outside the computer.

FIG. 10 is a block diagram showing an example of the hardware configuration of each device (information processing device) in the embodiment described above. Each device includes, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76. may be implemented as a computer 7 integrated with the

The computer 7 in FIG. 10 has one component, but may have a plurality of the same components. Also, in FIG. 10, one computer 7 is shown. good too. In this case, it may be in the form of distributed computing in which each computer communicates via the network interface 74 or the like to execute processing. In other words, each device (information processing device) in the above-described embodiments may be configured as a system in which functions are realized by one or more computers executing instructions stored in one or more storage devices. good. Alternatively, the information transmitted from the terminal may be processed by one or more computers provided on the cloud, and the processing result may be transmitted to the terminal.

Various operations of each device (information processing device) in the above-described embodiments may be executed in parallel using one or more processors or using multiple computers via a network. Also, various operations may be distributed to a plurality of operation cores in the processor and executed in parallel. Also, part or all of the processing, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud capable of communicating with the computer 7 via a network. Thus, each device in the above-described embodiments may be in the form of parallel computing by one or more computers.

Processor 71 may be at least electronic circuitry (processing circuitry, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that either controls or performs computations on a computer. Also, processor 71 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific operation, or a semiconductor device including both a general-purpose processor and dedicated processing circuit, or the like. Also, the processor 71 may include an optical circuit, or may include an arithmetic function based on quantum computing.

The processor 71 may perform arithmetic processing based on data and software input from each device or the like of the internal configuration of the computer 7, and may output the arithmetic result or control signal to each device or the like. The processor 71 may control each component of the computer 7 by executing the OS (Operating System) of the computer 7, applications, and the like.

Each device (information processing device) in the above-described embodiments may be implemented by one or more processors 71 . Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. You can point When multiple electronic circuits are used, each electronic circuit may communicate by wire or wirelessly.

The main memory device 72 may store instructions and various data executed by the processor 71 , and the information stored in the main memory device 72 may be read by the processor 71 . Auxiliary storage device 73 is a storage device other than main storage device 72 . These storage devices mean any electronic components capable of storing electronic information, and may be semiconductor memories. Semiconductor memory may be either volatile memory or non-volatile memory. A storage device for storing various data in each device (information processing device) in the above-described embodiments may be realized by the main storage device 72 or the auxiliary storage device 73, and may be realized by the built-in memory built into the processor 71. may be implemented. For example, the storage unit 102 in the above-described embodiment may be realized by the main storage device 72 or the auxiliary storage device 73.

When each device (information processing device) in the above-described embodiments is composed of at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, storage device 1 At least one processor may be connected to one. At least one storage device may be connected to one processor. Also, a configuration may be included in which at least one processor among the plurality of processors is connected to at least one storage device among the plurality of storage devices. This configuration may also be implemented by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. As for the network interface 74, an appropriate interface such as one conforming to existing communication standards may be used. The network interface 74 may exchange information with the external device 9A connected via the communication network 8. FIG. Note that the communication network 8 may be any one of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), etc., or a combination thereof, and is used to connect the computer 7 and the external device 9A. It is sufficient if information can be exchanged between them. Examples of WAN include the Internet, examples of LAN include IEEE 802.11 and Ethernet (registered trademark), and examples of PAN include Bluetooth (registered trademark) and NFC (Near Field Communication).

The device interface 75 is an interface such as USB for direct connection with the external device 9B.

The external device 9A is a device connected to the computer 7 via a network. External device 9B is a device that is directly connected to computer 7 .

As an example, the external device 9A or the external device 9B may be an input device. The input device is, for example, a device such as a camera, microphone, motion capture, various sensors, keyboard, mouse or touch panel, and provides the computer 7 with acquired information. Alternatively, a device such as a personal computer, a tablet terminal, or a smartphone including an input unit, a memory, and a processor may be used.

Also, the external device 9A or the external device 9B may be an output device as an example. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker or the like for outputting sound. Alternatively, a device such as a personal computer, a tablet terminal, or a smartphone including an output unit, a memory, and a processor may be used.

Also, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

Also, the external device 9A or the external device 9B may be a device having a part of the functions of the constituent elements of each device (information processing device) in the above-described embodiments. That is, the computer 7 may transmit part or all of the processing result to the external device 9A or the external device 9B, or may receive part or all of the processing result from the external device 9A or the external device 9B. .

In this specification (including claims), the expression "at least one (one) of a, b and c" or "at least one (one) of a, b or c" (including similar expressions) Where used, includes any of a, b, c, a-b, a-c, b-c or a-b-c. It may also contain multiple instances of any element, such as a - a, a - b - b, a - a - b - b - c - c. It also includes adding other elements besides the listed elements (a, b and c), such as having d such as a - b - c - d.

In the present specification (including claims), when expressions such as "using data as input/using/based on/according to/according to data" (including similar expressions) are used, If there is no data, use the data itself, or use the data that has been processed in some way (e.g., noise added, normalized, features extracted from the data, intermediate representation of the data, etc.) . In addition, if it is stated that some result can be obtained "using data as input/using/based on/according to data" (including similar expressions), unless otherwise specified, This includes cases where the results are obtained based solely on the data, and cases where the results are obtained under the influence of other data, factors, conditions and/or conditions. In addition, if it is stated that "data will be output" (including similar expressions), if there is no particular notice, if the data itself is used as output, or if the data has undergone some processing (for example, noise (addition, normalization, features extracted from data, intermediate representation of data, etc.) are used as output.

In this specification (including claims), when the terms "connected" and "coupled" are used, they refer to direct connection/coupling, indirect connection/coupling , electrically connected/coupled, communicatively connected/coupled, operatively connected/coupled, physically connected/coupled, etc. intended as a term. The term should be interpreted appropriately according to the context in which the term is used, but any form of connection/bonding that is not intentionally or naturally excluded is not included in the term. should be interpreted restrictively.

In this specification (including claims), when the expression "A configured to B" is used, the physical structure of element A is such that it is capable of performing action B. has a configuration, including that a permanent or temporary setting / configuration of element A is configured / set to actually perform action B good. For example, if element A is a general-purpose processor, the processor has a hardware configuration that can execute operation B, and operation B can be executed by setting a permanent or temporary program (instruction). It just needs to be configured to actually run. In addition, when the element A is a dedicated processor or a dedicated arithmetic circuit, etc., regardless of whether or not control instructions and data are actually attached, the circuit structure etc. of the processor is designed to actually execute the operation B. should be implemented in .

In the present specification (including the claims), when terms denoting containing or possessing (e.g., "comprising / including" and "having", etc.) are used, by the object of the terms Intended as an open-ended term, including the case of containing or possessing things other than the indicated object. Where the object of these terms of inclusion or possession is an expression which does not specify a quantity or implies a singular number (expressions with the articles a or an), the expression shall be construed as not being limited to a particular number. It should be.

In the specification (including the claims), expressions such as "one or more" or "at least one" are used in some places and quantities are specified in other places. Where no or singular-indicating expressions are used (with the articles a or an), the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or imply a singular number (indicative of the articles a or an) should be construed as not necessarily being limited to a particular number.

In this specification, when it is stated that a particular configuration of an embodiment has a particular advantage / result, unless there is a reason to the contrary, other one or more having that configuration It should be understood that this effect can also be obtained with the embodiment of However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and/or states, and that the configuration does not always provide the effect. The effect is only obtained by the configuration described in the embodiment when various factors, conditions and/or states are satisfied, and in the claimed invention defining the configuration or a similar configuration, The said effect is not necessarily acquired.

In this specification (including claims), when terms such as "maximize/maximization" are used, they refer to finding a global maximum, or approximating a global maximum. Including determining, determining a local maximum, and determining an approximation of a local maximum, should be interpreted appropriately depending on the context in which the term is used. It also includes probabilistically or heuristically approximating these maximum values. Similarly, when terms such as "minimize/minimization" are used, they refer to finding a global minimum, finding an approximation of a global minimum, and finding a local minimum. Including finding and finding an approximation of a local minimum, should be interpreted appropriately depending on the context in which the term is used. It also includes stochastically or heuristically approximating these minimum values. Similarly, when terms such as "optimize/optimization" are used, they refer to finding a global optimum, finding an approximation of a global optimum, and finding a local optimum. Including seeking, and finding an approximation of a local optimum, should be interpreted appropriately depending on the context in which the term is used. It also includes stochastically or heuristically approximating these optimum values.

In this specification (including claims), when a plurality of pieces of hardware perform a given process, each piece of hardware may work together to perform the given process, or a part of the hardware may perform the given process. You may do all of Also, some hardware may perform a part of the predetermined processing, and another hardware may perform the rest of the predetermined processing. In this specification (including claims), expressions such as "one or more pieces of hardware perform the first process, and the one or more pieces of hardware perform the second process" (including similar expressions) If used, the hardware that performs the first process and the hardware that performs the second process may be the same or different. In other words, the hardware that performs the first process and the hardware that performs the second process may be included in the one or more pieces of hardware. Note that hardware may include an electronic circuit or a device including an electronic circuit.

In this specification (including claims), when a plurality of storage devices (memories) store data, each of the plurality of storage devices may store only part of the data. , may store the entire data. Further, a configuration may be included in which some of the plurality of storage devices store data.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, replacements, partial deletions, etc. are possible without departing from the conceptual idea and spirit of the present disclosure derived from the content defined in the claims and equivalents thereof. For example, where numerical values or mathematical formulas are used in the descriptions of the above-described embodiments, they are provided for illustrative purposes and are not intended to limit the scope of the disclosure. Also, the order of each operation shown in the embodiment is an example and does not limit the scope of the present disclosure.

Claims (14)

one or more memories storing parameters of trained first, second and third models;
one or more processors capable of forming the first model, the second model, or the third model based on parameters stored in the one or more memories;
with
The one or more processors are
dividing a first image and a second image one unit time before the first image into tiles;
The tiles of the first image and the tiles of the second image are input into the first model to obtain discretized latent variables of the tiles of the first image and discretized latent variables of the tiles of the second image. get information about
inputting information of the discretized latent variables of the tiles of the second image into the third model to obtain a probability distribution of the discretized latent variables for the latent variables of the tiles of the first image;
encoding the discretized latent variables of tiles of the first image based on the probability distribution to obtain a code;
storing the code information in the one or more memories in association with the time information of the first image;
Compressor. The one or more processors are
inputting the first image and the second image into a second model to obtain a hyperpliance code;
inputting the hyperpliance code and information of the discretized latent variables of the tiles of the second image into the third model to obtain the probability distribution;
Compression apparatus according to claim 1 . When the first image is a keyframe, the one or more processors
dividing the first image into tiles;
inputting information of the tile into the first model to acquire information of a discretized latent variable for the tile;
storing the information of the latent variables in the one or more memories in association with information about the time of the first image and a flag indicating a key frame;
If the first image is not a keyframe, the one or more processors
dividing the first image and a second image one unit time before the first image into tiles;
inputting the first image and the second image into a second model to obtain a hyperpliance code;
inputting the tiles of the first image and the tiles of the second image into the first model to obtain information of the discretized latent variables of the tiles of the first image and the tiles of the second image;
inputting the hyperplier code and information of the discretized latent variables of the tiles of the second image into the third model, and calculating a probability distribution of the discretized latent variables for the latent variables of the tiles of the first image; and get
encoding the discretized latent variable of a tile of the first image based on the probability distribution to obtain a code;
storing the code information in the one or more memories in association with the time information of the first image;
3. Compression apparatus according to claim 2. The one or more processors are
encoding the discretized latent variables of tiles of the first image by entropy coding using the probability distribution;
A compression device according to any one of claims 1 to 3. one or more memories storing parameters of the third model and the fourth model trained by the trained trainer;
one or more processors capable of forming the third model or the fourth model based on parameters stored in the one or more memories;
with
The one or more processors are
receiving compressed data containing information encoded in the discretized latent variables of the tiles of the first image;
The discretized latent variables of the second image one unit time before the first image, which have already been acquired, are input to the third model, and the discretized latent variables of the tiles of the first image are input to the third model. get the probability distribution,
decoding the information encoded with the discretized latent variables of the tiles of the first image contained in the compressed data using the probability distribution;
inputting the decoded latent variables into the fourth model to obtain the color values of the tiles of the first image;
restoration device. The one or more processors are
Inputting the already obtained discretized latent variables of the second image and the hyperplier code of the first image obtained from the compressed data into the third model to obtain the probability distribution. 6. The restoration device according to claim 5. If the compressed first image is a keyframe, the one or more processors:
obtaining discretized latent variables of the compressed first image from compressed data;
inputting the latent variables into the fourth model to reconstruct the first image;
If the compressed first image is not a keyframe, the one or more processors:
The discretized latent variables of the second image one unit time before the first image, which have already been acquired, and the hyperprior code of the first image obtained from the compressed data are input to the third model. and obtaining a probability distribution of the discretized latent variables for the discretized latent variables of the tiles of the first image;
decoding encoded information of the discretized latent variables of tiles of the first image obtained from the compressed data using the probability distribution;
inputting the decoded latent variables into the fourth model to obtain the color values of the first image;
7. The restoration device according to claim 6. The one or more processors are
decoding the latent variable information of the tiles of the first image encoded by entropy coding using the probability distribution;
8. The restoration device according to any one of claims 5 to 7. dividing a first image and a second image one unit time before the first image into tiles;
estimating a discretized latent variable of the tile using a first model;
estimating the color value of the pixel corresponding to the position from the position in the image of the tile of the first image and the latent variable using a fourth model;
estimating, from the first image and the second image, a hyperprior code for the probability distribution of the latent variable corresponding to the first image using a second model;
estimating the probability distribution of the latent variables of the tile images of the first image from the hyperpliance code and the latent variables of the tile images of the second image using a third model;
the color value estimated using the fourth model, the correct color value, and the probability distribution obtained using the third model according to the probability distribution estimated using the second model; , the probability of the hyperplier code estimated using the second model, and training the first model, the second model, the third model and the fourth model. ,
Thus, a method of generating at least one model of said first model, said second model, said third model and said fourth model. Evaluation of restoration accuracy based on the difference between the color value and the correct color value, and the probability distribution obtained using the third model according to the probability distribution estimated using the second model. and an evaluation of the compression ratio based on the probability of the hyperplier code estimated using the second model, the first model, the second model, the third model, and the train a fourth model and
10. The method of claim 9. The first model is a model that estimates a probability distribution for a plurality of predefined embedding vectors for the latent variables of the tile.
11. A method according to claim 9 or claim 10. The second model is a model that assumes the first image as a posterior distribution and estimates the hyperprior code, which is a vector relating to changes from the second image, which is the prior distribution.
12. A method according to any of claims 9-11. The third model, based on the hyperplier code and the latent variables of the tiles of the second image, converts the tiles of the second image to the tiles of the first image into the tiles of the first image. estimating the latent variable as the probability and distribution of the latent variable in the second image;
13. A method according to any of claims 9-12. The fourth model is a model that obtains the color value of a pixel at an arbitrary position in the tile from the latent variable.
14. A method according to any of claims 9-13.

Images