JP7475842B2 - Image decoding device, control method, and program - Google Patents

Description

The present invention relates to an image decoding device, a control method, and a program for decoding encoded data.

In recent years, imaging devices such as digital cameras and digital camcorders have been used that have imaging elements that use image sensors such as CCD sensors and CMOS sensors. In such image sensors, one pixel constitutes one color component due to a color filter array (CFA) provided on the sensor surface. By using a CFA, image data (hereinafter sometimes referred to as RAW data) in a Bayer array in which color components (R (red), G0 (green), B (blue), G1 (green)) are arranged in a periodic pattern as shown in Figure 1 can be obtained.

Human vision has a relatively high sensitivity to luminance components. Based on the above knowledge, in a typical Bayer array, as shown in Figure 1, the number of pixels for the green component, which contains more luminance components, is allocated to be twice as many as the number of pixels for the red component and the number of pixels for the blue component.

Therefore, each pixel of the RAW data only has information for one color component. In image processing by the imaging device, demosaic processing is performed on the RAW data, so that each pixel has red, blue, and green color components. Generally, image data that encodes the RGB signal obtained by demosaic processing of the RAW data, or the YUV signal converted from the RGB signal, is recorded in the imaging device.

Image data in which each pixel has three color components, RGB or YUV, due to the above demosaic processing requires three times the amount of data as the original RAW data. Therefore, a method has been proposed to reduce the data volume by directly encoding and recording the RAW data itself that has not been subjected to demosaic processing. For example, Patent Document 1 discloses a method in which RAW data is separated into four color planes, R, G0, B, and G1, and then encoded.

JP 2003-125209 A

Regarding the above-mentioned encoding, when data is encoded for each frequency band (e.g., subband) by frequency transformation such as wavelet transform, there are cases where the degree of quantization differs for each frequency band. If the encoded data as described above is processed collectively regardless of the frequency band, there is a problem that the original data cannot be restored with high accuracy.

In view of the above, the present invention aims to provide an image decoding device, a control method, and a program that can restore encoded data with higher accuracy.

In order to achieve the above object, an image decoding device of the present invention is an image decoding device that decodes encoded data obtained by quantizing and encoding a plurality of subband data obtained by performing a frequency transform on image data , and includes a decoding means for decoding the encoded data, an inverse quantization means for inverse quantizing the data decoded by the decoding means to obtain a plurality of subband data, and an inference means for performing inference on the plurality of subband data obtained by the inverse quantization means to obtain a plurality of subband data in which data deteriorated by quantization has been restored, the inference means performing inference on the subband data corresponding to each of the plurality of subbands using inference parameters corresponding to the subband , the inference means performing a first inference on the plurality of subband data using first inference parameters which are inference parameters learned to correspond to each of the plurality of subbands, and performing a second inference on the subband data restored by the first inference using the learned second inference parameters .

The present invention makes it possible to restore encoded data with higher accuracy.

FIG. 2 is an explanatory diagram of a Bayer array. 1 is a block diagram illustrating a configuration of an image encoding device according to a first embodiment of the present invention. FIG. 4 is an explanatory diagram of plane conversion in the first embodiment of the present invention. 1 is an explanatory diagram of a reversible 5-3 DWT transformation in a first embodiment of the present invention. FIG. 2 is an explanatory diagram of subband decomposition in the first embodiment of the present invention. 1 is a block diagram illustrating a configuration of an image decoding device according to a first embodiment of the present invention. FIG. 2 is an explanatory diagram of a reversible 5-3 inverse DWT transformation in the first embodiment of the present invention. 2 is a block diagram illustrating a configuration of a frequency degradation restoration unit according to the first embodiment of the present invention; FIG. FIG. 1 is a diagram illustrating a configuration of a neuron according to a first embodiment of the present invention. FIG. 1 is a diagram illustrating a configuration of a neural network according to a first embodiment of the present invention. FIG. 2 is a diagram illustrating another configuration of the neural network according to the first embodiment of the present invention. FIG. 2 is an explanatory diagram of a learning process of a neural network according to the first embodiment of the present invention. FIG. 11 is a block diagram illustrating the configuration of an image decoding device according to a second embodiment of the present invention. FIG. 2 is an explanatory diagram of information loss of DWT coefficients due to quantization. FIG. 11 is an explanatory diagram of a learning process of a neural network according to a second embodiment of the present invention. 13 is a flowchart showing a restoration process according to a compression ratio in a second embodiment of the present invention. FIG. 11 is a block diagram illustrating another configuration of the image decoding device according to the second embodiment of the present invention. FIG. 13 is a block diagram illustrating the configuration of an image decoding device according to a third embodiment of the present invention. FIG. 11 is an explanatory diagram of a quantization matrix of 4×4 DCT and learning/inference groups in a modified example of the present invention.

The following describes in detail the embodiments of the present invention with reference to the accompanying drawings. Each embodiment described below is merely one example of a configuration that can realize the present invention. Each of the following embodiments can be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. Furthermore, not all of the combinations of elements included in each of the following embodiments are necessarily essential to realize the present invention, and some of the elements can be omitted as appropriate. Therefore, the scope of the present invention is not limited to the configurations described in each of the following embodiments. Furthermore, a configuration that combines multiple configurations described in the embodiments can be adopted as long as there are no mutual contradictions.

First Embodiment
Prior to describing the image decoding device 60 according to the first embodiment of the present invention, the generation of encoded data to be decoded by the image decoding device 60 will be described with reference to FIGS. 2 to 6. FIG.

FIG. 2 is a block diagram illustrating a configuration of an image encoding device 20 according to an embodiment of the present invention. As shown in FIG. 2, the image encoding device 20 includes a plane transform unit 200, a frequency transform unit 201, a quantization unit 202, a quantization parameter setting unit 203, and an entropy encoding unit 204.

In this embodiment, JPEG2000 is exemplified as an encoding method, and reversible 5-3 DWT is exemplified as a frequency transform method. However, the present invention is not limited to the above methods, and any encoding method and frequency transform method may be adopted. In addition, the quantization parameter in this embodiment is set for each subband (frequency band) divided by the DWT transform.

The plane conversion unit 200 performs color separation on image data (RAW data) input from an imaging means including an image sensor. That is, as shown in FIG. 3, the plane conversion unit 200 separates the RAW data including color elements of the Bayer array into independent plane data for each color plane (R, G0, G1, B) and outputs the data to the frequency conversion unit 201.

The frequency transform unit 201 performs a wavelet transform on each piece of plain data input from the plane transform unit 200, generates transform coefficients (subband coefficients) for each subband, and outputs them to the quantization unit 202. The transform coefficients are values that indicate the correlation between the target data (i.e., the plain data) and the wavelet. As described above, the frequency transform unit 201 performs a reversible 5-3 DWT transform as a wavelet transform.

The reversible 5-3 DWT transform (hereinafter simply referred to as "DWT transform") will be described with reference to Figures 4 and 5. Figure 4 is an explanatory diagram explaining transform coefficients (DWT coefficients) generated by DWT transform of pixel data (plane data) output from the plane transform unit 200. Figure 5 is an explanatory diagram of subband decomposition.

When the frequency transform unit 201 performs DWT transform on the pixel data a, b, c, d, and e, DWT coefficients b' and d' of high frequency components are generated. More specifically, the frequency transform unit 201 applies the following equation (1) to the pixel data a, b, and c to generate a DWT coefficient b', and applies the following equation (2) to the pixel data c, d, and e to generate a DWT coefficient d'. Note that equations (1) and (2) for generating the DWT coefficients of high frequency components are of the same type, although they use different pixel data.
b'=b-(a+c)/2 ......Equation (1)
d'=d-(c+e)/2 ... Equation (2)

Next, the frequency transform unit 201 performs a DWT transform on the pixel data c and the DWT coefficients b', d' of the high frequency components to generate a DWT coefficient c" of the low frequency components. More specifically, the DWT coefficient c" is generated by applying the following equation (3) to the pixel data c and the DWT coefficients b', d'.
c" = c + (b' + d' + 2) / 4 ... formula (3)

Note that the frequency transform unit 201 may apply the following equation (4) to the pixel data a, b, c, d, and e, instead of equation (3), to generate the DWT coefficient c″.
c" = (a + 2b + 6c + 2d - e) / 8 ... formula (4)

The frequency transform unit 201 performs the above-mentioned one-dimensional DWT transform on the plane data in the vertical and horizontal directions to obtain signals of four subband images at decomposition level 1 as shown in FIG. 5(a). That is, one plane image is decomposed into four subband images 1LL, 1HL, 1LH, and 1HH by the above-mentioned DWT transform. In FIG. 5, "H" indicates high-frequency components and "L" indicates low-frequency components. For example, "1HH" in the lower right of FIG. 5(a) indicates a subband at decomposition level 1 in which both the horizontal and vertical directions are high-frequency components (H).

As shown in FIG. 5(a), the number of horizontal and vertical coefficients in each subband of decomposition level 1 is half the number of coefficients in the input pixel data. Also, as shown in FIG. 5(b), the number of horizontal and vertical coefficients in each subband of decomposition level 2, which is obtained by further performing a DWT transform on subband 1LL, is half the number of coefficients in the subbands of decomposition level 1.

The quantization unit 202 quantizes the DWT coefficients (transformation coefficients) for each subband input from the frequency transformation unit 201 according to the quantization parameters set by the quantization parameter setting unit 203 described later, and outputs the quantized coefficients to the entropy coding unit 204.

The quantization parameter setting unit 203 specifies the quantization parameters for quantizing the DWT coefficients of each subband according to the compression ratio set by the user or the like, and sets them in the quantization unit 202. In general, the higher the frequency of the subband or the lower the decomposition level of the subband, the stronger the quantization is set, thereby improving image quality with the same amount of code. This is because the more the subband is quantized, the smaller the visual impact is when quantized. In this example, when frequency conversion is performed up to decomposition level 2, it is preferable for the quantization parameter setting unit 203 to set the parameters so that the quantization strength is 2HH>2HL≒2LH>1HH>1HL≒1LH>1LL.

The entropy coding unit 204 generates coded data by coding the DWT coefficients and quantization parameters quantized by the quantization unit 202, and outputs the coded data as a coded stream. In the coding described above, for example, entropy coding such as EBCOT (Embedded Block Coding with Optimized Truncation) is performed.

Next, the configuration and decoding method of the image decoding device 60 according to the first embodiment of the present invention will be described. The image decoding device 60 decodes the encoded data (subband encoded image data) generated by the image encoding device 20 described above as described below, and restores the original image data (raw data in the Bayer array).

Figure 6 is a block diagram illustrating the configuration of an image decoding device 60 according to the first embodiment of the present invention. As shown in Figure 6, the image decoding device 60 includes, as functional blocks, an entropy decoding unit 600, an inverse quantization unit 601, a frequency degradation restoration unit 602, an inverse frequency transformation unit 603, a Bayer transformation unit 604, and a frequency parameter setting unit 605.

The following processing of this embodiment, which is executed by the above functional blocks, is realized by one or more control processors of the image decoding device 60 expanding a program in a non-volatile memory such as a ROM into a volatile memory such as a RAM and executing it. The above-described processing of this embodiment includes learning and inference using a learning model (neural network) according to this embodiment, which will be described later.

The entropy decoding unit 600 decodes the DWT coefficients and quantization parameters that have been coded using an entropy coding technique such as EBCOT to obtain the decoded DWT coefficients, which it then outputs to the inverse quantization unit 601.

The inverse quantization unit 601 inverse quantizes the decoded DWT coefficients input from the entropy decoding unit 600 using the quantization parameter to obtain the inverse quantized DWT coefficients, and outputs them to the frequency degradation restoration unit 602.

The frequency parameter setting unit 605 selects the neural network parameters (weights and biases) learned for each subband and compression ratio according to the compression ratio of the encoded stream to be decoded, and sets them in the frequency degradation restoration unit 602. Hereinafter, the neural network may be abbreviated as "NN."

The frequency degradation restoration unit 602 (inference means) applies a neural network having inference parameters set by the frequency parameter setting unit 605 to the inversely quantized DWT coefficients input from the inverse quantization unit 601, and restores the DWT coefficients degraded by quantization by inference. The configuration of the frequency degradation restoration unit 602 and the parameter learning process will be described in detail later.

The inverse frequency transform unit 603 performs an inverse frequency transform (inverse DWT transform) on the restored DWT coefficients input from the frequency degradation restoration unit 602 to reconstruct independent plane data for each color plane (R, G0, G1, B). The reconstructed plane data is output to the Bayer transform unit 604.

With reference to FIG. 7, the reversible 5-3 DWT inverse transform, which is the inverse DWT transform executed by the inverse frequency transform unit 603, will be described. In FIG. 7, DWT coefficients a', c', and e' are high-frequency frequency transform coefficients, and DWT coefficients b", and d" are low-frequency frequency transform coefficients. When the inverse frequency transform unit 603 executes the inverse DWT transform on the DWT coefficients a', b", c', d", and e', pixel data b, c, and d are generated. More specifically, the inverse frequency transform unit 603 applies the following equation (5) to the DWT coefficients a', b", and c' to generate pixel data b, and applies the following equation (6) to the DWT coefficients c', d", and e' to generate pixel data d. Note that equations (5) and (6) for generating pixel data are of the same type, although the DWT coefficients used are different from each other. The pixel data b and d in the second row of FIG. 7 indicate even-numbered pixel data in each plane when the pixel at the start position of the DWT transformation is the 0th pixel.
b=b″-(a′+c′+2)/4 …Equation (5)
d=d″-(c′+e′+2)/4 …Equation (6)

Next, the inverse frequency transform unit 603 performs an inverse DWT transform on the DWT coefficients c' and the pixel data b and d to generate pixel data c. More specifically, the pixel data c is generated by applying the following equation (7) to the DWT coefficients c' and the pixel data b and d. The pixel data c in the third row of Fig. 7 indicates odd-numbered pixel data in each plane when the pixel at the DWT transform start position is the 0th pixel.
c=c′+(b+d)/2 ……Equation (7)

The inverse frequency transform unit 603 reconstructs the pixel data of each plane by repeatedly performing the above-mentioned inverse DWT transform in the horizontal and vertical directions.

The Bayer conversion unit 604 recomposes the independent plane data for each color plane (R, G0, G1, B) reconstructed by the inverse frequency conversion unit 603 into a RAW image in a Bayer array, and outputs RAW data equivalent to the RAW image.

Next, the inference-based restoration of degraded DWT coefficients using the learning model (neural network) in the frequency degradation restoration unit 602 will be described in detail with reference to Figs. 8 to 12. The restoration of degraded DWT coefficients at decomposition level 1 as shown in Fig. 5(a) will be described as an example.

As shown in FIG. 8, the frequency degradation restoration unit 602 has a 1LL restoration unit 800, a 1HL restoration unit 801, a 1LH restoration unit 802, and a 1HH restoration unit 803. That is, the frequency degradation restoration unit 602 has four restoration units 800, 801, 802, and 803 corresponding to the four subbands 1LL, 1HL, 1LH, and 1HH, respectively.

As shown in the figure, the 1LL restoration unit 800 applies an NN with corresponding parameters to the DWT coefficients (subband data) of the 1LL subband input from the inverse quantization unit 601 to perform inference, and outputs the 1LL subband with the degradation due to quantization restored. Similarly, the other restoration units 801, 802, and 803 each apply an NN with corresponding parameters to the DWT coefficients of the subbands from the inverse quantization unit 801, and output the DWT coefficients of the subbands with the degradation due to quantization restored.

The configurations of the NNs in the 1LL restoration unit 800, 1HL restoration unit 801, 1LH restoration unit 802, and 1HH restoration unit 803 may be the same or different from each other. On the other hand, the parameters (weights, biases) of the NNs in the 1LL restoration unit 800, 1HL restoration unit 801, 1LH restoration unit 802, and 1HH restoration unit 803 are different from each other.

9 is a diagram illustrating the configuration of a neuron 900, which is the computation unit of the neural network in the restoration units 800 to 803. The NN of this embodiment includes a plurality of neurons 900. The neuron 900 performs computations on a plurality of input values x ₁ to x _N using weights w ₁ to w _N , bias b, and an activation function, and outputs an output value y. More details are as follows.

Neuron 900 calculates value x' using weights _w1 to _wN and bias b as shown in the following formula (8). Weights _w1 to _wN and bias b are values that are variably determined by a learning process described later, and are parameters that can take different values for each of restoration units 800 to 803 as described above.

Next, the neuron 900 inputs the calculated value x' into an activation function to calculate the output y. The activation function is a nonlinear function such as a sigmoid function or a ReLU (Rectified Linear Unit) function.

When the value x' is given to the sigmoid function, the output value y is calculated using the following formula (9).

When the value x' is given to the ReLU function, the output value y is calculated using the following formula (10).

Figure 10 is a diagram illustrating the configuration of the NN in the restoration units 800 to 803. As shown in Figure 10, the NN of this embodiment has a four-layer structure with an input layer 1000, a first intermediate layer 1001, a second intermediate layer 1002, and an output layer 1003.

Two consecutive layers are connected by one or more neurons 900. The output value of the previous layer is input to the neuron 900, and the output value from the aforementioned calculation process is output to the subsequent layer.

The number of data in ₀ to in _N input to the input layer 1000 is equal to the number of data out ₀ to out _N output from the output layer 1003. On the other hand, the number of data mid ₀₀ to mid _0p in the first hidden layer 1001 and the number of data mid ₁₁ to mid _1q in the second hidden layer 1002 do not have to be equal to the number of data in the input layer 1000 and the output layer 1003. Therefore, the number of neurons 900 connecting the two layers may be any number equal to or greater than one.

The data in ₀ to in _N input to the input layer 1000 are DWT coefficients for each subband, and the data out ₀ to out _N output from the output layer 1003 are DWT coefficients for each subband with the degradation restored. That is, the DWT coefficients of the 1LL subband are input to NN of the 1LL restoration unit 800, and the DWT coefficients of the 1LL subband with the degradation restored by inference are output. Similarly, the DWT coefficients of the 1HL subband, the 1LH subband, and the 1HH subband are input to the other restoration units 801, 802, and 803, respectively, and the DWT coefficients of the 1HL subband, the 1LH subband, and the 1HH subband with the degradation restored are output.

Figure 11 is a diagram illustrating another configuration of the NN in the restoration units 800 to 803. As shown in Figure 11, another NN in this embodiment has a four-layer structure having an input layer 1100, a first intermediate layer 1101, a second intermediate layer 1102, and an output layer 1103.

The NN in Fig. 11 includes skip connections that directly connect distant layers (input layer 1100 and second hidden layer 1102). The dashed arrow between input layer 1100 and first hidden layer 1101 indicates the skipped portion. As shown in the figure, data _in0 and _in1 of input layer 1100 are output directly to second hidden layer 1102, skipping first hidden layer 1101. As described above, restoration units 800 to 803 may be NNs that include skip layers.

At least one of the restoration units 800 to 803 may have an NN with the structure shown in FIG. 10, and at least another of the restoration units 800 to 803 may have an NN with the structure shown in FIG. 11.

12 is an explanatory diagram of the learning process of the neural network according to the first embodiment of the present invention. In this embodiment, as described below, parameters for restoration (weights w ₁ to w _N , bias b) are learned for each subband. In FIG. 12, the learning process for the 1HH subband is described, but the same learning process can be applied to other subbands (1HL subband, 1LH subband, 1HH subband).

Overall, in the learning process of this embodiment, the input data are DWT coefficients for each subband, the output data are restored DWT coefficients in which degradation due to quantization has been restored, and the training data are undegraded DWT coefficients of the original image without quantization degradation.

The DWT coefficients 1200 of the 1HH subband quantized by the quantization unit 202 of the image encoding device 20 are input to the 1HH restoration unit 803 of the frequency degradation restoration unit 602. The frequency parameter setting unit 605 sets frequency parameters (weight, bias) for the 1HH subband in the 1HH restoration unit 803. The initial values of the frequency parameters set in the 1HH restoration unit 803 are set arbitrarily, for example, to values determined by random numbers.

The 1HH restoration unit 803 uses a neural network to which the set frequency parameters are applied to restore the degradation caused by quantization of the DWT coefficients 1200 of the 1HH subband, and outputs the restored DWT coefficients 1201 of the 1HH subband. The output restored DWT coefficients 1201 are input to the parameter update unit 1203.

In addition, the parameter update unit 1203 receives undegraded DWT coefficients 1202 corresponding to the original image of the 1HH subband as training data. The undegraded DWT coefficients 1202 are DWT coefficients output by the frequency transform unit 201 of the image encoding device 20, and are DWT coefficients that are not degraded by quantization.

The parameter update unit 1203 obtains an index indicating the comparison result between the input restored DWT coefficients 1201 and the undegraded DWT coefficients 1202, and updates the frequency parameters for the 1HH subband according to an update method such as backpropagation. For example, the peak signal-to-noise ratio (PSNR) or the sum of absolute differences (SAD) can be used as the index. When the peak signal-to-noise ratio is used as the index, the parameter update unit 1203 updates the frequency parameters so that the value of the index increases. When the sum of absolute differences is used as the index, the parameter update unit 1203 updates the frequency parameters so that the value of the index decreases.

The same processing as above is performed for the restoration units 800 to 802 corresponding to the other subbands.

By performing the above learning process using a large amount of image data (a set of restored DWT coefficients 1201 and undegraded DWT coefficients 1202), the frequency parameters that the frequency parameter setting unit 605 sets in the frequency degradation restoration unit 602 are determined.

In general, the degree of quantization varies depending on the frequency and decomposition level of the subband. For example, high-frequency and low-level subbands are quantized relatively strongly to reduce visual degradation, while low-frequency and high-level subbands are quantized relatively weakly. According to the configuration of the present embodiment described above, by performing a learning process for each of multiple subbands with different degrees of quantization, the parameters of the frequency degradation restoration unit 602 are adjusted for each subband, thereby enabling the image to be restored with higher accuracy.

The above learning process may be performed for each compression ratio (recording image quality in the case of a digital camera) that can be selected by the user. By performing learning for each compression ratio and setting the frequency parameters, it is possible to achieve more appropriate restoration that reflects the degree of degradation for each compression ratio.

Second Embodiment
A second embodiment of the present invention will be described below. In each of the embodiments exemplified below, elements that have the same actions and functions as those of the first embodiment will be designated by the same reference numerals as those in the above description, and the description of each element will be omitted as appropriate.

Figure 13 is a block diagram illustrating the configuration of an image decoding device 130 according to the second embodiment of the present invention. As shown in Figure 13, the image decoding device 130 includes, as functional blocks, a pixel degradation restoration unit 1300 and a pixel parameter setting unit 1301 in addition to an entropy decoding unit 600 through a frequency parameter setting unit 605. As in the first embodiment, the following processing of this embodiment executed by the above functional blocks is realized by one or more control processors possessed by the image decoding device 130 expanding a program in a non-volatile memory such as a ROM into a volatile memory such as a RAM and executing it.

The pixel parameter setting unit 1301 selects the pixel parameters (weights, biases) learned for each compression ratio according to the compression ratio of the encoded stream to be decoded, and sets them in the pixel degradation restoration unit 1300.

The pixel degradation restoration unit 1300 (inference means) applies a neural network having inference parameters set by the pixel parameter setting unit 1301 to the raw image of the Bayer array output by the Bayer conversion unit 604, and restores pixel data degraded by quantization by inference.

The reason why a two-stage restoration process in the frequency domain and the pixel domain is preferable will be described with reference to Fig. 14. Fig. 14 illustrates an example of the change in value before and after quantization when relatively strong quantization is performed in units (quantization steps) of 511 (=2 ⁹ -1) when the maximum value of the DWT coefficient is 1023 (=2 ¹⁰ -1). Fig. 14(a) shows the DWT coefficient before quantization, and Fig. 14(b) shows the DWT coefficient after quantization.

As shown, if the pre-quantization DWT coefficient is less than 511, the quantization will result in the DWT coefficient becoming 0 (i.e., the DWT coefficient is lost). Also, even if the pre-quantization DWT coefficient is 511 or greater and the post-quantization DWT coefficient is not 0, an error will occur in the pre-quantization DWT coefficient with a value in the range 0 to 510. When the compression ratio is relatively high (e.g., for the 1HH subband, which has a low decomposition level and high frequency), relatively strong quantization is applied, which may make restoration in the frequency domain difficult.

Therefore, in this embodiment, image quality degradation that could not be completely restored in the frequency domain is inferred and complemented based on a pixel domain constructed with reference to other subbands, thereby achieving a more accurate restoration (closer to the original image).

Figure 15 is an explanatory diagram of the learning process of a neural network according to the second embodiment of the present invention.

The DWT coefficients 1500 of the 1LL, 1HL, 1LH, and 1HH subbands quantized by the quantization unit 202 of the image encoding device 20 are input to the frequency degradation restoration unit 602 (inference means). As in the first embodiment, the frequency parameter setting unit 605 sets the frequency parameters (weights, biases) updated by learning for each subband to the frequency degradation restoration unit 602.

The frequency degradation restoration unit 602 uses the NN to which the frequency parameters are applied as described above to restore the DWT coefficients 1500 that have been degraded by quantization, and outputs them to the inverse frequency transform unit 603.

The inverse frequency transform unit 603 performs an inverse frequency transform (inverse DWT transform) on the restored DWT coefficients input from the frequency degradation restoration unit 602 to reconstruct independent plane data for each color plane (R, G0, G1, B). The reconstructed plane data is output to the Bayer transform unit 604.

The Bayer conversion unit 604 recomposes the independent plane data for each color plane (R, G0, G1, B) reconstructed by the inverse frequency conversion unit 603 into a RAW image in a Bayer array, and outputs RAW data equivalent to the RAW image. The output RAW data is input to the pixel degradation restoration unit 1300.

The pixel degradation restoration unit 1300 uses a neural network to which the pixel parameters set by the pixel parameter setting unit 1301 are applied to restore pixel data in a raw image that has been degraded by quantization, and outputs a restored raw image 1501. The restored raw image 1501 is input to the pixel parameter update unit 1502.

In addition, an original raw image 1503 is input to the pixel parameter update unit 1502 as training data. The original raw image 1503 is raw image data input to the image encoding device 20, and is raw image data that is not subject to degradation due to quantization.

The pixel parameter update unit 1502 obtains an index indicating the comparison result between the input restored raw image 1501 and the original raw image 1503, and updates the pixel parameters according to an update method such as the backpropagation method. For example, the peak signal-to-noise ratio (PSNR) or the sum of absolute differences (SAD) can be used as the index. When the peak signal-to-noise ratio is used as the index, the pixel parameter update unit 1502 updates the pixel parameters so that the value of the index increases. When the sum of absolute differences is used as the index, the pixel parameter update unit 1502 updates the pixel parameters so that the value of the index decreases.

By performing the above learning process using a large amount of image data (a pair of restored RAW images 1501 and original RAW images 1503), the pixel parameters that the pixel parameter setting unit 1301 sets in the pixel degradation restoration unit 1300 are determined.

The configuration of this embodiment described above provides the same technical effect as the first embodiment. In addition, image quality degradation that may be difficult to restore using restoration processing in the frequency domain is compensated for by restoration processing in the pixel domain, allowing images to be restored with higher accuracy.

The above learning process may be performed for each compression ratio (recording image quality in the case of a digital camera) that can be selected by the user. By performing learning for each compression ratio and setting pixel parameters, it is possible to achieve more appropriate restoration that reflects the degree of degradation for each compression ratio.

In this embodiment, a configuration may be adopted in which the processing (learning/inference) in one or more of the restoration units 800-803 included in the frequency degradation restoration unit 602 is skipped. As described above, relatively strong (relatively high compression rate) quantization is performed on subbands with low decomposition levels and high frequencies, so that it may be difficult to properly perform learning and inference in the frequency domain. In this configuration, the restoration of subbands (frequency bands) where learning and inference in the frequency domain are difficult is covered by restoration processing in the pixel domain, so that the image can be restored with higher accuracy. In addition, since learning and inference in the frequency domain for a specified subband is skipped, the time required to perform the learning and inference processing and the frequency parameters to be learned and stored can be reduced.

In the above configuration, processing (learning and inference) for a specific subband may be skipped only when a compression rate exceeding a specific compression rate α (threshold value) is applied. FIG. 16 is a flowchart showing the processing of the frequency degradation restoration unit 602 when the processing by the 1HH restoration unit 803 of the frequency degradation restoration unit 602 is skipped when the compression rate to be applied exceeds the specific compression rate α.

In step S1600, the frequency degradation restoration unit 602 determines whether the compression ratio set by the user is smaller than a predetermined compression ratio α.

If the set compression ratio is smaller than α (S1600: YES), the process proceeds to step S1601. In step S1601, the frequency degradation restoration unit 602 performs restoration processing for each subband, including restoration processing of the DWT coefficients of the 1HH subband using the NN (1HH restoration unit 803), and outputs the restored DWT coefficients of each subband.

If the set compression rate is equal to or greater than α (S1600: NO), the process proceeds to step S1602. In step S1602, the frequency degradation restoration unit 602 skips the restoration process of the DWT coefficients of the 1HH subband using the NN (1HH restoration unit 803), and outputs the unprocessed DWT coefficients of the 1HH subband. For the other subbands, the frequency degradation restoration unit 602 (restoration units 800 to 802) executes restoration processing as in the first embodiment, and outputs the restored DWT coefficients.

In the above-described embodiment, pixel degradation restoration is performed on Bayer RAW image data, but pixel degradation restoration may also be performed for each color plane after inverse frequency transformation. FIG. 17 is a block diagram illustrating another configuration of an image decoding device 130 that performs pixel degradation restoration after inverse frequency transformation. As shown in FIG. 17, in this configuration, the pixel degradation restoration unit 1300 is placed after the inverse frequency transformation unit 603, not after the Bayer transformation unit 604. Note that the learning process and inference process in this configuration can be achieved by simply applying the process related to RAW images in the above-described embodiment to each color plane (R, G0, G1, B).

Third Embodiment
FIG. 18 is a block diagram illustrating the configuration of an image decoding device 180 according to the third embodiment of the present invention. As shown in FIG. 18, the image decoding device 180 includes an entropy decoding unit 600, an inverse quantization unit 601, an inverse frequency transform unit 603, and a Bayer transform unit 604 as functional blocks. In addition, the image decoding device 180 includes a pixel/frequency degradation restoration unit 1802 and a pixel/frequency parameter setting unit 1805 as functional blocks. The pixel/frequency degradation restoration unit 1802 (inference means) is an element that integrates the frequency degradation restoration unit 602 and the pixel degradation restoration unit 1300 of the above-mentioned embodiment. The pixel/frequency parameter setting unit 1805 is an element that integrates the frequency parameter setting unit 605 and the pixel parameter setting unit 1301. As in the above-mentioned embodiment, the following processing of this embodiment executed by the above functional blocks is realized by one or more control processors of the image decoding device 180 expanding a program in a non-volatile memory such as a ROM into a volatile memory such as a RAM and executing it.

The learning of frequency parameters in the frequency domain and the learning of pixel parameters in the pixel domain are performed in the same manner as in the previously described embodiment, and therefore a detailed description is omitted.

The pixel/frequency parameter setting unit 1805 selects the frequency parameters (weights, biases) learned for each subband and compression ratio according to the compression ratio of the encoded stream to be decoded, and sets them in the pixel/frequency degradation restoration unit 1802.

The pixel/frequency degradation restoration unit 1802 applies a neural network having inference parameters set by the pixel/frequency parameter setting unit 1805 to the inversely quantized DWT coefficients input from the inverse quantization unit 601, and restores the DWT coefficients degraded by quantization by inference.

After restoring the DWT coefficients as described above, the pixel/frequency parameter setting unit 1805 selects the pixel parameters (weights, biases) learned for each compression ratio according to the compression ratio of the encoded stream to be decoded, and sets them in the pixel/frequency degradation restoration unit 1802.

The inverse frequency transform unit 603 performs an inverse frequency transform (inverse DWT transform) on the restored DWT coefficients input from the pixel/frequency degradation restoration unit 1802 to reconstruct independent plane data for each color plane (R, G0, G1, B). The reconstructed plane data is output to the Bayer transform unit 604.

The Bayer conversion unit 604 recomposes the independent plane data for each color plane (R, G0, G1, B) reconstructed by the inverse frequency conversion unit 603 into a Bayer array RAW image, and outputs RAW data equivalent to the RAW image. The output RAW data is input to the pixel/frequency degradation restoration unit 1802.

The pixel and frequency degradation restoration unit 1802 applies a neural network having pixel parameters set by the pixel and frequency parameter setting unit 1805 to the raw image of the Bayer array output by the Bayer conversion unit 604, and restores pixel data degraded by quantization through inference.

The configuration of this embodiment described above provides the same technical effects as the first and second embodiments. In addition, this configuration uses a pixel/frequency degradation restoration unit 1802 that integrates the frequency degradation restoration unit 602 and the pixel degradation restoration unit 1300, and a pixel/frequency parameter setting unit 1805 that integrates the frequency parameter setting unit 605 and the pixel parameter setting unit 1301. As a result, the circuit scale of the neural network used in the restoration process can be reduced compared to the second embodiment.

<Modification>
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

The neural networks in the frequency degradation restoration unit 602, pixel degradation restoration unit 1300, and pixel and frequency degradation restoration unit 1802 may each have any network configuration capable of executing the above-mentioned processing. For example, each of the above-mentioned restoration units (inference means) may have a CNN (Convolution Neural Network) or a DBP (Deep Brief Network). In addition, in the above-mentioned embodiment, a four-layer neural network is exemplified, but a neural network with any number of layers capable of executing the above-mentioned processing may be adopted.

The frequency degradation restoration unit 602 in the first and second embodiments has an NN for each subband (1LL, 1HL, 1LH, 1HH restoration units 800, 801, 802, 803). However, the frequency degradation restoration unit 602 may have only one NN (restoration unit) and perform learning and inference so as to switch parameters (weights, bias) for each subband. With the above configuration, since the frequency degradation restoration unit 602 has only one NN, the circuit scale can be reduced. On the other hand, in a configuration in which the frequency degradation restoration unit 602 has an NN for each subband, restoration processing for multiple subbands can be performed in parallel, so the processing time can be reduced compared to the configuration of this modified example.

In the configuration of the above embodiment, wavelet transform is used as the frequency transform method for RAW images, but other frequency transform methods may be used. For example, 4x4 DCT (discrete cosine transform) used in the H.264 standard may be adopted. Figure 19 is an explanatory diagram of the quantization matrix and learning/inference groups of 4x4 DCT.

In the H.264 standard, an image to be coded is divided into 16x16 macroblocks, and then coded by performing DCT in 4x4 units. The H.264 standard employs a quantization matrix as shown in FIG. 19 to quantize high-frequency components that have a small visual impact relatively strongly, while quantizing low-frequency components that have a large visual impact relatively weakly. The values shown in FIG. 19 are the initial values of the quantization matrix during intra prediction. The values shown in FIG. 19, i.e., the values included in the quantization matrix, are values multiplied by the quantization step obtained from the quantization parameter, and the higher the frequency, the larger the value. Frequency components that correspond to the same value are quantized with the same intensity. Therefore, by grouping the same values as one group or by frequency band, and performing learning and inference for each group, it is possible to achieve the same technical effect as the learning and inference for each subband as in the above embodiment. That is, in this modified example, learning and inference processes are performed for each of the frequency band groups (a) to (g) surrounded by dotted lines in FIG. 19.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of the computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

60 Image decoding device 130 Image decoding device 180 Image decoding device 600 Entropy decoding unit 601 Inverse quantization unit 602 Frequency degradation restoration unit (inference means)
603 Inverse frequency conversion unit 604 Bayer conversion unit 605 Frequency parameter setting unit 1203 Parameter update unit 1300 Pixel degradation restoration unit (inference means)
1301 pixel parameter setting unit 1502 pixel parameter update unit 1802 pixel/frequency degradation restoration unit (inference means)
1805 Pixel and frequency parameter setting unit

Claims (12)

1. An image decoding device that decodes encoded data obtained by quantizing and encoding a plurality of subband data obtained by performing a frequency transform on image data , the image decoding device comprising:
A decoding means for decoding the encoded data;
an inverse quantization means for inverse quantizing the data decoded by the decoding means to obtain a plurality of subband data;
an inference means for performing inference on the plurality of subband data acquired by the inverse quantization means to acquire a plurality of subband data in which data deteriorated by quantization has been restored, the inference means performing inference on the subband data corresponding to each of the plurality of subbands by using an inference parameter corresponding to the subband ;
The inference means includes:
performing a first inference on the plurality of subband data using first inference parameters which are inference parameters trained to correspond to the plurality of subbands respectively;
11. An image decoding device, comprising: a decoder for executing a second inference on subband data restored by the first inference using learned second inference parameters . the first inference parameters are weights and biases learned by a neural network to correspond to the plurality of subbands , respectively;
2. The image decoding apparatus according to claim 1, wherein the inference means comprises a neural network in which the weights and biases are set. 2. The image decoding apparatus according to claim 1 , wherein the inference means comprises a first inference means for executing the first inference, and a second inference means for executing the second inference. The image decoding device according to claim 1, characterized in that the inference means performs the first inference using the first inference parameters, and then switches the first inference parameters to the second inference parameters and performs the second inference. 5. The image decoding device according to claim 1 , wherein the second inference is an inference using the second inference parameters learned using pixel data of image data before it is encoded. 5. An image decoding device as claimed in claim 1, characterized in that the second inference is an inference on raw image data obtained after the first inference, using the second inference parameters learned using raw image data before it is encoded. An image decoding device as described in any one of claims 1 to 4, characterized in that the second inference is an inference on image data for each color plane obtained after the first inference, using the second inference parameters learned using image data for each color plane before encoding. 8. The image decoding device according to claim 1, wherein in the first inference, inferences corresponding to one or more subbands are skipped. 9. The image decoding apparatus according to claim 8 , wherein the inferences to be skipped are inferences corresponding to subbands having a compression ratio exceeding a predetermined value. 10. The image decoding device according to claim 1, wherein the inference parameters are learned for each compression ratio. A control method for an image decoding device that decodes encoded data obtained by quantizing and encoding a plurality of subband data obtained by performing a frequency transform on image data , the method comprising the steps of:
a decoding step of decoding the encoded data;
an inverse quantization step of inverse quantizing the data decoded in the decoding step to obtain a plurality of subband data;
an inference step of performing inference on the plurality of subband data acquired in the inverse quantization step to acquire a plurality of subband data in which data deteriorated by quantization is restored, the inference step performing inference on the subband data corresponding to each of the plurality of subbands by using an inference parameter corresponding to the subband ;
The inference step includes:
performing a first inference on the plurality of subband data using first inference parameters which are inference parameters trained to correspond to the plurality of subbands respectively;
A control method comprising : performing a second inference on subband data after it has been restored by the first inference, using the learned second inference parameters . A program for causing a computer to function as each of the means of the image decoding device according to any one of claims 1 to 10 .

Images