JP2018182531A - Division shape determining apparatus, learning apparatus, division shape determining method, and division shape determining program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a division shape determining apparatus, a learning apparatus, a division shape determining method, and divided shape determining program capable of determining a division shape of a CU for efficiently encoding an encoding target image even when a calculation amount for determining the division shape of the CU is reduced.SOLUTION: A division shape determining apparatus has a plurality of nodes that hold division probabilities that are probabilities related to division in a hierarchical structure and includes: a learning unit that updates a learning parameter of a learning model that is a set of nodes in accordance with a division probability of a node associated with a block that divides an encoding target image and outputs a division probability obtained as an output of a learning model in which the learning parameter is updated in association with the node; and a determination unit that determines whether to divide the block associated with the block on the basis of the division probability output in association with the node.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a divided shape determination device, a learning device, a divided shape determination method, and a divided shape determination program.

  H.264 / AVC (Advanced Video Coding) (hereinafter referred to as "AVC") is a standard of moving picture coding. As a new standard next to AVC, H.265 / HEVC (High Efficiency Video Coding) (hereinafter referred to as "HEVC") was standardized in 2013. HEVC boasts twice as much compression performance as AVC at comparable image quality. However, the amount of computation of HEVC is enormous compared to the amount of computation of AVC.

  In HEVC, an image to be encoded is divided in units of CTU (Coding Tree Unit), which is a block having a size of 64 pixels × 64 pixels. The image coding apparatus performs coding processing for each CTU. The division shape determination device can recursively divide the CTU into four blocks called a coding unit, called a CU (Coding Unit). In HEVC, four types of CU sizes are defined: 64 pixels × 64 pixels, 32 pixels × 32 pixels, 16 pixels × 16 pixels, and 8 pixels × 8 pixels. Hereinafter, n pixels × n pixels are described as “n × n”.

  FIG. 6 is a diagram illustrating an example of divided shapes of CUs. Each CU shares parameters such as intra prediction and inter prediction. If the distribution of luminance values in the CU is flat, the size of the CU is determined large. If the distribution of luminance values in the CU is complex, the size of the CU is set smaller. By determining the size of the CU as described above, the image coding apparatus of HEVC can improve the coding efficiency.

  FIG. 7 is a diagram illustrating an example of a quadtree data structure for representing a divided shape of a CU. The split shape of CU is expressed using a quadtree data structure. The quadtree data structure has a hierarchical structure. Each node of the quadtree data structure is associated with each CU. Each CU is classified by hierarchy (division depth) of the quadtree data structure. In each node of the quadtree data structure, a flag regarding division of a CU (block) associated with the node is defined as a label of the node. In HEVC, a flag relating to division is represented by a binary value of 1 representing division and 0 representing non-division.

  The split shape determination device of HEVC determines the split shape of CU based on rate distortion optimization defined in reference software such as HEVC test model (HM). The divided shape determination device calculates the divided shape and prediction mode of the CU that minimize the rate distortion cost function J (= D + λR) based on the rate distortion optimization defined in the reference software. In the rate distortion cost function J, D represents the amount of distortion generated in response to the selection of the parameter. R represents the amount of generated bits. λ represents a constant called Lagrange multiplier. The split shape determination device of HEVC determines and determines the split shape and prediction mode of a CU by performing a full search in rate distortion optimization. For this reason, the amount of computation of rate distortion optimization is enormous.

  Therefore, as a method of determining the divided shape of the CU without performing the rate distortion optimization, the divided shape determination device determines the divided shape of the CU using a learning model of a neural network using the divided shape of the CU as teaching data. A method has been proposed. In learning using supervised data (supervised learning), a large number of CTUs, which are inputs of a learning model, and correct answer labels representing divided shapes (division patterns) of CUs, which are outputs of the learning models, are prepared.

  The divided shape determination device causes the learning model to learn the divided shape of the CU by repeatedly using the teaching data for each CTU. The divided shape determination device updates the learning parameter of the learning model so that the divided shape of the CU obtained as a result of repetitively inputting the training data for each CTU into the learning model approaches the correct answer label.

  FIG. 8 is a diagram illustrating an example of correct answer labels indicating divided shapes of CUs. When the divided shape determination device learns the divided shape of the CU for each CTU, the correct shape label (classification model) which receives the original image of the coding target image of the CTU unit as an input and outputs the divided shape of the CU is used as the divided shape determination device The method of learning is the simplest method. However, if all divided shapes of a CU are covered in CTU units, the number of correct labels will exceed 80,000 and become enormous. Therefore, the division shape determination device can not learn the division shape of the CU unless a huge number of teacher data are prepared.

  Therefore, as a method by which the divided shape determination device can learn the divided shapes of CUs without using a large number of teacher data, a learning model is used that determines division or non-division of CUs for each hierarchy of CUs. The following method has been proposed (see Non-Patent Document 1). In Non-Patent Document 1, instead of preparing a large number of teacher data, the division shape determination device prepares a plurality of learning models that determine division or non-division of CUs for each hierarchy of CUs. Divided shapes can be learned.

  In Non-Patent Document 1, the divided shape determination device determines a divided shape of a CU by sequentially applying a learning model for each hierarchy of a quadtree data structure. Hereinafter, a block to be divided or not divided is referred to as a “target block”. Hereinafter, the probability regarding division of a CU (block) associated with a node is referred to as “division probability”. The learning model (probability distribution model) outputs a label representing the division probability for each target block associated with the node. The division probability value representing division (positive example) is 1. The division probability value representing non-division (negative example) is zero. The division probability may be a value (ambiguous value) within a predetermined range including 0.5 which is an average value of 0 and 1. If the division probability is ambiguous, the division shape determination device of Non-Patent Document 1 determines the division shape of the CU, which is the target block, based on rate distortion optimization defined in the HEVC test model.

F. Duanmu, Z. Ma, Y. Wang: “Fast CU Partition Decision Using Machine Learning for Screen Content Compression,” IEEE International Conference of Image Processing, Sept. 2015.

  FIG. 9 is a diagram showing an example of a plurality of learning models prepared to determine the divided shape of a CU of Non-Patent Document 1. As shown in FIG. FIG. 10 is a flowchart showing an example of the operation of the divided shape determination device of Non-Patent Document 1. As shown in FIG. 9 and FIG. 10, when the divided shape determination device of Non-Patent Document 1 determines the divided shapes of CUs, a plurality of learning models prepared for each hierarchy of the quadtree data structure ((1) Use a split judgment model).

  When the divided shape determination device uses a plurality of learning models, the amount of operation of processing for extracting the feature amount of the image increases, so the amount of operation for determining the divided shape of the CU becomes enormous. Further, when the divided shape determination device uses a plurality of learning models, the divided shape of the CU is determined independently without considering the correlation between adjacent CUs, so the divided shape determination device It is not possible to determine the division shape of the CU for efficiently encoding the image to be digitized.

  As described above, the conventional divided shape determination device determines the divided shape of the CU for efficiently encoding the encoding target image when the amount of operation for determining the divided shape of the CU is reduced. There was a problem that I could not do it.

  In view of the above circumstances, according to the present invention, it is possible to determine the CU division shape for efficiently encoding the encoding target image even when the amount of operation for determining the CU division shape is reduced. An object of the present invention is to provide a divided shape determination device, a learning device, a divided shape determination method, and a divided shape determination program.

  In one aspect of the present invention, a plurality of nodes holding a division probability, which is a probability relating to division, form a hierarchical structure, and learning parameters of a learning model which is a set of nodes are divided into blocks for dividing an image to be coded. A learning unit that updates according to the division probability of the associated node, and outputs the division probability obtained as an output of the learning model in which the learning parameter is updated, in association with the node; It is a division | segmentation shape determination apparatus provided with the determination part which determines whether the block matched with the said node is divided | segmented based on the said division | segmentation probability matched with the node.

  One embodiment of the present invention is the split shape determination device described above, wherein the learning unit is configured to split the child node that is a lower node of the node according to the split probability held by the node. Determine whether to refer to.

  One aspect of the present invention is the divided shape determination device described above, wherein the hierarchical structure is a quadtree data structure, and the learning unit is configured to determine that the division probability held by the node is 0. It is determined that when the learning parameter is updated, the division probability of the child node is not referred to.

  One aspect of the present invention is the divided shape determination device described above, wherein the determination unit is associated with the node based on the division probability held by a child node that is a subordinate node of the node. Decide whether to divide.

  One aspect of the present invention is the above-described split shape determination device, wherein the split probability is a probability represented by three or more values.

  According to an aspect of the present invention, when a plurality of nodes holding probabilities form a hierarchical structure, a learning parameter of a learning model which is a set of the nodes is a learning parameter of the nodes when the probability of the nodes is a predetermined value. It is a learning apparatus provided with the learning part updated without being based on the probability of a child node.

  One aspect of the present invention is a division shape determination method executed by a division shape determination device that determines a division shape of a block that divides an encoding target image, and a plurality of nodes holding division probabilities that are probabilities related to division are hierarchical The learning parameter of a learning model that is structured and is a set of nodes, is updated according to the division probability of the nodes associated with the block, and the learning parameters are updated. Whether or not to divide the block associated with the node based on the step of outputting the division probability obtained as output in association with the node, and the division probability output in association with the node And determining the division shape.

  According to one aspect of the present invention, in a computer, a plurality of nodes holding division probabilities, which are probabilities related to division, form a hierarchical structure, and learning parameters of a learning model, which is a set of nodes, are encoded target images. Updating according to the division probability of the node associated with the division block, and outputting the division probability obtained as an output of the learning model in which the learning parameter is updated in association with the node It is a division | segmentation shape determination program for performing the procedure which determines whether the block matched with the said node is divided | segmented based on the said division probability matched with the said node.

  According to the present invention, it is possible to determine the divided shape of the CU for efficiently encoding the image to be encoded, even when the amount of calculation for determining the divided shape of the CU is reduced.

It is a figure which shows the example of a structure of the image coding apparatus 1 in 1st Embodiment. It is a figure which shows the example of a quadtree data structure and an output label in 1st Embodiment. It is a figure which shows the example of a structure of a division | segmentation shape determination apparatus in 1st Embodiment. It is a flowchart which shows the example of operation | movement of a division | segmentation shape determination apparatus in 1st Embodiment. It is a flowchart which shows the example of operation | movement of a division | segmentation shape determination apparatus in 2nd Embodiment. It is a figure which shows an example of the division | segmentation shape of CU. It is a figure which shows the example of the quadtree data structure for showing the division | segmentation shape of CU. It is a figure which shows the example of the correct answer label showing the division | segmentation shape of CU. It is a figure which shows the example of the several learning model prepared in order to determine the division | segmentation shape of CU. It is a flowchart which shows the example of operation | movement of a division | segmentation shape determination apparatus.

Embodiments of the present invention will be described in detail with reference to the drawings.
First Embodiment
FIG. 1 is a diagram showing an example of the configuration of the image coding device 1. The image encoding device 1 is, for example, an information processing device such as a personal computer device, a smartphone terminal, a tablet terminal, or a server device. The image encoding device 1 encodes a plurality of images (frames) constituting a moving image as an image to be encoded. The image to be encoded is divided into blocks of CTU units each having a size of 64 pixels × 64 pixels.

  The image coding device 1 includes a division shape determination device 10, a subtractor 11, an orthogonal transformation / quantization unit 12, a variable length encoding unit 13, an inverse quantization / inverse orthogonal transformation unit 14, and an adder 15. , A loop filter unit 16, a decoded picture memory 17, an intra prediction unit 18, an inter prediction unit 19, and an intra / inter switching switch 20. The image coding apparatus 1 may further include, for example, a non-volatile recording medium (non-temporary recording medium) such as a magnetic hard disk device or a semiconductor storage device as a storage unit.

  Divided shape determination device 10, subtractor 11, orthogonal transform / quantization unit 12, variable length coding unit 13, inverse quantization / inverse orthogonal transform unit 14, adder 15, loop filter unit 16, intra prediction unit 18, inter Part or all of the prediction unit 19 and the intra / inter switching switch 20 may be realized, for example, by a processor such as a central processing unit (CPU) executing a program stored in the storage unit. It may be realized using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

  The divided shape determination device 10 is an information processing device (learning device) that performs learning using a single learning model. The learning model is a model in which a plurality of nodes holding division probabilities form a hierarchical structure. The learning model is not limited to a specific learning model as long as the learning model outputs a label of each node of the quadtree data structure. The learning model may be a learning model of a neural network or a learning model other than a neural network. The learning model other than the neural network may be, for example, a learning model of genetic programming. The split shape determination device 10 learns a determination method regarding general-purpose data represented by a quadtree data structure. The divided shape determination device 10 outputs the learned result to a predetermined functional unit.

  Below, the division | segmentation shape determination apparatus 10 acquires an encoding object image for every CTU as an example. The split shape determination device 10 learns, as an example, a method of determining a split shape of a CU represented by a quadtree data structure. The divided shape determination device 10 learns the divided shape of the CU using the learning model. The division shape determination device 10 can recursively divide the CTU into four blocks into units of CU. The division shape determination device 10 determines the division shape (division pattern) of the CU based on the result of learning the division shape of the CU. The division shape determination device 10 determines the division shape of a CU such as HEVC for each CTU.

  The subtractor 11 acquires an image to be encoded from the divided shape determination device 10 for each CTU for which the divided shape of the CU has been determined. The subtractor 11 obtains an output label representing the division probability of each node from the division and shape determination device 10 for each CTU. The subtractor 11 obtains a CTU predicted image from the intra prediction unit 18 or the inter prediction unit 19. The subtractor 11 outputs the difference between the CTU of the image to be encoded and the prediction image to the orthogonal transformation / quantization unit 12.

  The orthogonal transformation / quantization unit 12 performs orthogonal transformation processing and quantization processing on the difference between the CTU and the predicted image. The orthogonal transformation / quantization unit 12 outputs the quantization coefficient, which is the result of the orthogonal transformation process and the quantization process, to the variable length coding unit 13 and the inverse quantization / inverse orthogonal transformation unit 14.

  The variable-length coding unit 13 is a coding unit that executes variable-length coding processing. The variable-length coding unit 13 outputs, to an image decoding apparatus or the like, coded data including the result of performing variable-length coding processing on the quantization coefficient. The variable-length coding unit 13 may output coded data including coding parameters such as a motion vector to an image decoding apparatus or the like. The coding parameters are determined, for example, based on the result of rate distortion optimization.

  The inverse quantization / inverse orthogonal transform unit 14 outputs, to the adder 15, an image which is a result of performing the inverse quantization process and the inverse orthogonal transform process on the quantization coefficient. The adder 15 obtains, from the inverse quantization / inverse orthogonal transformation unit 14, an image which is a result of performing the inverse quantization process and the inverse orthogonal transformation process on the quantization coefficient. The adder 15 obtains a predicted image of CTU from the intra prediction unit 18 or the inter prediction unit 19 via the intra / inter switching switch 20. The adder 15 outputs, to the loop filter unit 16 and the intra prediction unit 18, a result obtained by adding the image which is the result of performing the inverse quantization process and the inverse orthogonal transformation process to the quantization coefficient and the prediction image.

  The loop filter unit 16 applies a loop filter to the result of the adder 15 adding the image which is the result of performing the inverse quantization process and the inverse orthogonal transformation process on the quantization coefficient and the predicted image. The loop filter unit 16 outputs the result to which the loop filter is applied to the decoded picture memory 17.

  The decoded picture memory 17 is, for example, a volatile recording medium such as a random access memory (RAM). The decoded picture memory 17 may be, for example, a non-volatile recording medium (non-temporary recording medium) such as a semiconductor storage device. The decoded picture memory 17 stores a plurality of images (frames) as a result of storing the result (reconstructed signal) to which the loop filter is applied to the result added by the adder 15. The decoded picture memory 17 outputs the result obtained by applying the loop filter to the result added by the adder 15 to the inter prediction unit 19.

  The intra prediction unit 18 acquires, from the adder 15, a result in which the loop filter is applied to the result added by the adder 15. The intra prediction unit 18 uses the result obtained by applying the loop filter to the result added by the adder 15 as a reference image. The intra prediction unit 18 generates a predicted image of CTU of the image to be encoded by intra prediction based on the reference image acquired from the adder 15.

  The inter prediction unit 19 acquires the encoding target image from the division shape determination device 10 for each CTU for which the division shape of the CU is determined. The inter prediction unit 19 acquires from the decoded picture memory 17 the result of the application of the loop filter to the result added by the adder 15. The inter prediction unit 19 uses a result obtained by applying the loop filter to the result added by the adder 15 as a reference image. The inter prediction unit 19 generates a predicted image of the CTU of the image to be encoded by inter prediction based on the reference image acquired from the decoded picture memory 17.

  The intra / inter switching switch 20 outputs the predicted image generated by the intra prediction unit 18 to the subtractor 11 and the adder 15 when the prediction mode of the CTU is intra prediction. The intra / inter switching switch 20 outputs the predicted image generated by the inter prediction unit 19 to the subtractor 11 and the adder 15 when the prediction mode of the CTU is the inter prediction.

Next, examples of quadtree data structures and output labels are described.
FIG. 2 is a diagram showing an example of a quadtree data structure and an output label. The split shape of CU in one CTU is represented using one quadtree data structure. For each node of the quadtree data structure, a probability (division probability) regarding division of a CU associated with the node is defined as a label of the node. One quadtree data structure represents the division probability of each CU of one CTU.

  The learning model takes as input the CTU of the image to be encoded. The learning model outputs a label representing the division probability of each node of the quadtree data structure, based on the inputted CTU CU division shape and learning parameters. The number of elements of the label (hereinafter referred to as “output label”) output by the learning model is equal to the maximum number of nodes of the quadtree data structure in one CTU. The output label consists of the division probability y [n] (n is an integer from 0 to 20) of each CU of the CTU. In the quadtree data structure of FIG. 2, the output label consists of y [0], y [1],..., Y [20], corresponding to the number of output units of the learning model being 21.

  In the output label, the division probability of a 64 × 64 sized CU in the shallowest hierarchy is y [0]. The division probability of each 32 × 32 sized CU in a hierarchy one hierarchy lower than the 64 × 64 sized CU is y [1] to y [4]. The division probability of each of the 16 × 16 sized CUs in a hierarchy one hierarchy level below the 32 × 32 sized CUs is y [5] to y [20].

  The number of elements of the correct answer label of the learning model is equal to the maximum number of nodes of the quadtree data structure in one CTU. The correct answer label consists of the division probability t [n] of each CU of the CTU. In the quadtree data structure of FIG. 2, the correct answer label t corresponds to the output label y (= y [0], y [1],..., Y [20]), t [0], t [1]. ], ..., t [20].

  In the correct answer label, the division probability of a 64 × 64 sized CU is t [0]. The division probability of each 32 × 32 sized CU in a hierarchy one hierarchy lower than the 64 × 64 sized CU is t [1] to t [4]. The division probability of each of the 16 × 16 sized CUs in the hierarchy one hierarchy level below the 32 × 32 sized CUs is t [5] to t [20]. The divided shape determination device 10 illustrated in FIG. 1 updates the learning parameters of the learning model during learning so that the output label representing the divided shape of the CU approaches the correct answer label.

  The division shape determination device 10 determines that the division probability of the node holding the division probability exceeding the threshold of the division probability is 1 based on the output label output from the learning model in which the learning parameter is updated. That is, the division shape determination device 10 determines to divide the CU associated with the node holding the division probability exceeding the division probability threshold.

  The division shape determination device 10 determines that the division probability of the node holding the division probability not exceeding the threshold of the division probability is 0, based on the output label outputted by the learning model in which the learning parameter is updated. That is, the division shape determination device 10 determines not to divide the CU associated with the node holding the division probability not exceeding the threshold of the division probability.

  When the division probability of the parent node of the quadtree data structure represents non-division (is 0), the division shape determination device 10 does not determine the division probability of a child node which is a subordinate node of the parent node. That is, the division | segmentation shape determination apparatus 10 does not determine the division | segmentation probability of the child node of the parent node matched with CU which is not divided | segmented.

Next, an example of the configuration of the divided shape determination device 10 will be described.
FIG. 3 is a diagram showing an example of the configuration of the divided shape determination device 10. As shown in FIG. The divided shape determination device 10 includes the feature extraction unit 100 as a single learning model. The division shape determination device 10 further includes a determination unit 110.

  The feature extraction unit 100 (learning unit) acquires, for each CTU, the original image or feature amount of the encoding target image. The feature extraction unit 100 outputs the division probability of each node of the quadtree data structure as an output label of a single learning model based on the original image or the feature amount of the encoding target image. The feature extraction unit 100 updates the learning parameters of the learning model so that the output label approaches the correct answer label as a result of repeated learning. The feature extraction unit 100 calculates a division probability of each node of the quadtree data structure based on a learning model in which learning parameters are updated as a result of learning. The determination unit 110 outputs an output label including the division probability determined for each node of the quadtree data structure to the subtractor 11.

  In FIG. 3, the learning model is, as an example, a learning model of a convolutional neural network. The feature extraction unit 100 includes a convolution layer 101, a pooling layer 102, a convolution layer 103, a pooling layer 104, and an all coupling layer 105.

  The convolution layer 101 (Convolution Layer) (updating unit) updates learning parameters such as filter coefficients as a result of learning. The convolution layer 101 may apply an activation function to each value of the two-dimensional array. The pooling layer 102 performs downsampling using the maximum value, the average value, and the like in the kernel. That is, the pooling layer 102 leaves valid values of each value of the two-dimensional array that is the output result of the convolutional layer 101.

  The convolutional layer 103 (update unit) updates learning parameters such as filter coefficients as a result of learning. The convolution layer 103 may apply an activation function to each value of the two-dimensional array that is the output result of the pooling layer 102. The pooling layer 104 performs downsampling using the maximum value, the average value, etc. in the kernel. That is, the pooling layer 104 leaves valid values of each value of the two-dimensional array that is the output result of the convolutional layer 103. The fully connected layer 105 (split probability output unit) outputs an output label representing the split probability for each node by combining the outputs of the pooling layer 104.

  The determination unit 110 (division probability determination unit) determines the division probability of the target block associated with the node based on the output labels of all the combined layers 105. That is, based on the output labels of all the combined layers 105, the determination unit 110 determines whether to divide the target block associated with the node. The determination unit 110 outputs an output label including the division probability determined for each node of the quadtree data structure to the subtractor 11 illustrated in FIG. 1 for each CTU.

Next, the learning method of the learning model in the feature extraction unit 100 will be described.
When learning the correct answer label of the divided shape of the CU, the feature extraction unit 100 acquires, for each CTU, the original image or the feature amount of the encoding target image. The total coupling layer 105 outputs an output label y. The output label y represents the division probability of each node of the quadtree data structure. The output label y is expressed as equation (1). The correct answer label t corresponding to the output label y is expressed as equation (2).

y = [y [0], y [1], ..., y [20]] ^T (1)

t = [t [0], t [1], ..., t [20] ^T (2)

  The convolution layer 101 and the convolution layer 103 calculate the value of the error function E representing the error between the output label y and the correct label t. The error function E is defined using the cross entropy of the output label y and the correct label t, the mean square error, and the like. The convolution layer 101 and the convolution layer 103 update the learning parameters w of the convolution layer 101 and the convolution layer 103 by an error back propagation method or the like so that the value of the error function E becomes small.

  The convolution layer 101 and the convolution layer 103 may use the gradient descent method in order to update the learning parameter w of the learning model in the direction in which the value of the error function E decreases when the error back propagation method is performed. That is, the convolutional layer 101 updates the learning parameter w of the convolutional layer 101 in the direction in which the value of the error function E becomes smaller as in the equation (4) using the equation (3) representing the gradient ∇E. The convolutional layer 103 updates the learning parameter w of the convolutional layer 103 in the direction in which the value of the error function E becomes smaller as in the equation (4) using the equation (3) representing the gradient ∇E. In equation (3), M represents the number of elements of the learning parameter w. In equation (4), ε represents a learning rate.

Gradient ∇ E
= ∂ E / ∂ w
= [∂ E / ∂ w ₁ , ∂ E / ∂ w ₂ , ..., ∂ E / ∂ w _M ] ^T ... (3)

  w w w-ε E E (4)

  In the first embodiment, each element of the correct answer label t is represented using a division probability obtained by rate distortion optimization in reference software such as HEVC test model (HM). In the first embodiment, the division probability of the node in the correct answer label t is represented by a binary value (division or non-division).

  The convolutional layer 101 and the convolutional layer 103 do not refer to the division probability of the child node of the parent node indicating non-division in the correctness label t when learning the correctness label t of the divided shape of the CU. For example, when the division probability of the node of the correct answer label t [1] represents non-division (the division probability is a predetermined value = 0), the convolution layer 101 and the convolution layer 103 are nodes of the correct answer label t [1] It does not refer to the division probabilities of the correct labels t [5] to t [8] of the child nodes of.

  The convolution layer 101 and the convolution layer 103 do not use the division probability not referenced in the correct answer label t for learning. That is, the convolution layer 101 and the convolution layer 103 update the learning parameters of the learning model based on the result of learning that there is no division probability of the child node of the parent node whose division probability indicates non-division.

Next, an example of the operation of the divided shape determination device 10 will be described.
FIG. 4 is a flowchart showing an example of the operation of the divided shape determination device 10. The feature extraction unit 100 acquires an encoding target image for each CTU. The feature extraction unit 100 extracts feature amounts such as luminance values from the CTU of the image to be encoded. The all coupling layer 105 calculates the division probability of each node based on the learning model in which the learning parameter is updated (step S101). The determination unit 110 executes processing by giving priority to a target block corresponding to a node having a shallow hierarchy in the quadtree data structure.

  The determination unit 110 determines whether the hierarchy in the quadtree data structure of the node corresponding to the target block is the deepest hierarchy (step S102). When the hierarchy in the quadtree data structure of the node corresponding to the target block is not the deepest hierarchy (step S102: NO), the determination unit 110 determines whether or not the division probability exceeds the threshold of the division probability for the target block. It determines (step S103). If the division probability exceeds the threshold of the division probability (step S103: YES), the determination unit 110 determines to divide the target block. The determination unit 110 determines that the division probability of the node corresponding to the target block is 1 (step S104). The determination unit 110 sets the next block as a target block in the processing order such as Z scan for the next lower layer (step S105). The determination unit 110 returns the process to step S102.

  If the hierarchy in the quadtree data structure of the node corresponding to the target block is the deepest hierarchy (step S102: YES), the determination unit 110 proceeds with the process to step S106. If the division probability does not exceed the threshold of the division probability (step S103: NO), the determination unit 110 determines that the target block is not divided. The determination unit 110 determines that the division probability of the node corresponding to the target block is 0 (step S106).

  The determination unit 110 determines whether or not the determination unit 110 has determined division or non-division for all blocks (CUs) in the CTU (step S107). If the determination unit 110 has not determined division or non-division for any block (CU) in the CTU (step S107: NO), the determination unit 110 sets the next block in the processing order as the target block (step S108). ). The determination unit 110 returns the process to step S102. When the determination unit 110 determines division or non-division for all blocks (CUs) in the CTU (step S107: YES), the determination unit 110 ends the process.

  As described above, the divided shape determination apparatus 10 according to the first embodiment includes the feature extraction unit 100 as a learning unit and the determination unit 110. The plurality of nodes holding the division probability form a hierarchical structure. The feature extraction unit 100 updates the learning parameter w of the learning model, which is a set of nodes, according to the division probability of the node associated with the block that divides the encoding target image. The feature extraction unit 100 outputs the division probability obtained as an output of the learning model in which the learning parameter has been updated, in association with the node. The determination unit 110 determines whether to divide the block associated with the node based on the division probability output in association with the node.

  As a result, the division shape determination apparatus 10 according to the first embodiment determines the division shape of the CU for efficiently encoding the encoding target image even when the amount of operation for determining the division shape of the CU is reduced. It is possible to make a decision.

  The feature extraction unit 100 according to the first embodiment determines whether to refer to the division probability of a child node that is a subordinate node of the node, according to the division probability held by the node. When the division probability held by the node is 0, the feature extraction unit 100 of the first embodiment determines that the division probability of the child node is not referred to when updating the learning parameter. The feature extraction unit 100 according to the first embodiment uses the learning parameters of a learning model in which a plurality of nodes holding probabilities form a hierarchical structure to the probability of child nodes of the node when the probability of the nodes is a predetermined value. Update without being based.

  Generally, when the learning model learns correct labels for all divided shapes of CU in CTU, the number of correct labels of divided shapes of CU is enormous, so the divided shapes of CUs can be efficiently used. I can not learn. In Non-Patent Document 1, a learning model can be efficiently learned to a certain extent. However, in the process of determining the divided shape of the CU before the encoding process, the divided shape determination device of Non-Patent Document 1 extracts feature quantities from the original image by using a plurality of learning models (division determination models) in series. Repeat the process. For this reason, in Non-Patent Document 1, the amount of operation of processing for extracting feature quantities from an original image is enormous. In addition, the learning model of Non-Patent Document 1 can not learn the divided shape of CU based on the correlation of spatial position in CTU.

  On the other hand, since the single learning model learns the division probability of the nodes of the quadtree data structure in the division shape determination apparatus 10 of the first embodiment, the encoding target image can be obtained even if the amount of calculation is small. It is possible to determine the division shape of the CU for encoding efficiently. Since the division shape determination apparatus 10 according to the first embodiment determines the division shape of the CU using a single learning model, it is not necessary to determine the division shape of the CU using two or more learning models in series. . Since the division shape determination apparatus 10 according to the first embodiment determines the division shape of a CU using a single learning model, the number of output units (number of elements) of the learning model can be reduced to a realistic number. It is. The divided shape determination device 10 according to the first embodiment can reduce the amount of operation for extracting feature quantities such as luminance values from the image to be encoded. The single learning model of the first embodiment extracts the feature quantities of the input image together, so that the divided shape of the CU can be learned based on the correlation of the spatial position in the CTU. Since the divided shape determination apparatus 10 according to the first embodiment determines the divided shape of the CU using a single learning model, it is possible to determine the divided shape of the CU based on the correlation of the spatial position in the CTU. It is. In the divided shape determination device 10 according to the first embodiment, since the learning model does not refer to the element of the correct answer label that does not contribute to the learning error during learning, the division probability of the child node of the parent node of the division probability representing non-division is defined It does not have to be done. Note that child nodes exist even if the division probability is not defined.

Second Embodiment
The second embodiment is different from the first embodiment in that the division shape determination device 10 evaluates the division probability of the child node of the parent node when the division probability of the parent node is ambiguous. In the second embodiment, only differences from the first embodiment will be described.

  When the division probability of the node associated with the target block is ambiguous (value within a predetermined range including 0.5), the determination unit 110 is one hierarchy lower than the hierarchy of the node associated with the target block. The split probability of the child node is compared with a predetermined split probability threshold. The determination unit 110 may compare the division probability of the child node with the threshold of the division probability by using an average value, a maximum value, or a minimum value of the division probability for a plurality of child nodes of the parent node. The determination unit 110 may select a division probability to be used for comparison from among the average value, the maximum value, the minimum value, and the like of the division probabilities of a plurality of child nodes.

  When the division probability of the child node exceeds the threshold of the division probability, the determination unit 110 determines to divide the target block associated with the parent node one hierarchy higher than the hierarchy of the child node. If the division probability of the child node does not exceed the threshold of the division probability, the determination unit 110 determines not to divide the target block associated with the parent node one hierarchy higher than the hierarchy of the child node.

Next, an example of the operation of the divided shape determination device 10 will be described.
FIG. 5 is a flowchart showing an example of the operation of the divided shape determination apparatus 10. Steps S201 to S202 are the same as steps S101 to S102 in FIG. The determination unit 110 determines whether the division probability is ambiguous for the target block. That is, the determination unit 110 determines whether or not the division probability is a value close to 0.5 for the target block (step S203). If the division probability is not ambiguous (step S203: NO), the determination unit 110 proceeds to step S04. Steps S204 to S206 are the same as steps S103 to S105 in FIG.

  If the division probability is ambiguous (step S203: YES), the determination unit 110 acquires the division probability of a child node one level lower than the level of the target block (step S207). The determination unit 110 determines whether or not the division probability of the child node exceeds the threshold of the division probability (step S208). When the division probability of the child node exceeds the threshold of the division probability (step S208: YES), the determination unit 110 proceeds with the process to step S205. When the division probability of the child node does not exceed the threshold of the division probability (step S208: NO), the determination unit 110 proceeds with the process to step S209. Steps S209 to S211 are the same as steps S106 to S108 in FIG.

  As described above, the determination unit 110 of the second embodiment determines whether or not to perform division associated with a node based on the division probability held by a child node that is a subordinate node of the node. As a result, when the division probability of the output label is ambiguous, the division shape determination apparatus 10 of the second embodiment can efficiently encode the coding target image even when the amount of calculation for determining the division shape of the CU is reduced. It is possible to determine the division shape of the CU to be encoded in an orderly manner.

  The split shape determination device 10 according to the second embodiment can obtain the split probability of each node of all the layers of the quadtree data structure representing one CTU in parallel as the output label y of the learning model. Since the division shape determination apparatus 10 of the second embodiment can obtain in parallel the division probabilities of each node of all the layers of the quadtree data structure, the division probability of the parent node of the parent node corresponding to the target block It can be acquired. As a result, even when the division probability of the output label is ambiguous, the division shape determination apparatus 10 of the second embodiment can execute the determination processing with high probability without executing rate distortion optimization.

Third Embodiment
The third embodiment is different from the first embodiment in that the division probability of the node represented in the correct answer label is a multivalue of three or more. In the third embodiment, only differences from the first embodiment will be described.

  The determination unit 110 determines the difference between the value of the rate distortion cost function J when the division probability of the node is 1 and the value of the rate distortion cost function J when the division probability of the node is 0 with respect to the correct label t. calculate. If the calculated difference is equal to or greater than a predetermined cost threshold, the determination unit 110 sets the weighting factor equal to or greater than the predetermined coefficient threshold. Thereby, the determination unit 110 can include the division probability far from the threshold of the division probability in the element of the correct answer label t. If the calculated difference is less than the predetermined cost threshold, the determination unit 110 sets the weighting factor to less than the predetermined coefficient threshold. Thus, the determination unit 110 can include the division probability close to the division probability threshold in the element of the correct answer label t.

  The determination unit 110 changes the division probability of each node in the correct answer label t using the weighting factor according to the calculated difference. In this manner, the determination unit 110 sets the division probability of the node represented by the correct answer label t to a multivalue of three or more. For example, the division probability of the node represented by the correct answer label t may be a continuous value between 0 and 1.

  The convolution layer 101 and the convolution layer 103 calculate the value of the error function E representing the error between the output label y and the correct label t. The error function E is defined using the mean square error of the output label y and the correct label t. The convolution layer 101 and the convolution layer 103 update the learning parameter w of each layer of the quadtree data structure by the error back propagation method so that the value of the error function E becomes smaller.

  As mentioned above, the division | segmentation probability of 3rd Embodiment is a probability represented by three or more values. Thus, the division shape determination apparatus 10 according to the third embodiment divides the CU division shape for encoding the encoding target image more efficiently, even when the amount of operation for determining the division shape of the CU is reduced. It is possible to determine

  When designing the correct answer label, the divided shape determination apparatus 10 according to the third embodiment includes, in the elements of the correct answer label, the division probability that is a multiple value multiplied by the weighting factor according to the rate distortion cost function J. By this, the division | segmentation shape determination apparatus 10 of 3rd Embodiment can determine the division | segmentation shape of CU in consideration of the influence which the division | segmentation shape of CU gives to encoding efficiency. The divided shape determination apparatus 10 according to the third embodiment can fill in the difference between the output obtained by machine learning in the learning model based on the feature amount and the output obtained by the full search in rate distortion optimization.

  At least a part of the image coding device, the divided shape determination device, and the learning device in the embodiments described above may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. Here, the “computer system” includes an OS and hardware such as peripheral devices. The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include one that holds a program for a certain period of time, such as volatile memory in a computer system that becomes a server or a client in that case. Further, the program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system. It may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within the scope of the present invention.

  The present invention is applicable to a division shape determination device that determines the division shape of a block that divides an image, a learning device that learns general-purpose data represented by a quadtree data structure, and an image coding device.

  DESCRIPTION OF SYMBOLS 1 ... Image coding apparatus, 10 ... Division | segmentation shape determination apparatus, 11 ... Subtractor, 12 ... Orthogonal transformation and quantization part, 13 ... Variable-length encoding part, 14 ... Dequantization and inverse orthogonal transformation part, 15 ... Addition , 16: loop filter unit, 17: decoded picture memory, 18: intra prediction unit, 19: inter prediction unit, 20: intra / inter switching switch, 100 ... feature extraction unit, 101 ... convolution layer, 102 ... pooling layer, 103 ... convolutional layer, 104 ... pooling layer, 105 ... all coupling layer, 110 ... determination unit

Claims (8)

A plurality of nodes that hold a division probability, which is a probability related to division, form a hierarchical structure, and learning parameters of a learning model that is a set of nodes are associated with a block that divides a coding target image. A learning unit that updates according to the division probability, and outputs the division probability obtained as an output of the learning model with the learning parameter updated, in association with the node;
A determination unit that determines whether to divide a block associated with a node based on the division probability output in association with the node.   The divided shape according to claim 1, wherein the learning unit determines whether or not to refer to the division probability of a child node which is a subordinate node of the node according to the division probability held by the node. Decision device. The hierarchical structure is a quadtree data structure,
The divided shape according to claim 2, wherein, when the division probability held by the node is 0, the learning unit determines not to refer to the division probability of the child node when updating the learning parameter. Decision device.   The determination unit determines whether or not to perform division associated with the node, based on the division probability held by a child node that is a subordinate node of the node. The split shape determination device according to any one of the above.   The divided shape determination device according to any one of claims 1 to 4, wherein the division probability is a probability represented by three or more values. The plurality of nodes holding the probability form a hierarchical structure, and the learning parameter of the learning model which is a set of the nodes is based on the probabilities of the child nodes of the node when the probability of the node is a predetermined value. A learning device provided with a learning unit that updates without updating. A division shape determination method executed by a division shape determination device that determines a division shape of a block that divides an image to be encoded.
A plurality of nodes holding a division probability, which is a probability related to division, form a hierarchical structure, and learning parameters of a learning model which is a set of the nodes are determined according to the division probability of the node associated with the block. Outputting the division probability obtained as an output of the learning model in which the learning parameter is updated, in association with the node; and
Determining whether or not to divide the block associated with the node based on the division probability output in association with the node. On the computer
A plurality of nodes that hold a division probability, which is a probability related to division, form a hierarchical structure, and learning parameters of a learning model that is a set of nodes are associated with a block that divides a coding target image. A procedure of updating according to the division probability and outputting the division probability obtained as an output of the learning model with the learning parameter updated, in association with the node;
A division shape determination program for executing the steps of determining whether to divide a block associated with a node based on the division probability output in association with the node.

Images