JP2024076475A - Video encoding device, video decoding device - Google Patents

Abstract

[Problem] To provide a video coding device and a decoding device that improve image recognition accuracy even at a low rate by coding and decoding additional auxiliary information without significantly changing the framework of video coding and decoding methods. [Solution] In a video transmission system 1 consisting of a video coding device 10, a network 21, a video decoding device 30, an image display device 41, and an image recognition device 51, the video decoding device 30 has an image decoding device 31 that decodes images from encoded data Te to generate a decoded video Td, and an auxiliary information decoding device 91 that decodes attention information having multiple values for each region as auxiliary information for the decoded video Td decoded by the image decoding device 31. [Selected Figure] Figure 1

Description

Embodiments of the present invention relate to video encoding devices and video decoding devices.

To efficiently transmit or record video, video encoding devices are used that generate encoded data by encoding video, and video decoding devices are used that generate decoded images by decoding the encoded data.

Specific examples of video encoding methods include H.264/AVC and H.265/HEVC (High-Efficiency Video Coding).

In such video coding methods, images (pictures) that make up a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded/decoded for each CU.

In such video coding methods, a predicted image is usually generated based on a locally decoded image obtained by encoding/decoding an input image, and the prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating predicted images include inter-prediction and intra-prediction.

Another recent example of video encoding and decoding technology is Non-Patent Document 1.

Non-Patent Document 1 describes a video encoding and decoding method with extremely high coding efficiency.

Non-patent document 2 discusses a method for integrating the description of video analysis results with video coding.

Non-Patent Document 3 provides a means of performing neural network post-filter processing for the method described in Non-Patent Document 1.

ITU-T Recommendation H.266 L.-Y. Duan, J. Liu, W. Yang, T. Huang and W. Gao, “Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics,” IEEETrans. Image Processing, vol.29, pp.8680-8695, Aug.2020. Text of ISO/IEC 23002-7:202x (2nd Ed.) DAM1 Additional SEI messages, Nov. 2022.

However, although Non-Patent Document 1 describes a video encoding and decoding method with high coding efficiency, there is a problem in that when performing image recognition on the decoded video, if the transmission rate is low, coding distortion reduces the accuracy of image recognition.

Non-Patent Document 2 discloses a method for integrating the description of video analysis results with video coding, but this method is not sufficient in terms of coding efficiency, and there is an issue that a low transmission bit rate cannot be achieved.

Non-Patent Document 3 provides a means for performing image processing in a neural network compared to the method in Non-Patent Document 1, but has the problem that it does not support image recognition technology.

A video decoding device according to one aspect of the present invention is characterized by having an image decoding device that decodes an image from encoded data, and an auxiliary information decoding device that decodes attention information having multiple values for each region as auxiliary information for the image decoded by the image decoding device.

The auxiliary information decoding device is also characterized in that it decodes information that specifies the neural network to be applied with respect to attention information having multiple values for each region.

A video coding device according to one aspect of the present invention is characterized by having an image coding device that codes an input image, and an auxiliary information coding device that codes attention information having multiple values for each region of the input image as auxiliary information.

The auxiliary information encoding device is also characterized in that it encodes, as auxiliary information, information that specifies the neural network to be applied with respect to attention information having multiple values for each region.

By configuring in this way, the problem of improving image recognition accuracy even at low rates can be solved by encoding and decoding additional auxiliary information without making major changes to the framework of the video encoding and decoding method.

1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. FIG. 2 is a diagram showing a hierarchical structure of encoded data. FIG. 1 is a schematic diagram showing a configuration of an image decoding device. 11 is a flowchart illustrating a schematic operation of an image decoding device. FIG. 1 is a block diagram showing a configuration of an image encoding device. FIG. 1 is a diagram showing an overview of the syntax of the neural network post-filter characteristic (NNPFC) SEI. A diagram showing the syntax of Neural Network Post Filter Activation (NNPFA) SEI. 11 is a diagram showing an example of the configuration of a syntax table of activation information SEI that specifies auxiliary information in one embodiment. FIG. A diagram showing a process of creating an activation map from syntax elements of activation information SEI in one embodiment. A diagram showing the syntax of an SEI payload, which is a container for an SEI message. FIG. 10 is a flowchart showing the processing of the image post-processing device 1002. FIG. 1 is a diagram showing an encoding device and a decoding device of an NNC.

First Embodiment
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

Figure 1 is a schematic diagram showing the configuration of another video transmission system according to this embodiment.

The video transmission system 1 is a system that transmits coded data that encodes an image, decodes and displays the transmitted coded data, and performs image recognition. The video transmission system 1 is composed of a video encoding device 10, a network 21, a video decoding device 30, an image display device 41, and an image recognition device 51.

The video encoding device 10 is composed of an image encoding device (image encoding unit) 11, an image analysis device (image analysis unit) 61, an auxiliary information creation device (auxiliary information creation unit) 71, an auxiliary information encoding device (auxiliary information encoding unit) 81, and a pre-image processing device (pre-image processing unit) 1001.

The video decoding device 30 is composed of an image decoding device (image decoding unit) 31, an auxiliary information decoding device (auxiliary information decoding unit) 91, and a post-image processing device (host image processing unit) 1002.

The pre-image processing device 1001 performs pre-image processing on the input video image T and sends the pre-processed image Tp to the image encoding device 11 and the auxiliary information creation device 71.

As an example of a specific embodiment, the activation information output by the auxiliary information creation device 71 may be input to the pre-image processing device 1001, and low-pass filter processing may be performed on areas other than the candidate recognition targets, reducing the difficulty of encoding and relatively improving the image quality of the candidate recognition target areas.

The image encoding device 11 compresses and encodes the output Tp of the pre-image processing device 1001.

The image analysis device 61 analyzes the input video image T, and in the image recognition device 51, analyzes information on which areas in the picture should be noted, and sends the analysis results to the auxiliary information creation device 71.

The auxiliary information creation device 71 generates useful attention information for the picture based on the analysis results of the image analysis device 61 and the pre-image processing Tp of the pre-image processing device 1001, and sends it to the auxiliary information encoding device 81.

The auxiliary information encoding device 81 encodes the auxiliary information created by the auxiliary information creation device 71 according to a predetermined syntax. The output of the image encoding device 11 and the output of the auxiliary information encoding device 81 are sent to the network 21 as encoded data Te.

The video encoding device 10 receives an input image T, compresses and encodes the image, analyzes the image, generates auxiliary information to be input to the image recognition device 51, encodes the image, generates encoded data Te, and transmits the encoded data to the network 21.

In FIG. 1, the auxiliary information encoding device 81 is not connected to the image encoding device 11, but the auxiliary information encoding device 81 and the image encoding device 11 may communicate necessary information as appropriate.

The network 21 transmits the encoded auxiliary information and the encoded data Te to the image decoding device 31. A part or all of the encoded auxiliary information may be included in the encoded data Te as auxiliary extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

The video decoding device 30 inputs the encoded data Te sent from the network 21, decodes the image, decodes the auxiliary information, and sends them to the image display device 41 and the image recognition device 51. It also decodes the auxiliary information and outputs it to the image recognition device 51.

The image decoding device 31 decodes each of the encoded data Te transmitted by the network 21, generates a decoded video image Td, and supplies it to the post-image processing device 1002.

The auxiliary information decoding device 91 decodes the encoded auxiliary information transmitted by the network 21 to generate auxiliary information, which it then transmits to the image recognition device 61.

In FIG. 1, the auxiliary information decoding device 91 is illustrated separately from the image decoding device 31, but the auxiliary information decoding device 91 may be included in the image decoding device 31. For example, the auxiliary information decoding device 91 may be included in the image decoding device 31 separately from each functional unit of the image decoding device 31. Also, although not connected to the image decoding device 31 in FIG. 1, the auxiliary information decoding device 91 and the image decoding device 31 may communicate necessary information as appropriate.

The post-image processing device 1002 performs post-image processing on the image decoding Td, which is the output of the image decoding device 31, and outputs the post-image processing To.

As an example of a specific embodiment, post-image processing using a neural network may be performed to improve the image quality of the decoded video image Td. At this time, network parameters for improving the image quality are input as auxiliary information from the auxiliary information decoding device 91 and used for the post-image processing.

The image display device 41 displays all or part of the post-processed image To output from the post-image processing device 1002. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Display forms include stationary, mobile, and HMD. When the image decoding device 31 has high processing power, it displays a high-quality image, and when it has only low processing power, it displays an image that does not require high processing power or display power.

The image recognition device 51 uses the post-processed image To output from the post-image processing device 1002 and the auxiliary information decoded by the auxiliary information decoding device 91 to perform image object detection, object region segmentation, object tracking, action recognition, human action evaluation, etc.

This configuration provides a framework that can maintain image recognition accuracy even at low rates by encoding and decoding additional auxiliary information without significantly changing the framework of the video encoding and decoding method.

This configuration provides a framework that can maintain image recognition accuracy even at low rates by encoding and decoding additional auxiliary information without significantly changing the framework of the video encoding and decoding method.

<Operator>
The operators used in this specification are listed below.

>> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

x ? y : z is a ternary operator that takes y if x is true (non-zero) and z if x is false (0).

Clip3(a,b,c) is a function that clips c to a value between a and b, and returns a if c<a, returns b if c>b, and returns c otherwise (where a<=b).

abs(a) is a function that returns the absolute value of a.

Int(a) is a function that returns the integer value of a.

floor(a) is a function that returns the largest integer less than or equal to a.

ceil(a) is a function that returns the smallest integer greater than or equal to a.

a/d represents the division of a by d (truncated to an integer).

<Structure of encoded data Te>
Before describing in detail the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

The coded data Te consists of multiple CVSs (Coded Video Sequences) and EoBs (End of Bitstream NAL units). CVSs consist of multiple AUs (Access Units) and EoSs (End of Sequence NAL units). The AU at the beginning of a CVS is called a CVSS (Coded Video Sequence Start) AU. A CVS is divided into layers and units called CLVSs (Coded Layer Video Sequences). An AU consists of one or more layered PUs (Picture Units) with the same output time. If the multilayer coding method is not adopted, an AU consists of one PU. A PU is a unit of coded data for one decoded picture consisting of multiple NAL units. A CLVS consists of PUs of the same layer, and the PU at the beginning of a CLVS is called a CLVSS (Coded Layer Video Sequence Start) PU. CLVSS PUs are limited to PUs that are randomly accessible IRAPs (Intra Random Access Pictures) or GDRs (Gradual Decoder Refresh Pictures). A NAL unit consists of a Nal unit header and RBSP (Raw Byte Sequence Payload) data. The Nal unit header consists of 2 bits of 0 data, followed by a 6-bit nuh_layer_id that indicates the layer value, a 5-bit nuh_unit_type that indicates the NAL unit type, and a 3-bit nuh_temporal_id_plus1 that is the Temporal ID value plus 1.

Figure 2 is a diagram showing a hierarchical structure of data in the coded data Te in units of PU. The coded data Te illustratively includes a sequence and a number of pictures constituting the sequence. Figure 2 shows a diagram showing a coded video sequence that defines the sequence SEQ, a coded picture that specifies the picture PICT, a coded slice that specifies the slice S, coded slice data that specifies the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

(Coded Video Sequence)
In the coded video sequence, a set of data to be referred to by the image decoding device 31 in order to decode the sequence SEQ to be processed is specified. As shown in Fig. 2, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

In a video parameter set (VPS), a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and individual layers included in the video, are specified.

The sequence parameter set SPS specifies a set of coding parameters that the image decoding device 31 references to decode the target sequence. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

Here, the sequence parameter set SPS includes the following syntax elements:
ref_pic_resampling_enabled_flag: A flag that specifies whether or not to use a function that changes the resolution (resampling) when decoding each image included in a single sequence that references the target SPS. In other words, this flag indicates that the size of the reference picture referenced in generating a predicted image changes between each image indicated by a single sequence. If the value of this flag is 1, the resampling is applied, and if the value is 0, it is not applied.
pic_width_max_in_luma_samples: A syntax element that specifies the width of the image with the largest width among the images in a single sequence, in units of luminance blocks. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY), where MinCbSizeY is a value determined by the minimum size of a luminance block.
pic_height_max_in_luma_samples: A syntax element that specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY).

The picture parameter set PPS specifies a set of coding parameters that the image decoding device 31 references in order to decode each picture in the target sequence. Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

Here, the picture parameter set PPS includes the following syntax elements:
pps_pic_width_in_luma_samples: A syntax element that specifies the width of the target picture. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_width_max_in_luma_samples.
pps_pic_height_in_luma_samples: A syntax element that specifies the height of the target picture. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_height_max_in_luma_samples.
pps_conformance_window_flag: a flag indicating whether conformance (cropping) window offset parameters will be signaled subsequently and where the conformance window should be displayed. If this flag is 1, the parameters will be signaled, if it is 0, the conformance window offset parameters are not present.
pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, pps_conf_win_bottom_offset: offset values for specifying the left, right, top, and bottom positions of a picture output by decoding processing with respect to a rectangular area specified by the output picture coordinates. In addition, when the value of pps_conformance_window_flag is 0, the values of pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, and pps_conf_win_bottom_offset are estimated to be 0.

Here, the variable ChromaFormatIdc of the chrominance format is the value of sps_chroma_format_id, and the variables SubWidthC and SubHightC are values determined by this ChromaFormatIdc. In the case of a monochrome format, SubWidthC and SubHightC are both 1, in the case of a 4:2:0 format, SubWidthC and SubHightC are both 2, in the case of a 4:2:2 format, SubWidthC is 2 and SubHightC is 1, and in the case of a 4:4:4 format, SubWidthC and SubHightC are both 1.
pps_init_qp_minus26 is information for deriving the quantization parameter SliceQpY of the slice referenced by the PPS.

(Subpicture)
A picture may be further divided into rectangular sub-pictures. The size of a sub-picture may be a multiple of the CTU. A sub-picture is defined as a set of tiles that are an integer number of consecutive tiles vertically and horizontally. In other words, a picture is divided into rectangular tiles, and a sub-picture is defined as a set of rectangular tiles. A sub-picture may be defined using the IDs of the top left tile and the bottom right tile of the sub-picture.

Figure 5 is a conceptual diagram of an image to be processed in the video transmission system 1, showing changes in the resolution of the image over time. However, in Figure 5, no distinction is made as to whether the image is encoded or not. Figure 5 shows an example of transmitting an image to the image decoding device 31 while adaptively changing the resolution using a picture parameter set PPS during processing in the video transmission system 1.

(Encoded Picture)
A coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode a picture PICT to be processed. As shown in Fig. 2, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

In the following, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. The same applies to other data that are included in the encoded data Te described below and have subscripts.

The picture header contains the following syntax elements:

pic_temporal_mvp_enabled_flag is a flag that specifies whether or not temporal motion vector prediction is used for inter prediction of the slice associated with the picture header. If the value of the flag is 0, the syntax elements of the slice associated with the picture header are restricted so that temporal motion vector prediction is not used in decoding the slice. If the value of the flag is 1, it indicates that temporal motion vector prediction is used in decoding the slice associated with the picture header. If the flag is not specified, the value is presumed to be 0.

(Coding Slice)
A coded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a current slice S. As shown in Fig. 2, a slice includes a slice header and slice data.

The slice header includes a set of coding parameters that the image decoding device 31 refers to in order to determine the decoding method for the target slice. Slice type specification information (slice_type) that specifies the slice type is an example of a coding parameter included in the slice header.

Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction during encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice that includes a block for which inter prediction can be used.

Note that the slice header may also contain a reference to the picture parameter set PPS (pic_parameter_set_id).

(Encoded slice data)
The coded slice data specifies a set of data to be referenced by the image decoding device 31 in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 2. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

(coding tree unit)
2 specifies a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU is divided into coding units CU, which are basic units of encoding processing, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). BT division and TT division are collectively called multi tree division (MT (Multi Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of a quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether CT splitting is performed, a QT split flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

Different trees may be used for luminance and chrominance. The type of tree is indicated by treeType. For example, when using a common tree for luminance (Y, cIdx=0) and chrominance (Cb/Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. When using two different trees (DUAL trees) for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

(Encoding Unit)
2 specifies a set of data to be referenced by the image decoding device 31 in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header specifies a prediction mode, etc.

Prediction processing may be performed on a CU basis, or on a sub-CU basis, which is a further division of a CU. If the size of the CU and sub-CU are the same, there is one sub-CU in the CU. If the size of the CU is larger than the size of the sub-CU, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (e.g., between display times or between layer images).

The transformation and quantization processes are performed on a CU basis, but the quantized transformation coefficients may be entropy coded on a subblock basis, such as 4x4.

(Prediction parameters)
The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

The prediction parameters of inter prediction are explained below. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indexes refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when the term "flag indicating whether or not XX is true" is used, a flag other than 0 (for example, 1) is XX, and 0 is not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (same below). However, in actual devices and methods, other values can be used as true and false values.

Syntax elements for deriving inter prediction parameters include, for example, the affine flag affine_flag used in merge mode, the merge flag merge_flag, the merge index merge_idx, the MMVD flag mmvd_flag, the inter prediction identifier inter_pred_idc for selecting a reference picture to be used in AMVP mode, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx for deriving a motion vector, the difference vector mvdLX, and the motion vector precision mode amvr_mode.

(Configuration of the image decoding device)
The configuration of an image decoding device 31 (FIG. 3) according to this embodiment will be described.

The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, an addition unit 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image encoding device 11 described below, the image decoding device 31 may also be configured not to include the loop filter 305.

The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data when a prediction error is included in the TU.

When in a mode other than skip mode (skip_mode==0), the TU decoding unit 3024 decodes the QP update information and the quantized prediction error from the encoded data. More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp indicating whether or not the current block contains a quantized prediction error, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the encoded data, it derives it as 0.

The TU decoding unit 3024 decodes an index mts_idx indicating the transformation base from the encoded data. The TU decoding unit 3024 also decodes an index stIdx indicating the use of a secondary transformation and the transformation base from the encoded data. When stIdx is 0, it indicates no application of a secondary transformation, when it is 1, it indicates one transformation of a set (pair) of secondary transformation bases, and when it is 2, it indicates the other transformation of the pair.

The TU decoding unit 3024 may also decode the sub-block transform flag cu_sbt_flag. When cu_sbt_flag is 1, the CU is divided into multiple sub-blocks, and the residual of only one specific sub-block is decoded. The TU decoding unit 3024 may further decode a flag cu_sbt_quad_flag indicating whether the number of sub-blocks is 4 or 2, cu_sbt_horizontal_flag indicating the division direction, and cu_sbt_pos_flag indicating a sub-block that includes a non-zero transform coefficient.

The predicted image generation unit 308 includes an inter predicted image generation unit 309 and an intra predicted image generation unit 310.

In the following, an example will be described in which CTU and CU are used as processing units, but this is not limiting and processing may be performed in sub-CU units. Alternatively, CTU and CU may be interpreted as blocks and sub-CU as sub-blocks, and processing may be performed in block or sub-block units.

The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from outside, and decodes each code (syntax element). There are two types of entropy coding: one is to perform variable-length coding of syntax elements using a context (probability model) adaptively selected according to the type of syntax element and the surrounding circumstances, and the other is to perform variable-length coding of syntax elements using a predefined table or formula. The former CABAC (Context Adaptive Binary Arithmetic Coding) stores the CABAC state of the context (probability state index pStateIdx that specifies the type (0 or 1) and probability of the most probable symbol) in memory. The entropy decoding unit 301 initializes all CABAC states at the beginning of a segment (tile, CTU row, slice). The entropy decoding unit 301 converts the syntax elements into a binary string (Bin String) and decodes each bit of the Bin String. When a context is used, a context index ctxInc is derived for each bit of the syntax element, the bit is decoded using the context, and the CABAC state of the used context is updated. Bits that do not use a context are decoded with equal probability (EP, bypass), and the ctxInc derivation and CABAC state are omitted. The decoded syntax elements include prediction information for generating a predicted image and prediction error for generating a difference image.

The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, the prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

(Basic flow)
FIG. 4 is a flowchart illustrating a schematic operation of the image decoding device 31.

(S1100: Decoding parameter set information) The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS from the encoded data.

(S1200: Slice information decoding) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

The image decoding device 31 then repeats the processes from S1300 to S5000 for each CTU included in the target picture to derive a decoded image for each CTU.

(S1300: Decoding CTU information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

(S1400: CT information decoding) The CT information decoding unit 3021 decodes the CT from the encoded data.

(S1500: CU decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

(S1510: CU information decoding) The CU decoding unit 3022 decodes CU information, prediction information, the TU split flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, etc. from the encoded data.

(S1520: TU information decoding) When a prediction error is included in a TU, the TU decoding unit 3024 decodes the QP update information, the quantization prediction error, and the transform index mts_idx from the encoded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

(S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the target CU based on the prediction information.

(S3000: Inverse quantization and inverse transform) The inverse quantization and inverse transform unit 311 performs inverse quantization and inverse transform processing on each TU included in the target CU.

(S4000: Decoded image generation) The adder 312 generates a decoded image of the target CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311.

(S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

Non-Patent Document 1 describes a video encoding and decoding method with extremely high coding efficiency, but when performing image recognition on the decoded image of a compressed video, there is a problem in that the accuracy of image recognition decreases due to coding distortion when the transmission rate is low.

In addition, Non-Patent Item 2 discusses a method for integrating the description of video analysis results with video coding, but this is not sufficient in terms of coding efficiency, and there is an issue that low transmission bit rates cannot be achieved.

In addition, Non-Patent Document 3 provides a means for performing neural network post-filter processing for the method in Non-Patent Document 1, but has the problem that it does not support image recognition technology.

In this embodiment, a framework is provided that can maintain image recognition accuracy even at low rates by encoding and decoding additional auxiliary information without significantly changing the framework of the video encoding and decoding method.

(Neural network post-filter characteristic (NNPFC) SEI
Fig. 6 shows an outline of the syntax of the Neural Network Post-Filter Characteristics (NNPFC) SEI message of Non-Patent Document 3. The NNPFC SEI message specifies the neural network to be applied as a post-filter process. The application of a specific post-filter process to a specific picture is indicated by the Neural Network Post-Filter Activation SEI message (Fig. 7).

To apply this SEI message, the following variables must be defined:
The width and height of the picture decoded by the image decoding device 31 in units of luminance pixels. The luminance pixel array CroppedYPic[idx] and the chrominance pixel arrays CroppedCbPic[idx] and CroppedCrPic[idx], and the picture with idx in the range from 0 to numInputPics-1, which are used as inputs for post-filter processing. The pixel bit depth BitDepthY of the luminance pixel array.
・Pixel bit length of the chrominance pixel array BitDepthC
Variables SubWidthC and SubHeightC indicating the chrominance format indicated by the chrominance format ChromaFormatIdc of the picture decoded by the image decoding device 31. When the chrominance format is 4:2:0, the variables SubWidthC and SubHeightC are both 2, when the chrominance format is 4:2:2, the variables SubWidthC and SubHeightC are both 2 and 1, when the chrominance format is 4:4:4, the variables SubWidthC and SubHeightC are both 1.
nnpfc_auxiliary_inp_idc indicates the presence of auxiliary information in the input tensor of the neural network postfilter, and if its value is equal to 1, input the deblocking filtering strength control value StrengthControlVal, which is a real number ranging from 0 to 1, as auxiliary information.

The syntax element nnpfc_id indicates an identification number that can be used to identify the post-filter process.

If the NNPFC SEI message is the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value within the current CLVS, the following applies:
- Indicates that this SEI message is a basic post-filter process.
This SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer until the end of the current CLVS. If an NNPFC SEI message is a repetition of a previous NNPFC SEI message in the current CLVS in decoding order, the subsequent semantics apply as if this SEI message was the only NNPFC SEI message with the same content in the current CLVS.

The NNPFC SEI message is the first NNPFC in decoding order with a particular nnpfc_id value in the current CLVS.
If it is not an SEI message, the following applies:
Indicates that this SEI message is an update relative to the previous elementary post-filter in decoding order using the same nnpfc_id value. This SEI message pertains to the current decoded picture and all subsequent decoded pictures in the current layer, until the end of the current CLVS or until the next NNPFC SEI message with the particular nnpfc_id value in the current layer.
If nnpfc_mode_idc is 0, it indicates that this SEI message contains an ISO/IEC 15938-17 bit stream that specifies post-filtering or is an update relative to an underlying post-processing filter with the same nnpfc_id value.

nnpfc_mode_idc equal to 1 indicates that the post-filter associated with the nnpfc_id value is a neural network identified by the URI indicated by nnpfc_uri, of the form identified by the tag URI nnpfc_tag_uri.

nnpfc_reserved_zero_bit_a indicates 0.

nnpfc_tag_uri contains a tag URI with syntax and semantics specified in IETF RFC4151. It is used to store format and related information about the neural network used as the base postfilter process, or for updates related to postfilter processes with the same nnpfc_id value. Note that nnpfc_tag_uri allows the format of the neural network data specified by nnrpf_uri to be uniquely identified without the need for a registration authority. If nnpfc_tag_uri is "tag:iso.org,2023:15938-17", it indicates that the neural network data identified by nnpfc_uri is ISO/IEC 15938-17 compliant and encoded with NNC (Neural Network Coding).

nnpfc_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 to identify the neural network to be used as a postfilter or update associated with a postfilter with the same nnpfc_id value.

When nnpfc_formatting_and_purpose_flag is 1, it indicates that syntax elements related to the filter's purpose, input format, output format, and complexity are present. When nnpfc_formatting_and_purpose_flag is 0, it indicates that syntax elements related to the filter's purpose, input format, output format, and complexity are not present.

This SEI message is the first NNPFC in decoding order with a particular nnpfc_id value in the current CLVS.
If this SEI message is a NN PFC SEI message, then nnpfc_formatting_and_purpose_flag shall be equal to 1. If this SEI message is not the first NN PFC SEI message in decoding order, then the value of nnpfc_formatting_and_purpose_flag shall be 0 for the particular nnpfc_id value in the current CLVS.

nnpfc_purpose indicates the purpose of the postfilter processing. The value of nnpfc_purpose is 1 for image quality improvement, 2 for chrominance upsampling from 4:2:0 chrominance format to 4:2:2 or 4:4:4, or chrominance upsampling from 4:2:2 chrominance format to 4:4:4, 3 for increasing the width or height of the cropped decoded output image without changing the chrominance format, 4 for increasing the width or height of the decoded output image and upsampling the chrominance format, and 5 for picture rate upsampling.

nnpfc_reserved_zero_bit_b is set to 0.

nnpfc_payload_byte[i] is the i-th byte of the bitstream encoded in NNC in accordance with ISO/IEC 15938-17.

(Neural Network Post Filter Activation (NNPFA) SEI)
7 shows the syntax of the Neural Network Post-Filter Activation (NNPFA) SEI message of Non-Patent Document 3. The Neural Network Post-Filter Activation NNPFA SEI message activates or deactivates the application of a target neural network post-filtering identified by nnpfa_target_id for post-filtering of a sequence of pictures.

nnpfa_target_id indicates the target picture neural network postfiltering. It identifies one or more NNPFC SEI messages with nnpfc_id equal to nnfpa_target_id for the current picture.

An NNPFA SEI message with a particular value of nnpfa_target_id must not be present on the current PU unless one or both of the following conditions are true:
- There is an NNPFC SEI message in the current CLVS with nnpfc_id equal to a particular value of nnpfa_target_id present in a PU that precedes the current PU in decoding order.
- There is an NNPFC SEI message with nnpfc_id equal to a specific value of nnpfa_target_id of the current PU.

If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFA SEI message with nnpfa_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFA SEI message in decoding order.

nnpfa_cancel_flag, when set to 1, indicates that the continuity of the target neural network postfilter process set by the previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is cancelled. That is, the target neural network postfilter process is not executed.

If it has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 0, it will not be used unless activated by another NNPFA SEI message. If nnpfa_cancel_flag is 0, it indicates that nnpfa_persistence_flag will persist.

nnnpfa_persistence_flag indicates the continuity of the target neural network postfilter processing for the current layer.

When nnpfa_persistence_flag is 0, it indicates that the target neural network postfiltering applies to postfiltering of the current image only.

nnpfa_persistence_flag, when set to 1, indicates that the target neural network postfiltering should be applied to the current image and all subsequent pictures of the current layer until one or more of the following conditions become true:
A new CLVS for the current layer begins. The bitstream ends. The picture for the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 1 is output after the current picture in output order.

Note that neural network postfiltering is not applied to subsequent pictures of the current layer associated with an NNPFA SEI message if the current SEI message has the same nnpfa_target_id and nnpfa_cancel_flag as the current SEI message and both have the same nnpfa_target_id and nnpfa_cancel_flag set to 1.

(Attention Information SEI)
8 is a diagram showing one form of the syntax of auxiliary information encoded and decoded by the auxiliary information encoding device 81 and the auxiliary information decoding device 91 of this embodiment. In this example, an SEI called attention_info is shown. This SEI is transmitted in units of PU, and is intended to improve the recognition accuracy and reduce the amount of processing when the image recognition device processes the picture. For this purpose, it is an SEI message that is encoded and decoded as auxiliary information for generating attention information used in the recognition processing for the picture, and has the number of bytes of the value of payloadSize.

The syntax, syntax elements, and semantics of the attention information SEI in this embodiment are explained below.

To apply this SEI message, the following variables must be defined:
Variables SubWidthC and SubHeightC indicating the chrominance format indicated by the chrominance format ChromaFormatIdc of the picture decoded by the image decoding device 31. When the chrominance format is 4:2:0, the variables SubWidthC and SubHeightC are both 2, when the chrominance format is 4:2:2, the variables SubWidthC and SubHeightC are both 2 and 1, when the chrominance format is 4:4:4, the variables SubWidthC and SubHeightC are both 1.

The syntax element attention_target_id specifies the neural network of the target picture. If there is one or more neural network post-filter characteristics SEI messages with nnpfc_id equal to nnfpa_target_id, attention information is input to nnpfc_purpose of the NNPFC SEI and neural network post-filter processing is performed. In this case, the attention information is input to the post image processing device 1002 and used as attention information.

As a specific example of an embodiment, when the value of nnpfc_auxliary_inp_idc is 2, attention information is input as an input tensor of the neural network postfilter, and postfilter processing is performed on areas that are attracting high attention in the attention information. The postfilter processing at this time may be optimized based on the criterion of improving image recognition rate, rather than simply improving image quality.

As another example of implementation, when performing post-filtering of a picture activated by the NNPFA SEI, attention information may be used to adjust the strength of post-filtering of regions of high interest.

This configuration improves image recognition accuracy and reduces the amount of processing required for image recognition.

If the value of attention_target_id is equal to the ID value of the neural network of the image recognition device 51 that is outside the value of nnpfc_id, the attention information is input to the image recognition device 51. In this case, the attention information is input to the image recognition device 51 and used as attention information for image recognition. With this configuration, it is possible to improve the image recognition accuracy and reduce the amount of processing required for image recognition.

attention_width_in_luma_samples and attention_height_in_luma_samples represent the number of pixels in the horizontal and vertical directions for the luminance of the attention information. Note that the value of attention_width_in_luma_samples is a multiple of the variable SubWidthC to accommodate 4:2:2 and 4:2:0 chrominance formats, where the number of chrominance pixels is different from the number of luminance pixels, and the value of attention_height_in_luma_samples is a multiple of the variable SubHeightC.

attention_bit_depth_minus8 is a syntax element that indicates the attention value minus 8. The attention value shall take values from 0 to 2 to the power of (attention_bit_depth_minus8+8) minus 1.

attention_number_of_region_minus1 is a syntax element that represents the number of regions for describing attention information minus 1. The number of regions for describing attention information is equal to the value of attention_number_of_region_minus1 plus 1.

Attention_region_x and attention_region_y are syntax elements that indicate the position of the top left corner of a rectangle for generating attention. Attention_region_x is the x coordinate value (horizontal direction) of the luminance of the top left corner of the rectangular region. The value of attention_region_x is a multiple of the variable SubWidthC. Attention_region_y is the y coordinate value (vertical direction) of the luminance of the top left corner of the rectangular region. The value of attention_region_y is a multiple of the variable SubHeightC. In addition, attention_region_x and attention_region_y may be relative positions within the screen.

attention_region_width and attention_region_hight are syntax elements that indicate the size of the rectangle for generating attention. attention_region_width is the number of pixels of horizontal luminance in the rectangular region, and is a multiple of the value of the variable SubWidthC. Note that the value of attention_region_x+attention_region_width must not exceed the number of pixels in the horizontal direction of the picture. attention_region_hight is the number of pixels of vertical luminance in the rectangular region, and is a multiple of the value of the variable SubHightC. Note that the value of attention_region_y+attention_region_height must not exceed the number of pixels in the vertical direction of the picture.

In this embodiment, attention information is generated using a rectangle, and is expressed by the coordinate value of the top left corner of the rectangle and the number of pixels in the horizontal and vertical directions. However, other methods are also possible. For example, the position information (attention_region_x, attention_region_y) may be the top right, bottom left, bottom right, or center of gravity instead of the top left corner of the rectangle. Furthermore, the size of the rectangle (attention_region_width, attention_region_height) may be limited to a square other than a rectangle, and only the number of pixels on one side (region_size) may be specified. Alternatively, the position and size may be specified in 4x4 or 16x16 units, or the CTU address and number of CTUs, which are the encoding units, instead of in pixel units.

attention_region_weight_y indicates the luminance value for generating attention. attention_region_weight_cb indicates the chrominance Cb value for generating attention. attention_region_weight_cr indicates the chrominance Cr value for generating attention. attention_region_weight_y, attention_region_weight_cb, and attention_region_weight_cr shall take values from 0 to 2 to the power of (attention_bit_depth_minus8+8) minus 1.

Figure 9 shows an example of a procedure for creating attention from the values of syntax elements of the attention information SEI in Figure 8.

The arrays AttentionY[][], AttentionCb[][], and AttentionCr[][] are two-dimensional array structures that represent attention information, and attention_width_in_luma_samples and attention_height_in_luma_samples represent the number of luminance pixels in the horizontal and vertical directions. The number of chrominance pixels of the attention information is attention_width_in_luma_samples/SubWidthC in the horizontal direction and attention_height_in_luma_samples/SubHeightC in the vertical direction.

First, the arrays AttentionY[][], AttentionCb[][], and AttentionCr[][] are initialized. The luminance attention array AttentionY[][] is initialized to 0, and the chrominance attention arrays AttentionCb[][] and AttentionCr[][] are initialized to the value 1 << (attention_bit_depth_minus8 + 7), which is 2 to the power of (attention_bit_depth_minus8 + 7).

Next, loop attention_number_of_region_minus1 plus once and input attention_region_weight_y to AttentionY[y][x], which has a size of attention_region_width and attention_region_height, from the point indicated by attention_region_x and attention_region_y. attention_number_of_region_minus1 is a syntax element that represents the number of rectangular regions for describing the attention information set in the attention information SEI minus 1.

attention_region_x is the x coordinate value (horizontal) of the luminance of the top left of the rectangular region. attention_region_y is the y coordinate value (vertical) of the luminance of the top left of the rectangular region. attention_region_width is the number of pixels in the horizontal direction, and attention_region_height is the number of pixels in the vertical direction. AttentionY[y][x] is the attention array of the luminance of the rectangular region. attention_region_weight_y is the attention value.

For AttentionCb[y][x] and AttentionCr[y][x], valCb and valCr are input, respectively, from the points indicated by attention_region_x/SubWidthC and attention_region_y/SubHeightC to rectangular regions with pixel sizes of attention_region_width/SubWidthC and attention_region_height/SubHeightC.

AttentionCb[y][x] and AttentionCr[y][x] are the attention arrays for chrominance. attention_region_x/SubWidthC is the x coordinate value (horizontal direction) of the luminance at the top left of the rectangular region. attention_region_y/SuabHeightC is the y coordinate value (vertical direction) of the luminance at the top left of the rectangular region. attention_region_width/SubWidthC is the number of horizontal pixels, and attention_region_height/SubHeightC is the number of vertical pixels.

valCb is the sum of attention_region_weight_cb and 2 to the power of (attention_bit_depth_minus8 + 7), and valCr is the sum of attention_region_weight_cr and 2 to the power of (attention_bit_depth_minus8 + 7).

Note that in the method for creating attention information in this embodiment, if the rectangular areas for describing attention information overlap, the value will be overwritten with the value of the area defined later, but the average of the original value and this value may also be taken.

Using this syntax and semantics, attention information can be described concisely and with a small amount of code.

The image analysis device 61 analyzes the input video image T to detect candidates for recognition. Here, in order to reduce the amount of processing, it is sufficient that the accuracy of the candidates for recognition is sufficient. Furthermore, when the position of the recognition target in the picture can be predicted, such as in the case of a fixed camera image, the image analysis device 61 may set the detection target area in advance as the candidate recognition area.

The auxiliary information creation device 71 converts the attention information detected by the image analysis device 61 into information on the position within the picture and the size of the rectangle, and sends it to the auxiliary information encoding device 81.

Alternatively, the output of the auxiliary information creation device 71 may be input to the image coding device 11. In this case, the image coding device 11 may control image quality using attention information created by the auxiliary information creation unit 71. For example, high image quality may be achieved by using a quantization parameter for the area of the candidate recognition target that is smaller than that for other areas in the picture. By doing this, it is possible to improve the recognition accuracy.

In addition to the decoded video image Td, attention information is input to the image recognition device 51 as auxiliary information. As a result, it is possible to process the candidate recognition target area with priority over all information in the picture, significantly reducing the amount of processing and improving recognition accuracy. Furthermore, if the image quality of the decoded image of the recognition target area is improved, the recognition accuracy will also improve.

According to this embodiment, even when using decoded images encoded at a low rate, it is possible to improve the image recognition accuracy of the image recognition device 51 and reduce the amount of processing required for image recognition.

In addition, the auxiliary information creation device 71, the auxiliary information encoding device 81, and the auxiliary information decoding device 91 may commonly hold generic network parameters. The auxiliary information creation device 71 uses a framework such as the neural network postfilter characteristic SEI to create network parameters as auxiliary information that partially update the commonly held generic network. The auxiliary information may then be encoded by the auxiliary information encoding device 81 and decoded by the auxiliary information decoding device 91. With this configuration, the amount of code for the auxiliary information can be reduced, and auxiliary information corresponding to the input image T can be created, encoded, and decoded.

In order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) indicating the format may be sent. Furthermore, the actual auxiliary information following the identifier may be transmitted as a byte string.

The auxiliary information of the network parameters decoded by the auxiliary information decoding device 91 is input to the post-image processing device 1002.

The post-image processing device 1002 uses the decoded auxiliary information (neural network post-filter characteristics SEI, neural network post-filter activation SEI, and attention information SEI) to perform post-image processing using a neural network to restore the decoded video image Td.

For example, post-image processing may be performed only on specific regions using the attention information in the attention information SEI shown in Figure 8. The post-filter processing in this case may be optimized based on improving the image recognition rate, rather than simply improving image quality.

This improves the image quality of the decoded video image Td on the decoded image side, and also improves the recognition accuracy of the image recognition device.

The encoding and decoding of activation information is not limited to SEI, and syntax such as SPS, PPS, APS, and slice header may also be used.

The auxiliary encoding device 81 encodes the auxiliary information based on the syntax tables in Figures 6, 7, 8, and 10. The auxiliary information is encoded as auxiliary extension information SEI, multiplexed into the encoded data Te output by the image encoding device 11, and output to the network 21.

The auxiliary information decoding device 91 decodes the auxiliary information from the encoded data Te based on the syntax tables in Figures 6, 7, 8, and 10, and sends the decoded result to the post-image processing device 1002 and the image recognition device 51. The auxiliary information decoding device 91 decodes the auxiliary information encoded as auxiliary extension information SEI.

The post-image processing device 1002 performs post-image processing on the decoded video Td using the decoded video Td and auxiliary information to generate post-image processing To.

In addition, the auxiliary information creation device 71, the auxiliary information encoding device 81, and the auxiliary information decoding device 91 may commonly hold general-purpose network parameters. The auxiliary information creation device 71 may create network parameters that partially update the commonly held general-purpose network as auxiliary information, which may then be encoded by the auxiliary information encoding device 81 and decoded by the auxiliary information decoding device 91. With this configuration, the amount of coding for the auxiliary information can be reduced, and auxiliary information corresponding to the input image T can be created, encoded, and decoded.

In addition, in order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) indicating the format may be sent. Furthermore, the actual auxiliary information following the identifier may be transmitted as a byte string.

The auxiliary information of the network parameters decoded by the auxiliary information decoding device 91 is input to the post-image processing device 1002.

Note that in the example of this embodiment, the syntax is shown as SEI, but it is not limited to SEI, and syntax such as SPS, PPS, APS, slice header, etc. may also be used.

By configuring in this way, the problem of maintaining image recognition accuracy even at low rates can be solved by encoding and decoding additional auxiliary information without making major changes to the framework of the video encoding and decoding method.

In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the control function. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or devices that store a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the above-mentioned program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

In addition, part or all of the image encoding device 11 and image decoding device 31 in the above-mentioned embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and image decoding device 31 may be individually made into a processor, or part or all of them may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI appears due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the present invention.

[Application example]
The above-mentioned video encoding device 10 and video decoding device 30 can be mounted and used in various devices that transmit, receive, record, and play back videos. Note that the video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

(SEI payload)
FIG. 10 is a diagram showing the syntax of the SEI payload, which is a container of the SEI message.

Called when nal_unit_type is PREFIX_SEI_NUT. PREFIX_SEI_NUT indicates that the SEI is located before the slice data.

When payloadType is 210, the neural network post-filter characteristics SEI is called.

When payloadType is 211, the neural network post-filter activation SEI is called.

When payloadType is 212, the attention information SEI is called.

(SEI decoding and post-filtering)
The header decoding unit 3020 reads the SEI payload, which is a container of the SEI message, and decodes the neural network post-filter characteristics SEI message. For example, the header decoding unit 3020 decodes nnpfc_id, nnpfc_mode_idc, nnpfc_formatting_and_purpose_flag, nnpfc_purpose, nnpfc_reserved_zero_bit_a, nnpfc_uri_tag[i], nnpfc_uri[i], nnpfc_reserved_zero_bit_b, and nnpfc_payload_byte[i].

Figure 11 is a diagram showing a flowchart of the processing of the post-image processing device 1002. The post-image processing device 1002 performs the following processing according to the parameters of the above SEI message.

S6001: Read processing volume and accuracy from SEI.

S6002: If the complexity exceeds the level that the post-image processing device 1002 can process, the process ends. If not, the process proceeds to S6003.

S6003: If the accuracy exceeds the processing capability of the post-image processing device 1002, the process ends. If not, the process proceeds to S6004.

S6004: Identify the network model from the SEI and set the topology of the post image processing device 1002.

S6005: Derive network model parameters from SEI update information.

S6006: The derived network model parameters are loaded into the post-image processing device 1002.

S6007: Executes filter processing in the post-image processing device 1002 and outputs to the outside.

However, the SEI is not necessarily required to construct luma and chroma samples in the decoding process.

(Details of the post-imaging processing device 1002)
The NN filter unit performs filtering using a neural network model, using the input image inputTensor and input parameters (e.g., QP, bS, etc.). The input image may be an image for each component, or an image with multiple components as channels. In addition, the input parameters may be assigned to a channel different from the image.

The NN filter section may repeatedly apply the following process:

The NN filter section performs a convolution operation (conv, convolution) on the inputTensor with the kernel k[m][i][j], and derives the output image outputTensor by adding bias. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the sum over mm, i, and j, respectively.

outputTensor[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][xx+i-of][yy+j-of]+bias[nn])
For 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, of=0 is set. For 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, of=1 is set. n is the number of channels of outSamples, m is the number of channels of inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. of is the size of the padding area around inputTensor to make the sizes of inputTensor and outputTensor the same. In the following, when the output of the NN filter part is a value (correction value) rather than an image, the output is represented as corrNN instead of outputTensor.

Note that if you write inputTensor and outputTensor in CHW format instead of inputTensor and outputTensor in CWH format, it is equivalent to the following process.

outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn])
In addition, a process called Depth-wise Conv, shown in the following formula, may be performed. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for i and j, respectively. n is the number of channels of outputTensor and inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor.

outputTensor[nn][xx][yy]=ΣΣ(k[nn][i][j]*inputTensor[nn][xx+i-of][yy+j-of]+bias[nn])
Also, a nonlinear process called Activate, for example, ReLU, may be used.
ReLU(x) = x >= 0 ? x : 0
Alternatively, leakyReLU shown in the following formula may be used.

leakyReLU(x) = x >= 0 ? x : a * x
Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the values of k, bias, and a above may be integers, and a right shift may be performed after conv.

With ReLU, 0 is always output for values less than 0, and the input value is output as is for values greater than or equal to 0. On the other hand, with leakyReLU, linear processing is performed for values less than 0 with the gradient set by a. With ReLU, the gradient for values less than 0 disappears, which can make it difficult for learning to progress. With leakyReLU, the gradient for values less than 0 remains, making the above problem less likely to occur. Also, of the above leakyReLU(x), PReLU, which uses a parameterized value of a, can be used.

(NNC)
Neural Network Coding (NNC) is an international standard ISO/IEC15938-17 for efficiently compressing neural networks (NNs). Compressing trained NNs makes it possible to store and transmit NNs more efficiently.

The following provides an overview of NNC's encoding and decoding processes.

Figure 12 shows the NNC encoding and decoding devices.

The NN coding device 801 has a pre-processing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN coding device 801 inputs an uncompressed NN model O, and the quantization unit 8012 quantizes the NN model O to obtain a quantized model Q. The NN coding device 801 may repeatedly apply parameter reduction techniques such as pruning and sparsification in the pre-processing unit 8011 before quantization. Then, the entropy coding unit 8013 applies entropy coding to the quantized model Q to obtain a bit stream S for storing and transmitting the NN model.

The NN decoding device 802 has an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first inputs the transmitted bit stream S, and the entropy decoding unit 8021 performs entropy decoding of S to obtain an intermediate model RQ. If the operating environment of the NN model supports inference using the quantized representation used in RQ, the RQ may be output and used for inference. If not, the parameter restoration unit 8022 restores the parameters of RQ to their original representation to obtain an intermediate model RP. If the sparse tensor representation used can be processed in the operating environment of the NN model, the RP may be output and used for inference. If not, a reconstructed NN model R that does not include a tensor or structural representation different from the NN model O is obtained and output.

The NNC standard includes decoding methods for the numerical representation of specific NN parameters, including integers and floating point.

The NNR_PT_INT decoding method decodes a model with integer-valued parameters. The NNR_PT_FLOAT decoding method extends NNR_PT_INT by adding a quantization step size delta. This delta is multiplied by the integer value above to produce a scaled integer. Delta is derived from the integer quantization parameter qp and the granularity parameter qp_density of delta as follows:

mul = 2^(qp_density) + (qp& (2^(qp_density)-1))
delta = mul * 2^((qp >> qp_density) - qp_density)
(Format of trained NN)
The representation of a trained NN consists of two elements: a topological representation, such as the size of layers and the connections between layers, and a parameter representation, such as weights and biases.

Topology representation is covered by native formats such as Tensorflow and PyTorch, but to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX) and Neural Network Exchange Format (NNEF) exist.

The NNC standard also transmits topology information nnr_topology_unit_payload as part of the NNC bitstream that contains the compressed parameter tensors. This allows interoperability with topology information expressed in native formats as well as exchange formats.

(Configuration of the Image Encoding Device)
Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 5 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a predicted image generating unit 101, a subtraction unit 102, a transformation/quantization unit 103, an inverse quantization/inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

The predicted image generation unit 101 generates a predicted image for each CU.

The subtraction unit 102 subtracts the pixel values of the predicted image of the block input from the predicted image generation unit 101 from the pixel values of image T to generate a prediction error. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

The transform/quantization unit 103 calculates transform coefficients by frequency transforming the prediction error input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction error. The transform/quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization/inverse transform unit 105.

The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (Figure 5) in the image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit 106.

The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The general operation of each module is explained below.

The header encoding unit 1110 performs encoding processing of parameters such as header information, splitting information, prediction information, and quantization transformation coefficients.

The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information, etc.

The CU encoding unit 1112 encodes CU information, prediction information, split information, etc.

When a TU contains a prediction error, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111.

The entropy coding unit 104 receives the quantized transform coefficients and coding parameters from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

The prediction parameter derivation unit 120 derives inter-prediction parameters and intra-prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter-prediction parameters and intra-prediction parameters are output to the parameter encoding unit 111.

The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in the reference picture memory 109.

The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

The prediction parameter memory 108 stores the prediction parameters generated by the encoding parameter determination unit 110 in a predetermined location for each target picture and CU.

The reference picture memory 109 stores the decoded image generated by the loop filter 107 in a predetermined location for each target picture and CU.

The coding parameter determination unit 110 selects one set from among multiple sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the control function. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or devices that store a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the above-mentioned program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

In addition, part or all of the image encoding device 11 and image decoding device 31 in the above-mentioned embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and image decoding device 31 may be individually made into a processor, or part or all of them may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI appears due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the present invention.

This embodiment will be described with reference to FIG. 1. A video coding device 10 is characterized by having an image coding device 11 that codes an input image, and an auxiliary information coding device 81 that codes, as auxiliary information, attention information having multiple values for each region of the input image. The auxiliary information coding device 81 is also characterized by encoding, as auxiliary information, information that specifies a neural network to be applied with respect to the attention information having multiple values for each region.

The video decoding device 30 is characterized by having an image decoding device 31 that decodes an image from encoded data, and an auxiliary information decoding device 91 that decodes attention information having multiple values for each region as auxiliary information for the image decoded by the image decoding device 31. The auxiliary information decoding device 91 is also characterized by decoding information specifying a neural network to be applied with respect to the attention information having multiple values for each region.

The embodiments of the present invention are not limited to the above-mentioned embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

Embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded. They can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device.

1. Video transmission system
30 Video Decoding Device
31 Image Decoding Device
301 Entropy Decoding Unit
302 Parameter Decoding Unit
305, 107 Loop Filter
306, 109 Reference Picture Memory
307, 108 Prediction parameter memory
308, 101 Prediction image generation unit
311, 105 Inverse quantization and inverse transformation section
312, 106 Addition section
320 Prediction Parameter Derivation Unit
10 Video Encoding Device
11 Image Encoding Device
102 Subtraction section
103 Transformation and Quantization Section
104 Entropy coding unit
110 Encoding parameter determination unit
111 Parameter Encoding Unit
120 Prediction parameter derivation part
71 Auxiliary information creation device
81 Auxiliary information coding device
91 Auxiliary Information Decoding Device
1001 Pre-image processing device
1002 Post image processing device

Claims (4)

an image decoding device that decodes an image from encoded data;
A video decoding device comprising an auxiliary information decoding device that decodes attention information having a plurality of values for each region as auxiliary information for an image decoded by the image decoding device. The auxiliary information decoding device comprises:
Regarding attention information with multiple values for each region,
2. The video decoding device according to claim 1, further comprising: decoding information for specifying a neural network to be applied. an image encoding device for encoding an input image;
A video encoding device comprising an auxiliary information encoding device for encoding attention information having a plurality of values for each region of the input image as auxiliary information. The auxiliary information encoding device comprises:
Regarding attention information with multiple values for each region,
4. The video encoding device according to claim 3, wherein information for specifying a neural network to be applied is encoded as auxiliary information.

Images