KR20240072961A - Method and apparatus for processing feature map of images for machine vision - Google Patents

Abstract

A method and device for processing image feature maps for machine vision are disclosed. A method of processing a feature map for machine vision according to an embodiment includes arranging multi-channel multi-scale feature maps extracted from the image into single-scale feature maps, and selecting one or more single-scale feature maps from among the single-scale feature maps. It includes generating a converted feature map by performing feature channel conversion and encoding the converted feature map.

Description

{Method and apparatus for processing feature map of images for machine vision}

The present invention relates to image processing technology, which extracts a feature map of an input image and encodes and/or decodes the extracted feature map so that the feature map of the input image can be efficiently processed or transmitted to perform machine vision tasks. It is about image processing technology.

Deep learning or machine learning is advancing and various artificial intelligence platforms and applications are being developed. Accordingly, as the data processed by machines or people through deep learning or machine learning becomes vast, the demand for encoding and decoding technology for images and/or their characteristics for efficient information transmission and real-time signal processing increases. there is.

Conventional image (particularly video) compression technology using deep learning or machine learning extracts features/feature map/feature vector/latent vector from the input image, The extracted feature map may be transmitted as is, or the feature map may be encoded and transmitted. At this time, the size of the extracted feature map may be defined depending on the designed deep learning or machine learning network. That is, the feature map may be defined by one or more parameters among various scales, multiple channels, and various data types.

However, the feature map defined in this way may have a small size compared to the input image, but the feature map with a small size causes loss of information compared to the original image. Therefore, in order to reduce the loss of information, a feature map of as large a size as possible is required. In this case, because the amount of data increases, transmission or real-time data processing is difficult. Therefore, a method has been proposed that not only reduces information loss but also reduces the size of data to be transmitted or processed by performing entropy encoding and decoding on a larger-sized feature map.

For example, in Korea Patent Publication No. 2018-0087264, “Image processing method and device using feature map compression” (Patent Document 1), a feature map generated by performing a convolution operation on an input image is stored in at least one line unit. A technology has been disclosed to reduce the amount of computation by reducing the number of filter parameters by processing or compressing the filter. And in Korean Patent Publication No. 2020-0026026, “Electronic device and control method for high-speed compression processing of feature maps of CNN utilization system” (Patent Document 2), feature maps for input images are acquired, and the acquired features are A technology has been disclosed that converts a feature map through a lookup table corresponding to the map and then compresses the converted feature map through a corresponding compression mode.

Korean Patent Publication No. 2018-0087264 Korean Patent Publication No. 2020-0026026

As a representative machine vision application, according to the network structure used for object detection/segmentation, multi-scale features are extracted from a single image/video to distinguish objects of various scales and used for machine vision. The set of multi-scale feature maps is sorted, channels are reduced, converted to single-scale features of an efficient size for compression, and minimum and maximum normalization is performed for compression using an existing video compression codec.

In the case of compression and restoration of a feature map using an existing video compression codec, the minimum and maximum values generated by applying minimum and maximum normalization when compressing the feature map are transmitted and used during restoration. In this case, a separate single stream design is required to add and transmit the minimum and maximum values of the feature map.

One embodiment provides an apparatus and method for compressing and/or restoring feature maps having different resolutions and/or number of channels when encoding and/or decoding feature maps for machine vision.

One embodiment provides an apparatus and method for improving compression performance by encoding and/or decoding a smaller amount of data for a similar task performance result using a conventional method of encoding and/or decoding a machine vision image.

One embodiment provides an apparatus and method that does not require transmission of the minimum and maximum values of the feature map resulting from minimum and maximum normalization required when restoring a multi-scale feature map from a single-scale feature map decoded using a video compression codec. .

One embodiment provides an apparatus and method for extracting a feature map considering data loss of an image/feature map that may occur due to neural network and compression, and encoding/decoding the extracted feature map.

A method of processing a feature map of an image for machine vision according to an embodiment of the present invention to solve the above problem includes arranging multi-channel, multi-scale feature maps extracted from the image into single-scale feature maps; It includes generating a converted feature map by performing feature channel conversion on one or more single-scale feature maps among the single-scale feature maps, and performing encoding on the converted feature map.

A feature map processing method according to another embodiment of the present invention to solve the above problem includes the steps of performing size restoration on the converted feature maps and performing channel number restoration on the converted feature maps. and generating feature maps, wherein the reconstructed feature maps include feature maps having different resolutions or numbers of channels.

According to one embodiment, when encoding and/or decoding a feature map for machine vision, an apparatus, method, and recording medium may be provided for compressing and/or restoring feature maps having different resolutions and/or number of channels. there is.

According to one embodiment, an apparatus, method, and recording medium are provided for improving compression performance by encoding and/or decoding a smaller amount of data while achieving similar mission performance results by using an encoding and/or decoding method for a machine vision image. can be provided.

According to one embodiment, when restoring a multi-scale feature map from a single-scale feature map decoded using an existing video compression codec, a device that does not need to transmit the minimum and maximum values of the feature map resulting from the required minimum and maximum normalization , a method, and a recording medium may be provided.

According to one embodiment, even if data loss occurs, the impact on network performance can be minimized through learning that takes into account the data loss of the feature map extracted from the network.

Figure 1 is a block diagram showing the configuration of an encoding device according to an embodiment.
Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment.
Figure 3 is a flowchart showing a method of processing a feature map according to an embodiment of the present invention.
Figure 4 schematically shows an example of aligning multi-scale feature maps into a single-scale feature map of the same size.
Figure 5 is an example of channel reduction in step S310 of Figure 3.
Figure 6 is an example of residual restoration of a multiscale feature map.
Figure 7 is another example of residual restoration of a multiscale feature map.
Figure 8 is an example of restoration for a multiscale feature map.
Figure 9 shows an example of denormalizing a feature map.
Figure 10 is a flowchart showing a feature map processing method according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The terms and words used in this specification are terms selected in consideration of their functions in the embodiments, and the meaning of the terms may vary depending on the intention or custom of the invention. Accordingly, terms used in the embodiments described below, if specifically defined in the present specification, shall follow the definition, and if there is no specific definition, they shall be interpreted as meanings generally recognized by those skilled in the art.

Additionally, a module in this specification may mean hardware that can perform functions and operations according to each name described in this specification, or it may mean computer program code that can perform specific functions and operations. Alternatively, it may mean an electronic recording medium loaded with computer program code that can perform specific functions and operations, for example, a processor or microprocessor. In other words, a module may mean a functional and/or structural combination of hardware for carrying out the technical idea of the present invention and/or software for driving the hardware.

Figure 1 is a block diagram showing the configuration of an encoding device according to an embodiment. Referring to FIG. 1, the encoding device 100 includes a processing unit 110, a memory 130, a user interface (UI) input device 150, and a UI output device 160 that communicate with each other through a bus 190. ) and storage (storage) 140. Additionally, the encoding device 100 may further include a communication unit 120 connected to the network 199.

The processing unit 110 may be a semiconductor device that executes processing instructions stored in a central processing unit (CPU), memory 130, or storage 140. The processing unit 110 may be at least one hardware processor.

The processing unit 110 may generate and process signals, data, or information that are input to the encoding device 100, output from the encoding device 100, or used inside the encoding device 100. Inspection, comparison, and judgment related to data or information can be performed. That is, in an embodiment, generation and processing of data or information, and inspection, comparison, and judgment related to the data or information may be performed by the processing unit 110.

The processing unit 110 may include an inter prediction unit, an intra prediction unit, a switch, a subtractor, a transform unit, a quantization unit, an entropy encoding unit, an inverse quantization unit, an inverse transform unit, an adder, a filter unit, and a reference picture buffer. At least some of the inter prediction unit, intra prediction unit, switch, subtractor, transform unit, quantization unit, entropy encoding unit, inverse quantization unit, inverse transform unit, adder, filter unit, and reference picture buffer may be program modules, and may be external devices. Or you can communicate with the system. Program modules may be included in the encoding device 100 in the form of an operating system, application program module, and other program modules.

Program modules may be physically stored on various known storage devices. Additionally, at least some of these program modules may be stored in a remote memory device capable of communicating with the encoding device 100.

Program modules are routines, subroutines, programs, objects, components, and data that perform a function or operation according to an embodiment or implement an abstract data type according to an embodiment. It may include data structures, etc., but is not limited thereto.

Program modules may be composed of instructions or codes that are executed by at least one processor of the encoding device 100.

The processing unit 110 may execute instructions or codes of an inter prediction unit, an intra prediction unit, a switch, a subtractor, a transform unit, a quantizer, an entropy encoder, an inverse quantizer, an inverse transform unit, an adder, a filter unit, and a reference picture buffer. .

The storage unit may represent memory 130 and/or storage 140. Memory 130 and storage 140 may be various types of volatile or non-volatile storage media. For example, the memory 130 may include at least one of a read-only memory (ROM) 131 and a random access memory (RAM) 132.

The storage unit may store data or information used for the operation of the encoding device 100. In an embodiment, data or information held by the encoding device 100 may be stored in a storage unit. For example, the storage unit can store pictures, blocks, lists, motion information, inter prediction information, and bitstreams.

The encoding device 100 may be implemented in a computer system that includes a recording medium that can be read by a computer. The recording medium may store at least one module required for the encoding device 100 to operate. The memory 130 may store at least one module, and the at least one module may be configured to be executed by the processing unit 110.

Functions related to communication of data or information of the encoding device 100 may be performed through the communication unit 320. For example, the communication unit 120 may transmit a bitstream to the decoding device 200, which will be described later.

Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment.

The decryption device 200 includes a processing unit 210, a memory 230, a user interface (UI) input device 250, a UI output device 260, and storage that communicate with each other through the bus 290. It may include (240). Additionally, the decryption device 200 may further include a communication unit 220 connected to the network 299.

The processing unit 210 may be a semiconductor device that executes processing instructions stored in a central processing unit (CPU), memory 230, or storage 240. The processing unit 210 may be at least one hardware processor.

The processing unit 210 may generate and process signals, data, or information that are input to the decoding device 200, output from the decoding device 200, or used inside the decoding device 200. Inspection, comparison, and judgment related to data or information can be performed. That is, in an embodiment, generation and processing of data or information, and inspection, comparison, and judgment related to the data or information may be performed by the processing unit 210.

The processing unit 210 may include an entropy decoder, an inverse quantization unit, an inverse transform unit, an intra prediction unit, an inter prediction unit, a switch, an adder, a filter unit, and a reference picture buffer. At least some of the entropy decoder, inverse quantization unit, inverse transform unit, intra prediction unit, inter prediction unit, switch, adder, filter unit, and reference picture buffer may be program modules and may communicate with an external device or system. Program modules may be included in the decryption device 200 in the form of an operating system, application program module, and other program modules.

Program modules may be physically stored on various known storage devices. Additionally, at least some of these program modules may be stored in a remote memory device capable of communicating with the decoding device 200. Program modules are routines, subroutines, programs, objects, components, and data that perform a function or operation according to an embodiment or implement an abstract data type according to an embodiment. It may include data structures, etc., but is not limited thereto. Program modules may be composed of instructions or codes that are executed by at least one processor of the decoding device 200.

The processing unit 210 may execute instructions or codes of an entropy decoder, inverse quantization unit, inverse transform unit, intra prediction unit, inter prediction unit, switch, adder, filter unit, and reference picture buffer.

The storage unit may represent memory 230 and/or storage 240. Memory 230 and storage 240 may be various types of volatile or non-volatile storage media. For example, the memory 230 may include at least one of a read-only memory (ROM) 231 and a random access memory (RAM) 232.

The storage unit may store data or information used for the operation of the decoding device 200. In an embodiment, data or information held by the decoding device 200 may be stored in the storage unit. For example, the storage unit can store pictures, blocks, lists, motion information, inter prediction information, and bitstreams.

The decryption device 200 may be implemented in a computer system that includes a recording medium that can be read by a computer. The recording medium may store at least one module required for the decoding device 200 to operate. The memory 230 may store at least one module, and the at least one module may be configured to be executed by the processing unit 210.

Functions related to communication of data or information of the decryption device 200 may be performed through the communication unit 220. For example, the communication unit 220 may receive a bitstream from the encoding device 100.

Hereinafter, the processing unit may refer to the processing unit 110 of the encoding device 100 and/or the processing unit 210 of the decoding device 200. For example, in functions related to prediction, the processor may represent a switch. For functions related to inter prediction, the processing unit may represent an inter prediction unit, a subtractor, and/or an adder. For functions related to intra prediction, the processing unit may represent an intra prediction unit, a subtractor, and/or an adder. In functions related to conversion, the processing unit may represent a conversion unit and/or an inverse conversion unit. In functions related to quantization, the processing unit may represent a quantization unit and/or an inverse quantization unit. In functions related to entropy encoding and/or decoding, the processing unit may represent an entropy encoding unit and/or an entropy decoding unit. In functions related to filtering, the processing unit may represent a filter unit. For functions related to reference pictures, the processing unit may indicate a reference picture buffer.

Embodiments of the present invention described later relate to feature map image encoding/decoding for efficiently transmitting features of image information for machines or people. As deep learning/machine learning develops and various artificial intelligence platforms and applications are being developed, and as the data processed by machines or people increases due to deep learning/machine learning, video or video for efficient information transmission and real-time signal processing A method for encoding/decoding features is required.

According to the network structure used for object detection/segmentation as a representative machine vision, multi-scale features are extracted from one image/video to distinguish objects of various scales and are used in machine vision. Because the multi-scale feature has various resolutions and multi-channel features, the size of the data is enormous, making it difficult to reduce the size of the data.

The present invention sorts the feature set subject to compression, readjusts the sorted feature set, and reduces its size, efficiently converting the size of existing massive features. Additionally, when restoring compressed features, effective restoration is possible by adding a separate processing step to each restoration process.

In the case of existing feature compression and restoration, not only are complex network configurations and difficult learning processes required to reduce features of a large size, but also restoration is performed on various features in a single stream during restoration, which reduces restoration efficiency and affects performance. It's crazy.

The present invention improves restoration ability by compressing features through simple feature alignment, readjustment, and compression processes, and configuring separate processing processes corresponding to the features to restore various features.

In feature alignment and channel conversion, not only can channel conversion be optimized by considering the compression rate, but also content-adaptive feature compression can be performed by adaptively determining the degree of channel conversion during the channel readjustment process. And through the present invention, not only can features be effectively compressed, but the efficiency of machine vision performance can be increased compared to compression performance by improving the effect of feature restoration.

And according to one embodiment of the present invention, the process of transmitting the minimum and maximum values of the features is excluded from the process of compressing the features using the video compression codec. By this, by excluding transmission of the feature minimum and maximum values, which is an essential process of feature compression using a video compression codec, the design of a single stream for transmission of the minimum and maximum values is omitted and encoding efficiency equivalent to the size of the minimum and maximum values is obtained. You can.

Figure 3 is a flowchart showing a method of processing a feature map according to an embodiment of the present invention. In Figure 3, step S342 is a step of denormalizing the feature map, thereby eliminating the need to encode the minimum and maximum values of the feature map and include them in the bitstream or transmit them to the decoding device. According to one embodiment, step S342 can be omitted as an optional step.

Referring to FIG. 3, the feature map processing method includes a feature alignment step (S310). In the feature alignment step (S310), at least one feature map among the multi-scale feature maps extracted from the input image may be aligned into a feature map of the same scale. Figure 4 schematically shows an example of aligning multi-scale feature maps into a single-scale feature map of the same size.

The feature map may be expressed in at least one of (H, W), (C, H, W), and (B, C, H, W). Here, C may be the number of channels of the feature map, H may be the height of the feature map, and W may be the width of the feature map, each of which may be an integer greater than or equal to 0. And B is the number of feature maps with a size of (C, H, W) and can be an integer greater than 0.

The set of feature maps may be a set of one or more feature maps having the same (C, H, W). For example, p1 = (C, H, W), p2 = (C, H, W), p3 = (C, H, W), … It may be a set of one or more feature maps with the same number of channels, height, and width.

Here, the set of multiscale feature maps may be a set of one or more feature maps with different (C, H, W). For example, p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), … As shown, the number of channels is the same as C, and it can be a set of one or more feature maps whose height and width increase by two. Or, p1 = (C, H, W), p2 = (C, H', W'), p3 = (C, H'', W''), … As shown, the number of channels is the same as C, and it can be a set of one or more feature maps with different heights and widths. Or, for example, p1 = (C, H, W), p2 = (C', 2H, 2W), p3 = (C'', 4H, 4W), … It may be a set of one or more feature maps where the number of different channels is the same and the height and width are increased by two.

Among the set of multi-scale feature maps, at least one may be aligned with at least one resolution. For example, the smallest feature map may be scaled and sorted into feature maps of the same size.

For example, if p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), p4 = (C, 8H, 8W), the smallest feature map Based on p1, the features of p2 to p4 can be downscaled to the same size as p1. The downscaled feature maps can be combined and aligned to size (4C, H, W).

For example, the largest feature map may be scaled and sorted into feature maps of the same size. For example, if p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), p4 = (C, 8H, 8W), the largest feature map Based on p4, the features of p1 to p3 can be upscaled to the same size as p4. Each upscaled feature map can be combined and aligned to a size of (4C, 8H, 8W).

As another example, a feature map of the same size may be scaled and sorted based on a feature of a medium size. For example, if p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), p4 = (C, 8H, 8W), based on p2 p1 can be upscaled, and p3~4 can be downscaled. Each scaled feature map can be combined and sorted to size (4C, 2H, 2W).

The above example is an example of scaling to one size, and when scaling to two or more resolutions, the resolution can be aligned after scaling in the same manner as the example.

Continuing to refer to FIG. 3, the feature map processing method includes a feature channel conversion process for single-scale feature maps (S320). More specifically, in this step, multi-channel single-scale feature maps can be converted to at least one of channel increase/decrease/maintenance into feature maps for at least one channel.

A multi-channel feature map refers to a feature map with at least one channel. For example, the size of a multi-channel feature map can be expressed as (C, H, W), where C can be any positive integer. The multi-channel feature map may be a combination of at least one of the feature map set and the multi-scale feature map set in a single resolution.

In converting the channels of the feature map, readjustment between channels can be performed. The inter-channel rebalancing network may be composed of at least one layer among pooling, fully-connected, convolution, and activation function layers.

Re-adjustment between channels can be done by at least one of the methods below. For example, when re-adjusting between channels, at least one representative value can be extracted from a single-resolution feature map.

For example, the representative value of each channel may be at least one of {average value, maximum value, minimum value, center value, mode value} of the pixels of each channel. For example, representative values of each channel information may be extracted through at least one convolution layer. Alternatively, if the size of the single resolution feature map is (C, H, W), it can be expressed as (C, 1, 1) by the representative value of each channel. Alternatively, when the size of a single resolution feature is (C, H, W), two representative values can be extracted from each channel and expressed as (2C, 1, 1).

For example, when readjusting between channels, the representative value of each channel may be adjusted through at least one fully connected layer. For example, the representative value of the feature map of (C, 1, 1) expressed by the representative value may be adjusted through at least one fully connected layer to become a feature map of (C', 1, 1). Alternatively, the representative value of the feature map of (2C, 1, 1) expressed by the two representative values can be adjusted through at least one fully connected layer to become a feature map of (C', 1, 1). . At this time, C' may be the same as or different from C.

For example, in inter-channel readjustment, a single resolution feature map can be readjusted using the adjusted representative value. The feature map can be readjusted through at least one operation among (+, -, *, /) on the adjusted representative value. For example, if the size of the adjusted representative value of a single-resolution feature map of size (C, H, W) is (C, 1, 1), the single-resolution feature map can be readjusted by multiplying each channel. Alternatively, if the size of the adjusted representative value of the single-resolution feature map of size (C, H, W) is (C, 1, 1), the single-resolution feature map can be readjusted by the sum of each channel. Alternatively, if the size of the adjusted representative value is (C, 1, 1), the single-resolution feature map of size (C, H, W) can be readjusted by the difference for each channel.

According to one embodiment, when converting the channel of the feature map, at least one of the channels of the feature map can be converted to increase/decrease/maintain one or more channels. In this case, the channel conversion network of the feature map may be composed of at least one layer among pooling, fully-connected, convolution, and activation function layers.

Channel conversion of the feature map may be performed on at least one of the feature map, a set of feature maps, a set of multi-scale feature maps, and a readjusted feature map. For example, a rescaled feature map of size (C, H, W) can become a channel transformed feature map of size (C', H, W) by one or more convolution layers. At this time, C' may be any positive integer. And C' can be equal to, greater than, or less than C.

This feature map channel conversion can be performed on the feature map formed by at least one of the feature map alignment step (S310) and inter-channel readjustment. For example, a set of feature maps might look like p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), p4 = (C, 8H, 8W). In this case, it can be scaled and aligned into a feature map F with a size of (4C, H, W), and can become a feature map F' through inter-channel readjustment. The readjusted feature map F' can become a feature map F'' with a size of (C', H, W) through channel conversion. Figure 5 illustrates channel reduction in step S310.

According to one embodiment, in feature map channel conversion, the degree of channel conversion can be adjusted by considering the compression rate. For example, for high quality compression (high bitrate), the bitrate generated can be minimized by increasing the degree of channel reduction. Alternatively, for low quality compression (low bitrate), the amount of information lost can be reduced by lowering the degree of channel reduction. For example, the relationship between the degree of channel reduction and the bitrate generated during compression can be optimized.

According to the above example, optimization can be achieved by configuring a function between parameters that determine the degree of channel conversion and the degree of compression. For example, by configuring an optimized function between the degree of channel reduction and the quantization parameter (QP) of the compression codec, the QP can be adjusted according to channel reduction. At this time, parameters related to the function may be transmitted additionally. As another example, the QP can be adjusted according to the increase in channels by configuring an optimized function between the degree of channel increase and the QP of the compression codec. At this time, parameters related to the function may be transmitted additionally. As another example, the QP can be adjusted according to channel maintenance by configuring an optimized function between channel maintenance and QP of the compression codec. At this time, parameters related to the function may be transmitted additionally.

According to one embodiment, when converting feature map channels, important channels can be determined during inter-channel readjustment and the number of channels can be adaptively converted. For example, in the process of rebalancing between channels, the importance of each channel can be determined through an activation function and channels with low importance can be excluded to perform adaptive channel reduction. At this time, the number of channels for restoration can be transmitted as additional information according to the adaptively changing number of channels. And, the activation function may use at least one of sigmoid, relu, prelu, and leaky-relu.

Continuing to refer to FIG. 3, the feature map processing method includes an encoding process for the converted feature map (S330). More specifically, in this step, encoding (compression) may be performed on the feature map on which at least one of the above-described feature map alignment step (S310) and feature map channel conversion step (S320) has been performed.

Compression of the feature map in this step includes encoding and decoding processes. When encoding/decoding a feature map, at least one of an existing video compression codec or a neural network-based video compression codec may be used to encode/decode the feature map.

When using at least one of an existing video compression codec or a neural network-based video compression codec, the feature map can be converted into a form suitable for video compression.

For example, a multi-channel feature map of size (C, H, W) can be converted into a black and white feature map of one frame of size nH'×mW'. Here, n×m may be equal to or different from c. And H' and W' may be the same as or different from H and W.

As another example, multi-channel features of size (C, H, W) can be converted into a color feature map of one frame of size 3 × nH' × mW'. Here, n×m×3 may be the same as or different from C. And H' and W' may be the same as or different from H and W.

As another example, a multi-channel feature map of size (C, H, W) can be converted into a video of C' frame with resolution H' × W'. Here, the video may be black and white or color video. And, C', H', and W' may be the same as or different from C, H, and W.

According to one embodiment, channel rearrangement may be performed to increase compression efficiency by increasing correlation between channels. When performing channel rearrangement, the correlation between channels can be identified and rearranged. At this time, the correlation between channels can be determined using at least one of the MSE between channels, the channel average value, and the channel center value. And, order information of rearranged channels can be additionally transmitted. Additionally, channel rearrangement may perform at least one of spatial rearrangement and temporal rearrangement.

According to one embodiment, when converting a feature map into a form suitable for image compression, information necessary for conversion may be additionally transmitted. For example, in order to convert a feature map into image form, information necessary for the normalization process may be additionally transmitted. Alternatively, when performing maximum and minimum normalization, the maximum and minimum values may be transmitted additionally. Alternatively, when performing mean and standard deviation normalization, the mean and standard deviation values may be additionally transmitted.

In the above-described step S330, when converting the multi-channel feature map into a feature map of one frame when using at least one of an existing video compression codec and a neural network-based video compression codec, the compression ratio due to the boundary between channels To reduce efficiency, tile-based encoding/decoding may be performed. For example, when a multi-channel feature map is converted into one frame of size nH' × mW', the size of the tile can be specified as H' × W'.

Additionally, when performing tile-based encoding/decoding, the degree of quantization can be performed differently depending on the region of interest. For example, for an area where an important feature map is placed, a low quantization parameter can be set, and for an area where an unimportant feature map is placed, a high quantization parameter can be set. At this time, information about areas with different degrees of quantization may be additionally transmitted.

According to one embodiment, when encoding/decoding a feature map, encoding/decoding may be performed in units of at least one of sample, line, block, and frame. At this time, encoding/decoding may be performed in units of at least one of sample, line, block, and frame, depending on at least one of the shape, size, and dimension of the input feature map.

At this time, information about the unit can be encoded/decoded according to the encoded unit. For example, when performing block-level encoding/decoding, at least one piece of information among the size, division, and shape of the block can be encoded/decoded. For example, when performing line-by-line encoding/decoding, at least one piece of information among the length, number, and shape of the line can be encoded/decoded. For example, when encoding/decoding on a frame basis, at least one piece of information among the shape, size, number, and division information of the frame can be encoded/decoded.

According to one embodiment, when encoding/decoding a feature map, prediction-based encoding/decoding may be performed. At this time, when performing prediction, prediction may be performed on at least one unit among sample, line, block, and frame. At this time, the prediction method may vary depending on the data characteristics of the feature map. The prediction method may use at least one of spatial prediction, temporal prediction, and channel prediction. In the case of spatial prediction, prediction of the coding unit can be performed according to the coding unit from already encoded/decoded surrounding samples or predicted values. At this time, the neighboring sample may be at least one of a sample adjacent to the current coding unit and a sample that is not adjacent to the current coding unit. At this time, the surrounding sample may be at least one of a sample or a set of samples.

When performing prediction according to coding units from already encoded/decoded neighboring samples, prediction may be performed using at least one of directional prediction, template prediction, dictionary prediction, and matrix product prediction. For example, in the case of directional prediction, prediction of the current unit can be performed by copying or interpolating the neighboring samples according to the direction using neighboring samples that have already been encoded/decoded. For example, in the case of template prediction, prediction can be performed by searching for a template most similar to the current prediction unit from surrounding samples that have already been encoded/decoded. For example, in the case of dictionary prediction, already encoded/decoded neighboring samples or unit prediction blocks are stored in a separate memory, and the prediction of the current block can be predicted from the stored dictionary. For example, in the case of matrix product prediction, the current block can be predicted by multiplying already encoded/decoded surrounding samples with an arbitrary matrix. For example, in the case of temporal prediction, prediction of the coding unit can be performed according to the coding unit from the temporally preceding/following frame. For example, in the case of channel prediction, when the feature map has a multi-channel form, prediction of the coding unit can be performed according to the coding unit from a channel other than the current channel.

At this time, the coding unit may vary depending on the configuration of the feature map. If the feature map is composed of multiple channels and arranged by correlation between channels, a coding unit using correlation can be specified. For example, if channels are arranged by a specific average value, a coding unit can be formed as a bundle of similar average values. For example, if the channels are arranged based on similarity between channels, a coding unit can be formed as a bundle of similarities.

When performing prediction-based encoding/decoding, the difference between the original signal and the prediction signal can be encoded/decoded. At this time, at least one of the above prediction methods may be used as the prediction signal. At this time, the residual signal may be the difference between the predicted signal and the original signal by at least one of the above prediction methods.

According to one embodiment, at least one piece of information about features can be entropy encoded/decoded. For example, the information about the feature may include at least one of the following information: maximum value, minimum value, average value, mode, and standard deviation of the feature map. Maximum, minimum, average, mode, and standard deviation for each channel of the feature map. Index representing the channel of the feature map. Index to the region of interest in the feature map. Index to the channel of interest in the feature map. Feature pixels and residual pixels. Parameters related to the channel conversion degree and compression degree function.

When entropy encoding/decoding at least one of the information about the above features, at least one of the following binarization methods can be used: Truncated Rice binarization method. K-th order exp-golomb binarization method. Fixed length binarization method. Unary binarization method. A truncated unary binarization method. Truncated binary binarization method.

In performing entropy encoding/decoding on the information about the feature or the binary information generated through the binarization, at least one of the following methods can be used: Context-adaptive binary arithmetic (CABAC) coding). Context-adaptive variable length coding (CAVLC). Bypass encoding.

When performing the entropy encoding/decoding, the entropy encoding/decoding may be adaptively performed using at least one encoding information of the channel type, size/shape of the current feature map, and encoding information of weekly features.

Continuing to refer to FIG. 3, the feature map processing method includes a decoding process for the encoded feature map (S340). More specifically, when the original feature map is converted in at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map encoding step (S333), the original feature map is converted to the original feature map. Restoration can be performed.

According to one embodiment, the feature map to be restored may be expressed in at least one of (H, W), (C, H, W), and (B, C, H, W). C may be the number of channels of the feature, H may be the height of the feature, W may be the width of the feature, and may be an integer greater than or equal to 0. B is the number of features with a size of (C, H, W) and can be an integer greater than 0.

The set of feature maps to be restored may be a set of one or more feature maps having the same (C, H, W). For example, p1 = (C, H, W), p2 = (C, H, W), p3 = (C, H, W), … It may be a set of one or more feature maps with the same number of channels, height, and width.

The set of multiscale feature maps to be restored may be a set of one or more feature maps with different (C, H, W). For example, p1 = (C, H, W), p2 = (C, 2H, 2W), p3 = (C, 4H, 4W), … As shown, the number of channels is the same as C, and it can be a set of one or more feature maps whose height and width increase by two. For example, p1 = (C, H, W), p2 = (C, H', W'), p3 = (C, H'', W''), … As shown, the number of channels is the same as C, and it can be a set of one or more feature maps with different heights and widths. For example, p1 = (C, H, W), p2 = (C', 2H, 2W), p3 = (C'', 4H, 4W), … It may be a set of one or more feature maps where the number of different channels is the same and the height and width are increased by two.

According to one embodiment, when the original feature is converted in at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map compression step (S330), upscaling and downscaling are performed. , the original feature map can be restored through at least one of the convolution layer, fully connected layer, and activation function.

When restoring the original feature map, if the spatial size needs to be restored, it can be restored through at least one of upscaling, downscaling, and convolution layers. For example, a feature map of size (C, H', W') needs to be restored to an original feature map of size (C, H, W), and if H' < H, W' < W, upscaling is performed. It can be restored through Here, upscaling may be at least one of linear interpolation, latest value interpolation, and bi-cubic interpolation.

For example, a feature map of size (C, H', W') needs to be restored to the original feature map of size (C, H, W), and if H' > H, W' > W, upscaling is performed. It can be restored through Here, upscaling may be at least one of linear interpolation, latest value interpolation, and bi-cubic interpolation.

For example, a feature map of size (C, H', W') needs to be restored to an original feature map of size (C, H, W), and if H' < H, W' < W, the convolution layer It can be restored through . Here, the convolution layer may be at least one of a transposed convolution layer and a convolution layer.

For example, a feature map of size (C, H', W') needs to be restored to an original feature map of size (C, H, W), and if H' > H, W' > W, the convolution layer It can be restored through . Here, the convolution layer may be at least one of a transposed convolution layer and a convolution layer.

According to one embodiment, when restoring the original feature map, if the size of the channel needs to be restored, it can be restored through at least one process of a convolution layer, a fully connected layer, and an activation function. For example, a feature map of size (C', H, W) must be restored to an original feature map of size (C, H, W), and can be restored through a convolution layer. Here, the convolution layer may be at least one of a transposed convolution layer and a convolution layer. Here, the result of passing the convolution layer can go through an activation function process.

According to one embodiment, when restoring the feature map, if the feature map to be restored is a multi-scale feature map, it may be restored in a residual reconstruction format.

Figures 6 and 7 are examples of residual restoration of a multiscale feature map. Residual restoration may be in at least one of the following forms: top-down as shown in FIG. 6 or bottom-up as shown in FIG. 7. This example is an example of restoring four original feature maps from input feature maps of size [C', H', W'], and the number of feature maps to be restored may be N. Here, N may be a positive integer.

And in this example, "processM" may be a different processing process, depending on at least one of the feature maps to be restored, features, upscaling, downscaling, convolutional layer, transposed convolutional layer, and activation function. It may consist of at least one or more. For example, in Figure 6, if the sizes of p1 to p4 have the same number of channels, but the height and width are doubled, {Process 2, Process 3, Process 4} is used for upscaling, downscaling, convolution layer, and transposition. By consisting of at least one of the convolution layer and activation function, a processing process suitable for the growing height and width can be configured. For example, in Figure 6, if the number of channels of the input feature map is C' and the number of channels of the feature maps p1 to p4 to be restored are all C, {Process 2, Process 3, Process 4} changes the number of channels to C' → The same process of converting to C may be included.

According to one embodiment, when the size of the feature map is converted by at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map compression step (S330), the restoration process Additional information may be requested. At this time, additional information can be transmitted and used in the restoration step.

For example, if the size of the original feature map is not restored during a fixed restoration process, the size information of the original feature map can be transmitted and a restoration process corresponding to the size of the original feature map can be transmitted.

According to one embodiment, when restoring the feature map, if the feature to be restored is a multi-scale feature map, it can be restored through a simple processing process. Figure 8 is an example of restoration for a multiscale feature map. This example is an example of restoring four original feature maps from input feature maps of size [C', H', W'], and the number of feature maps to be restored may be N. Here, N may be a positive integer.

In this example, “processM” may be a different processing process, and depending on at least one of the features to be restored, at least one of upscaling, downscaling, convolution layer, transposed convolution layer, and activation function. It may consist of the above.

For example, in Figure 6, if the sizes of p1 to p4 have the same number of channels, but the height and width are doubled, {Process 2, Process 3, Process 4} is used for upscaling, downscaling, convolution layer, and transposition. By consisting of at least one of the convolution layer and activation function, a processing process suitable for the growing height and width can be configured. For example, in Figure 6, if the number of channels of the input feature is C' and the number of channels of the features p1 to p4 to be restored are all C, {Process 2, Process 3, Process 4} changes the number of channels from C' → C. The same conversion process may be included.

According to one embodiment, when the size of the feature map is converted by at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map compression step (S330), the restoration process Additional information may be requested. At this time, additional information can be transmitted and used in the restoration step. For example, if the size of the original feature map is not restored during a fixed restoration process, the size information of the original feature map can be transmitted and a restoration process corresponding to the size of the original feature map can be transmitted.

Continuing to refer to FIG. 3, the decoding step (S340) for the encoded feature map may include a denormalization process (S342) for the decoded feature map. More specifically, denormalization may be performed on a feature map on which at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map compression step (S330) have been performed. This denormalization process (S342) is performed only when normalization of the feature map is performed in at least one of the feature map alignment step (S310), feature channel conversion step (S320), and feature map compression step (S330). It's okay too.

The feature map on which denormalization has been performed may be expressed in at least one of (H, W), (C, H, W), and (B, C, H, W). C may be the number of channels in the feature map, H may be the height of the feature map, and W may be the width of the feature map, and may be an integer greater than or equal to 0. B is the number of feature maps with a size of (C, H, W) and can be an integer greater than or equal to 0.

The feature map generated during the denormalization process can be expressed in at least one of (2,1,1) and (B,2,1,1). B is the number of feature maps with a size of (2,1,1) and can be an integer greater than or equal to 0.

According to one embodiment, when the original feature map is converted in at least one of the feature map alignment step (S310), the feature channel conversion step (S320), and the feature map compression step (S330), a fully connected layer; Denormalization of the feature map can be performed by generating the minimum and maximum values of the feature map through the process of at least one of the convolution layers.

In generating the minimum and maximum values required for denormalization of the feature map, the process of inter-channel readjustment and channel reduction of the feature map can be performed. Re-adjustment between channels can be done by at least one of the methods below.

When readjusting between channels, at least one representative value can be extracted from a single resolution feature map. For example, the representative value of each channel may be at least one of {average value, maximum value, minimum value, center value} of the pixels of each channel. For example, representative values of each channel may be extracted through at least one convolution layer. For example, if the size of the single resolution feature map is (C, H, W), it can be expressed as (C, 1, 1) by the representative value of each channel.

In the process of channel reduction, the minimum and maximum values of the representative value of each channel can be generated through at least one fully connected layer. For example, the representative value of the feature map of (C,1,1) expressed by the representative value can be adjusted through at least one fully connected layer to generate the minimum and maximum value of (2,1,1). there is.

The minimum and maximum values generated in the channel reduction process can be generated by configuring a separate neural network loss function. For example, when compressing the feature map and configuring a loss function using the minimum and maximum values extracted from the feature map, the parameters that determine the generation of the minimum and maximum values for denormalization can be optimized. Here, the loss function can be at least one of {MSE, MAE}.

Figure 9 is a diagram showing an example of denormalizing the feature map.

Process 1 in Figure 9 refers to channel readjustment of denormalization, global max pooling, global average pooling, global min pooling, and global median pooling. It may be in at least one of the following formats. For example, in Figure 9, when a feature map of size [C, H, W] is input, the feature map is [ C,1,1] can be resized.

Process 2 in FIG. 9 refers to channel reduction and may be in the form of at least one of a convolutional neural network and a fully connected neural network. For example, in the case of a feature map of size [C,1,1] by the channel readjustment in Figure 9, features of size [2,1,1] are generated through one or more layers of a convolutional neural network or a fully connected neural network. Channel axes can be performed using a map. At this time, the feature values of the reduced feature map mean the minimum and maximum values, respectively.

Process 3 in Figure 9 means performing denormalization on the feature map using the generated minimum and maximum values. For example, denormalization can be performed using the minimum and maximum values generated through the channel readjustment and channel reduction process of the feature map of the size [C, H, W] initially input in Figure 9.

Figure 10 is a flowchart showing a feature map processing method according to another embodiment of the present invention. The processing method shown in FIG. 10 is different from the processing method shown in FIG. 3 in that it further includes a step (S405) of extracting a feature map of the image. That is, in the processing method shown in FIG. 10, steps S410, S420, S430, and S440 (including S442) are respectively In the processing method shown in FIG. 3, steps S310, S320, S330, and S340 (including S342) corresponds to). Hereinafter, in order to avoid unnecessary duplicate description, detailed description of steps S410, S420, S430, and S440 (including S442) is omitted.

Referring to Figure 10, first, a feature map is extracted from the image (S405). At this time, in this embodiment, the feature map is extracted considering data loss.

The feature map extracted in this step may be at least one of a feature/feature map/feature vector/feature set extracted from an arbitrary layer from a general image through a machine vision network.

The feature map may be expressed in at least one of [H, W], [C, H, W], and [B, C, H, W]. C may be the number of channels in the feature map, H may be the height of the feature map, and W may be the width of the feature map, and may be expressed as an integer greater than or equal to 0. B is the number of feature maps with a size of [C, H, W] and can be expressed as an integer greater than 0. A feature set may be a set of one or more feature maps that have the same [C, H, W] or different [C, H, W].

The feature map extracted in this step is a feature map extracted by defining a loss function that takes into account data loss that may occur due to at least one of dimensionality reduction, quantization, and channel cutting.

Dimensionality reduction can refer to any method in which the spatial or temporal size can be reduced by downsampling, pooling, arbitrary network layers, etc. For example, in an image or feature map with a size of [C, H, W], this may mean that at least one of C, H, and W is reduced from the existing size by at least one of the dimension reduction methods.

Quantization can refer to any method that can reduce the data expression range by uniform quantization, non-uniform quantization, vector quantization, compression using existing video codecs, etc. For example, this may mean that an image or feature map expressed in n bits is expressed in m bits and is reduced. At this time, n may be a positive integer larger than m.

Channel cutting can refer to any method in which an image or feature map expressed in three dimensions [C, H, W] is cut on a channel basis. By channel cutting, an arbitrary number of channels of the upper index of an image or feature map expressed in three dimensions [C, H, W] can be cut. For example, the n channels of the upper index of a feature map with c channels can be cut to create a feature map with c-n channels. For example, the n lower index channels of a feature map with c channels can be cut to create a feature map with c-n channels. For example, a feature map with c channels can be made into a feature map with c-n channels by cutting n channels with random indices.

In this step, a loss function can be defined in consideration of data loss caused by at least one of the data loss methods, and a feature map can be extracted by applying the defined loss function. Equation 1 is an example of the loss function defined in this step.

[Equation 1]

Here, L _task refers to an arbitrary loss function defined for neural network learning, means the parameters of the existing network. The term considering data loss in Equation 1 is am. At this time, d(A,B) means the difference between A and B. refers to the existing network loss function. means a loss function based on network parameters learned including data lost by the data loss method. λ may be any real number that adjusts the coefficients of the terms included in the present invention.

Therefore, in this step, the existing loss function and defines a loss function L that can minimize the difference from the loss function by network parameters including data lost by the data loss method.

For example, a term obtained by multiplying the difference between the loss calculated by the feature cut by the channel cutting method and the loss calculated by the uncut feature by an arbitrary coefficient is added to the loss calculated by the uncut feature to obtain the final result. It is composed of a loss function. For example, by the quantization method, a term obtained by multiplying the difference between the loss calculated by the quantized feature and the loss calculated by the unquantized feature by an arbitrary coefficient is added to the loss calculated by the unquantized feature to obtain the final loss. Composed of functions.

Due to this, the data loss of the feature map to be extracted can be minimized by minimizing the difference between the data loss that may occur by the data loss method and the loss calculated when no data loss occurs.

Although the present invention has been described in detail with reference to preferred embodiments, the present invention is not limited to the above-described embodiments, and various modifications may be made by those skilled in the art within the scope of the technical idea of the present invention. possible.

Claims (20)

As a method of processing an image feature map for machine vision,
Sorting multi-channel, multi-scale feature maps extracted from the image into single-scale feature maps;
generating a converted feature map by performing feature channel transformation on one or more single-scale feature maps among the single-scale feature maps; and
A feature map processing method comprising the step of performing encoding on the converted feature map. According to paragraph 1,
The sorting step is a method of processing a feature map, characterized in that the feature maps having different resolutions are scaled to the same resolution. According to paragraph 2,
The method of processing a feature map, wherein the step of aligning includes combining the single-scale feature maps. According to paragraph 1,
After performing the encoding, it further includes the step of decoding the encoded feature map,
The step of performing the encoding includes normalizing the converted feature map,
The performing of the decoding includes the step of denormalizing the decoded feature map. According to paragraph 1,
Further comprising extracting multi-scale feature maps from the image before the aligning step,
In the extracting step, multi-scale feature maps are extracted from the image in consideration of data loss. According to paragraph 1,
The step of generating the converted feature map includes extracting representative values for each channel for the single-scale feature maps. According to clause 6,
A method of processing a feature map, characterized in that the representative value for each channel is adjusted through a fully-connected layer. According to clause 6,
A feature map processing method, characterized in that the single-scale feature maps are adjusted by the representative value for each channel. According to paragraph 1,
The step of generating the converted feature map includes one or more layers from a pooling layer, a fully-connected layer, a convolution layer, and an activation function layer. A method of processing a feature map, characterized in that it is performed based on a transformation network. According to paragraph 1,
The converted feature maps have a different number of channels than the multiscale feature maps,
The step of generating the converted feature maps includes adjusting quantization coefficients based on the number of channels of the converted feature maps. According to paragraph 1,
The step of generating the converted feature maps includes calculating the importance of each channel of the multi-scale feature maps and converting the number of channels based on the importance of each channel. According to paragraph 1,
The step of performing the encoding includes performing channel rearrangement of the converted feature maps based on channel-specific correlation of the converted feature maps. According to clause 12,
The correlation for each channel is calculated based on the average value for each channel, the center value for each channel, or the mean square error for each channel. According to paragraph 1,
The step of performing the encoding includes converting the converted feature maps into feature maps of one frame. According to clause 14,
The step of performing the encoding includes performing tile-based encoding on the converted feature maps of one frame. performing size restoration on the converted feature maps; and
Generating restored feature maps by performing channel number restoration on the converted feature maps,
A method of processing a feature map, wherein the restored feature maps include feature maps having different resolutions or numbers of channels. According to clause 16,
The step of performing size restoration is a method of processing a feature map, characterized in that configuring layers for restoration differently depending on the resolution of the feature map to be restored. According to clause 16,
The reconstructed feature maps are performed based on a transformation network including one or more layers from a pooling layer, a fully connected layer, a convolution layer, and an activation function layer. According to clause 16,
A feature map processing method, wherein the step of performing size restoration uses size information of the original feature map. According to clause 16,
A method of processing a feature map, wherein the size restoration or the channel number restoration is performed using residual-based restoration.