KR102589551B1 - Multi-scale object detection method and device - Google Patents

Abstract

A multi-scale object detection method and device are disclosed. The multi-scale object detection device includes a backbone network module having a plurality of encoders with a bottom-up hierarchical structure; A plurality of decoders with a top-down hierarchical structure; A plurality of DCN (deformable convolution network) modules that apply a deformable convolutional network algorithm to the feature maps output from the plurality of encoders and transmit them to a decoder; and a plurality of up-sampling modules, each located between the plurality of decoders, which outputs an up-sampling result by applying global average pooling to a feature map output from a higher layer decoder, wherein the plurality of decoders each perform the up-sampling. After receiving the result of element summing the result and the lower layer feature maps delivered through the plurality of DCN modules, a deconvolution operation is performed to output a multi-scale feature map.

Description

Multi-scale object detection method and device {Multi-scale object detection method and device}

The present invention relates to a multi-scale object detection method and device.

Although object detection in images is performed by default, it is still one of the difficult problems. In computer vision, deep learning-based object detection methods are widely used in various fields, such as medical imaging, face detection, object tracking, pedestrian detection, and autonomous driving.

CNN (Convolution Neural Network) has developed rapidly in recent years, starting with Alexnet, and shows greatly improved results in the field of object detection.

The pyramid method for detecting multi-scale objects uses multi-scale feature maps from the upper layer to the lower layer at the expense of increased computational load and memory space, but meaningful information loss occurs during the multi-scale feature map fusion process. There is a problem.

The present invention is to provide a multi-scale object detection method and device.

Additionally, the present invention is intended to provide a multi-scale object detection method and device that can reduce meaningful information loss that occurs during the up-sampling process when detecting a multi-scale object using a pyramid network structure.

According to one aspect of the present invention, an object detection device is provided that can reduce meaningful information loss in a multi-scale feature map fusion process in a pyramid network structure.

According to one embodiment of the present invention, a backbone network module having a plurality of encoders with a bottom-up hierarchical structure; A plurality of decoders with a top-down hierarchical structure; A plurality of DCN (deformable convolution network) modules that apply a deformable convolutional network algorithm to the feature maps output from the plurality of encoders and transmit them to a decoder; and a plurality of up-sampling modules, each located between the plurality of decoders, which outputs an up-sampling result by applying global average pooling to a feature map output from a higher layer decoder, wherein the plurality of decoders each perform the up-sampling. An object detection device may be provided that receives a result of summing the result and a lower-layer feature map delivered through the plurality of DCN modules, performs a deconvolution operation, and outputs a multi-scale feature map.

The plurality of up-sampling modules each include a first branch module that generates an attention map by applying global average pooling and convolution operations to the feature map output from the upper layer decoder; a second branching module that adjusts the size of the feature map output from the higher layer decoder by applying neighborhood interpolation; and a fusion module that outputs an up-sampling result by fusing the attention map, which is the output of the first branching module, and the resized feature map, which is the output of the second branching module.

The nth (n is a natural number) DCN module applies a deformable convolution operation to the nth feature map, which is the output of the nth encoder, and transmits it to the nth decoder. The sampling results and the feature map delivered through the nth DCN module can be fused.

The nth DCN module can fix the number of channels to 256 through point-wise conversion and output the result of the deformable convolution operation to the nth decoder.

According to another aspect of the present invention, an object detection method that can reduce meaningful information loss during the multi-scale feature map fusion process in a pyramid network structure is provided.

According to one embodiment of the present invention, (a) an encoding step of generating a plurality of multi-scale feature maps by applying a convolution operation through a plurality of encoders having a bottom-up hierarchical structure; (b) applying a deformable convolutional network algorithm to each feature map that is the output of each encoder; (c) an up-sampling step that is located between a plurality of decoders having a top-down hierarchical structure and outputs an up-sampling result by applying global average pooling to the feature map that is the output of the upper-layer decoder; and (d) a decoding step of applying the result of element sum operation of the up-sampling result and the result of applying the deformable convolution operation to a decoder and outputting the result of the deconvolution operation. there is.

In step (b), the result of the deformable convolution operation can be output by fixing the number of channels to 256 through point-wise conversion.

The step (c) includes generating an attention map by applying global average pooling and convolution operations to the feature map output from the higher layer decoder; adjusting the size by applying a neighborhood interpolation algorithm to the feature map output from the upper layer decoder; And it may include fusing the attention map and the resized feature map to output the up-sampling result.

In step (d), a deconvolution operation can be applied by receiving the result of fusing the nth up-sampling result with the result of applying a deformable convolutional network algorithm to the nth feature map, which is the output of the nth encoder. .

By providing a multi-scale object detection method and apparatus according to an embodiment of the present invention, meaningful information loss occurring during the up-sampling process can be reduced when detecting a multi-scale object using a pyramid network structure.

Through this, the present invention has the advantage of enabling accurate multi-scale object detection.

1 is a block diagram schematically showing the configuration of a multi-scale object detection device according to an embodiment of the present invention;
Figure 2 is a diagram showing a pyramid network structure according to an embodiment of the present invention.
Figure 3 is a diagram showing an up-sampling structure according to an embodiment of the present invention.
Figure 4 is a diagram comparing multi-scale object detection according to the prior art and an embodiment of the present invention.
5 is a flowchart showing a multi-scale object detection method according to an embodiment of the present invention;

As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “consists of” or “comprises” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps may be included in the specification. It may not be included, or it should be interpreted as including additional components or steps. In addition, terms such as "... unit" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a block diagram schematically showing the configuration of a multi-scale object detection device according to an embodiment of the present invention, FIG. 2 is a diagram showing a pyramid network structure according to an embodiment of the present invention, and FIG. 3 is a This is a diagram illustrating an up-sampling structure according to an embodiment of the present invention, and Figure 4 is a diagram comparing multi-scale object detection according to the prior art and an embodiment of the present invention.

The multi-scale object detection apparatus 100 described below is an apparatus for detecting multi-scale objects based on a pyramid network structure as shown in FIG. 2. The backbone network module and decoder module described below should be understood as generating feature maps in a bottom-up hierarchical structure and a top-down hierarchical structure, respectively.

Referring to Figure 1, the multi-scale object detection device 100 according to an embodiment of the present invention includes a backbone network module 110, a plurality of DCN modules 120, a decoder module 130, and a plurality of up-sampling modules ( 140), an object detection unit 145, a memory 150, and a processor 160.

The backbone network module 110 includes a plurality of encoders with a bottom-up hierarchical structure.

As shown in FIG. 2, the backbone network module 110 has a plurality of encoders with a bottom-up hierarchical structure, and each encoder extracts a multi-scale feature map by applying a convolution operation to the input image or feature map. can do.

For convenience of understanding and explanation, the description will be made assuming that the backbone network module 110 has a first encoder, a second encoder, and a third encoder.

The first encoder may extract a first feature map with a first resolution by applying a convolution operation to the input image. Additionally, the second encoder may apply a convolution operation to the first feature map to extract a second feature map with a second resolution. Additionally, the third encoder may apply a convolution operation to the second feature map to extract a third feature map with a third resolution. The first feature map may have the highest resolution, and the third feature map may have the lowest resolution.

A plurality of DCN (deformable convolution network) modules 120 apply a deformable convolution operation to the feature map that is the output of each encoder, and fix the number of channels to 256 through point-wise conversion to decoder. Output as

For example, the plurality of DCN modules 120 will be referred to as a first DCN module, a second DCN module, and a third DCN module.

The first DCN module may apply a deformable convolution operation to the first feature map that is the output of the first encoder and then transfer it to the first decoder. Additionally, the second DCN module may apply a deformable convolution operation to the second feature map that is the output of the second encoder and transmit it to the second decoder. Additionally, the third DCN module can apply a deformable convolution operation to the third feature map, which is the output of the third encoder, and transmit it to the third decoder.

By applying a deformable convolution operation to the feature maps generated by each encoder, salient context information is generated for each feature map and delivered to the decoder, thereby reducing the loss of semantic information for the upper layer feature maps.

In one embodiment of the present invention, it is assumed that DCNv2 is applied, and since this is obvious to those skilled in the art, a detailed description of DCN will be omitted.

Additionally, each DCN module (i.e., the first DCN module to the third DCN module) can expand and change the number of channels to 256 when applying a deformable convolution operation to the feature map of the encoder.

The results of each DCN module can be expressed as Equation 1.

here, and represents the offset and weight for the kth position, respectively, and represents the characteristic of position p in the output feature map and the input feature map, class represents the learnable offset and modulation scalar for the kth position, respectively.

These DCN modules can improve deformable convolutions by fine-tuning the spatial support region with a modulation scalar in the range [0, 1] for offsets with an unlimited range of real numbers. These DCN modules use a flexible kernel that can be easily changed, and a modulation scalar allows fine tuning of the receptive field. Therefore, the DCN module can focus on sparse spatial locations more powerfully and efficiently than the fixed kernel.

By using the DCN module to transfer the feature map, which is the output of the encoder, to the decoder, the expression ability of the feature map generated at each step can be improved, and object detection performance can be improved by extracting high-level features with a feature map that is more robust to geometric deformation. can be improved.

The decoder module 130 includes a plurality of decoders with a top-down hierarchical structure. Each decoder receives the result of fusing the upper layer feature map and the lower layer feature map and applies a deconvolution operation.

Let us explain each of these.

This will be described in more detail with reference to FIG. 2 .

The output value of the first encoder is fused with the output value (up-sampling result) of the first attention module after the first DCN module is applied and then input to the first decoder. Accordingly, the first decoder can apply a deconvolution operation to the input fused result.

That is, the first decoder creates a feature map that fuses the result of up-sampling the feature map of the upper layer decoder through the first up-sampling module and the lower layer feature map by applying a deformable convolution operation to the feature map of the first encoder. You can take input and apply deconvolution operation.

The operation of each decoder can be applied identically. However, since there is no upper layer decoder, the deconvolution operation can be applied by receiving the result of applying the feature map of the highest layer encoder to the DCN module.

The up-sampling module 140 is located between the decoders and outputs up-sampling results by applying global average pooling to the feature map that is the output of the upper layer decoder.

The detailed structure of the up-sampling module 140 is as shown in FIG. 3.

The up-sampling module 140 includes a first branch module 310, a second branch module 320, and a fusion module 330.

The first branch module 310 generates an attention map by applying a global average pooling operation and a convolution operation to the output value (feature map) of the upper layer decoder.

In this way, by applying a global average pooling operation to the feature map output from the upper layer decoder through the first branch module 310, globally important features can be extracted by considering the entire image and summarizing and reflecting spatial information. . In this way, edges and contours can be emphasized by applying the global average pooling operation to the feature map.

In other words, a global average pooling operation can be performed on the feature map output from the upper layer decoder, global context information can be extracted through a 1 x 1 x C convolution operation, and then an attention map can be generated through batch normalization and ReLU operation.

The second branch module 320 may adjust the size of the feature map by applying an interpolation operation to the feature map output from the upper layer decoder.

For example, the second branching module 320 may adjust the size of the feature map by applying a neighbor interpolation operation.

The fusion module 330 multiplies the output value (attention map) of the first branch module 310 and the output value (scaled feature map) of the second branch module 320 and up-samples by focusing on the semantic information of the upper layer feature map. can produce results.

The conventional pyramid network structure has the problem of deteriorating performance because checkerboard artifacts due to overlap cannot be avoided when up-sampling the feature maps of the upper layer and fusing them with the feature maps of the lower layer.

To solve this, as in an embodiment of the present invention, a global average pooling operation is performed on the feature map of the upper layer decoder, a convolution operation is performed to generate an attention map, and then the up-sampled map (i.e., through interpolation) By generating an up-sampling result by multiplying it with the resized feature map of the upper-layer decoder, there is an advantage of generating an up-sampling result that focuses on the semantic information of the upper-layer feature map.

Each decoder can receive the upsampling result for the upper layer feature map and the result of element sum calculation of the lower layer feature map delivered through the DCN module and then output the deconvolution result.

The object detector 145 may extract each object using the multi-scale feature map output through the decoder module 130.

The memory 150 stores program code for providing an object detection method that can reduce meaningful information loss in the multi-scale feature map fusion process in a pyramid network structure according to an embodiment of the present invention.

The processor 160 includes internal components (e.g., a backbone network module 110, a plurality of DCN modules 120, a decoder module 130, It is a means for controlling a plurality of up-sampling modules 140, object detection unit 145, memory 150, etc.).

Figure 4 is a diagram showing comparison results according to an embodiment of the present invention and the prior art. The first and second rows of FIG. 4 compare the results of object detection in images containing objects of various sizes, and it can be seen that the object detection device according to the present invention can successfully detect even small objects.

In addition, the third and fourth rows of FIG. 4 compare object detection results in images containing large objects. While the conventional method incorrectly detects several objects, the object detection device according to an embodiment of the present invention detects global objects. It can be seen that large objects are successfully detected because feature fusion is performed by paying attention to context information.

Figure 5 is a flowchart showing a multi-scale object detection method according to an embodiment of the present invention. The multi-scale object detection apparatus 100 described below should be interpreted in an extended manner as operating based on a pyramid network structure.

In step 510, the multi-scale object detection apparatus 100 applies the input image to a backbone network module composed of a plurality of encoders with a bottom-up hierarchical structure to extract a plurality of feature maps with multi-scale.

In step 515, the multi-scale object detection device 100 applies a deformable convolutional network algorithm to each encoder feature map of the backbone network module and outputs it to the corresponding decoder.

That is, the result of applying the DCN module to the feature map of the first encoder with the first resolution can be transmitted to be used as an input to the first decoder with the first resolution. The result of applying the DCN module is not directly input to the first decoder, but as shown in FIG. 2, it can be input to the first decoder by being fused with the result of up-sampling the output of the upper layer decoder.

However, as shown in Figure 2, the result of applying the DCN module to the feature map of the top layer encoder is that the top decoder does not go through an up-sampling process because there is no decoder located at the top. Therefore, the result of applying the DCN module to the feature map of the corresponding top layer encoder can be directly used as the input of the top layer decoder.

In step 520, the multi-scale object detection device 100 applies global average pooling to the output (feature map) of the upper layer decoder to the up-sampling module located between a plurality of decoders with a top-down hierarchical structure and outputs an up-sampling result. .

As described above, the multi-scale object detection device 100 applies global average pooling through the first branch module to the output (feature map) of the upper layer decoder and applies a convolution operation to generate an attention map, and The size can be adjusted by interpolating the output (feature map) of the upper layer decoder through the second branch module.

Next, the multi-scale object detection apparatus 100 may multiply the attention map and the resized feature map and output the up-sampling result to the lower layer decoder.

In step 525, the multi-scale object detection device 100 performs a deconvolution operation through the decoder by adding the result of up-sampling the feature map of the upper layer decoder and the result of applying the DCN module to the feature map of the encoder of the same layer (resolution). Prints a result.

The multi-scale object detection apparatus 100 detects multi-scale objects using the results (multi-resolution feature maps) output by a plurality of decoders having this top-down hierarchical structure.

Devices and methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on a computer-readable medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes magneto-optical media and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been examined focusing on its embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

100: object detection device
110: backbone network module
120: DCN module
130: decoder module
140: Upsampling module
145: object detection unit
150: memory
160: processor

Claims (9)

A backbone network module having a plurality of encoders with a bottom-up hierarchical structure;
A plurality of decoders with a top-down hierarchical structure;
A plurality of DCN (deformable convolution network) modules that apply a deformable convolutional network algorithm to the feature maps output from the plurality of encoders and transmit them to a decoder; and
Includes a plurality of up-sampling modules, each located between the plurality of decoders, which outputs an up-sampling result by applying global average pooling to the feature map that is the output of the upper layer decoder,
The plurality of decoders each receive a result of element summing the up-sampling result and the lower layer feature map transmitted through the plurality of DCN modules, then perform a deconvolution operation and output a multi-scale feature map,
Each of the plurality of upsampling modules:
a first branch module that generates an attention map by applying global average pooling and convolution operations to the feature map output from the upper layer decoder;
a second branching module that adjusts the size of the feature map output from the higher layer decoder by applying neighborhood interpolation;
It includes a fusion module that outputs an up-sampling result by fusing the attention map, which is the output of the first branch module, and the resized feature map, which is the output of the second branch module,
The nth (n is a natural number) DCN module applies a deformable convolution operation to the nth feature map, which is the output of the nth encoder, and transmits it to the nth decoder.
The n-th decoder is an object detection device characterized in that it fuses the up-sampling result, which is the output of the n-th up-sampling module, and the feature map transmitted through the n-th DCN module.
delete delete According to claim 1,
The n-th DCN module fixes the number of channels to 256 through point-wise conversion and outputs the result of the deformable convolution operation to the n-th decoder.
(a) an encoding step of generating a plurality of multi-scale feature maps by applying a convolution operation through a plurality of encoders with a bottom-up hierarchical structure;
(b) applying a deformable convolutional network algorithm to each feature map that is the output of each encoder;
(c) an up-sampling step that is located between a plurality of decoders having a top-down hierarchical structure and outputs an up-sampling result by applying global average pooling to the feature map that is the output of the upper-layer decoder; and
(d) a decoding step of applying the result of element sum operation of the up-sampling result and the result of applying the deformable convolution operation to a decoder to output the result of the deconvolution operation,
In step (c),
generating an attention map by applying global average pooling and convolution operations to the feature map output from the upper layer decoder;
adjusting the size by applying a neighborhood interpolation algorithm to the feature map output from the upper layer decoder; and
A step of fusing the attention map and the resized feature map to output the up-sampling result,
In step (d),
An object detection method characterized by applying a deconvolution operation by receiving the result of fusing the result of applying a deformable convolutional network algorithm to the nth feature map, which is the output of the nth encoder, and the nth up-sampling result.
According to clause 5,
In step (b),
An object detection method characterized by outputting the result of the deformable convolution operation by fixing the number of channels to 256 through point-wise conversion.
delete delete A computer-readable recording medium recording program code for performing the method according to claim 5 or 6.