KR102512151B1 - Method and apparatus for object detection - Google Patents

Abstract

An object detection method and apparatus using a high-performance, low-computational object detection model are provided. This enables robust object detection against various object shapes and size changes. In addition, by providing an efficient post-processing process, the accuracy of object area determination is improved.

Description

Object detection method and apparatus {METHOD AND APPARATUS FOR OBJECT DETECTION}

The present invention relates to a method and apparatus for detecting an object, and more particularly, to a method and apparatus for detecting an object from an input image using an object detection model.

The contents described below are only described for the purpose of providing background information related to an embodiment of the present invention, and the contents described do not naturally constitute prior art.

Deep learning-based object detection models are largely divided into two-stage detection models and one-stage detection models.

The two-stage detection model consists of a module that generates a Region of Interest (RoI) and a module that estimates object class information and location information of RoI through RoI sampling. The one-stage detection model consists of one module that estimates the class and location information of an object at once without the RoI sampling process.

In terms of performance, the two-stage detection model shows excellent results, but in terms of speed, the one-stage detection model shows excellent results.

Meanwhile, a conventional object detection model performs object detection by defining an aspect ratio and basic object size values (Anchors) as detection templates in advance in order to detect objects of various sizes. In recent one-stage models, research on multi-scale feature convergence techniques shows excellent object detection performance even without applying anchor, but many learning parameters/operations are required for multi-scale convergence.

In order to develop a high-performance, low-computational object detection deep learning model, a one-stage deep learning model structure that does not require anchors is required. In addition, a feature decoding method that can express features that are robust to various size/shape changes and a learning method based on the feature decoding method are required. Furthermore, there is a need for an efficient NMS post-processing method capable of removing region redundancy for predicted object regions.

Republic of Korea Patent Publication (A) 10-2019-0005045 (published date: 2019.01.15)

An object of the present invention is to provide an object detection method and apparatus based on an object detection model that does not require an anchor.

One object of the present invention is to provide an object detection model that is robust to changes in the size and shape of an object.

An object of the present invention is to provide an efficient NMS post-processing method capable of removing region redundancy with respect to predicted object regions.

The object of the present invention is not limited to the problems mentioned above, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

An object detection method according to an embodiment of the present invention provides at least one object information for at least one candidate object from an input image for each decoding layer of a plurality of decoding layers by using an object detection model including a plurality of decoding layers. Obtaining, performing an E-NMS (Efficient Non-Maximum Suppression) operation for each decoding layer based on at least one piece of object information obtained from each decoding layer, and E-NMS operation for each decoding layer. and re-performing the E-NMS operation based on the result.

For example, the object information may include object domain prediction information, object intersection over union (IOU) prediction information, and object classification prediction information.

For example, the E-NMS operation is based on the first operation for determining the initial value of the confidence score for the object region prediction information of at least one candidate object, the object region prediction information and the object IOU prediction information of the at least one candidate object A second operation for updating the confidence score and a third operation for determining a final object region of at least one candidate object based on the updated confidence score may be included.

An object detection apparatus according to an embodiment of the present invention includes a memory for storing an object detection model including a plurality of decoding layers and one or more processors, the processor using the object detection model to each of the plurality of decoding layers. At least one object information about at least one candidate object is obtained from an input image for each decoding layer, and E-NMS (Efficient Non-Maximum Suppression for each decoding layer) is performed based on the at least one object information obtained for each decoding layer. ) operation, and to re-perform the E-NMS operation based on the result of the NMS operation for each decoding layer.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, a high-performance, low-computational object detection model that does not require an anchor is provided.

According to an embodiment of the present invention, it is possible to detect objects robustly to various object shapes and sizes.

According to an embodiment of the present invention, the accuracy of object area determination is improved by providing an efficient post-processing process (E-NMS).

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary diagram schematically illustrating an object detection process according to an embodiment.
2 is a block diagram of an object detection device according to an embodiment.
3 is a flowchart of an object detection method according to an embodiment.
4 is a schematic illustration of an object detection model according to an embodiment.
5 is a configuration diagram of an object detection model according to an embodiment.
6 is a configuration diagram of a common decoder of an object detection model according to an embodiment.
7 is a configuration diagram of a header and a decoder of an object detection model according to an embodiment.
8 is a schematic illustration of E-NMS operation according to an embodiment.
9 is a flowchart of E-NMS operation according to an embodiment.
10 is a flowchart of a second operation of E-NMS operation according to an embodiment.
11 is an exemplary diagram of an IOU matrix according to an embodiment.
12 is a diagram showing an object detection result of an object detection model according to an embodiment by way of example.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The invention may be embodied in many different forms and is not limited to the embodiments set forth herein. In the following embodiments, parts not directly related to the description are omitted in order to clearly describe the present invention, but this does not mean that the omitted configuration is unnecessary in implementing a device or system to which the spirit of the present invention is applied. . In addition, the same reference numbers are used for the same or similar elements throughout the specification.

In the following description, terms such as first and second may be used to describe various components, but the components should not be limited by the terms, and the terms refer to one component from another. Used only for distinguishing purposes. Also, in the following description, singular expressions include plural expressions unless the context clearly indicates otherwise.

In the following description, terms such as "comprise" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that it does not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof.

The present invention will be described in detail with reference to the drawings below.

1 is an exemplary diagram schematically illustrating an object detection process according to an embodiment.

The object detection device 100 according to the embodiment receives an input image. For example, the input image may be directly obtained using a camera connected to the object detection device 100 or may be received from an external device through a communication network. Here, the input image includes still images and moving images.

The object detection apparatus 100 analyzes the received input image using an object detection model. The object detection model is a deep learning model for obtaining object information from an input image, and is an image-based object detection model. The object detection model will be described later with reference to FIGS. 4 to 7 .

In one example, the object information may include object domain prediction information, intersection over union (IOU) prediction information, and object classification prediction information.

Object region prediction information means prediction information about an image region corresponding to an object in an input image.

Object IOU prediction information means information about the degree of overlap between object area prediction information and Ground Truth (GT) information. For example, in object IOU prediction information, the size of an area, which is the intersection of object area prediction information and ground truth (GT) information about the area of the object, is calculated as object area prediction information and ground truth (GT) information about the area of the object. It may be determined based on a value divided by the size of an area that is a union of information.

Object classification prediction information refers to information probabilistically predicting what kind of object a predicted object region will be.

Additionally, the object information may further include object size prediction information and object classification accuracy prediction information. This will be described later with reference to FIG. 7 .

The object detection device 100 corresponds to various types of electronic devices capable of executing an object detection model.

In one example, the object detection device 100 may be mounted on a vehicle. In one example, the object detection device 100 may be an electronic device capable of communicating with a vehicle. For example, the object detection apparatus 100 may include a server connected to a vehicle through a network. In one example, the object detection device 100 may include a terminal device such as a robot and a smart phone, but is not limited thereto.

2 is a block diagram of an object detection device according to an embodiment.

The object detection device 100 according to the embodiment may include a processor 110 and a memory 120 . Components shown in FIG. 2 are exemplary, and the object detection apparatus 100 may further include additional components.

The object detection device 100 may include a processor 150 .

The processor 110, as a kind of central processing unit, may control the operation of the object detection device 100 by executing one or more commands stored in the memory 120. The processor 110 may include all types of devices capable of processing data by executing instructions.

The processor 110 may mean, for example, a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but is not limited thereto. Processor 110 may include one or more processors.

Additionally, the object detection device 100 may include a memory 120 .

The memory 120 may store an object detection model for acquiring object information. The memory 120 may store instructions for a process in which the object detection apparatus 100 executes an object detection model.

The processor 110 may execute an object detection process according to an embodiment based on a program, instructions, and an object detection model stored in the memory 120 .

The memory 120 may further store intermediate data, calculation results, and the like that are calculated by an algorithm and generated during an operation process for detecting an object.

The memory 120 may include built-in memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, and NAND. Non-volatile memory such as flash memory or NOR flash memory, flash drives such as SSD, compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or memory stick; Alternatively, it may include a storage device such as a HDD. The memory 120 may include magnetic storage media or flash storage media, but is not limited thereto.

3 is a flowchart of an object detection method according to an embodiment.

An object detection method according to an embodiment includes obtaining at least one object information about at least one candidate object from an input image for each decoding layer of a plurality of decoding layers by using an object detection model including a plurality of decoding layers ( S10), performing an E-NMS (Efficient Non-Maximum Suppression) operation for each decoding layer based on at least one object information obtained from each decoding layer (S20) and E-NMS operation for each decoding layer A step of re-performing the E-NMS operation based on the result of (S30) may be included.

In step S10, the processor 110 obtains at least one object information about at least one candidate object from an input image for each decoding layer of the plurality of decoding layers by using an object detection model including a plurality of decoding layers. can

Here, the plurality of decoding layers correspond to a bottom-up path merging layer of the common decoder 2 to be described later with reference to FIG. 5 .

In one example, object information may include object domain prediction information (pred_box_posts), IOU (Intersection Over Union) prediction information (pred_reward_iou), and object classification prediction information (pred_confidence).

In step S20, the processor 110 may perform an E-NMS operation for each decoding layer based on at least one piece of object information acquired in each decoding layer in step S10.

In step S20, the processor 110 performs an E-NMS operation for each decoding layer. That is, in step S20, the processor 110 performs a layer-wise E-NMS operation.

The E-NMS operation includes a first operation for determining an initial value of the confidence score of at least one candidate object, and a second operation for updating the confidence score based on object region prediction information and object IOU prediction information of the at least one candidate object. and a third operation for determining a final object region of at least one candidate object based on the updated confidence score. An E-NMS operation according to an embodiment will be described later in detail with reference to FIGS. 8 to 11 .

In step S30, the processor 110 may re-perform the E-NMS calculation based on the result of the E-NMS calculation for each decoding layer in step S20.

That is, in step S30, the processor 110 may combine the result of the E-NMS operation performed for each decoding layer in step S20, and re-perform the E-NMS operation based on the combined object information.

Hereinafter, an object detection model according to an embodiment will be described with reference to FIGS. 4 to 7 .

4 is a schematic illustration of an object detection model according to an embodiment.

The object detection model according to the embodiment includes a common encoder 1, a common decoder 2, a header 3 including an object domain header 3-1 and an object classification header 3-2, and an object domain decoder 4 -1) and a decoder 4 including an object classification decoder 4-2.

The object detection model is a fully-convolutional neural network-based object detection deep learning model.

The common encoder (1) and common decoder (2) of the object detection model are located in front of the header (3) and decoder (4) implemented for each task (eg, object region prediction task and object classification prediction task) to input the image. A common feature map is extracted from

That is, the object detection model can remove redundancy of operations for encoding and decoding features of an input image through a structure of a common encoder 1 and a common decoder 2 shared between tasks.

In the object detection model, header 3 and decoder 4 may be implemented for each task related to object detection. That is, the header 3 and the decoder 4 are implemented with layers and operations suitable for object information to be detected by each task.

The header 3 converts all or part of the feature maps output through the common encoder 1 and the common decoder 2 into feature maps suitable for the decoder 4. The decoder 4 receives the feature map converted from the header 3 and outputs object information to be detected.

The object detection model according to the embodiment easily plugs additional tasks in addition to the aforementioned object region prediction task and object classification prediction task by connecting additional headers 3 and decoders 4 to the common encoder 1 and common decoder 2. -It is designed in a structure that enables plug-in/out.

Hereinafter, the structure of the object detection model will be described in more detail with reference to FIG. 5 .

5 is a configuration diagram of an object detection model according to an embodiment.

A common encoder 1 may include a series of encoding layers (Conv1, Conv2, Conv3, Conv4, Conv5, Conv6 and Conv7). The common encoder 1 outputs a plurality of first common feature maps obtained by encoding features of an input image through a series of encoding layers. The plurality of first common feature maps correspond to feature maps output from each layer of a series of encoding layers.

The common encoder 1 may include Conv1 to Conv5 encoding layers. A structure such as VGGNet, ResNet, XceptionNet, ResnetXT, SuffleNet or MobileNet may be applied to the Conv1 to Conv5 encoding layers.

The common encoder 1 may additionally include Conv6 and Conv7 encoding layers. This can express the large object/region features of the input image well, and is a necessary structure for multi-scale feature convergence in the common decoder (2).

A common decoder 2 may include multiple decoding layers. The multi-decoding layer may include a feature pyramid network (FPN) structure, which is a decoder structure for object detection, and a bottom-up path aggregation layer (BPA).

The common decoder 2 outputs a plurality of second common feature maps P3, P4, P5, P6 and P7 through multiple decoding layers. The plurality of second common feature maps correspond to feature maps output from each layer of the BPA layer.

The plurality of second common feature maps P3 , P4 , P5 , P6 , and P7 output from the common decoder 2 have characteristics capable of robust feature extraction against changes in the size and shape of various objects. This means that the common decoder 2 generates a plurality of second common feature maps P3, P4, P5, P6, and P7 in a multiscale feature fusion scheme, and such a plurality of second common feature maps P3, P4, P5 , P6 and P7), it is possible because the header 3 and the decoder 4, which will be described later, operate. The header 3 constructs an input feature map based on at least a part of the plurality of second common feature maps P3, P4, P5, P6, and P7, and provides the configured input feature map to the decoder 4.

The plurality of second common feature maps P3, P4, P5, P6, and P7 output from the common decoder 2 include a header 3-1 implemented for object region prediction and a header implemented for object classification prediction ( 3-2) respectively. The header 3 and the decoder 4 will be described later with reference to FIG. 7 .

Hereinafter, the common decoder 2 will be described in detail with reference to FIG. 6 .

6 is a configuration diagram of a common decoder of an object detection model according to an embodiment.

The common decoder 2 includes multiple decoding layers for extracting a plurality of first common feature maps through multi-scale feature fusion.

The multi-decoding layer upsamples and adds parts (C3, C4, C5) of the plurality of first common feature maps (C3, C4, C5, C6, and C7) output from the common encoder (1). The top-down layer (2-1) output by the top-down layer (2-1), a plurality of first common feature maps (C3, C4, C5, C6 and C7) in the common encoder (1) except for some (C6 , C7) and the middle layer 2-2 performing the convolution operation on the output of the top-down layer 2-1, and the outputs of the middle layer 2-2 are convolved and summed to form a plurality of second layers. A bottom-up path aggregation layer 2-3 outputting common feature maps P3, P4, P5, P6, and P7 may be included.

The top-down layer 2-1 upsamples and adds a part of a plurality of first common feature maps output from the common encoder 1, and outputs them.

In one example, the top-down layer includes three feature maps (C3, C4, and C4) corresponding to medium resolution among a plurality of first common feature maps (C3, C4, C5, C6, and C7) generated by the common encoder (1). C5) is performed.

The first step of the top-down layer (2-1) is a feature map (C5') generated by 1x1x256 convolution of the first common feature map (C5) output from the fifth encoding layer (Conv5) of the common encoder (1). ) is output. The second stage of the top-down layer (2-1) is the result of upsampling the output (C5') of the first stage by a factor of 2 and the first common feature map output from the fourth encoding layer (Conv4) of the common encoder (1). The feature map (C4') generated by 1x1x256 convolution of (C4) is summed (C4'+C5') and output.

In the same way, the third stage of the top-down layer (2-1) is the result of upsampling the output (C4' + C5') of the second stage by a factor of 2 and the output from the third encoding layer (Conv3) of the common encoder (1). The first common feature map C3 is summed (C3'+C4'+C5') with the feature map C3' generated by 1x1x256 convolution and output. This allows cumulative fusion of low-resolution and high-resolution features in the top-down direction.

The middle layer (2-2) is the remainder (C6 and C7) except for some of the plurality of first common feature maps (C3, C4, C5, C6 and C7) in the common encoder (1) and the top-down layer (2-2). Convolution is performed on the output of 1).

The middle layer 2-2 includes two unused top-down layers 2-1 among a plurality of first common feature maps C3, C4, C5, C6, and C7 output from the common encoder 1. 3x3x256 convolutions on feature maps C6 and C7, respectively. That is, the intermediate layer 2-2 performs a 3x3x256 convolution of the first common feature map C7 output from the last encoding layer C7 of the common encoder 1 to output an intermediate feature map M7. Similarly, the intermediate layer 2-2 performs a 3x3x256 convolution of the first common feature map C6 output from the sixth encoding layer C6 of the common encoder 1 to output an intermediate feature map M6.

The intermediate layer 2-2 performs 3x3x256 convolution on the output of the top-down layer 2-1, respectively, and outputs intermediate feature maps M5, M4, and P3.

The bottom-up path merging layer 2-3 convolves and sums the outputs of the middle layer 2-2 to output a plurality of second common feature maps P3, P4, P5, P6, and P7. The bottom-up path merging layer 2 - 3 generates a plurality of second common feature maps P3 , P4 , P5 , P6 , and P7 while proceeding from the lowest layer to the highest layer of the pyramid.

The bottom-up path merge layer (2-3) generates a low-resolution feature map (P3 ') by convolving the P3 intermediate feature map output from the intermediate layer (2-2) with 3x3x256, stride = 2, and the intermediate layer ( The second common feature map P4 is generated by summing (P3'+M4) with the intermediate feature map M4 output in 2-2). The remaining second common feature maps P4, P6, and P7 are generated in the same manner with respect to the generated second common feature map P4.

In this specification, a plurality of decoding layers means the final decoding layer of the common decoder 2. For example, the plurality of decoding layers may correspond to the bottom-up path merging layer 2-3.

The plurality of second common feature maps P3, P4, P5, P6, and P7 include a series of multi-scale feature maps output from the bottom-up path merging layer 2-3. That is, a series of multi-scale feature maps can be generated by accumulatively fusing low-resolution and high-resolution features in the bottom-up direction by the bottom-up path merging layer 2-3, whereby various objects are complexly included. It is possible to express characteristics that are robust to changes in size and shape of images and objects.

7 is a configuration diagram of a header and a decoder of an object detection model according to an embodiment.

In the object detection model according to the embodiment, a plurality of second common feature maps (P3, P4, P5, P6, P7) is input to the header 3 with reference to FIG.

The object detection model according to the embodiment includes an object domain header 3-1 and an object classification header 3-2.

The object region header 3-1 includes a domain adaptation layer that converts a plurality of second common feature maps P3, P4, P5, P6, and P7 output from a plurality of decoding layers into an input feature map for object region detection. it means.

The object classification header 3-2 includes a kind of domain adaptation layer that converts a plurality of second common feature maps (P3, P4, P5, P6, and P7) generated in a plurality of decoding layers into an input feature map for object classification. it means.

As described above, the plurality of decoding layers correspond to the bottom-up path merging layer 2 - 3 of the common decoder 2 described with reference to FIG. 6 .

In the header 3 for object detection, a plurality of second common feature maps P3, P4, P5, P6, and P7 are respectively input to the decoder 4 for object detection through a domain adaptation layer. Convert to input feature map. Here, the input feature map includes an object domain feature map and an object classification feature map.

The header 3 is composed of an object domain header 3-1 and an object classification header 3-2. The object region header (box_feature) and the object classification header (class_feature) are a plurality of second common feature maps (P3, P4, P5, P6, and P7) output from the common decoder 2 using the same parameter (Sharing Layer) An object domain feature map expressing object domain features and an object classification feature map expressing object classification features are created.

In one example, the object region header 3-1 and the object classification header 3-2 may include a plurality of convolutional layers that share weights.

In one example, the object domain header 3-1 and the object classification header 3-2 are a plurality of second common feature maps output from the bottom-up path merging layer 2-3 of the common decoder 2, respectively. For (P3, P4, P5, P6, and P7), a 3x3x256 convolution is performed four times to generate an object domain feature map and an object classification feature map, respectively.

The object region header 3-1 and the object classification header 3-2 use the same parameters to generate a plurality of second common feature maps P3, P4, P5, P6, and P7 output from a plurality of decoding layers. Multi-scale features are extracted into object domain feature maps and object classification feature maps.

The decoder 4 of the object detection model predicts at least one piece of object information using the input feature map generated in the header 3.

In one example, the decoder 4 may use an anchorless Multi-Scale Fully-Convolutional Neural Network (MS-FCN), and may secure high performance and efficiency by removing redundant and additional operations.

The object domain feature map is used as an input of the object domain decoder 4-1 that predicts object domain prediction information (pred_box_posts), object IOU prediction information (pred_reward_iou), and object size prediction information (pred_reward_scale).

The object region decoder (4-1) performs a 3x3x4 (x1, y1, x2, y2: position where the object is away from the current grid position) convolution with respect to the object region feature map to generate object boxes (pred_boxes), and generate Object region prediction information (pred_box_posts) is generated through multiplication of the predefined object boxes (pred_boxes) and scale factors.

In one example, the scale factor is a constant learned to increase an object region predicted size value, and may be learned based on object size prediction information (pred_reward_scale).

The object domain decoder 4-1 generates object IOU prediction information (pred_reward_iou) and object size prediction information (pred_reward_scale) through 1x1x256 convolution and 1x1x1 convolution with respect to the object domain feature map, respectively.

The object IOU prediction information (pred_reward_iou) predicts how much the predicted object area (pred_box_posts) matches the actual correct answer object area, thereby increasing object detection accuracy during learning, and the object IOU prediction information (pred_reward_iou) is used after object detection. It is possible to improve object area detection performance by applying E-NMS, which is a process.

The object classification feature map is used as an input of the object classification decoder 4-2 that predicts object classification prediction information (pred_confidence) and object classification accuracy prediction information (pred_reward_identificatoin).

The object classification decoder 4-2 generates object classification prediction information (pred_confidence) through 3x3xN (N: background + number of classified objects) convolution with respect to the object classification feature map.

Meanwhile, object classification accuracy prediction information (pred_reward_identification) and object size prediction information (pred_reward_scale) are used for learning the object detection model, but are not used in the actual inference process of the object detection model.

That is, object classification accuracy prediction information (pred_reward_identification) and object size prediction information (pred_reward_scale) are virtual auxiliary tasks to help the object detection model learn, and are not used in an actual inference process.

Here, the virtual auxiliary task calculates a loss function through prediction during learning, reflects it in the update of the learning parameter to help the object information learn well, and does not predict the object information in the execution mode. This means a method that does not increase the computational load.

Object size prediction information (pred_reward_scale) is information that predicts whether the size of an object is accurately predicted. By learning object IOU prediction information (pred_reward_iou) and object size prediction information (pred_reward_scale) together during learning, object area prediction performance is strengthened and object size prediction information is strengthened. Detection accuracy is improved.

Object classification accuracy prediction information (pred_reward_identification) is information for predicting whether object classification information is accurately predicted, and can enhance object classification prediction performance during learning.

The processor 110 may determine a final object region through N-NMS-based post-processing of the object information obtained using the decoder 4.

In one example, the processor 110 inputs object region prediction information (pred_box_posts), object IOU prediction information (pred_reward_iou), and object classification prediction information (pred_confidence) to an N-NMS operation to be described later with reference to FIG. can choose

8 is a schematic illustration of E-NMS operation according to an embodiment.

The object detection method according to the embodiment proposes E-NMS operation for post-processing of object information output from the object detection model.

The E-NMS operation determines the final object region for at least one candidate object predicted to be included in the input image based on object classification prediction information (pred_confidence), object IOU prediction information (pred_reward_iou), and object region prediction information (pred_box_posts). Perform a post-processing process to determine.

The object detection method according to the embodiment executes the E-NMS operation in two steps. That is, referring to FIG. 3, the processor 110 performs the E-NMS operation in steps S20 and S30.

In step S20, an E-NMS operation is performed for each decoding layer (Layer-wise E-NMS). In step S30, the entire E-NMS calculation is performed based on the result of the E-NMS calculation for each layer in step S20.

Hereinafter, the E-NMS operation will be described in more detail with reference to FIGS. 9 and 10 .

9 is a flowchart of E-NMS operation according to an embodiment.

The E-NMS operation is based on the first operation (S31) for determining the initial value of the confidence score of at least one candidate object, the object area prediction information (pred_box_posts) and the object IOU prediction information (pred_reward_iou) of the at least one candidate object. A second operation ( S32 ) of updating the confidence score and a third operation ( S33 ) of determining a final object region of at least one candidate object based on the updated confidence score may be included.

In the first operation S31, the processor 110 determines an initial value of the confidence score of at least one candidate object.

In one example, when the processor 110 performs the E-NMS operation in step S20, in the first operation S31, the initial value of the confidence score based on the function for object IOU prediction information (pred_reward_iou). can determine

In this case, the initial value of the confidence score can be expressed by Equation 1 below.

Here, the confidence of the right term means object classification prediction information (pred_confidence) output from the object classification decoder 4-2 of the object detection model.

f(pred_reward_iou) is a weight factor based on object IOU prediction information (pred_reward_iou), and various functions may be applied. For example, f(x)= x ^0.8 can be used as f(pred_reward_iou).

Such initial value setting enables E-NMS to be performed based on fusion information of object classification prediction information (pred_confidence) and object IOU prediction information (pred_reward_iou).

In one example, when the processor 110 performs the E-NMS operation in step S30, in the first operation S31, based on the E-NMS operation result for each decoding layer in step S20, An initial value of the confidence score may be determined.

For example, while performing the second operation (S32) of the E-NMS operation for each decoding layer in step (S20), the processor 110 converts the updated confidence score into the first operation of the E-NMS operation (S30). It can be determined as the initial value of the confidence score in operation S31.

In the second operation S32, the processor 110 updates the confidence score based on object region prediction information (pred_box_posts) and object IOU prediction information (pred_reward_iou) of at least one candidate object. The second operation ( S32 ) will be described with reference to FIG. 10 .

In a third operation (S33), the processor 110 determines a final object region of at least one candidate object based on the updated confidence score.

For example, the processor 110 may determine a region having the largest updated confidence score among regions corresponding to the object region prediction information (pred_box_posts) as the final object region of the corresponding object.

As a result, the processor 110 may determine a final object region for at least one candidate object predicted to be included in the input image by executing the E-NMS operation.

10 is a flowchart of a second operation of E-NMS operation according to an embodiment.

The processor 110 executes the following operation to execute the second operation (S32) of the E-NMS operation. That is, the second operation (S32) is based on the initial value determined in the first operation (S31). Based on a predefined weight function and determining the confidence score of at least one candidate object based on the re-determined IOU (S32-3).

In operation S32-1, the processor 110 sorts object region prediction information (pred_box_posts) for at least one candidate object based on the confidence score determined in operation S31. For example, object region prediction information (pred_box_posts) for at least one candidate object may be sorted based on the descending order of confidence scores of the candidate objects.

In one example, the processor 110 performs the operation of determining the IOU (S32-2), the operation of re-determining the IOU (S32-3), and the operation of determining the confidence score (S32-4) in parallel using the IOU matrix. can be performed with

The calculation method using the IOU matrix is a method of generating and calculating a descending triangular IOU matrix between predicted object areas, and is a method of improving overlap ambiguity and parallel operation efficiency between different objects.

The IOU matrix is a square matrix having as many rows and columns as the number of object region prediction information (pred_box_posts). The E-NMS calculation based on the IOU matrix is described below.

In operation S32-2, the processor 110 determines an IOU between two different pieces of object area prediction information (pred_box_posts) for at least one candidate object.

1) In operation (S32-2), the processor 110 determines the IOU between the i-th object region prediction information Box _i and the j-th object region prediction information Box _j , and determines the IOU as j of the i-th row of the IOU matrix It is stored in b _ij , which is the matrix element of the th row. Here, the IOU value of b _ii whose row and column numbers match is 1.

2) The value of b _ij (i<j) located in an ascending triangle based on b _ii is set to 0. The IOU value of b _ii whose row and column numbers match is also set to 0.

3) In order to eliminate performance degradation due to overlapping of different candidate objects, if the vertical/horizontal objects for each IOU matrix are not the same object, the corresponding matrix element is set to 0.

In operation S32-3, the processor 110 re-determines the IOU based on the predefined weight function.

Equation 2 below is an exemplary predefined weight function.

Here, n is a linear weight function, the IOU of the right term is the IOU currently stored in the IOU matrix, and the IOU of the left term represents the IOU re-determined according to Equation 2.

In operation S32-4, the processor 110 determines a confidence score based on the re-determined IOU.

The processor 110 determines a confidence score for an object region corresponding to object region prediction information (pred_box_posts) of at least one candidate object based on Equation 3 below.

Here, conf _n means the confidence score of the nth object area (Box _n ).

11 is an exemplary diagram of an IOU matrix according to an embodiment.

The figure on the left exemplarily shows an IOU matrix for storing the IOU re-determined in operation S32-3.

The diagram on the right shows that bi is calculated by multiplying the value of IOU in the i-th row to determine the confidence score for each object region ( _bi ) according to Equation 3.

12 is a diagram showing an object detection result of an object detection model according to an embodiment by way of example.

It can be seen that objects of various shapes and sizes are well detected, and the object area is accurately specified even if it is very adjacent or occluded.

Hereinafter, a learning process of an object detection model according to an embodiment will be described.

The loss function for learning the object detection model according to the embodiment includes an object classification error (class), an object region estimation error (box), an IOU estimation error (iou), an object size accuracy prediction error (scale), and Object classification accuracy prediction (id) was considered. In addition, the final loss for learning can be obtained as a weighted sum of these four errors.

(1) Class error calculation uses the method using Focal loss as shown in Equation 4.

(2) The object area prediction error calculation uses GIOU loss as shown in Equation 5.

(3) The IOU estimation error iou uses Equation 6.

(4) Object size prediction accuracy The prediction error (scale) can be obtained using Equation 7.

(5) Object Classification Accuracy The prediction error (id) can be obtained using Equation 8.

where u is a uni-step function.

(6) Finally, the final error function required for model learning is calculated as a weighted sum of the error values obtained in (1) to (4) as shown in Equation 9.

Hereinafter, artificial intelligence technology related to an embodiment of the present invention will be described.

Artificial intelligence (AI) is a field of computer science and information technology that studies ways to enable computers to do thinking, learning, and self-development that human intelligence can do. This means that behavior can be imitated.

Also, artificial intelligence does not exist by itself, but is directly or indirectly related to other fields of computer science. In particular, in modern times, attempts to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in those fields are being actively made.

Machine learning is a branch of artificial intelligence, a field of study that gives computers the ability to learn without being explicitly programmed.

Specifically, machine learning can be said to be a technology that studies and builds a system that learns based on empirical data, makes predictions, and improves its own performance, as well as algorithms for it. Machine learning algorithms build specific models to make predictions or decisions based on input data, rather than executing rigidly defined, static program instructions.

The term 'machine learning' may be used interchangeably with the term 'machine learning'.

In machine learning, many machine learning algorithms have been developed regarding how to classify data. Representative examples include decision trees, Bayesian networks, support vector machines (SVMs), and artificial neural networks (ANNs).

A decision tree is an analysis method that performs classification and prediction by charting decision rules in a tree structure.

A Bayesian network is a model that expresses a stochastic relationship (conditional independence) among multiple variables in a graph structure. Bayesian networks are suitable for data mining through unsupervised learning.

A support vector machine is a supervised learning model for pattern recognition and data analysis, and is mainly used for classification and regression analysis.

An artificial neural network is an information processing system in which a plurality of neurons called nodes or processing elements are connected in the form of a layer structure by modeling the operating principle of biological neurons and the connection relationship between neurons.

An artificial neural network is a model used in machine learning, a statistical learning algorithm inspired by neural networks in biology (particularly the brain in the central nervous system of animals) in machine learning and cognitive science.

Specifically, an artificial neural network may refer to an overall model that has problem-solving ability by changing synapse coupling strength through learning of artificial neurons (nodes) that form a network by synapse coupling.

The term artificial neural network may be used interchangeably with the term neural network.

An artificial neural network may include a plurality of layers, and each of the layers may include a plurality of neurons. In addition, the artificial neural network may include neurons and synapses connecting neurons.

Artificial neural networks generally use the following three factors: (1) connection patterns between neurons in different layers, (2) a learning process that updates the weights of connections, and (3) an output value from the weighted sum of the inputs received from the previous layer. It can be defined by the activation function you create.

Artificial neural networks may include network models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Bidirectional Recurrent Deep Neural Network (BRDNN), Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN). , but not limited thereto.

In this specification, the term 'layer' may be used interchangeably with the term 'layer'.

Artificial neural networks are classified into single-layer neural networks and multi-layer neural networks according to the number of layers.

A typical single-layer neural network consists of an input layer and an output layer.

In addition, a general multilayer neural network is composed of an input layer, one or more hidden layers, and an output layer.

The input layer is a layer that accepts external data. The number of neurons in the input layer is the same as the number of input variables. The hidden layer is located between the input layer and the output layer. do. The output layer receives a signal from the hidden layer and outputs an output value based on the received signal. The input signal between neurons is multiplied by each connection strength (weight) and then summed. If this sum is greater than the neuron's threshold, the neuron is activated and outputs the output value obtained through the activation function.

Meanwhile, a deep neural network including a plurality of hidden layers between an input layer and an output layer may be a representative artificial neural network implementing deep learning, which is a type of machine learning technology.

Meanwhile, the term 'deep learning' may be used interchangeably with the term 'deep learning'.

The artificial neural network may be trained using training data. Here, learning may refer to a process of determining parameters of an artificial neural network using learning data in order to achieve a purpose such as classification, regression analysis, or clustering of input data. can As representative examples of parameters of an artificial neural network, a weight assigned to a synapse or a bias applied to a neuron may be cited.

An artificial neural network learned from training data may classify or cluster input data according to a pattern of the input data.

Meanwhile, an artificial neural network trained using training data may be referred to as a trained model in this specification.

Next, the learning method of the artificial neural network will be described.

Learning methods of artificial neural networks can be largely classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

Supervised learning is a method of machine learning to infer a function from training data.

Among the inferred functions, outputting a continuous value is called regression analysis, and predicting and outputting a class of an input vector is called classification.

In supervised learning, an artificial neural network is trained under a given label for training data.

Here, the label may mean a correct answer (or a result value) to be inferred by the artificial neural network when training data is input to the artificial neural network.

In this specification, when training data is input, an answer (or a result value) to be inferred by an artificial neural network is referred to as a label or labeling data.

Also, in this specification, setting labels on training data for learning of an artificial neural network is referred to as labeling labeling data on training data.

In this case, training data and labels corresponding to the training data constitute one training set, and may be input to the artificial neural network in the form of a training set.

Meanwhile, the training data represents a plurality of features, and labeling the training data with a label may mean that a label is attached to a feature represented by the training data. In this case, the training data may represent the characteristics of the input object in the form of a vector.

The artificial neural network may use the training data and the labeling data to infer a function for a correlation between the training data and the labeling data. In addition, parameters of the artificial neural network may be determined (optimized) through evaluation of the function inferred from the artificial neural network.

Unsupervised learning is a type of machine learning in which labels are not given to the training data.

Specifically, unsupervised learning may be a learning method for learning an artificial neural network to find and classify a pattern in training data itself rather than an association between training data and a label corresponding to the training data.

Examples of unsupervised learning include clustering or independent component analysis.

In this specification, the term 'clustering' may be used interchangeably with the term 'clustering'.

Examples of artificial neural networks using unsupervised learning include a Generative Adversarial Network (GAN) and an Autoencoder (AE).

A generative adversarial network is a machine learning method in which two different artificial intelligences, a generator and a discriminator, compete to improve performance.

In this case, the generator is a model that creates new data and can generate new data based on original data.

In addition, the discriminator is a model that recognizes data patterns and can play a role in discriminating whether input data is original data or new data generated by a generator.

The generator learns by receiving data that has not deceived the discriminator, and the discriminator can learn by receiving deceived data from the generator. Accordingly, the generator can evolve to deceive the discriminator as best as possible, and the discriminator can evolve to distinguish well between the original data and the data generated by the generator.

An autoencoder is a neural network that aims to reproduce the input itself as an output.

An auto-encoder includes an input layer, at least one hidden layer, and an output layer.

In this case, since the number of nodes in the hidden layer is smaller than the number of nodes in the input layer, the dimensionality of data is reduced, and compression or encoding is performed accordingly.

Also, the data output from the hidden layer goes into the output layer. In this case, since the number of nodes in the output layer is greater than the number of nodes in the hidden layer, the dimensionality of data increases, and accordingly, decompression or decoding is performed.

On the other hand, the autoencoder adjusts the connection strength of neurons through learning, so that input data is expressed as hidden layer data. In the hidden layer, information is expressed with fewer neurons than in the input layer, and being able to reproduce input data as an output may mean that the hidden layer discovered and expressed a hidden pattern from the input data.

Quasi-supervised learning is a type of machine learning and may refer to a learning method using both labeled training data and unlabeled training data.

As one of the techniques of semi-supervised learning, there is a technique of inferring the label of unlabeled training data and then performing learning using the inferred label. This technique is useful when the cost required for labeling is high. can

Reinforcement learning is a theory that if an agent is given an environment in which it can judge what action to take every moment, it can find the best way through experience without data.

Reinforcement learning may be performed mainly by a Markov Decision Process (MDP).

To explain the Markov decision process, first, an environment in which the information necessary for the agent to take the next action is given, second, how the agent will behave in that environment, and third, if the agent does well, a reward ( Fourth, the optimal policy is derived by repeating experience until the future reward reaches the highest point.

The structure of an artificial neural network is specified by model configuration, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and hyperparameters are set in advance before learning. After setting, the model parameter (Model Parameter) is set through learning, so that the content can be specified.

For example, factors determining the structure of an artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, an input feature vector, a target feature vector, and the like.

Hyperparameters include various parameters that must be initially set for learning, such as initial values of model parameters. And, the model parameters include several parameters to be determined through learning.

For example, the hyperparameters may include an initial value of weight between nodes, an initial value of bias between nodes, a mini-batch size, a number of training iterations, a learning rate, and the like. In addition, model parameters may include weights between nodes, biases between nodes, and the like.

The loss function may be used as an index (reference) for determining optimal model parameters in the learning process of an artificial neural network. In an artificial neural network, learning means a process of manipulating model parameters to reduce a loss function, and the purpose of learning can be seen as determining model parameters that minimize a loss function.

The loss function may mainly use mean squared error (MSE) or cross entropy error (CEE), but the present invention is not limited thereto.

Cross entropy error can be used when the correct answer label is one-hot encoded. One-hot encoding is an encoding method in which the correct answer label value is set to 1 only for neurons corresponding to the correct answer, and the correct answer label value is set to 0 for neurons with no correct answer.

In machine learning or deep learning, learning optimization algorithms can be used to minimize the loss function, and learning optimization algorithms include Gradient Descent (GD), Stochastic Gradient Descent (SGD), Momentum ), NAG (Nesterov Accelerate Gradient), Adagrad, AdaDelta, RMSProp, Adam, and Nadam.

Gradient descent is a technique that adjusts model parameters in the direction of reducing the value of the loss function by considering the slope of the loss function in the current state.

A direction for adjusting model parameters is called a step direction, and a size for adjusting the model parameters is called a step size.

In this case, the step size may mean a learning rate.

In the gradient descent method, a gradient may be obtained by partial differentiation of a loss function with respective model parameters, and the model parameters may be updated by changing the model parameters in the direction of the obtained gradient by a learning rate.

Stochastic gradient descent is a technique that increases the frequency of gradient descent by dividing training data into mini-batches and performing gradient descent for each mini-batch.

Adagrad, AdaDelta, and RMSProp are techniques that increase optimization accuracy by adjusting the step size in SGD. In SGD, momentum and NAG are techniques that increase optimization accuracy by adjusting the step direction. Adam is a technique that increases optimization accuracy by adjusting the step size and step direction by combining momentum and RMSProp. Nadam is a technique that increases optimization accuracy by adjusting the step size and step direction by combining NAG and RMSProp.

The learning speed and accuracy of an artificial neural network are characterized by being largely dependent on hyperparameters as well as the structure of the artificial neural network and the type of learning optimization algorithm. Therefore, in order to obtain a good learning model, it is important to set appropriate hyperparameters as well as to determine an appropriate artificial neural network structure and learning algorithm.

Typically, hyperparameters are experimentally set to various values to train the artificial neural network, and as a result of learning, the optimal values are set to provide stable learning speed and accuracy.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the media include hard disks, solid state disks (SSDs), silicon disk drives (SDDs), magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, floptical disks and and hardware devices specially configured to store and execute program instructions, such as a magneto-optical medium, such as ROM, RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

The description of the embodiments of the present invention described above is for illustrative purposes, and those skilled in the art can easily modify them into other specific forms without changing the technical spirit or essential features of the present invention. you will understand that Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present invention.

100 object detection device
110 processor
120 memory

Claims (20)

obtaining at least one object information about at least one candidate object from an input image for each decoding layer of the plurality of decoding layers by using an object detection model including a plurality of decoding layers;
performing an Efficient Non-Maximum Suppression (E-NMS) operation for each decoding layer based on the at least one piece of object information obtained from each decoding layer; and
Re-performing the E-NMS operation based on object information obtained by combining the result of the E-NMS operation for each decoding layer.
including,
The object information,
Object domain prediction information, object intersection over union (IOU) prediction information, and object classification prediction information
including,
The E-NMS calculation,
a first operation for determining an initial value of a confidence score for object region prediction information of the at least one candidate object;
a second operation of updating the confidence score based on object region prediction information and object IOU prediction information of the at least one candidate object; and
a third operation for determining a final object region of the at least one candidate object based on the updated confidence score;
including,
object detection method.
According to claim 1,
The object detection model,
a common encoder including a series of encoding layers for extracting a plurality of first common feature maps from the input image; and
A common decoder including multiple decoding layers generating a plurality of second common feature maps by multi-scale feature fusion of the plurality of first common feature maps.
containing
object detection method.
According to claim 2,
The multiple decoding layers,
a top-down layer that upsamples and adds a portion of the plurality of first common feature maps and outputs them;
an intermediate layer that performs a convolution operation on outputs of the top-down layer and other than some of the plurality of first common feature maps; and
A bottom-up path aggregation layer configured to output the plurality of second common feature maps by convolving and summing outputs of the middle layer
including,
The plurality of decoding layers correspond to the bottom-up path merging layer,
object detection method.
According to claim 1,
The object detection model,
an object region header for converting the plurality of second common feature maps output from the plurality of decoding layers into an object region feature map for object region detection; and
An object classification header for converting the plurality of second common feature maps into an object classification feature map for object classification.
including,
object detection method.
According to claim 4,
The object domain header and the object classification header include a plurality of convolutional layers that share weights with each other.
object detection method.
According to claim 1,
In the step of performing the E-NMS operation for each decoding layer, the first operation,
Determining the initial value based on a function for object IOU prediction information of the candidate object;
object detection method.
According to claim 1,
The second calculation is,
sorting object region prediction information of the at least one candidate object based on the initial value;
an operation for determining an IOU between two different pieces of object area prediction information for the at least one candidate object;
an operation to re-determine the IOU based on a predefined weight function; and
An operation for determining the confidence score based on the re-determined IOU
including,
object detection method.
According to claim 7,
The operation of determining the IOU, the operation of re-determining the IOU, and the operation of determining the confidence score can be performed in parallel using an IOU matrix,
object detection method.
According to claim 4,
The object detection model is
outputting object size prediction information (pred_reward_scale) based on the object region feature map;
Outputting object classification accuracy prediction information (pred_reward_identification) based on the object classification feature map,
object detection method.
According to claim 9,
The object size prediction information and the object classification accuracy prediction information are information used in the learning process of the object detection model and not used in the inference process.
object detection method.
a memory for storing an object detection model including a plurality of decoding layers; and
one or more processors
including,
the processor,
Obtaining at least one object information about at least one candidate object from an input image for each decoding layer of the plurality of decoding layers by using the object detection model;
Performing E-NMS (Efficient Non-Maximum Suppression) operation for each decoding layer based on the at least one object information obtained for each decoding layer,
configured to re-perform the E-NMS operation based on object information obtained by combining the result of the E-NMS operation for each decoding layer;
The object information,
Object domain prediction information, object IOU prediction information, and object classification prediction information
including,
The E-NMS calculation,
a first operation for determining an initial value of a confidence score for object region prediction information of the at least one candidate object;
a second operation of updating the confidence score based on object region prediction information and object IOU prediction information of the at least one candidate object; and
a third operation for determining a final object region of the at least one candidate object based on the updated confidence score;
including,
object detection device.
According to claim 11,
The object detection model,
a common encoder including a series of encoding layers for extracting a plurality of first common feature maps from the input image; and
A common decoder including multiple decoding layers generating a plurality of second common feature maps by multi-scale feature fusion of the plurality of first common feature maps.
containing
object detection device.
According to claim 12,
The multiple decoding layers,
a top-down layer that upsamples and adds a portion of the plurality of first common feature maps and outputs them;
an intermediate layer that performs a convolution operation on outputs of the top-down layer and other than some of the plurality of first common feature maps; and
A bottom-up path aggregation layer configured to output the plurality of second common feature maps by convolving and summing outputs of the middle layer
including,
The plurality of decoding layers correspond to the bottom-up path merging layer,
object detection device.
According to claim 11,
The object detection model,
An object region header for converting a plurality of second common feature maps output from the plurality of decoding layers into an object region feature map for object region detection; and
An object classification header for converting the plurality of second common feature maps into an object classification feature map for object classification.
including,
object detection device.
15. The method of claim 14,
The object domain header and the object classification header include a plurality of convolutional layers that share weights with each other.
object detection device.
According to claim 11,
the processor,
In order to perform the E-NMS operation for each decoding layer,
In the first operation, the initial value is determined based on a function for object IOU prediction information of the candidate object.
object detection device.
According to claim 11,
The second calculation is,
sorting object region prediction information of the at least one candidate object based on the initial value;
an operation for determining an IOU between two different pieces of object area prediction information for the at least one candidate object;
an operation to re-determine the IOU based on a predefined weight function; and
An operation for determining the confidence score based on the re-determined IOU
including,
object detection device.
18. The method of claim 17,
the processor,
Capable of parallelly performing an operation for determining the IOU, an operation for re-determining the IOU, and an operation for determining the confidence score using an IOU matrix,
object detection device.
15. The method of claim 14,
The processor executes the object detection model,
Extracting object size prediction information (pred_reward_scale) based on the object region feature map;
Is configured to extract object classification accuracy prediction information (pred_reward_identification) based on the object classification feature map,
object detection device.
According to claim 19,
the processor,
The object size prediction information and the object classification accuracy prediction information are used in a learning process of the object detection model and configured not to be used in an inference process.
object detection device.

Images