KR20220069336A - Method and apparatus for object detection - Google Patents

Abstract

An object detection method and apparatus using a high-performance, low-computational object detection model are provided. Accordingly, robust object detection can be performed in various object shapes and size changes. In addition, by providing an efficient post-processing process, the accuracy of object area determination is improved. The object detection method may include the steps of: obtaining at least one piece of object information; performing an effective non-maximum suppression (ENMS) operation; and re-performing the ENMS operation.

Description

Object detection method and apparatus

The present invention relates to a method and apparatus for detecting an object, and more particularly, to a method and apparatus for detecting an object from an input image using an object detection model.

The content to be described below is only provided for the purpose of providing background information related to an embodiment of the present invention, and the content to be described does not naturally constitute the prior art.

Deep learning-based object detection models are largely divided into two-stage detection models and one-stage detection models.

The two-stage detection model consists of a module for generating Region of Interest (RoI) and a module for estimating object class information and location information of RoI through RoI sampling. One-stage detection model consists of one module that estimates object class and location information at once without RoI sampling process.

In terms of performance, the two-stage detection model shows excellent results, but in terms of speed, the one-stage detection model shows excellent results.

On the other hand, the conventional object detection model performs object detection by defining the aspect ratio and basic object size values (Anchors) as a detection template in advance in order to detect objects of various sizes. Recently, the one-stage model shows excellent object detection performance even without applying an anchor through research on the multi-scale feature fusion technique, but many learning parameters/operations for multi-scale fusion are required.

To develop a high-performance, low-computational object detection deep learning model, a one-stage deep learning model structure that does not require an anchor is required. In addition, there is a need for a feature decoding method that can express features robust to various size/shape changes, and a learning method based on it. Furthermore, there is a need for an efficient NMS post-processing method that can remove region redundancy for predicted object regions.

An object of the present invention is to provide an object detection method and apparatus based on an object detection model that does not require an anchor.

An object of the present invention is to provide an object detection model that is robust to changes in the size and shape of an object.

It is an object of the present invention to provide an efficient NMS post-processing method capable of removing region redundancy for predicted object regions.

The object of the present invention is not limited to the above-mentioned problems, and other objects and advantages of the present invention that are not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be appreciated that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations thereof indicated in the claims.

In an object detection method according to an embodiment of the present invention, at least one object information on at least one candidate object from an input image for each decoding layer of a plurality of decoding layers using an object detection model including a plurality of decoding layers obtaining, performing an Efficient Non-Maximum Suppression (E-NMS) operation for each decoding layer based on at least one object information obtained from each decoding layer, and E-NMS operation for each decoding layer It may include re-performing the E-NMS operation based on the result.

For example, the object information may include object region prediction information, object intersection over union (IOU) prediction information, and object classification prediction information.

For example, the E-NMS operation is based on a first operation for determining an initial value of a confidence score for object region prediction information of at least one candidate object, object region prediction information of at least one candidate object, and object IOU prediction information. and a second operation of updating the confidence score and a third operation of determining a final object region of the at least one candidate object based on the updated confidence score.

An object detection apparatus according to an embodiment of the present invention includes a memory for storing an object detection model including a plurality of decoding layers and one or more processors, and the processor includes: each of the plurality of decoding layers using the object detection model At least one object information on at least one candidate object is obtained from an input image for each decoding layer, and Efficient Non-Maximum Suppression (E-NMS) for each decoding layer is obtained based on the at least one object information obtained for each decoding layer. ) operation, and re-performing the E-NMS operation based on the result of the NMS operation for each decoding layer.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims and detailed description.

According to an embodiment of the present invention, a high-performance, low-computational object detection model that does not require an anchor is provided.

According to an embodiment of the present invention, object detection robust to changes in various object shapes and sizes is possible.

According to an embodiment of the present invention, the accuracy of object region determination is improved by providing an efficient post-processing process (E-NMS).

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary diagram schematically illustrating an object detection process according to an embodiment.
2 is a block diagram of an object detecting apparatus according to an embodiment.
3 is a flowchart of a method for detecting an object according to an embodiment.
4 is a schematic illustration of an object detection model according to an embodiment.
5 is a block diagram of an object detection model according to an embodiment.
6 is a block diagram of a common decoder of an object detection model according to an embodiment.
7 is a configuration diagram of a header and a decoder of an object detection model according to an embodiment.
8 is a schematic illustration of an E-NMS operation according to an embodiment.
9 is a flowchart of an E-NMS operation according to an embodiment.
10 is a flowchart of a second operation of an E-NMS operation according to an embodiment.
11 is an exemplary diagram of an IOU matrix according to an embodiment.
12 is a diagram illustrating an object detection result of an object detection model according to an exemplary embodiment.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The present invention may be implemented in several different forms, and is not limited to the embodiments described herein. In the following embodiments, parts not directly related to the description are omitted in order to clearly explain the present invention, but it does not mean that the omitted configuration is unnecessary in implementing the device or system to which the spirit of the present invention is applied . In addition, the same reference numerals are used for the same or similar elements throughout the specification.

In the following description, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms, and the terms refer to one component from another component. It is used only for distinguishing purposes. Also, in the following description, the singular expression includes the plural expression unless the context clearly dictates otherwise.

In the following description, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, but one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is an exemplary diagram schematically illustrating an object detection process according to an embodiment.

The object detecting apparatus 100 according to an embodiment receives an input image. For example, the input image may be directly acquired using a camera connected to the object detection apparatus 100 or received from an external device through a communication network. Here, the input image includes a still image and a moving image.

The object detection apparatus 100 analyzes the received input image using the object detection model. The object detection model is a deep learning model for obtaining object information from an input image, and is an image-based object detection model. The object detection model will be described later with reference to FIGS. 4 to 7 .

In an example, the object information may include object region prediction information, IOU (Intersection Over Union) prediction information, and object classification prediction information.

The object region prediction information means prediction information on an image region corresponding to an object in the input image.

The object IOU prediction information means information about the degree of overlap between the object area prediction information and the ground truth (GT) information. For example, the object IOU prediction information is the size of the area that is the intersection of the object area prediction information and the ground truth (GT) information for the area of the object, the object area prediction information and the ground truth (GT) for the area of the object It may be determined based on a value divided by the size of a region that is a union of information.

The object classification prediction information refers to information probabilistically predicting what kind of object the predicted object area will be.

Additionally, the object information may further include object size prediction information and object classification accuracy prediction information. This will be described later with reference to FIG. 7 .

The object detection apparatus 100 corresponds to various types of electronic devices capable of executing an object detection model.

In an example, the object detection apparatus 100 may be mounted on a vehicle. In one example, the object detecting apparatus 100 may be an electronic device capable of communicating with a vehicle. For example, the object detection apparatus 100 may include a server connected to the vehicle through a network. In an example, the object detection apparatus 100 may include a robot and a terminal device such as a smart phone, but is not limited thereto.

2 is a block diagram of an object detecting apparatus according to an embodiment.

The object detecting apparatus 100 according to the embodiment may include a processor 110 and a memory 120 . The components shown in FIG. 2 are exemplary, and the object detecting apparatus 100 may further include additional components.

The object detection apparatus 100 may include a processor 150 .

The processor 110 is a kind of central processing unit, and may execute one or more instructions stored in the memory 120 to control the operation of the object detection apparatus 100 . The processor 110 may include any type of device capable of processing data by executing instructions.

The processor 110 may refer to, for example, a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing apparatus embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but is not limited thereto. The processor 110 may include one or more processors.

Additionally, the object detection apparatus 100 may include a memory 120 .

The memory 120 may store an object detection model for obtaining object information. The memory 120 may store a command for a process in which the object detection apparatus 100 executes an object detection model.

The processor 110 may execute an object detection process according to an embodiment based on a program, instructions, and an object detection model stored in the memory 120 .

The memory 120 may further store intermediate data and operation results that are calculated by an algorithm and are generated in the operation process for object detection.

Memory 120 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND Flash memory, or non-volatile memory such as NOR flash memory, SSD, compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or flash drive such as a memory stick; Alternatively, it may include a storage device such as an HDD. The memory 120 may include, but is not limited to, magnetic storage media or flash storage media.

3 is a flowchart of a method for detecting an object according to an embodiment.

An object detection method according to an embodiment includes obtaining at least one object information about at least one candidate object from an input image for each decoding layer of a plurality of decoding layers using an object detection model including a plurality of decoding layers ( S10), performing an Efficient Non-Maximum Suppression (E-NMS) operation for each decoding layer based on at least one object information obtained from each decoding layer (S20) and E-NMS operation for each decoding layer It may include a step (S30) of re-performing the E-NMS operation based on the result of .

In step S10, the processor 110 obtains at least one object information about at least one candidate object from the input image for each decoding layer of the plurality of decoding layers using an object detection model including a plurality of decoding layers. can

Here, the plurality of decoding layers correspond to the bottom-up path merge layer of the common decoder 2, which will be described later with reference to FIG. 5 .

In an example, the object information may include object region prediction information (pred_box_posts), IOU (Intersection Over Union) prediction information (pred_reward_iou), and object classification prediction information (pred_confidence).

In step S20 , the processor 110 may perform an E-NMS operation for each decoding layer based on at least one object information obtained from each decoding layer in step S10 .

In step S20, the processor 110 performs an E-NMS operation for each decoding layer. That is, in step S20 , the processor 110 performs a layer-wise E-NMS operation.

The E-NMS operation includes a first operation for determining an initial value of the confidence score of the at least one candidate object, and a second operation for updating the confidence score based on the object region prediction information and the object IOU prediction information of the at least one candidate object. and a third operation of determining a final object region of the at least one candidate object based on the updated confidence score. The E-NMS operation according to the embodiment will be described later in detail with reference to FIGS. 8 to 11 .

In step S30 , the processor 110 may re-perform the E-NMS operation based on the result of the E-NMS operation for each decoding layer in step S20 .

That is, in step S30 , the processor 110 may combine the results of the E-NMS operation performed for each decoding layer in step S20 , and re-perform the E-NMS operation based on the combined object information.

Hereinafter, an object detection model according to an embodiment will be described with reference to FIGS. 4 to 7 .

4 is a schematic illustration of an object detection model according to an embodiment.

The object detection model according to the embodiment includes a header 3 and an object region decoder 4 including a common encoder 1, a common decoder 2, an object region header 3-1, and an object classification header 3-2. -1) and a decoder 4 including an object classification decoder 4-2.

The object detection model is an object detection deep learning model based on a Fully-Convolutional Neural Network.

The common encoder 1 and the common decoder 2 of the object detection model are located in front of the header 3 and the decoder 4 implemented for each task (eg, an object region prediction task and an object classification prediction task), so that the input image Extract the common feature map from

That is, the object detection model can remove the computational redundancy for encoding and decoding the features of the input image through the common encoder (1) and common decoder (2) structures shared between tasks.

In the object detection model, the header 3 and the decoder 4 may be implemented for each task related to object detection. That is, the header 3 and the decoder 4 are implemented with layers and operations suitable for object information to be detected by each task.

The header 3 converts all or part of the feature map output through the common encoder 1 and the common decoder 2 into a feature map suitable for the decoder 4 . The decoder 4 receives the feature map converted from the header 3 and outputs object information to be detected.

The object detection model according to the embodiment connects the additional header 3 and the decoder 4 to the common encoder 1 and the common decoder 2 to easily plug additional tasks in addition to the above-described object region prediction task and object classification prediction task. - It is designed with a plug-in/out structure.

Hereinafter, the structure of the object detection model will be described in more detail with reference to FIG. 5 .

5 is a block diagram of an object detection model according to an embodiment.

The common encoder 1 may include a series of encoding layers Conv1, Conv2, Conv3, Conv4, Conv5, Conv6 and Conv7. The common encoder 1 outputs a plurality of first common feature maps obtained by encoding features of an input image through a series of encoding layers. The plurality of first common feature maps correspond to feature maps output from each layer of the series of encoding layers.

The common encoder 1 may include Conv1 to Conv5 encoding layers. Conv1 to Conv5 encoding layers may apply a structure such as VGGNet, ResNet, XceptionNet, ResnetXT, SuffleNet or MobileNet.

The common encoder 1 may additionally include Conv6 and Conv7 encoding layers. This is a structure that can express a large object/region feature for an input image well and is necessary for multi-scale feature fusion in the common decoder 2 .

The common decoder 2 may include multiple decoding layers. The multiple decoding layer may include a feature pyramid network (FPN) structure, which is a decoder structure for object detection, and a bottom-up path aggregation layer (BPA).

The common decoder 2 outputs a plurality of second common feature maps P3, P4, P5, P6 and P7 through multiple decoding layers. The plurality of second common feature maps correspond to feature maps output from each layer of the BPA layer.

The plurality of second common feature maps P3 , P4 , P5 , P6 , and P7 output from the common decoder 2 has a characteristic capable of robust feature extraction to changes in the size and shape of various objects. This causes the common decoder 2 to generate a plurality of second common feature maps P3, P4, P5, P6 and P7 in a multiscale feature fusion manner, and such a plurality of second common feature maps P3, P4, P5 , P6 and P7), it is possible because the header 3 and the decoder 4, which will be described later, operate. The header 3 constructs an input feature map based on at least a part of the plurality of second common feature maps P3 , P4 , P5 , P6 and P7 , and provides it to the decoder 4 .

The plurality of second common feature maps P3, P4, P5, P6 and P7 output from the common decoder 2 includes a header 3-1 implemented for object region prediction and a header implemented for object classification prediction ( 3-2), respectively. The header 3 and the decoder 4 will be described later with reference to FIG. 7 .

Hereinafter, the common decoder 2 will be described in detail with reference to FIG. 6 .

6 is a block diagram of a common decoder of an object detection model according to an embodiment.

The common decoder 2 includes multiple decoding layers for extracting a plurality of first common feature maps through multi-scale feature fusion.

The multiple decoding layer upsampling and adds a portion (C3, C4, C5) of a plurality of first common feature maps (C3, C4, C5, C6, and C7) output from the common encoder (1) A top-down layer 2-1, which is output by performing , C7) and the output of the intermediate layer 2-2 and the intermediate layer 2-2, which perform a convolution operation on the output of the top-down layer 2-1, by convolution and summing the output of the plurality of second A bottom-up path aggregation layer 2-3 for outputting the common feature maps P3, P4, P5, P6 and P7 may be included.

The top-down layer 2-1 upsampling and summing a portion of the plurality of first common feature maps output from the common encoder 1 and outputs the upsampling and summing.

In one example, the top-down layer includes three feature maps C3, C4 and C5) is the target.

Step 1 of the top-down layer 2-1 is a feature map C5' generated by 1x1x256 convolution of the first common feature map C5 output from the fifth encoding layer Conv5 of the common encoder 1 ) is output. In step 2 of the top-down layer 2-1, the result of upsampling the output C5' of step 1 is doubled and the first common feature map output from the fourth encoding layer Conv4 of the common encoder 1 The feature map (C4') generated by 1x1x256 convolution of (C4) is summed (C4' + C5') and output.

In the same way, step 3 of the top-down layer 2-1 is the result of upsampling the output (C4' + C5') of step 2 by 2 and output from the third encoding layer (Conv3) of the common encoder 1 The feature map C3' generated by 1x1x256 convolution of the first common feature map C3 is summed (C3' + C4' + C5') and output. Through this, low-resolution and high-resolution features can be cumulatively fused in the top-down direction.

The middle layer (2-2) includes the remainder (C6 and C7) and the top-down layer (2-) except for a part of the plurality of first common feature maps (C3, C4, C5, C6 and C7) in the common encoder (1) A convolution operation is performed on the output of 1).

The middle layer 2-2 includes two unused first common feature maps C3, C4, C5, C6, and C7 output from the common encoder 1 in the top-down layer 2-1. 3x3x256 convolutions for the feature maps C6 and C7, respectively. That is, the intermediate layer 2-2 3x3x256 convolves the first common feature map C7 output from the last encoding layer C7 of the common encoder 1 to output the intermediate feature map M7. Similarly, the intermediate layer 2-2 performs 3x3x256 convolution on the first common feature map C6 output from the sixth encoding layer C6 of the common encoder 1 to output the intermediate feature map M6.

The intermediate layer 2-2 outputs intermediate feature maps M5, M4, and P3 by performing 3x3x256 convolution on the output of the top-down layer 2-1, respectively.

The bottom-up path merging layer 2-3 outputs a plurality of second common feature maps P3, P4, P5, P6 and P7 by convolution and summing the output of the intermediate layer 2-2. The bottom-up path merging layer 2-3 generates a plurality of second common feature maps P3 , P4 , P5 , P6 and P7 while proceeding from the lowest layer to the highest layer of the pyramid.

The bottom-up path merging layer (2-3) generates a low-resolution feature map (P3') by 3x3x256, stride=2 convolution of the P3 intermediate feature map output from the intermediate layer (2-2), and the intermediate layer ( A second common feature map P4 is generated by summing (P3'+M4) with the intermediate feature map M4 output in 2-2). The remaining second common feature maps P4 , P6 and P7 are generated in the same manner with respect to the generated second common feature map P4 .

In this specification, a plurality of decoding layers means a final decoding layer of the common decoder 2 . For example, the plurality of decoding layers may correspond to the bottom-up path merging layer 2-3.

The plurality of second common feature maps P3 , P4 , P5 , P6 and P7 include a series of multi-scale feature maps output from the bottom-up path merging layer 2-3 . That is, a series of multi-scale feature maps can be generated by cumulatively merging low-resolution and high-resolution features in the bottom-up direction by the bottom-up path merging layer 2-3, which includes various objects in a complex manner. It becomes possible to express characteristics that are robust to changes in the size and shape of images and objects.

7 is a configuration diagram of a header and a decoder of an object detection model according to an embodiment.

In the object detection model according to the embodiment, the plurality of second common feature maps P3, P4, P5, P6, P7) is input to the header 3 with reference to FIG.

The object detection model according to the embodiment includes an object region header 3-1 and an object classification header 3-2.

The object region header 3-1 is a domain adaptation layer that converts a plurality of second common feature maps P3, P4, P5, P6 and P7 output from a plurality of decoding layers into an input feature map for object region detection. it means.

The object classification header 3-2 is a kind of domain adaptation layer that converts the plurality of second common feature maps P3, P4, P5, P6 and P7 generated in the plurality of decoding layers into an input feature map for object classification. it means.

As described above, the plurality of decoding layers corresponds to the bottom-up path merging layer 2-3 of the common decoder 2 described with reference to FIG. 6 .

The header 3 for object detection is to input a plurality of second common feature maps P3, P4, P5, P6 and P7 through a domain adaptation layer to the decoder 4 for object detection, respectively. Convert it to an input feature map. Here, the input feature map includes an object domain feature map and an object classification feature map.

The header 3 is composed of an object area header 3-1 and an object classification header 3-2. The object region header (box_feature) and the object classification header (class_feature) are the plurality of second common feature maps (P3, P4, P5, P6, and P7) output from the common decoder 2 using the same parameters (Sharing Layer). An object domain feature map expressing object domain characteristics and an object classification feature map representing object classification characteristics are generated.

In an example, the object region header 3 - 1 and the object classification header 3 - 2 may include a plurality of convolutional layers sharing a weight with each other.

In one example, the object region header 3-1 and the object classification header 3-2 are a plurality of second common feature maps output from the bottom-up path merging layer 2-3 of the common decoder 2, respectively. 3x3x256 convolution is performed 4 times for (P3, P4, P5, P6, and P7) to generate an object domain feature map and an object classification feature map, respectively.

The object region header 3-1 and the object classification header 3-2 use the same parameter to form a plurality of second common feature maps P3, P4, P5, P6 and P7 output from a plurality of decoding layers. Multi-scale features are extracted as an object domain feature map and an object classification feature map.

The decoder 4 of the object detection model predicts at least one piece of object information using the input feature map generated from the header 3 .

In one example, the decoder 4 may use an anchor-free Multi-Scale Fully-Convolutional Neural Network (MS-FCN), and may secure high performance and high efficiency by eliminating redundant and additional operations.

The object region feature map is used as an input of the object region decoder 4-1 that predicts the object region prediction information (pred_box_posts), the object IOU prediction information (pred_reward_iou), and the object size prediction information (pred_reward_scale).

The object area decoder 4-1 performs 3x3x4 (x1, y1, x2, y2: an object away from the current grid position) convolution on the object area feature map to generate object boxes (pred_boxes) and generate Object area prediction information (pred_box_posts) is generated through the multiplication operation of the object boxes (pred_boxes) and the scale factor.

In one example, the scale factor is a constant learned to increase the object region prediction size value, and may be learned based on the object size prediction information (pred_reward_scale).

The object region decoder 4-1 generates object IOU prediction information (pred_reward_iou) and object size prediction information (pred_reward_scale) through 1x1x256 convolution and 1x1x1 convolution with respect to the object region feature map, respectively.

The object IOU prediction information (pred_reward_iou) predicts how much the predicted object area (pred_box_posts) matches the actual correct answer object area, so it is possible to increase the object detection accuracy during learning. The object area detection performance can be improved by applying it to E-NMS processing.

The object classification feature map is used as an input of the object classification decoder 4-2 that predicts object classification prediction information (pred_confidence) and object classification accuracy prediction information (pred_reward_identificatoin).

The object classification decoder 4-2 generates object classification prediction information (pred_confidence) through 3x3xN (N: background + number of classification objects) convolution with respect to the object classification feature map.

Meanwhile, the object classification accuracy prediction information (pred_reward_identification) and the object size prediction information (pred_reward_scale) are used for learning the object detection model, but are not used in the actual reasoning process of the object detection model.

That is, the object classification accuracy prediction information (pred_reward_identification) and the object size prediction information (pred_reward_scale) are virtual auxiliary tasks for helping the learning of the object detection model, and are not used in the actual reasoning process.

Here, the virtual auxiliary task calculates a loss function through prediction during training and reflects it in the learning parameter update to help the object information learn well, and does not predict the object information in execution mode. It means a method that does not increase the computational load.

The object size prediction information (pred_reward_scale) is information that predicts whether the size of an object is accurately predicted. When learning, the object area prediction performance is strengthened and the object The detection accuracy is improved.

The object classification accuracy prediction information (pred_reward_identification) is information for predicting whether the object classification information is accurately predicted, and may enhance object classification prediction performance during learning.

The processor 110 may determine the final object region through N-NMS-based post-processing of the object information obtained using the decoder 4 .

In an example, the processor 110 inputs the object region prediction information (pred_box_posts), the object IOU prediction information (pred_reward_iou), and the object classification prediction information (pred_confidence) into an N-NMS operation to be described later with reference to FIG. 8 to input the final object region can be selected.

8 is a schematic illustration of an E-NMS operation according to an embodiment.

The object detection method according to the embodiment proposes an E-NMS operation for post-processing of object information output from the object detection model.

The E-NMS operation calculates the final object area for at least one candidate object predicted to be included in the input image based on the object classification prediction information (pred_confidence), the object IOU prediction information (pred_reward_iou), and the object area prediction information (pred_box_posts) Perform post-processing to determine.

The object detection method according to the embodiment executes the E-NMS operation in two steps. That is, with reference to FIG. 3 , the processor 110 performs the E-NMS operation in steps S20 and S30 .

In step S20, an E-NMS operation is performed for each decoding layer (Layer-wise E-NMS). In step S30, the entire E-NMS operation is performed based on the result of the E-NMS operation for each layer in step S20.

Hereinafter, the E-NMS operation will be described in more detail with reference to FIGS. 9 and 10 .

9 is a flowchart of an E-NMS operation according to an embodiment.

The E-NMS operation is based on a first operation ( S31 ) of determining an initial value of the confidence score of at least one candidate object, the object area prediction information (pred_box_posts) and the object IOU prediction information (pred_reward_iou) of the at least one candidate object It may include a second operation ( S32 ) of updating the confidence score and a third operation ( S33 ) of determining a final object region of the at least one candidate object based on the updated confidence score.

In the first operation ( S31 ), the processor 110 determines an initial value of the confidence score of at least one candidate object.

In one example, when the processor 110 performs the E-NMS operation for step S20 , in the first operation S31 , the initial value of the confidence score is based on a function for the object IOU prediction information (pred_reward_iou). can be decided

In this case, the initial value of the confidence score may be expressed by the following Equation (1).

Here, Confidence in the right term means object classification prediction information (pred_confidence) output from the object classification decoder 4-2 of the object detection model.

f(pred_reward_iou) is a weight factor based on the object IOU prediction information (pred_reward_iou), and various functions can be applied. For example, f(x)= x ^0.8 can be used as f(pred_reward_iou).

The initial value setting enables E-NMS to be performed based on the fusion information of the object classification prediction information (pred_confidence) and the object IOU prediction information (pred_reward_iou).

In one example, when the processor 110 performs the E-NMS operation with respect to step S30, in the first operation S31, based on the result of the E-NMS operation for each decoding layer of step S20, An initial value of the confidence score may be determined.

For example, the processor 110 calculates the updated confidence score while performing the second operation S32 of the E-NMS operation for each decoding layer in step S20, the first of the E-NMS operation of step S30. It can be determined as the initial value of the confidence score in the operation (S31).

In the second operation ( S32 ), the processor 110 updates the confidence score based on the object region prediction information (pred_box_posts) and the object IOU prediction information (pred_reward_iou) of the at least one candidate object. The second operation S32 will be described with reference to FIG. 10 .

In the third operation ( S33 ), the processor 110 determines a final object area of at least one candidate object based on the updated confidence score.

For example, the processor 110 may determine a region having the largest updated confidence score among regions corresponding to the object region prediction information pred_box_posts as the final object region of the corresponding object.

As a result, the processor 110 may determine the final object area for at least one candidate object predicted to be included in the input image by executing the E-NMS operation.

10 is a flowchart of a second operation of an E-NMS operation according to an embodiment.

The processor 110 executes the following operation to execute the second operation S32 of the E-NMS operation. That is, the second operation S32 is based on the initial value determined in the first operation S31. operation to align at least one candidate object (S32-1), an operation to determine the IOU between two different object region prediction information for at least one candidate object (S32-2), based on a predefined weight function and an operation (S32-3) of re-determining the IOU and an operation (S32-4) of determining the confidence score of at least one candidate object based on the re-determined IOU.

In operation S32-1, the processor 110 sorts the object region prediction information pred_box_posts for at least one candidate object based on the confidence score determined in the first operation S31. For example, the object region prediction information (pred_box_posts) for at least one candidate object may be sorted based on the descending order of the confidence scores of the candidate objects.

In one example, the processor 110 performs an operation for determining an IOU (S32-2), an operation for re-determining an IOU (S32-3), and an operation for determining a confidence score (S32-4) using an IOU matrix in parallel can be performed with

The calculation method using the IOU matrix is a method of generating and calculating a descending triangular IOU matrix between predicted object areas, and is a method that improves the ambiguity of overlap between different objects and the efficiency of parallel operation.

The IOU matrix is a square matrix having as many rows and columns as the number of object region prediction information (pred_box_posts). The E-NMS operation based on the IOU matrix will be described below.

In operation S32-2, the processor 110 determines an IOU between two pieces of different object region prediction information (pred_box_posts) for at least one candidate object.

1) In operation S32-2, the processor 110 determines an IOU between Box i, which is the i-th object area prediction information, and Box _j , which is the j-th object area prediction information, and sets the determined IOU to _j of the i-th row of the IOU matrix. It is stored in the matrix element b _ij of the th row. Here, the IOU value of b _ii with the same row and column number becomes 1.

2) The value of b _ij (i<j) located in the ascending triangle with respect to b _ii is set to 0. The IOU value of b _ii with the same row and column number is also set to 0.

3) To eliminate performance degradation due to overlap between different candidate objects, if the vertical/horizontal objects are not the same for each IOU Matrix, the corresponding matrix element is set to 0.

In operation S32-3, the processor 110 re-determines the IOU based on a predefined weight function.

Equation 2 below is an exemplary predefined weight function.

Here, n is a linear weight function, the IOU in the right term is the IOU stored in the current IOU matrix, and the IOU in the left term indicates the IOU re-determined according to Equation (2).

In operation S32-4, the processor 110 determines a confidence score based on the re-determined IOU.

The processor 110 determines a confidence score for an object region corresponding to the object region prediction information (pred_box_posts) of at least one candidate object based on Equation 3 below.

Here, conf _n means the confidence score of the nth object area (Box _n ).

11 is an exemplary diagram of an IOU matrix according to an embodiment.

The left diagram exemplifies the IOU matrix for storing the IOU re-determined in the operation (S32-3).

The right diagram shows that bi is calculated by multiplying the IOU value of the i-th row in order to determine the confidence score for each object area b _i according to Equation 3.

12 is a diagram illustrating an object detection result of an object detection model according to an exemplary embodiment.

It can be seen that objects of various shapes and sizes are detected well, and the object area is accurately specified even if there is very close proximity or occlusion.

Hereinafter, a learning process of an object detection model according to an embodiment will be described.

A loss function for learning an object detection model according to an embodiment includes an object classification error (class), an object area estimation error (box), an IOU estimation error (iou), an object size accuracy prediction error (scale), and Object classification accuracy prediction (id) was considered. In addition, the final loss for learning can be obtained as a weighted sum of these four errors.

(1) Class error calculation uses a method using focal loss as in Equation 4.

(2) The object region prediction error calculation uses the GIOU loss as in Equation 5.

(3) Equation (6) is used for the IOU estimation error (iou).

(4) Object size prediction accuracy The prediction error (scale) can be obtained using Equation (7).

(5) The object classification accuracy prediction error (id) can be obtained using Equation (8).

where u is a uni-step function.

(6) Finally, the final error function required for model learning is calculated as the weighted sum of the error values obtained in (1) to (4) as in Equation 9.

Hereinafter, artificial intelligence technology related to an embodiment of the present invention will be described.

Artificial intelligence (AI) is a field of computer engineering and information technology that studies how computers can do the thinking, learning, and self-development that can be done with human intelligence. This means that the behavior can be imitated.

In addition, artificial intelligence does not exist by itself, but is directly or indirectly related to other fields of computer science. In particular, in modern times, attempts are being made to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in that field.

Machine learning is a branch of artificial intelligence, a field of study that gives computers the ability to learn without an explicit program.

Specifically, machine learning can be said to be a technology to study and build a system and algorithms for learning based on empirical data, making predictions, and improving its own performance. Machine learning algorithms build specific models to make predictions or decisions based on input data, rather than executing strictly set static program instructions.

The term 'machine learning' may be used interchangeably with the term 'machine learning'.

With regard to how to classify data in machine learning, many machine learning algorithms have been developed. Decision trees, Bayesian networks, support vector machines (SVMs), and artificial neural networks (ANNs) are representative examples.

Decision tree is an analysis method that performs classification and prediction by charting decision rules in a tree structure.

The Bayesian network is a model that expresses the probabilistic relationship (conditional independence) between multiple variables in a graph structure. Bayesian networks are suitable for data mining through unsupervised learning.

The support vector machine is a model of supervised learning for pattern recognition and data analysis, and is mainly used for classification and regression analysis.

An artificial neural network is an information processing system in which a number of neurons called nodes or processing elements are connected in the form of a layer structure by modeling the operating principle of biological neurons and the connection relationship between neurons.

Artificial neural network is a model used in machine learning, and it is a statistical learning algorithm inspired by neural networks in biology (especially the brain in the central nervous system of animals) in machine learning and cognitive science.

Specifically, the artificial neural network may refer to an overall model having problem-solving ability by changing the bonding strength of synapses through learning in which artificial neurons (nodes) that form a network by combining synapses.

The term artificial neural network may be used interchangeably with the term neural network.

The artificial neural network may include a plurality of layers, and each of the layers may include a plurality of neurons. Also, the artificial neural network may include neurons and synapses connecting neurons.

In general, artificial neural networks calculate the output value from the following three factors: (1) the connection pattern between neurons in different layers (2) the learning process to update the weight of the connection (3) the weighted sum of the input received from the previous layer It can be defined by the activation function it creates.

The artificial neural network may include network models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Bidirectional Recurrent Deep Neural Network (BRDNN), Multilayer Perceptron (MLP), Convolutional Neural Network (CNN). , but is not limited thereto.

In this specification, the term 'layer' may be used interchangeably with the term 'layer'.

Artificial neural networks are divided into single-layer neural networks and multi-layer neural networks according to the number of layers.

A typical single-layer neural network consists of an input layer and an output layer.

In addition, a general multilayer neural network consists of an input layer, one or more hidden layers, and an output layer.

The input layer is a layer that receives external data. The number of neurons in the input layer is the same as the number of input variables, and the hidden layer is located between the input layer and the output layer. do. The output layer receives a signal from the hidden layer and outputs an output value based on the received signal. The input signal between neurons is multiplied by each connection strength (weight) and then summed. If the sum is greater than the neuron threshold, the neuron is activated and the output value obtained through the activation function is output.

Meanwhile, a deep neural network including a plurality of hidden layers between an input layer and an output layer may be a representative artificial neural network that implements deep learning, which is a type of machine learning technology.

Meanwhile, the term 'deep learning' may be used interchangeably with the term 'deep learning'.

The artificial neural network may be trained using training data. Here, learning refers to a process of determining parameters of an artificial neural network using learning data to achieve objectives such as classification, regression, or clustering of input data. can As a representative example of parameters of an artificial neural network, a weight applied to a synapse or a bias applied to a neuron may be mentioned.

The artificial neural network learned by the training data may classify or cluster input data according to a pattern of the input data.

Meanwhile, an artificial neural network trained using training data may be referred to as a trained model in the present specification.

The following describes the learning method of the artificial neural network.

Learning methods of artificial neural networks can be broadly classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

Supervised learning is a method of machine learning to infer a function from training data.

And among these inferred functions, outputting a continuous value is called regression, and predicting and outputting a class of an input vector can be called classification.

In supervised learning, an artificial neural network is trained in a state in which a label for training data is given.

Here, the label may mean a correct answer (or result value) that the artificial neural network should infer when training data is input to the artificial neural network.

In the present specification, when training data is input, the correct answer (or result value) that the artificial neural network must infer is called a label or labeling data.

Also, in this specification, setting a label on the training data for learning of the artificial neural network is called labeling the labeling data on the training data.

In this case, the training data and a label corresponding to the training data) constitute one training set, and may be input to the artificial neural network in the form of a training set.

On the other hand, training data represents a plurality of features, and labeling the training data may mean that the feature represented by the training data is labeled. In this case, the training data may represent the features of the input object in a vector form.

The artificial neural network may infer a function for the relationship between the training data and the labeling data by using the training data and the labeling data. In addition, parameters of the artificial neural network may be determined (optimized) through evaluation of the function inferred from the artificial neural network.

Unsupervised learning is a type of machine learning where no labels are given to training data.

Specifically, the unsupervised learning may be a learning method of learning the artificial neural network to find and classify patterns in the training data itself, rather than the association between the training data and the labels corresponding to the training data.

Examples of unsupervised learning include clustering or independent component analysis.

In this specification, the term 'clustering' may be used interchangeably with the term 'clustering'.

Examples of artificial neural networks using unsupervised learning include a generative adversarial network (GAN) and an autoencoder (AE).

A generative adversarial neural network is a machine learning method in which two different artificial intelligences, a generator and a discriminator, compete to improve performance.

In this case, the generator is a model that creates new data, and can generate new data based on the original data.

In addition, the discriminator is a model for recognizing patterns in data, and may play a role of discriminating whether input data is original data or new data generated by the generator.

And the generator learns by receiving the data that did not deceive the discriminator, and the discriminator can learn by receiving the deceived data from the generator. Accordingly, the generator may evolve to deceive the discriminator as best as possible, and the discriminator may evolve to distinguish the original data and the data generated by the generator well.

An autoencoder is a neural network that aims to reproduce the input itself as an output.

The auto-encoder includes an input layer, at least one hidden layer and an output layer.

In this case, since the number of nodes in the hidden layer is smaller than the number of nodes in the input layer, the dimension of data is reduced, and thus compression or encoding is performed.

Also, the data output from the hidden layer goes into the output layer. In this case, since the number of nodes of the output layer is greater than the number of nodes of the hidden layer, the dimension of data is increased, and decompression or decoding is performed accordingly.

On the other hand, the auto-encoder controls the neuron's connection strength through learning, so that the input data is expressed as hidden layer data. The hidden layer expresses information with fewer neurons than the input layer, and being able to reproduce the input data as an output may mean that the hidden layer found and expressed hidden patterns from the input data.

Semi-supervised learning is a type of machine learning, and may refer to a learning method using both labeled and unlabeled training data.

As one of the techniques of semi-supervised learning, there is a technique of inferring a label of unlabeled training data and then performing learning using the inferred label. can

Reinforcement learning is a theory that, given the environment in which the agent can decide what action to take at every moment, it can find the best way through experience without data.

Reinforcement learning may be mainly performed by a Markov Decision Process (MDP).

To explain the Markov decision process, first, an environment is given in which the information necessary for the agent to take the next action is given, secondly, how the agent behaves in the environment is defined, and thirdly, the agent is rewarded ( reward) and a penalty point for failure to do so, and fourthly, the optimal policy is derived by repeating experiences until the future reward reaches the highest point.

The structure of an artificial neural network is specified by the model configuration, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and hyperparameters are It is set, and then the model parameter is set through learning and the content can be specified.

For example, factors determining the structure of an artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, an input feature vector, a target feature vector, and the like.

Hyperparameters include several parameters that must be initially set for learning, such as initial values of model parameters. And, the model parameter includes several parameters to be determined through learning.

For example, the hyperparameter may include an initial weight value between nodes, an initial bias value between nodes, a mini-batch size, a number of learning repetitions, a learning rate, and the like. In addition, the model parameters may include inter-node weights, inter-node biases, and the like.

The loss function may be used as an index (reference) for determining the optimal model parameter in the learning process of the artificial neural network. In artificial neural networks, learning refers to the process of manipulating model parameters to reduce the loss function, and the purpose of learning can be seen to determine the model parameters that minimize the loss function.

The loss function may mainly use a mean squared error (MSE) or a cross entropy error (CEE), but the present invention is not limited thereto.

The cross-entropy error can be used when the correct answer label is one-hot encoded. One-hot encoding is an encoding method in which the correct label value is set to 1 only for neurons corresponding to the correct answer, and the correct answer label value is set to 0 for neurons that do not have the correct answer.

In machine learning or deep learning, a learning optimization algorithm can be used to minimize the loss function, and learning optimization algorithms include Gradient Descent (GD), Stochastic Gradient Descent (SGD), and Momentum. ), Nesterov Accelerate Gradient (NAG), Adagrad, AdaDelta, RMSProp, Adam, and Nadam.

Gradient descent is a technique that adjusts model parameters in a direction to reduce the loss function value by considering the gradient of the loss function in the current state.

The direction in which the model parameter is adjusted is referred to as a step direction, and the size to be adjusted is referred to as a step size.

In this case, the step size may mean a learning rate.

In the gradient descent method, a gradient is obtained by partial differentiation of the loss function into each model parameter, and the model parameters can be updated by changing the learning rate in the obtained gradient direction.

The stochastic gradient descent method is a technique in which the frequency of gradient descent is increased by dividing the training data into mini-batch and performing gradient descent for each mini-batch.

Adagrad, AdaDelta, and RMSProp are techniques to increase optimization accuracy by adjusting the step size in SGD. In SGD, momentum and NAG are techniques to increase optimization accuracy by adjusting the step direction. Adam is a technique to increase optimization accuracy by adjusting the step size and step direction by combining momentum and RMSProp. Nadam is a technique to increase optimization accuracy by adjusting the step size and step direction by combining NAG and RMSProp.

The learning speed and accuracy of an artificial neural network have a characteristic that it largely depends on hyperparameters as well as the structure of the artificial neural network and the type of learning optimization algorithm. Therefore, in order to obtain a good learning model, it is important not only to determine an appropriate artificial neural network structure and learning algorithm, but also to set appropriate hyperparameters.

Typically, hyperparameters are set to various values experimentally to train the artificial neural network, and as a result of learning, they are set to optimal values that provide stable learning speed and accuracy.

The above-described embodiment according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a solid state disk (SSD), a silicon disk drive (SDD), a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and a floppy disk. magneto-optical media, such as, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

The description of the embodiment of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can easily transform into other specific forms without changing the technical spirit or essential features of the present invention you will be able to understand that Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention.

100 object detection device
110 processor
120 memory

Claims (20)

obtaining at least one object information on at least one candidate object from an input image for each decoding layer of the plurality of decoding layers by using an object detection model including a plurality of decoding layers;
performing an Efficient Non-Maximum Suppression (E-NMS) operation for each decoding layer based on the at least one object information obtained from each decoding layer; and
Re-performing the E-NMS operation based on the result of the E-NMS operation for each decoding layer
including,
The object information is
Object area prediction information, object intersection over union (IOU) prediction information, and object classification prediction information
including,
The E-NMS operation is
a first operation for determining an initial value of a confidence score for object region prediction information of the at least one candidate object;
a second operation of updating the confidence score based on the object region prediction information and the object IOU prediction information of the at least one candidate object; and
A third operation for determining a final object area of the at least one candidate object based on the updated confidence score
containing,
Object detection method.
The method of claim 1,
The object detection model is
a common encoder including a series of encoding layers for extracting a plurality of first common feature maps from the input image; and
a common decoder including multiple decoding layers for generating a plurality of second common feature maps by multi-scale feature fusion for the plurality of first common feature maps;
containing
Object detection method.
3. The method of claim 2,
The multiple decoding layer,
a top-down layer for outputting a portion of the plurality of first common feature maps by upsampling and adding;
an intermediate layer that performs a convolution operation on the remainder except for a portion of the plurality of first common feature maps and the output of the top-down layer; and
Bottom-Up Path Aggregation Layer for outputting the plurality of second common feature maps by convolution and summing outputs of the intermediate layer
including,
The plurality of decoding layers correspond to the bottom-up path merging layer,
Object detection method.
The method of claim 1,
The object detection model is
an object region header for converting a plurality of second common feature maps output from the plurality of decoding layers into an object region feature map for object region detection; and
An object classification header for converting the plurality of second common feature maps into an object classification feature map for object classification
containing,
Object detection method.
5. The method of claim 4,
The object region header and the object classification header include a plurality of convolutional layers sharing a weight with each other,
Object detection method.
The method of claim 1,
In the step of performing the E-NMS operation for each decoding layer, the first operation is
determining the initial value based on a function for the object IOU prediction information of the candidate object,
Object detection method.
The method of claim 1,
The second operation is
an operation of aligning object region prediction information of the at least one candidate object based on the initial value;
an operation for determining an IOU between two different object region prediction information for the at least one candidate object;
an operation of re-determining the IOU based on a predefined weight function; and
Calculation to determine the confidence score based on the re-determined IOU
containing,
Object detection method.
8. The method of claim 7,
The operation for determining the IOU, the operation for re-determining the IOU, and the operation for determining the confidence score can be performed in parallel using an IOU matrix,
Object detection method.
5. The method of claim 4,
The object detection model is
Outputs object size prediction information (pred_reward_scale) based on the object region feature map,
Outputting object classification accuracy prediction information (pred_reward_identification) based on the object classification feature map,
Object detection method.
10. The method of claim 9,
The object size prediction information and the object classification accuracy prediction information are information used in the learning process of the object detection model and not used in the inference process,
Object detection method.
a memory for storing an object detection model including a plurality of decoding layers; and
one or more processors
including,
The processor is
obtaining at least one object information on at least one candidate object from an input image for each decoding layer of the plurality of decoding layers using the object detection model;
Efficient Non-Maximum Suppression (E-NMS) operation is performed on each decoding layer based on the at least one object information obtained for each decoding layer,
configured to re-perform the E-NMS operation based on the result of the NMS operation for each decoding layer,
The object information is
Object area prediction informationObject IOU prediction information and object classification prediction information
including,
The E-NMS operation is
a first operation for determining an initial value of a confidence score for object region prediction information of the at least one candidate object;
a second operation of updating the confidence score based on the object region prediction information and the object IOU prediction information of the at least one candidate object; and
A third operation for determining a final object area of the at least one candidate object based on the updated confidence score
containing,
object detection device.
12. The method of claim 11,
The object detection model is
a common encoder including a series of encoding layers for extracting a plurality of first common feature maps from the input image; and
a common decoder including multiple decoding layers for generating a plurality of second common feature maps by multi-scale feature fusion for the plurality of first common feature maps;
containing
object detection device.
13. The method of claim 12,
The multiple decoding layer,
a top-down layer for outputting a portion of the plurality of first common feature maps by upsampling and adding;
an intermediate layer that performs a convolution operation on the remainder except for a portion of the plurality of first common feature maps and the output of the top-down layer; and
Bottom-Up Path Aggregation Layer for outputting the plurality of second common feature maps by convolution and summing outputs of the intermediate layer
including,
The plurality of decoding layers correspond to the bottom-up path merging layer,
object detection device.
12. The method of claim 11,
The object detection model is
an object region header for converting a plurality of second common feature maps output from the plurality of decoding layers into an object region feature map for object region detection; and
An object classification header for converting the plurality of second common feature maps into an object classification feature map for object classification
containing,
object detection device.
15. The method of claim 14,
The object region header and the object classification header include a plurality of convolutional layers sharing a weight with each other,
object detection device.
12. The method of claim 11,
The processor is
In order to perform an E-NMS operation for each decoding layer,
configured to, in the first operation, determine the initial value based on a function for the object IOU prediction information of the candidate object;
object detection device.
12. The method of claim 11,
The second operation is
an operation of aligning object region prediction information of the at least one candidate object based on the initial value;
an operation for determining an IOU between two different object region prediction information for the at least one candidate object;
an operation of re-determining the IOU based on a predefined weight function; and
Calculation to determine the confidence score based on the re-determined IOU
containing,
object detection device.
18. The method of claim 17,
The processor is
By using the IOU matrix, the operation to determine the IOU, the operation to re-determine the IOU, and the operation to determine the confidence score can be performed in parallel,
object detection device.
15. The method of claim 14,
The processor executes the object detection model,
Extracting object size prediction information (pred_reward_scale) based on the object region feature map,
configured to extract object classification accuracy prediction information (pred_reward_identification) based on the object classification feature map,
object detection device.
20. The method of claim 19,
The processor is
configured to use the object size prediction information and the object classification accuracy prediction information in the learning process of the object detection model, and not to use it in the inference process,
object detection device.

Images