KR102340354B1 - Object Detection Method Using Multiscale Convolution Filter - Google Patents

Abstract

The present invention discloses a method of detecting an object by applying a plurality of convolution filters having different sizes. According to the present invention, an object can be accurately detected in one process regardless of the size of the object by performing convolution by applying a plurality of convolution filters having different preset sizes when detecting an object.

Description

Object Detection Method Using Multiscale Convolution Filter

The present invention relates to an object detection method, in a faster R-CNN (Convolutional Neural Network) based RPN (Regional Proposal Network), by applying a plurality of convolution filters having different sizes to detect an object it's about how

Recently developed detection and recognition techniques based on deep neural networks are very useful for digitizing ancient documents. As the first step of digitizing ancient documents, character detection and localization work is very important for further work of character recognition and translation. Character detection and localization in ancient documents is a difficult task. This is because the condition is bad in terms of form.

In particular, since Chinese characters are composed of several geometric components that can be mistaken for independent characters, the precise region of interest (RoI) of Chinese characters needs to be localized. The conventionally used detection system, Faster R-CNN, is composed of a CNN-based feature extractor and RPN, and the RPN is composed of a 3×3 convolution filter, a confidence score network, and a bounding box prediction network.

However, RPN using 3x3 single-size convolution filter in Faster R-CNN has limitations in localizing characters of various sizes. For example, when localizing larger characters than a 3×3 convolution filter as shown in FIG. In RPN, you see the background as well as the text area. Due to this, there is a problem in that the RPN using the conventional single-size convolution filter cannot localize the exact region of the character.

Also, due to inaccurate localization, such as omitting a part of a character, performance may be greatly reduced in an additional process for character recognition and translation. Therefore, there is a need for an object detection method that improves the conventional RPN in order to find the correct localization of the detected text, that is, the correct bounding coordinate box.

S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, Neural Information Processing System (NIPS), Montreal, Canada, December 2015, pp. .91-99

An object of the present invention is to provide an object detection method capable of accurately detecting an object regardless of the size of the object.

In order to achieve the above object, an object detection method according to an embodiment of the present invention includes the steps of receiving a first feature map extracted from a target image for detecting an object, preset in the input first feature map Forming a second feature map by performing convolution using a convolution filter of size, and detecting the object based on the second feature map, wherein the forming of the second feature map comprises: , characterized in that a plurality of convolution filters having different preset sizes are applied to the first feature map.

Here, in the forming of the second feature map, when the preset size of the convolution filter is N x N, N is an odd number. (here, the unit of N is a pixel)

In addition, in the forming of the second feature map, when the size of the largest object among the objects is L x L, and one pixel of the first feature map corresponds to M pixels of the target image, the plurality of The maximum value of the size of the convolution filter is K x K, and K is characterized as L/M or less. (Here, the units of L and K are pixels)

Also, in the forming of the second feature map, when the size of the smallest object among the objects is P x P and one pixel of the first feature map corresponds to M pixels of the target image, the plurality of The minimum value of the size of the convolution filter is Q x Q, and Q is characterized in that it is greater than or equal to P/M. (Here, the units of P and Q are pixels)

An object detection method according to another embodiment of the present invention includes the steps of: a) receiving a first feature map extracted from a target image to detect an object; b) forming a second feature map by performing convolution on the input first feature map by a plurality of convolution filters having different preset sizes; c) detecting the object based on the second feature map; d) measuring a detection rate and detection accuracy of all objects detected from the target image; e) comparing the detection rates and detection accuracies of all the detected objects with preset target detection rates and target detection accuracies, respectively; and f) as a result of the comparison, if the detection rates and detection accuracy of all the detected objects satisfy the target detection rate and the target detection accuracy or more, the detection of the object is terminated, and, as a result of the comparison, among all the detected objects When the detection rate and detection accuracy of one or more objects are less than the target detection rate and the target detection accuracy, the size of the object having the detection rate and detection accuracy that is the greatest difference from the target detection rate and the target detection accuracy is calculated, and the calculated and repeating steps b) to e) by adding a convolution filter having a size corresponding to the object size.

Here, in step b), when the preset size of the convolution filter is N x N, N is an odd number. (here, the unit of N is a pixel)

Also, in step b), when the size of the largest object among the objects is L × L and one pixel of the first feature map corresponds to M pixel of the target image, the maximum value of the size of the plurality of convolution filters is K x K, and K is characterized as L/M or less. (Here, the units of L and K are pixels)

Also, in step b), when the size of the smallest object among the objects is P x P and one pixel of the first feature map corresponds to M pixel of the target image, the minimum value of the size of the plurality of convolution filters is Q x Q, and Q is characterized in that P/M or more. (Here, the units of P and Q are pixels)

Meanwhile, the present invention is characterized in that it provides a computer-readable recording medium in which a program for implementing an object detection method is recorded.

According to the present invention, by performing convolution by applying a plurality of convolution filters having different preset sizes when detecting an object, it is possible to accurately detect an object in one process regardless of the size of the object.

1A and 1B are diagrams illustrating problems of a conventional object detection algorithm.
2 is a conceptual diagram of a conventional object detection algorithm.
3 is a conceptual diagram illustrating an object detection system in which an object detection method according to an embodiment of the present invention is performed.
4 is a flowchart of a method for detecting an object according to an embodiment of the present invention.
5 is a diagram illustrating a cumulative density function (CDF) of an intersection over union (IoU) for a bounding box having a confidence score of 0.5 in Comparative Example 1, Comparative Example 2, and Example 1, respectively.
6A and 6B are diagrams illustrating character detection accuracy in IoU in Comparative Example 1, Comparative Example 2, and Example 1, respectively.
7 shows the detection rate (true positive rate) according to the size of the character in each of Comparative Example 1, Comparative Example 2, and Example 1.

Hereinafter, an object detection method according to the present invention will be described in detail with reference to the accompanying drawings. The accompanying drawings are provided by way of example only so that the technical idea of the present invention can be sufficiently conveyed to those skilled in the art, and the present invention is not limited to the drawings presented below and may be embodied in other forms. have. Detailed descriptions of well-known functions and configurations that may unnecessarily obscure the gist of the present invention will be omitted.

In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

2 is a conceptual diagram of a conventional object detection algorithm.

The conventional RPN 20 receives a feature map extracted from a target image as an input, and slides by a convolution filter 21 having a single size (generally 3x3, where the unit of size is a pixel). window) method, and input the feature maps proposed by object candidate regions into the bounding box prediction network and the confidence score network, respectively, to detect the object. .

At this time, as described above, the conventional RPN 20 searches for an object using only the convolution filter 21 of a single size, so that when the target image contains objects of various sizes, the exact area of the object can be localized. There is a problem that there is no

3 is a conceptual diagram illustrating an object detection system in which an object detection method is performed according to an embodiment of the present invention, and FIG. 4 is a flowchart of an object detection method performed by an RPN of the object detection system.

Referring first to FIG. 3 , an object detection system 1000 for implementing an object detection method according to an embodiment of the present invention includes a first feature map generation module 100 and an RPN 200 .

Here, the first feature map generation module 100 is configured to extract a first feature map from a target image for detecting an object, and in this case, the first feature map is extracted by convolution and pooling the target image. As the first feature map generation module 100, well-known Resnet34, VGG, googlenet, etc. may be used.

As shown in FIG. 4 , the RPN 200 includes a convolution layer composed of a plurality of convolution filters, a coordinate prediction network, and a confidence score network.

At this time, as described above, the object detection method according to an embodiment of the present invention is performed in the RPN 200 .

On the other hand, in order to learn the RPN 200, the coordinate loss and the score loss are used. Coordinate loss (L _coordinate ) normalizes the L2 distance between the predicted object's bounding box coordinates (x, y, w, h) and the ground truth bounding box coordinates (xt, yt, wt, ht) and Calculated by Equation 1.

[Equation 1]

The score loss (L _score ) is the binary cross entropy between the predicted confidence score p and the ground truth value pt (0 or 1), and is calculated by Equation 2 below.

[Equation 2]

The final objective function L total (Equation 3 below) for training the RPN 200 is calculated as the weighted sum _{of L coordinate} and L _score.

[Equation 3]

Hereinafter, an object detection method according to an embodiment of the present invention will be described in stages.

First, the RPN 200 receives the first feature map extracted from the target image to be detected by the first feature map generation module 100 (S110)

Here, the target image may be an image file of an ancient document including Chinese characters having various sizes.

The first feature map is generated by the first feature map generating module 100 convolves and pools the target image, and as mentioned above, the first feature map generating module may use Resnet34, VGG, or the like.

Next, the RPN 200 performs convolution on the input first feature map by a convolution filter of a preset size to form a second feature map (S120).

Here, in the present invention, when performing convolution on the RPN 200, convolution is not performed using only a convolution filter having a 3x3 single size as in the prior art, but a plurality of convolution filters having different sizes, that is, multi-scale A convolution filter is applied.

More specifically, a plurality of convolutional filters having a multi-scale form a plurality of second feature maps suggesting object candidate regions while passing the first feature map in a sliding window manner, and the plurality of second feature maps are channel-axis It is connected in the direction and becomes the input of the confidence score network and the coordinate prediction network in a later step.

In other words, as shown in FIG. 3, multi-scale convolution filters having various sizes such as 1x1, 3x3, 5x5, 7x7 are used, and through this, even if one target image includes objects having various sizes, the boundary area of the object can be found more precisely.

At this time, in this step (S120), when the preset size of the convolution filter is N x N, N may be an odd number. If the size of the convolution filter is an even number, it cannot have a center point, and there is a problem that asymmetric padding is required.

Also, in this step (S120), when the size of the largest object among the objects is L × L and 1 pixel of the first feature map corresponds to M pixel of the target image, the largest convolution filter among the plurality of convolution filters The maximum value of the filter, that is, the convolutional filter size is K x K, where K is preferably L/M or less.

Also, in the same way, when the size of the smallest object among the objects is P x P and 1 pixel of the first feature map corresponds to M pixel of the target image, the smallest convolution filter among the plurality of convolution filters, that is, the convolution The minimum value of the filter size is Q x Q, where Q is preferably P/M or more.

Of course, here, the unit indicating the size of L, M, N, K, O, P, Q, etc. is a pixel.

While applying the convolution filters of various sizes as described above, the object can be efficiently detected by determining the upper and lower limits of the size of the convolution filter according to the number of pixels of the target image per pixel of the first feature map and the size of the object.

Finally, the RPN 200 detects an object based on the second feature map (S130). In this case, while generating the second feature map, the confidence score and bounding box coordinates for the proposed object candidate region are output, and the object is detected based on this.

More specifically, the confidence score is output as a probability value of the existence of an object as an output of the confidence score network, and the bounding box coordinates are the output of the coordinate prediction network, including the central position (x, y) of the object and the width (w), which is the size of the object, and It is output as x,y,w,h with height (h).

Hereinafter, a detailed experiment performed to verify the effect of the method for detecting an object according to an embodiment of the present invention and a result thereof will be described. In the following experiments, an ancient document containing Chinese characters was used as the target image. At this time, an ancient document was created by synthesizing an ancient-style background image and a handwritten Chinese character document for testing.

Resnet34, pre-trained as an ImageNet data set, was used as the first feature map generator 100 . Resnet34 is known to have high classification performance and use fewer parameters compared to VGG.

The size of the largest Chinese character in the produced old document is 112x112, and 1 pixel of the first feature map corresponds to 16 pixels of the old document. Therefore, since 112/16=7, the maximum value of the convolution filter size becomes 7.

The first experiment analyzed the results of using a multi-scale convolution filter on 100 test images.

Table 1 below is an analysis result of the first experiment, and a detection rate representing the number of detected characters among all characters as a ratio was used as a performance index of the present invention.

It can be seen that the object detection method using a single size (3×3) convolution filter shows the lowest performance (0.57) and achieves an improvement of 9% by adding an additional 1×1 size convolution filter (0.66). ). The additional use of a larger size convolution filter maintains or slightly improves object detection performance. At this time, when the maximum size of the convolution filter, 7x7, is used, the detection rate is saturated.

[Table 1]

In the second experiment, the object detection accuracy was compared for Comparative Example 1, Comparative Example 2, and Example 1 using three different methods for object detection.

Comparative Example 1 uses an RPN including only a single-size convolution filter, Comparative Example 2 uses an RPN including only a single-size convolution filter and a regressor network together, and Example 1 is a multi-scale of the present invention Objects were detected using RPN with convolutional filters.

Here, the regression network of Comparative Example 2 is the difference (_x) between the bounding box coordinates (x, y, w, h) of the RPN for a given input and the ground truth bounding box coordinates (xt, yt, wt, ht) , _y, _w, _h) are output. This regression network combines the updated bounding box coordinates (xr = x + _x, yr = y + _y, wr = w + _w, hr = h + _h) and the intersection over union (IoU) loss of the ground truth box coordinates (mathematical below) (see Equation 4) is trained to minimize

[Equation 4]

In this case, IoU means a value obtained by dividing the area of the intersection of the two areas by the value of the sum area. It is used as one of the indicators to evaluate the accuracy of the predicted bounding box in object detection, and the IoU of the predicted bounding box and the ground truth bounding box is considered as the detection accuracy of the corresponding bounding box. Also in the present invention, this IoU is used as a detection accuracy performance index.

In this experiment, IoU between the detected bounding box coordinates of Comparative Examples 1, 2, and 1 and the ground truth bounding box coordinates were measured for 500 test documents.

5 is a diagram illustrating a cumulative distribution function (CDF) of an intersection over union (IoU) with respect to a bounding box coordinate having a confidence score of 0.5 in Comparative Example 1, Comparative Example 2, and Example 1, respectively.

where p = 0.5 is the character detection threshold. It can be seen that adding the regression network to the single-scale RPN as shown in Fig. 5 shows that the CDF has a higher value at IoU_0.6 than the single-scale RPN and is effective in increasing the IoU of the detected bounding box.

In the object detection method using the multi-scale convolution filter of Example 1 of the present invention, the CDF shows the steepest increase at IoU_0.8, and shows a large density value at IoU higher than those of Comparative Examples 1 and 2.

This means that the object detection method of the present invention has higher character detection performance than conventional known methods.

6A and 6B are diagrams showing character detection accuracy in IoU in Comparative Example 1, Comparative Example 2, and Example 1, respectively.

The numbers represent the IoUs of the bounding box of the same color, and the red box is the ground truth box.

As can be seen from Fig. 6(a), the object detection method (green box) using the multi-scale convolution filter of Example 1 of the present invention almost includes the boundary area of the character and has a large IoU (0.82), whereas the comparative example The object detection method using the single-scale convolution filter of 1 (blue box) and the object detection method using the regression network of Comparative Example 2 (orange box) miss some boundaries of the characters, resulting in relatively small IoUs (0.68 and 0.69). know that you have

In addition, when two characters overlap as shown in Fig. 6(b), Example 1 (green box) individually detected two characters with very large IoU (0.93 and 0.87), whereas Comparative Example 1 (blue box) and Comparative Example 2 (orange box) did not detect the character properly.

The reason that the present invention shows a large effect compared to other methods is that it uses convolution filters of various sizes, and it is possible to capture the entire regional characteristics using a large size convolution filter (see FIG. 6a ), and a small This is because a convolution filter can be used to avoid small areas where characters overlap (see Fig. 6b).

In the third experiment, Comparative Example 1, Comparative Example 2, and Example 1 identical to those of the second experiment were applied to 100 test images per character size to compare the object detection rate (true positive rate).

7 shows the detection rates according to the size of the characters in Comparative Example 1, Comparative Example 2, and Example 1, respectively.

As can be seen from FIG. 7 , since the 3x3 convolution filter corresponds to 48 pixels of the target image, Comparative Examples 1, 2, and 1 have the same detection rate for a character having a size of 48 pixels.

However, in the case of relatively small characters (16 or 32 pixels), it can be seen that in Comparative Examples 1 and 2 using only a 3×3 convolution filter corresponding to 48 pixels of the target image, character detection was omitted and the detection rate was lowered. have. On the other hand, in Example 1, since the RPN including the 1x1 convolution filter is used, the detection rate is significantly higher than that of Comparative Examples 1 and 2.

In addition, even in the case of a relatively large character (112 pixels), Example 1 uses an RPN including a 7x7 convolution filter that can cover the entire area of the character, so that the detection rate is higher than that of Comparative Examples 1 and 2 high can be seen.

Table 2 below shows the detection rates for each of all font sizes. It can be seen that the present invention achieves much higher performance for very small or very large characters (16 or 32 or 112 pixels) and, on average, the object detection method of the present invention uses RPNs comprising a single scale convolution filter. It can be seen that the method has a detection rate 3% higher than that of the method, and has a detection rate that is 1.4% higher than the method using the regression network and the RPN including a single-scale convolution filter.

[Table 2]

Hereinafter, an object detection method according to another embodiment of the present invention will be described step by step.

The method for detecting an object according to another embodiment of the present invention is also performed by the regional proposal network. The first feature map generating module receives a first feature map extracted from a target image for detecting an object (S210), the input Step b of forming a second feature map by performing convolution by a plurality of convolution filters having different preset sizes on the first feature map (S220) and detecting the object based on the second feature map Including step c ( S230 ) is the same as the object detection method according to an embodiment of the present invention.

However, here, after step c (S230) of detecting the object, step d (S240) of measuring the detection rate and detection accuracy of all objects detected from the target image, and the detection rate and the detection accuracy of all the objects are set to the target detection rate and target, respectively. It further includes step e (S250) of comparing with the detection accuracy.

Thereafter, when the detection rate and detection accuracy of all objects detected from the target image are greater than or equal to the target detection rate and the target detection accuracy, the detection of the object is terminated, and the comparison result is detected from the target image When the detection rate and detection accuracy of at least one object among all the objects are less than the target detection rate and the target detection accuracy, the size of the object having the largest difference between the target detection rate and the target detection accuracy and the detection accuracy is calculated do. Thereafter, steps b to e are repeated by adding a convolution filter having a size corresponding to the calculated object size to the plurality of convolution filters.

That is, the object detection method according to another embodiment of the present invention is a method of adding a convolution filter until the target detection rate and target detection accuracy are satisfied for all objects included in the target image. How to apply it selectively.

For example, preset convolution filters of different sizes in step b may be 1x1 and 7x7, and the detection rate and detection accuracy of all objects detected by performing up to step e are compared with the target detection rate and target detection accuracy, respectively. At this time, if the size of the convolution filter corresponding to the size of the object having the detection rate and detection accuracy that is the greatest difference from the target detection rate and the target detection accuracy is 3x3, add a 3x3 convolution filter to perform steps b to e do it one more time

Thereafter, the detection rates and detection accuracies of all objects detected again in step e are compared with the target detection rate and the target detection accuracy, respectively, and steps b to e are performed once more by adding a convolutional filter of a required size. After repeating this, when the detection rates and detection accuracy of all detected objects finally satisfy the target detection rate and the target detection accuracy or more, the object detection is terminated.

At this time, as in the object detection method according to an embodiment of the present invention, in step b, when the preset size of the convolution filter is N x N, N may be an odd number.

Also, in step b, when the size of the largest object among the objects is L × L and 1 pixel of the first feature map corresponds to M pixel of the target image, the size of the largest convolution filter among the plurality of convolution filters , that is, the maximum value of the convolution filter size is K x K, where K is less than or equal to L/M.

Also, in the same way, when the size of the smallest object among the objects is P x P and one pixel of the first feature map corresponds to M pixel of the image, the size of the smallest convolution filter among the plurality of convolution filters, that is, The minimum value of the convolution filter size is Q x Q, where Q is greater than or equal to P/M.

Of course, here too, the unit indicating the size of the convolution filter such as L, M, N, K, O, P, Q is a pixel.

According to the object detection method according to another embodiment of the present invention, an object is initially detected using only a convolution filter of the minimum size (eg 1x1) and a convolution filter of the maximum size (eg 7x7), and then the target In a method of adding convolution filters corresponding to the size of the object showing the greatest difference from the detection rate and detection accuracy one by one, only the convolution filters necessary for object search of the current target image can be selected and used.

In other words, according to the present invention, the minimum number and size of convolution filters necessary to satisfy the target detection accuracy and detection rate when detecting an object can be applied, so that an object can be detected more efficiently.

On the other hand, the object detection method according to the present invention described above may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable recording medium. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floppy disks. Included are magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments given as examples for the purpose of description of the present invention are merely one embodiment in which the present invention is embodied, and combinations are possible in various forms in order to realize the gist of the present invention. Therefore, the present invention is not limited to the above embodiments, and as claimed in the following claims, anyone with ordinary skill in the art to which the invention pertains can implement various modifications without departing from the gist of the present invention. It will be said that there are technical features of the present invention to the extent.

1000: object detection system
100: first feature map generation module
200: RPN

Claims (9)

receiving a first feature map extracted from a target image for detecting an object;
forming a second feature map by performing convolution on the input first feature map by a convolution filter having a preset size; and
detecting the object based on the second feature map;
In the forming of the second feature map, a plurality of convolution filters having different preset sizes are applied to the first feature map,
In the forming of the second feature map, when the size of the largest object among the objects is L x L, and one pixel of the first feature map corresponds to M pixels of the target image, the plurality of convolutions The maximum value of the filter size is K x K, and K is L/M or less.
delete delete According to claim 1,
In the forming of the second feature map, when the size of the smallest object among the objects is P x P, and one pixel of the first feature map corresponds to M pixels of the target image, the plurality of convolutions The minimum value of the filter size is Q x Q, and Q is P/M or more. (Here, the units of P and Q are pixels)
a) receiving a first feature map extracted from a target image for detecting an object;
b) forming a second feature map by performing convolution on the input first feature map by a plurality of convolution filters having different preset sizes;
c) detecting the object based on the second feature map;
d) measuring a detection rate and detection accuracy of all objects detected from the target image;
e) comparing the detection rates and detection accuracies of all the detected objects with preset target detection rates and target detection accuracies, respectively; and
f) If, as a result of the comparison, the detection rates and detection accuracy of all the detected objects satisfy the target detection rate and the target detection accuracy or more, the detection of the object is terminated, and, as a result of the comparison, any of the detected objects When the detection rate and detection accuracy of one or more objects are less than the target detection rate and the target detection accuracy, calculating the size of an object having a detection rate and detection accuracy that is the greatest difference from the target detection rate and the target detection accuracy, and the calculated object and repeating steps b) to e) by adding a convolution filter having a size corresponding to the size.
6. The method of claim 5,
In step b), when the preset size of the convolution filter is N x N, N is an odd number. (Where N is a pixel)
6. The method of claim 5,
In step b), when the size of the largest object among the objects is L × L and one pixel of the first feature map corresponds to M pixel of the target image, the maximum value of the size of the plurality of convolution filters is An object detection method, characterized in that K x K, and K is L/M or less. (Here, the units of L and K are pixels)
6. The method of claim 5,
In step b), when the size of the smallest object among the objects is P x P and one pixel of the first feature map corresponds to M pixel of the target image, the minimum value of the size of the plurality of convolution filters is Q x Q, and Q is an object detection method, characterized in that P/M or more. (Here, the units of P and Q are pixels)
A computer-readable recording medium in which a program for implementing the object detection method according to any one of claims 1 to 8 is recorded.

Images