KR101882743B1 - Efficient object detection method using convolutional neural network-based hierarchical feature modeling - Google Patents

Abstract

The present invention relates an efficient object detection method for suggesting a hierarchical deep feature-based training framework with a generalization capability using a hierarchical feature model (HFM) and a hierarchical classifier ensemble (HCE) based on facts that the performance of many object detectors is degraded due to the ambiguity of differences between classes as well as large changes in the class and that deep features extracted from visual objects exhibit strong hierarchical clustering properties.

Description

[0001] The present invention relates to an object detection method using convolutional neural network-based hierarchical feature modeling,

The present invention relates to an object detection method, and more particularly, to an efficient object detection method using convolutional neural network-based hierarchical feature modeling.

Object detection and object localization are becoming increasingly difficult to detect in the computer vision domain. Object detection is a very complex process due to the image ambiguity of the inter-class appearance and variations due to large intra category changes. In many studies, training samples have been divided into several components and methods have been dealt with to improve the performance degradation of object detection by independently training the components. Disassembly of the training data set can mitigate local variations and variations within the class. Some early pioneering research examines a clustering approach to training data in terms of object size, posture, aspect ratio, and component label. However, performance can be improved considering inter-class ambiguity, but most of them consider only intra-class variations and do not investigate inter-class ambiguity. Some progress in detection performance is based on more general subcategory models within semantic object categories.

For example, Gu et al. Classify samples into components using annotated key points and masks in "Proceedings of the IEEE European Conference Computer Vision, pp. 445-458 (2012) ", Aghazadeh et al. of the IEEE European Conference Computer Vision, pp. 115-128 (2012), used a similarity graph that represents information within a class to separate data into spectral clusters. Ruan et al., In "Signal Process. Lett., IEEE 22 (2), 244-248 (2015)," examined supervised multi-component model training for subcategories.

Although many studies use sub-category structures to improve the accuracy of object detection, most sub-categories are constructed based only on similarity information within the class. However, there are many objects that are confusing due to ambiguity between classes. Valuable interclass information can be used to solve the confusing subcategory problem of interclass samples. Recently, Dong et al. Proposed a subcategory mining approach for exploring intra-class diversity in IEEE Trans. Circuits Syst. Video Technol. 25 (8), 1322-1334 (2015). However, recently proposed deep feature-based object detection methods such as fast R-CNN and spatial pyramid pooling (SPP) in a convolution network have been proposed There is a problem that the performance is much lower than that of the first embodiment.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in order to solve the above-mentioned problems, and it is an object of the present invention to provide a method and an apparatus for analyzing a plurality of object detectors, Based hierarchical feature model (HFM) and a hierarchical classifier ensemble (HCE), based on the fact that the resulting deep feature exhibits strong hierarchical clustering characteristics, And to provide an efficient object detection method that presents a feature-based training framework.

According to another aspect of the present invention, there is provided an object detection method using hierarchical feature modeling for semantic objects included in an image in an object detection apparatus, A root node forming a semantic object category space including the semantic objects according to a feature of the region of interest, at least one super category node corresponding to each semantic object as a child of the root node and being trained using a training data set, One or more enhancement category nodes to be trained using the training data set as a child of the super category node, and one or more subcategory nodes to be trained using the training data set as a child of each enhancement category node ; Learning a multi-classifier according to a SVM (Support Vector Machine) ensemble algorithm for the super category node, the augmentation category node, and the subcategory node, and calculating a reliability score in each category; And collecting reliability scores of each category to detect a semantic object included in the image.

Wherein calculating the confidence score comprises: projecting the binary SVM classifiers pseudo-probability for the region of interest in each of the categories; And calculating a multi-class margin in the category based on the pseudo-probability in each category and calculating a normalized multi-class margin for the multi-class margin to calculate a reliability score in each category do.

(H | r) for the one or more super category nodes in the region of interest r, and calculates a confidence score CS _h ^(k) (r) for the kth super category node according to the following equation: Calculates a reliability score for each node in the super category,

은 k번째 수퍼 카테고리 노드 h^(k)에 대한 의사 확률,

은 h^(k)에 대한 다중 클래스 마진,

은 h^(k)에 대한 정규화된 다중 클래스 마진, |Ω_H|은 수퍼 카테고리 공간 Ω_H에서 h^(k)에 대응하는 최대수이다.here,

Is the pseudo-probability for the k-th super-category node h ^(k)

Is a multi-class margin for h ^(k)

Is the normalized multi-class margin for h ^(k) , | Ω _H | is the maximum number corresponding to h ^(k) in the super-category space Ω _H.

(H r) for the one or more enhancement category nodes in the region of interest r and calculates a confidence score CS _m ^{(k ')} (r) for the kth enhancement category node according to the following equation: Calculates a reliability score for each node in the augmentation category,

은 k'번째 증강 카테고리 노드 m^(k')에 대한 의사 확률,

은 m^(k')에 대한 다중 클래스 마진,

은 m^(k')에 대한 정규화된 다중 클래스 마진, |Ω_M|은 증강 카테고리 공간 Ω_M에서 m^(k')에 대응하는 최대수이다.here,

Is the pseudo probability for the k 'th enhancement category node m ^(k') ,

Is a multi-class margin for m ^{(k ')} ,

Is ^"normalized multiclass margin for, | Ω _M | is m ^(k in augmented category space Ω _M m ^(k), the maximum number corresponding to ^a).

(H " r) for the one or more sub-category nodes in the region of interest r and calculates a confidence score CS _l ^{(k ")} (r ), Calculates a reliability score for each node in the sub-category,

은 k"번째 서브 카테고리 노드 l^(k")에 대한 의사 확률,

은 l^(k")에 대한 다중 클래스 마진,

은 l^(k")에 대한 정규화된 다중 클래스 마진, |Ω_L|은 서브 카테고리 공간 Ω_L에서 l^(k")에 대응하는 최대수이다.here,

Is the pseudo-probability for the k "th sub-category node l ^(k") ,

Is a multi-class margin for l ^{(k ")} ,

Is ^"normalized multiclass margin for, | Ω _L | is l ^(k in the sub-category space Ω _L l ^(k)" is the maximum number corresponding to ^a).

According to another aspect of the present invention, there is provided a recording medium embodied in computer readable code comprising a root node forming a semantic object category space including semantic objects according to characteristics of an area of interest of an image, One or more super category nodes corresponding to each semantic object as a child and trained using a training data set, one or more enhancement category nodes to be trained using a training data set as a child of each super category node, a child of each enhancement category node Performing a configuration of a hierarchical feature model comprising one or more subcategory nodes to be trained using training data sets; Learning the multi-classifier according to the SVM ensemble algorithm for the super-category node, the enhancement category node, and the sub-category node to calculate a reliability score in each category; And a function of collecting reliability scores of each category in order to detect the semantic objects included in the image, wherein the object detection apparatus performs object detection using hierarchical feature modeling of the semantic objects included in the image And the like.

According to the object detection method using the convolutional neural network-based hierarchical feature modeling according to the present invention, it is possible to solve the problem of intra-class ambiguity and in-class change especially in the detection of a large scale object according to the concept of the enhanced object category, The method of the present invention can reduce the computational overhead because a restricted category is assigned instead of the entire assignment of the entire semantic category to the ROI, as seen in the state of the art, such as R-CNN.

In addition, a hierarchical feature model (HFM) can be used more effectively than a flat feature model and a subcategory based feature model in combination with a hierarchical classifier ensemble (HCE) that utilizes the clustering quality of the deep feature layer.

In addition, many complex data samples can be clustered correctly into subcategories by using interclass information, and simple sub-problems can be solved to improve overall search accuracy.

Figure 1 shows the concept of the enhancement category of the HFM of the present invention.
2 is a diagram for explaining an object detection system using a convolutional neural network-based hierarchical feature modeling (CNN-HFM) according to an embodiment of the present invention.
3 is a flowchart illustrating an object detection method of an object detection system according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are denoted by the same reference symbols as possible. In addition, detailed descriptions of known functions and / or configurations are omitted. The following description will focus on the parts necessary for understanding the operation according to various embodiments, and a description of elements that may obscure the gist of the description will be omitted. Also, some of the elements of the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore the contents described herein are not limited by the relative sizes or spacings of the components drawn in the respective drawings.

The present invention presents a hierarchical deep feature-based training framework with generalization capability instead of the existing algorithm-centric detection algorithm. This is caused by the following observations. (1) the performance of many object detectors is degraded due to the ambiguity of differences between classes as well as large changes in classes, and (2) the deep features extracted from visual objects exhibit strong hierarchical clustering properties.

The present invention proposes a new object detection method using a hierarchical feature model (HFM) and a hierarchical classifier ensemble (HCE). This technique features typical and flexible feature structures in terms of super-, augmented, and sub-categories. Wherein the augmented category is a segmentation of a semantic object category that takes into account the effects of the super-category using a latent topic model (LTM). Thus, each augmented category corresponds to one semantic object category.

Figure 1 shows the concept of the enhancement category of the HFM of the present invention. In Figure 1, the concept of an enhanced category compared to the original semantic object is shown. For each semantic object category, training samples are divided into unsupervised super categories. The augmented category is determined by dividing the semantic object category based on the super category. Therefore, each augmented category corresponds to one semantic object category.

For example, the person category can be divided into three enhancement categories: a sitting person (augmented category 2), a standing person (augmented category 3), and a rider (augmented category 4). In large object detection, a classifier constructed using an augmented class can provide better performance than a classifier constructed using a semantic object category because the category similarity errors are effectively reduced.

The method of the present invention is the first complete end-to-end approach to interactively build hierarchical functional structures and classifier ensembles to explore generalization capabilities in object classification and direction localization. Data Hierarchy In each node of the HFM, the existing multi-level classification ensemble is adopted but adaptive. Instead of using a flat linear SVM (Support Vector Machine) for all object classes, the hierarchical SVM ensemble HCE is used for interclass and intraclass decisions.

First, the HCE uses a one-against-all SVM to compute confidence for one class prediction made by the binary SVM classifier for each enhancement category label. Next, the multi-class reliability score (reliability score of each category) for object detection is calculated by combining multiple detectors. Next, each HCE tree is trained in different decision paths in the HFM and is used to calculate the total confidence score of the test image to minimize detection errors. Finally, in the detection phase, HCE-based object detection is performed by aggregating the confidence score (s) of each domain proposal derived by HFM. The reliability score of each category can be used to judge object detection, such as whether the target object is included in the image through ridge regression, non-maximum suppression, etc. have.

The main features of the present invention are summarized as follows.

1) The concept of augmented object categories can solve the ambiguity between classes and in-class changes, especially in large object detection. The method of the present invention can reduce the computational overhead because a limited category is assigned instead of the entire assignment of the entire semantic category to the ROI, as seen in state of the art techniques such as SPP, high speed R-CNN.

2) The Hierarchical Feature Model (HFM) can be used more effectively than the flat feature model and subcategory based feature models in combination with a hierarchical classifier ensemble (HCE) that takes advantage of the clustering quality of the deep feature layer.

3) Many complex data samples can be clustered correctly into subcategories by using interclass information, and simple sub-problems can be solved to improve overall search accuracy.

2 is a diagram for explaining an object detection system (apparatus) using a convolutional neural network-based hierarchical feature modeling (CNN-HFM) according to an embodiment of the present invention. Referring to FIG. 2, an object detection system according to an embodiment of the present invention includes a hierarchical feature model unit for constructing a hierarchical feature model (HFM), and a multi-classifier at each node of the HFM using an SVM ensemble algorithm And a hierarchical classifier ensemble (HCE) that trains and computes multiple grade confidence scores for object detection.

The components of the object detection system according to an embodiment of the present invention may be implemented by hardware such as a semiconductor processor, software such as an application program, or a combination thereof. In addition, the functions used in the object detection system according to an embodiment of the present invention can be implemented as computer-readable codes on a recording medium readable by an apparatus such as a computer. And can be implemented to input, output, and display data or information necessary for performing a function. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, hard disk, removable storage device and the like.

In general, as the number of object classes increases, the classification performance deteriorates. The placement of fully connected neural networks with a flat SVM is successful when the number of object classes is small or moderate, but the detection performance deteriorates as the number of object categories increases. Data imbalance is a common phenomenon due to increased noise and variations that contaminate real world image data. The proposed HFM based object detection method aims to provide robust object detection with generalization capability. Accuracy improvements based on HCE and HFM are new in a data-driven manner. HFM, which is the core of the proposed object detection method, consists of a three level cluster tree composed of super category, augmented object category and subcategory feature model. HFM uses feature information and semantic subcategories of unsupervised super categories for meaningless object category recognition. The region of interest (ROI) can be extracted using EdgeBoxes, the proposed algorithm. In the learning phase, the category hierarchy can be found by using a potential topic model (LTM) with features extracted from the pre-trained CNN. The HFM is fine-tuned to the hierarchical category and is constructed at the inter-class level and the intra-class level. The HCE is constructed by training multiple classifiers at each node of the HFM using the SVM ensemble algorithm. The region compensation model can be constructed using a hierarchical ridge regression algorithm.

In the detection step, the pool of augmented object categories is predicted in terms of the HCE subtree hypothesis for the ROI generated by the region proposal algorithm. Object detection is performed based on the above hypothesis following ridge regression and non-maximum suppression similar to Girshick's method. The scores of the HCE subtree are combined in terms of super categories, enhancement categories, and subcategories for the hypothetical ROI. Finally, the post-processing of non-maximum suppression is performed using the combined score and location information, and the object category is determined.

<Hierarchical Feature Model Unit (HFM)>

First, the function of the hierarchical feature model unit (HFM) will be described in detail.

Region Proposed Algorithm EdgeBoxes are used to find ROIs in images, and ROI features are generated using 16-layer CNNs. Using the normalized ROI function, three levels of the super level (inter-class), the M level (augmented class) and the sub-category (intra class) of the enhancement category associated with the semantic objects of the root node A deep feature layer HFM is constructed. The root node of the HFM has one or more super category nodes corresponding to each semantic object as a child at the H level. Each super category node has at least one augmented object category node as a child at the M level which is the original or divided semantic object class according to the inter-class characteristic. Each augmented object category node has one or more subcategory nodes as children at the L level.

The HCE is built by training a multi-category classifier at each node of the HFM, which is a collection of one-versus-all SVMs. Girshicks' flat feature structure may correspond to a special case of HFM with only the root and the entire semantic object category at the M level without the augmented category. This flat HFM structure is called a flat feature model (FFM). HFM is also a generalization of the sub-category based approach.

<Potential topic model for category hierarchy (LTM)>

We introduce an unsupervised approach to learning data-driven hierarchical categories. For the unsupervised learning phase, a super-category is built using the Potential Topic Model (LTM). As a mixed model, the LTM provides a new way of representing potential hybrid components for grouping data that is beneficial to learning a layered structure. In LTM, ROI is represented by a combination of potential topics. These learned topics correspond to super categories for building category hierarchies. For ROI representation, features extracted from deep CNN (convolutional neural network) are used. More specifically, the CNN model is fine-tuned in the training set using a pre-trained CNN model. A fixed length feature vector (feature vector) is then extracted from the last fully connected layer for each ROI. The extracted features (features) are encoded, quantized, and scaled. Considering that each ROI is represented as a vector, the goal is to learn the super-category by applying a mixed model to the expressed ROI. Specifically, LTM represents each ROI by combining K subjects. Each subject corresponds to one super category or a few super categories. At this time, a generation process similar to the original LTM can be used.

<Hierarchical Feature Model (HFM)>

The hierarchical feature modeling unit builds a hierarchical feature model (HFM) using the features of the ROI of the training data set D associated with the semantic object space? Containing the semantic objects contained in the image. The HFM consists of a root node, a super category node, an enhancement category node and a subcategory node. The root of the HFM is associated with the entire feature set D with category space? And is connected to the super category node as a child. Semantic objects with multiple super categories are divided into several enhancement categories according to their inter-class characteristics. The concept of an augmented category has been introduced to reduce the influence of ambiguity at the inter-class level as well as variations in the level of the class of the original semantic object class and is augmented to indicate the multi-label category and the occluded semantic object category . The super-category node h is associated with the training data set D _h , which is a subset of D and has a plurality of augmented object nodes as its children. LTM analysis allows one semantic object category to belong to several super categories. This is because different objects can share parts with similar shapes or characteristics. At the M level, semantic object categories belonging to a plurality of super categories are divided into a plurality of enhancement categories. The enhancement category node m has a training data set denoted by D _m divided from the training data set D _h . The training set of each augmented object category can be further subdivided into lower level subcategories using the LTM algorithm to minimize the impact of intra-class variation. The training data set of subcategory node l is denoted D _l , and this data is divided in the augmented training data set D _m .

<Hierarchical Classifier Ensemble (HCE)>

The hierarchical classifier ensemble (HCE) uses a SVM ensemble algorithm to train multiple classifiers at each node of the HFM to yield multiple grade confidence scores for object detection.

The node reliability function consists of a multi-class classifier constructed by a set of binary classifiers of individual child nodes. Reliability scores are needed to maintain linear relationships with predictive accuracy and to increase predictive reliability with several considerations.

As shown in FIG. 2, let? _H ,? _M, and? _L be the super-category space, the enhancement category space, and the sub-category space, respectively. Routes have one or more of the super-category node as a child comprising (| | Ω _H, the maximum number corresponding to the h ^(k)) and super-category node h is one or more of the enhanced object category node ^{_{(| Ω M |, m (}} k ^') ), And the enhancement category node m includes one or more sub-category nodes (the maximum number corresponding to | Q _L |, l ^(k'') ).

First, the root generates an SVM (Support Vector Machine) ensemble for calculating the reliability score of ROI (Region of Interest) used when moving to the super category node. | Ω _H | The binary SVM classifiers (φ ₁ , φ ₂ , ..., φ _{| ΩH |} ) are trained using the D _h at the root node, which ROI uses to determine the _{supercategory} nodes. Since the prediction used directly by the multi-class classification and estimated by the binary SVM is unreliable, the reliability function described below is introduced.

At a given ROI r, a linear SVM (classifier) φ _h for all super-category nodes _h is projected to a pseudo probability P (y = h | r), as in equation (1).

[Equation 1]

Here, the parameters? And? Are determined by logistic regression as shown in Equation (1a). The parameters a ', b' and a ", b "can be similarly determined in a similar manner.

Equation (1a)

Where w is the weight for ROI samples (each w _{i is} possible for each sample), y is the corresponding label. At a given ROI r, the multi-class prediction at the root node yields the highest scored first super-category node predicted value h ⁽¹⁾ ( ⁽ h)) for ROI r with pseudo-probability P r &lt; / RTI &gt;

&Quot; (2a) &quot;

Multiclass margin ζ _h ^{(1) (r)} is defined as shown in Equation 2b].

(2b)

The normalized multi-class margin φ _h ⁽¹⁾ (r) is multiplied with the pseudo probability P (y = h | r) using a sigmoid function as in Equation (2c) And the class margin? _H ⁽¹⁾ (r).

(2c)

Here, parameters A, B, and C are predetermined coefficients that can be determined through empirical fitting. The same applies to the parameters A ', B', C 'and A ", B" and C ".

Accordingly, the reliability function CS _h ⁽¹⁾ (r) for the predicted value h ⁽¹⁾ (r) for the first super-category node can be calculated as shown in equation (2d).

Equation (2d)

Accordingly, the reliability function CS _h ^(k) (r) for the predicted value h ^(k) (r) for the k-th super-category node h ^(k) is calculated as in Equation (2e) Can be used for reliability score calculation.

(2e)

Here, according to the equations (2a), (2b), (2c), the relationship as shown in the following equation (2f) is established. Here, the predicted value h ^(k) , the multiple class margin? _H ^(k) (r), and the normalized multiple class margin? _H ^(k) (r) for the kth super category node among the plurality of super category nodes are used.

&Quot; (2f) &quot;

The super category node has augmented object category set &lt; RTI ID = 0.0 &gt; OM, &lt; / RTI &gt; An SVM ensemble for each super category node is established and the confidence score for each augmented object category is calculated. An SVM ensemble is built up in each augmented object category.

| Ω _M | Binary SVM classifier _{(φ '1, φ' 2} ,, φ '|... ΩM |) , the training data set in the super-category nodes in order to determine the best best node (s) in augmented category space Ω _M D _m . The linear SVM (classifier)? ' _M for the enhancement category m is projected to a pseudo probability P (y = m ^(k') | r) defined as in (3a) (see Equation 1).

(3a)

Thus, the k 'th credit value ^{_{CS m (k') (r}} ) for the respective enhancement category of the plurality of enhancement category node calculation for the super-category node is calculated as shown in Equation 3b] Total Growth category node (See Equation (2e)). &Lt; / RTI &gt;

(3b)

Here, a predicted value m ^{(k ')} , a multi-class margin? _M ^(k') (r), and a normalized multiple class margin? _M ^{(k ')} (r) for a k'th enhancement category node are used.

Similarly, | Ω _L | Binary SVM classifier _{(... Φ '' 1} , φ '' 2,, φ '' | ΩL |) are trained in the enhanced category nodes in order to determine the best best node (s) in the sub-category space Ω _L The data set is trained using D _l . The linear SVM (classifier)? '' ₁ for subcategory l is projected to a pseudo probability P (y = l ^{(k '')} | r) defined as in Equation 4, calculated for, k for each sub-category of the plurality of sub-category node, "" second confidence value ^{_{CS l (k '') (}} r) , the [equation 4] are calculated as the total sub-category reliability for the node score Can be used for the calculation.

&Quot; (4) &quot;

Here, k "predicted value l ^{(k ''),} a multi-class margin ^{_{ζ l (k '(r)}} ') (r), the normalized multiclass margin φ _l ^{(k '')} for the second sub-category node is utilized .

Hereinafter, an object detection method of the object detection system according to an embodiment of the present invention will be described with reference to the flowchart of FIG.

3 is a flowchart illustrating an object detection method of an object detection system according to an embodiment of the present invention.

3, in an object detection method using hierarchical feature modeling for semantic objects included in an image in an object detection system (apparatus) of the present invention, first, a hierarchical feature model unit (HFM) in accordance with a feature of the region of interest (ROI), the root node to form a semantic object category space including the semantic object and corresponding to each of the semantic object as a child of the root node a to be trained using a training data set D _h One or more sub-category nodes to be trained using training data set D _l as a child of each enhancement category node, one or more enhancement category nodes to be trained using training data set D _m as a child of each super category node, (Step S100). In step S100, the hierarchical feature model HFM is constructed.

The hierarchical classifier ensemble (HCE) learns multiple classifiers according to the SVM ensemble algorithm for the super-category nodes, the enhancement category nodes, and the sub-category nodes, as described above, to calculate the confidence scores in each category (S200). In order to calculate a confidence score in each category, a hierarchical classifier ensemble (HCE) projects the binary SVM classifiers for the region of interest in each of the categories with a probability probability, and based on the pseudo probability in each category Category is calculated and a multi-class margin normalized for the multi-class margin is calculated to calculate a reliability score in each category. The Hierarchical Classifier Ensemble (HCE) projects the SVM classifiers to the pseudo-probability P (h | r) for each of the category nodes in the ROI r, as described above, K &quot; / k "as shown in Equation (2), Equation 2f, Equation 3b and Equation 4, The reliability score can be calculated.

The predetermined object detector collects reliability scores of each category in order to detect a semantic object included in the target image (S300). This HCE - based object detection is performed by statistically analyzing and analyzing the confidence scores of each domain suggestions derived and collected by HFM. The reliability score of each category can be used to judge object detection, such as whether the target object is included in the image through ridge regression, non-maximum suppression, etc. have.

<Contents of experiment>

The object detection system of the present invention based on a hierarchical feature model (HFM) with a hierarchical classifier ensemble (HCE) was evaluated in the PASCAL VOC 2007 and PASCAL VOC 2012 detection tasks. Each dataset contains thousands of images for the real world scene, and the goal is to predict the bounding boxes of all the objects in the image. If the predicted bounding box overlaps with the reality by more than 50%, it is considered to be true positive. The 16th floor VGG-Net was used as the base line of the system. Among the most advanced domain proposal algorithms, Edge Boxes were adopted because they provide fast and accurate domain proposals. In the first experiment on PASCAL VOC 2007, we trained the detection system in a set of training data. The second experiment was evaluated as PASCAL VOC 2007 with knowledge transfer learning for HFM. Finally, the method of the present invention is compared with a cutting-edge method on a PASCAL VOC 2012 public leaderboard.

<VOC 2007 results>

The experimental results of the PASCAL VOC 2007 data set are shown in Table 1 below. Each method is distinguished by a flat feature model (FFM), an HFM with an enhancement category level (M level HFM), and an HFM with a subcategory level (L level HFM). All the experiments in Table 1 were trained in the VOC 2007 training data set and evaluated with Mean Averagr Precision (mAP) in the test set using the standard Pascal evaluation tool [27]. The detection performance compared with the results of the state-of-the-art detector is shown in Table 1.

<FFM>

FFM is a special type of HFM with no enhancement categories and only root and full semantic object categories. FFM is used as a feature extractor in experiments. The public VGG16 CNN structure was chosen as the baseline according to the training protocol. To build the FFM, ImageNet's pre-trained CNN fine-tuned the data (D, Ω) at a training rate of 0.001 to 50K iterations. After 50K repetitions, the training speed decreased by 10 times for fine adjustment with 20K repetitions. During FFM fine tuning, only the weights from conv4_1 to fc7 were fine-tuned and the weights from conv1_1 to conv3_3 were fixed.

<M-level HFM>

인 수퍼 카테고리 i를 결정하여 각 ROI r에 허용되었다. T_θ는 경험적으로 0.3으로 결정되었다. HFM을 구축하기 위해 매우 큰 데이터 세트를 사용하여 훈련하기 위해 하드 네가티브 마이닝(hard negative mining)을 사용하여 post-hoc SVM 훈련을 구현하였다. HFM은 Sect. 3.2.로 구성된다. HCE를 학습 후에, 실현 가능한 HCE 서브 트리는 사후 처리에서 서로 경쟁하여 최종 객체 위치를 방향정위(localization)한다. 후 처리 과정에서 보충 자료에 설명된 가중치 처리된 non-maximum suppression은 물론 계층적 능형 회귀가 적용되었다. Table 1은 FFM에서 2.9 %의 HFM을 적용하고 70.8 %에 도달함으로써 현저한 개선이 달성될 수 있음을 보여준다.The M-level HFM is built without subcategories in the VOC 2007 training data set. To construct a category hierarchy, we set up K subjects in the LTM to find the K-dimensional super category distribution θ ^K for each ROI. In this experiment, K was set to 5, which was selected by a grid search of {1,2, ..., 9} or more. The isolated HFM H-level can be easily filled due to data sparsity and the HFM hierarchy misleads some candidates. In order to overcome the problem of overfitting and misleading, the super category Instead of

Gt; i &lt; / RTI &gt; was determined and allowed for each ROI r. T _θ was empirically determined to be 0.3. To implement HFM, we implemented post-hoc SVM training using hard negative mining to train using very large data sets. HFM is Sect. It consists of 3.2. After learning the HCE, the feasible HCE subtrees compete with each other in post-processing to localize the final object location. Hierarchical ridge regression was applied as well as the weighted non-maximum suppression described in the supplementary data in the post-processing. Table 1 shows that a significant improvement can be achieved by applying 2.9% HFM in FFM and reaching 70.8%.

<L-level HFM>

L-level HFM was constructed considering sub-category levels. After the M-level HFM was established, a sub-category was discovered by the LTM. Thus, for each augmented object category node at the M level, there are L level subcategory children. The learning process is the same as the M-level HFM. Improvement of L level HFM is 4.4% than FFM. As shown in Table 1, a total 72.3% mAP was achieved in the VOC 2007 data set, higher than the state-of-the-art methods such as Fast R-CNN with 66.9%.

<VOC 2007 results with domain adaptation>

The performance of the model for cross-domain transfer learning was evaluated using Microsoft's Common Objects in Context (COCO) 2014 and PASCAL VOC 2007 + VOC 2012 (VOC +) as two domains. First, the following two baselines were considered without transmission learning.

<VOC +>

To build FFM _{VOC +} , ImageNet pre-trained CNN was fine-tuned for data (D ^{VOC +} , Ω ^{VOC +} ) at 50K repetitions and a learning rate of 0.001, and then the training speed was reduced by a factor of 10 for 20K iterations. After the FFM _{VOC +} was fine tuned, the HFM was built into the VOC + using the category hierarchy obtained from the LTM. All parameters were fixed as described above, and the performance of the VOC 2007 test for the VOC + baseline was 75.6% as described in Table 2. Compared with high-speed R-CNN, the performance improvement was achieved even more dramatically by training additional data (3.1 to 4.8%). This result is mainly due to the HFM approach to building a hierarchical structure and its ability to be enhanced in larger data sets.

<COCO>

Since the COCO data sets differ in the number of classes compared to the VOC, but the VOC can be considered a subset of the COCO, all 80 classes were used to train the COCO baseline. First, FFM _COCO was built and fine tuned to 200K iterations and a learning rate of 0.001 at ImageNet pre-training CNN data (D ^COCO , Ω ^COCO ). Then the training speed was reduced by a factor of 10 for 80K repeats. The same procedure as described for VOC + was followed. The COCO baseline achieved a mAP of 73.9% in Table 2. This is lower than the VOC + baseline due to domain differences.

<COCO → VOC +>

The effect of knowledge transfer learning on HFM was verified. Instead of using the same domain to build the category hierarchy, we used an external domain for our previous knowledge. First, FFM _COCO was used as the COCO baseline. Second, the category hierarchy was constructed using LTM and FFM _COCO . The VOC hierarchy category is then obtained by transferring the appearance from the COCO data set. Finally, the FFM _COCO was fine-tuned to the data (D ^{VOC +} , Ω ^{VOC +} ) to build the HFM. The fine tuning options were set at a training speed of 0.001 to 50K repetitions and the training speed was reduced by a factor of 10 for 20K repetitions. The experimental parameters are as described above, and Table 2 shows that knowledge transfer learning based on HFM was performed 80.4% better than the baseline for COCO and VOC +.

<VOC 2012 results>

The detection performance of the VOC 2012 test set was evaluated in this experiment. For the end result of the VOC2012 data set, CNN was fine-tuned in the COCO train set and the domain adaptation method was performed in the VOC 2012 train set applying the same procedure as described above.

Table 3 uses VGG16 as the baseline and additional training data to compare with the HFM on the items in the VOC2012 leaderboard. Without using domain data, HFM is one of the excellent search methods of 71.0%. After fine tuning through the domain adaptation approach and configuring the L-level HFM, HFM achieved a state-of-the-art 77.5% mAP in VOC 2012 test results.

Thus, the present invention provides a new data-based hierarchical object detection framework. This framework outperforms the most advanced results for PASCAL VOC 2007 and VOC 2012 data sets. Deep features were segmented by building a hierarchical deep feature model HFM through the LTM algorithm. The classifier was assembled at each node of HFM and composed of HCE.

As described above, the object detection method using the convolutional neural network-based hierarchical feature modeling in the object detection system according to the present invention is based on the concept of the enhanced object category, in particular, The problem can be solved and the method of the present invention is able to solve computational overheads because the ROI is assigned a limited category instead of the entire assignment of the entire semantic category, as can be seen in the state of the art SPP, high-speed R- Can be reduced. In addition, a hierarchical feature model (HFM) can be used more effectively than a flat feature model and a subcategory based feature model in combination with a hierarchical classifier ensemble (HCE) that utilizes the clustering quality of the deep feature layer. In addition, many complex data samples can be clustered correctly into subcategories by using interclass information, and simple sub-problems can be solved to improve overall search accuracy.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the essential characteristics of the invention. Therefore, the spirit of the present invention should not be construed as being limited to the embodiments described, and all technical ideas which are equivalent to or equivalent to the claims of the present invention are included in the scope of the present invention .

Inter-class ambiguity
Intra-class variations
Semantic object categories
Sub categories
Latent Topic Model (LTM)
Deep feature-based &lt; RTI ID = 0.0 &gt;
Hierarchical Feature Model (HFM)
Hierarchical Classifier Ensemble (HCE)
Hierarchical ridge regression algorithm

Claims (6)

A method for detecting an object using hierarchical feature modeling of semantic objects included in an image in an object detecting apparatus,
A root node forming a semantic object category space including the semantic objects according to a characteristic of an area of interest of the image, at least one super category corresponding to each semantic object as a child of the root node and being trained using a training data set A hierarchical feature model comprising one or more subcategory nodes to be trained using training data sets as children of each enhancement category node, one or more enhancement category nodes to be trained using training data sets as children of each super category node, Performing a configuration;
Learning a multi-classifier according to a SVM (Support Vector Machine) ensemble algorithm for the super category node, the augmentation category node, and the subcategory node, and calculating a reliability score in each category; And
Collecting reliability scores of each category to detect semantic objects included in the image
The method comprising the steps of: The method according to claim 1,
The step of calculating the confidence score comprises:
Projecting binary SVM classifiers for the ROI in each of the categories with a pseudo probability; And
Calculating a multi-class margin in the category based on the pseudo-probability in each category, calculating a normalized multi-class margin for the multi-class margin, and calculating a reliability score in each category
The method comprising the steps of: 3. The method of claim 2,
(H | r) for the one or more super category nodes in the region of interest r, and calculates a confidence score CS _h ^(k) (r) for the kth super category node according to the following equation: Calculates a reliability score for the one or more super category nodes,

Here, A, B, and C are predetermined coefficients,

Is the pseudo-probability for the k-th super-category node h ^(k)

Is a multi-class margin for h ^(k)

Is a normalized multi-class margin for h ^(k) , Ω _H 은 is the maximum number corresponding to h ^(k) in the super-category space Ω _H. 3. The method of claim 2,
(M | r) for the one or more enhancement category nodes in the region of interest r and calculates a confidence score CS _m ^{(k ')} (r ^') for the k'th enhancement category node according to the following equation: r ) To yield a confidence score for the one or more enhancement category nodes,

Here, A ', B', and C 'are predetermined coefficients,

Is the pseudo probability for the k 'th enhancement category node m ^(k') ,

Is a multi-class margin for m ^{(k ')} ,

Is m ^{(k ')} the normalized multiclass margin, for | Ω _M | is m ^(k in augmented category space Ω ^_M' maximum number of the object detecting method, characterized in that corresponding to ^a). 3. The method of claim 2,
(L | r) for the one or more sub-category nodes in the region of interest r and calculates a confidence score CS _l ^{(k ")} (r ) To yield a confidence score for the one or more sub-categories,

Here, A &quot;, B &quot;, and C "are predetermined coefficients,

Is the pseudo-probability for the k "th sub-category node l ^(k") ,

Is a multi-class margin for l ^{(k ")} ,

Is ^"normalized multiclass margin for, | Ω _L | is l ^(k in the sub-category space Ω _L l ^(k)" The maximum number of the object detecting method, characterized in that corresponding to ^a). One or more super category nodes corresponding to each semantic object as a child of the root node and to be trained using a training data set, according to a characteristic of a region of interest of an image, a root node forming a semantic object category space including semantic objects, One or more enhancement category nodes to be trained using the training data set as a child of each super category node, one or more subcategory nodes to be trained using the training data set as a child of each enhancement category node Function to perform;
Learning the multi-classifier according to the SVM ensemble algorithm for the super-category node, the enhancement category node, and the sub-category node to calculate a reliability score in each category; And
And collecting reliability scores of each category to detect a semantic object included in the image,
A computer-readable recording medium storing computer-readable code for performing object detection using hierarchical feature modeling of semantic objects contained in an image in an object detection apparatus.

Images