KR20220125008A - Rollback and Deep Learning Method for Rapid Adaptive Object Detection - Google Patents

Abstract

The present invention relates to a rollback deep learning method for fast adaptive object detection. In the rollback deep learning method for fast adaptive object detection, by utilizing active learning (AL) and rollback-based semi-supervised learning (SSL), the AL can select more informative and representative samples that measure uncertainty and variability, and the SSL divides the selected streaming image samples into bins, wherein each bin repeatedly transmits identification information of EER and CNN models to the next bin until convergence and integration with an EER rollback learning algorithm are achieved. The EER model provides fast short-term myopic adaptation, and the CNN model provides gradual long-term performance improvement, which can overcome noisy and biased labels in various data distributions.

Description

Rollback and Deep Learning Method for Rapid Adaptive Object Detection

The present invention relates to a rollback deep learning method for rapid adaptive object detection, and more particularly, to improve the learning performance when detecting streaming objects, reliable label estimation and performance by combining rollback learning-based EER and ASSL It relates to a rollback deep learning method for fast adaptive object detection that can be improved.

Recently, there have been notable advances in artificial intelligence and machine learning. Pattern classification techniques, pattern extraction, and the task of making predictions based on knowledge acquired from patterns remain challenging, especially in the complex and changing real world. Object detection is one of the important sub-areas of computer vision. In the past few years, deep learning techniques have been successfully applied in many computer vision fields since Krischesky et al broke the barrier in 2012. Most performance gains depend on the availability of large datasets that are correctly labeled assuming a simple static data distribution. However, the underlying distribution in the real world is highly variable and disproportionate, and rebuilding the system requires difficult and time-consuming labor and effort. Also, some collected samples are prone to bias or misrepresentation and may lead to poor performance. Fully labeled retraining is practically impossible due to cost and time constraints, but collecting unlabeled data is relatively inexpensive. Being able to utilize both labeled and unlabeled data can be of great practical value. Semi-supervised learning (SSL), which allows unlabeled data to be used with labeled data, can significantly improve learning performance.

Active learning (AL) can be thought of as a special case of semi-supervised learning. Active learning (AL) uses minimal queries to Oracle to obtain labels for unknown data to optimize model performance. In most cases, the intuition of experienced experts is important for successful model convergence in the active learning (AL) process. Human labeling is always costly as it is labor intensive, time consuming and error prone. Therefore, how to effectively utilize general human effort is one of the main concerns of active learning (AL) research. The advantage of active learning (AL) is that it explores unlabeled samples, taking into account the potential of each unlabeled sample. Active learning (AL) uses selective sampling to search for the most informative samples with minimal labeling cost instead of relying on labeled data in a passive way. In active learning (AL), the labeling effort of training data is greatly reduced compared to the existing supervised learning method. When unlabeled data samples are labeled, they are used for forward learning operations, but can lead to poor performance of the model if there are noisy samples or labels. Noise from mislabeled samples or outliers negatively impacts generalization ability building. In addition, it can have a negative effect, so Bi-Directional Active Learning (BDAL) has been recently proposed to improve model generalization ability. Bidirectional active learning (BDAL) combines forward and backward learning processes based on EER-based uncertainty formulas and achieves superior performance compared to unidirectional efforts. However, BDAL is still time-consuming to redo in data selection and undo in data labeling, and cannot be used in a changing streaming data environment, so it needs fast adaptive learning capabilities. Motion information from weakly labeled videos can be used to learn high-precision object suggestions. Krishna et al. show that integrating candidate object recognition with weak supervised learning improves detection performance. Deep Neural Networks such as CNNs are one of the most powerful hierarchical learning networks that provide generalization capabilities and have been doing so for 30 years. However, the slow convergence of deep neural networks is still a very difficult problem and a significant obstacle, especially in real-time data streams.

The combination of AL and SSL, called ASSL, can efficiently improve object detection performance in various data distributions, overcoming the limitations of CNN approaches that require large-scale label data sets and long training times. Recently, an incremental ASSL approach has been proposed and shows that CNN can efficiently handle time-varying problems due to its knowledge transfer ability and incremental learning. However, the combined method of AL and SSL still requires a long CNN deep learning step and consumes a trivial amount of training time to meet the fast adaptive learning function.

References 1 to 3 have been proposed in relation to the above.

Reference 1: Korean Patent No. 10-2174048 Reference 2: Korean Patent No. 10-1990123 Reference 3: Korean Patent No. 10-1882743

Therefore, in order to solve the problems and limitations of the prior art, the present invention provides a rollback deep learning method for rapidly adaptive object detection that can overcome noisy and biased labels in various unbalanced data distributions. have.

The present invention in order to solve this technical problem;

a first step of selecting streaming image samples that measure uncertainty and diversity in an input data stream through active learning (AL); And through semi-supervised learning (SSL), the selected streaming image sample is divided into bins, and each bin repeats the identification information of EER and CNN models to the next bin until convergence with the EER rollback learning algorithm. A second step of transmitting; provides a rollback deep learning method for fast adaptive object detection, characterized in that it includes.

In this case, the first step is characterized in that a batch of data samples is collected and a step of selecting streaming image samples through joint sampling by a joint sampling algorithm.

And the second step is characterized in that the sample is reselected and relabeled by performing forward learning and rollback learning based on EER through a semi-supervised learning (SSL) algorithm with a rollback function.

In addition, the rollback learning (rollback learning) is characterized in that the sample removal, label change and reselection in the bin (Bin).

According to the present invention, by combining ERR-based rollback learning and bin-based SSL for CNN object detectors when there is a noisy data distribution, it overcomes noisy and biased labels in actual various unbalanced data distributions, but compared to the existing state-of-the-art detectors. It can achieve higher accuracy and reduce human effort.

In particular, the present invention supports fast short-term myopia adaptation, and CNN can expect a gradual long-term performance improvement, so that it can work with high-speed feed-forward networks and extreme learning machines to build adaptive and improved deep learning architectures to perform better in noisy data distributions. A flexible and fast adaptive learning sequence can be achieved.

Figure 1 is to explain the rollback deep learning method for fast adaptive object detection according to the present invention, (a) bean-based SSL using only forward deep learning and (b) forward reselection combined with forward deep learning, rollback removal and an information flow snapshot of EER rollback learning consisting of rollback redirection.
2 is a diagram illustrating a rollback deep learning architecture for fast adaptive object detection according to the present invention.
3 is a graph of experimental results obtained by varying learning rates using Adam and the SGD optimizer.
4 is an information flow diagram of a rollback-based ASSL.
5 is a diagram illustrating a noisy image sample with a labeling result.
6 is a performance comparison graph measured by the average precision (AP) of the present invention (EER-ASSL) using a noisy local image.
7 is a diagram showing the detection results of IASSL for a local data set of a chair, a sofa, and a table.
8 is a graph comparing the performance of the state-of-the-art object detector and the present invention (EER-ASSL) when measuring mAP for a benchmark data set.

Hereinafter, the features of the rollback deep learning method for fast adaptive object detection according to the present invention will be understood by the embodiments described in detail with reference to the accompanying drawings.

Prior to this, the terms or words used in the present specification and claims should not be construed as being limited to conventional or dictionary meanings, and the inventor appropriately defines the concept of the term in order to best describe his invention. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be done.

Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention, so at the time of the present application, they can be replaced It should be understood that various equivalents and modifications may exist.

The rollback deep learning method for rapid adaptive object detection according to the present invention proposes a method (EER-ASSL) that combines EER learning, which is a rapid adaptive learning framework for object detection, and a deep learning SSL algorithm. In the present invention, like BDAL (Bi-Directional Active Learning), when a suspicious data sample is inspected in a labeled data set as in FIG. rebuild In rollback bin-based SSL, a selected batch sample is split into multiple bins, and each bin iteratively conveys differential knowledge of CNN deep learning and EER-based rollback learning for fast adaptation. In this case, the rollback learning method is built into bin-based SSL to quickly remove the influence of noisy, uncertain samples from an unbalanced distribution. The above process is repeated until convergence is achieved. The characteristics of the method (EER-ASSL) proposed in the present invention are as follows.

First, the ERR-ASSL of the present invention provides a fast adaptive learning framework for efficient object detection in a changing environment. This method does not rely on the assumption of static data distribution employed by most state-of-the-art detection techniques. It combines EER rollback learning with CNN deep learning to transfer the dynamic differential knowledge of the model from the current bin to the next until convergence. Therefore, the ERR-ASSL of the present invention can overcome noisy and biased labels in an unknown data distribution. Here, the EER model enables fast short-term adaptation and the CNN model helps to provide a gradual long-term performance improvement.

In addition, the rollback learning method using EER in ERR-ASSL of the present invention effectively utilizes the differential function by removing outliers or re-labeling mislabeled samples. While this provides a fast adaptive learning function, the one-way labeling and two-way approach as in Fig. 1(a) requires time-consuming redo to undo in the data selection and learning process. The EER-based ASSL of the present invention effectively utilizes the myopic adaptive learning function of the EER and overcomes the shortcomings of the long learning time of the deep learning object detector.

In addition, the ERR-ASSL method of the present invention is compared with state-of-the-art detection methods such as Faster RCNN , SSD300 and YOLOv2 using Pascal VOC 2007, 2012, MS COCO benchmark data set and local data set. improve the detection rate.

Hereinafter, the latest related work related to the present invention will be briefly described, the system overview of the present invention will be described in detail, and then detailed information about the EER-ASSL algorithm of the present invention will be mentioned, the results will be shown, and the conclusion will be described later.

First, a look at the latest related work related to the present invention.

First, let's look at ASSL (Active Semi-Supervised Learning). In many cases, it is expected that labeled data sets will be provided and fixed, but this is not always the case. Using specialized oracle or algorithm labels and using active learning (AL) can change the labeled dataset. Active learning (AL) can remove noisy data and label more data types correctly. Active learning (AL) plays an important role when smaller amounts of data need to be labeled, and algorithms can decide when to label and when not to label. Zhu et al. first introduced the idea of an effective combination of SSL (Semi-Supervised Learning) and Active Learning (AL) in 2003. He applied a Gaussian random field model with AL so that unlabeled data can minimize the risk of harmonic energy minimization functions. Pool-based active learning or selective sampling can be used to minimize the number of query choices. Combining AL and SSL together can leverage both labeled and unlabeled data to improve classification performance. Tong and Koller et al. conducted a similar study to reduce the version space size of SVM. Cohn et al. minimize the estimated generalization error by reducing the variance component. Once the query is chosen, most of these active learning methods are unlikely to exploit large amounts of unlabeled data. Some researchers applied SSL (Semi-Supervised Learning) during the training phase. Chaloner and Verdinelli applied the Bayesian approach in 1995. Gaussian random fields are well suited for harmonious functions due to the combination of SSL and AL. ASSL has many advantages because it can efficiently estimate query points, then randomly select an ambiguous sample and estimate the expected generalization error. Better label selection criteria can train object detectors while eliminating ambiguities and imbalances in sample distribution and overcoming incomplete labeling and selection bias.

Next, Expected Error Reduction (EER) reduces the generalization error as a selection criterion when labeling new samples based on retaining-based active learning (AL). The effectiveness of this method has been demonstrated in text classification applications. The rationale is to choose samples that minimize future generalization errors. The unlabeled pool represents the test distribution and serves as the validation set. Since there is no knowledge of the labels of unlabeled samples, EER estimates the average case potential loss in terms of average case, worst case, or best case. Such an EER explores changes in the predicted error model by selecting samples that lead to the largest change in the current model. A similar approach can be found in variance reduction methods that seek to minimize output variance. After adopting variance and logistic regression, it is extended to the expected variance reduction method for logistic regression. In minimum-maximum active learning, the worst-case criterion is adjusted to select the sample that minimizes the objective function after labeling a new sample. This method is extended to consider all unlabeled data when calculating the objective function. However, the EER-based retraining approach requires a lot of computation because it explores all unlabeled data and all possible labels.

Hereinafter, a system overview of the present invention will be described in detail with reference to FIG. 2 .

The system for the present invention consists of joint sampling and EER-ASSL for the implementation of a rollback deep learning method for fast adaptive object detection. The method proposed in the present invention (EER-ASSL) integrates the rapid modeling function of the EER method and the precise and progressive learning function of the CNN. The method (EER-ASSL) proposed in the present invention uses the step-by-step utilization of the bin-based SSL algorithm with the search function and rollback function of the AL algorithm. This method minimizes training time as well as costly human effort while maintaining high-quality label datasets and high-precision object detectors in the adaptive learning process. Co-sampling methods that take into account uncertainty, variability, and reliability criteria can select more informative and reliable samples with less redundancy. Since the uncertainty criterion can lead to the selection of noisy or duplicate samples, the diversity criterion is applied to the clustering-based deduplication algorithm. In the present invention, the joint sampling based AL learning algorithm is integrated with the rollback bean based SSL algorithm. A simplified sketch of EER-ASSL for fast adaptive object detection in a dynamically changing environment is shown in FIG. 2 . Sample batches are selected based on the uncertainty and diversity sampling of the image stream rather than one image at a time. After co-sampling, the sampled batch stream is divided into bins for the rollback SSL algorithm. The samples in the image bins are unlabeled samples and are pseudo-labeled by bin-based SSL using both CNN detector and EER-based rollback learning methods. However, the pseudo-data set contains not only samples that are incorrectly labeled, but also labels that are biased from an unbalanced data distribution. Since these samples have a detrimental effect and do not contribute in any way to building a better object detector, noisy samples should be excluded from the reliable samples. The EER-based learning method proposed in the present invention was adopted for fast forward learning and rollback learning to enable more informative sampling and learning through reselection and relabelling.

Next, learning of the method (EER-ASSL) proposed in the present invention will be described.

In the present invention, we use a very limited number of labeled data for training the EER model and the fine-tuned CNN model. Fine-tuned CNN models and EER models are built using limited labeled data samples. The ensemble network consists of a CNN detector and an EER model to perform object detection. Incremental ASSL is employed so that batches of data samples are collected from the input data stream and selectively sampled by a joint sampling algorithm. The selected sample is divided into bins whose batch and bin size are determined according to the image capture quality. The dataset initially labeled by the CNN and EER ensemble and the pseudo-training dataset are used to train new CNN and EER models. That is, it is used to update the model for the next empty cycle. The new EER model is involved in a rollback learning process, which consists of removing samples from bins, changing labels, and reselecting them if necessary. Bin-based incremental learning is processed by integrating forward learning for sample reselection and rollback learning for removal and relabeling (see Fig. 3). The new CNN model is also used in the new joint sampling process. This process is repeated until convergence.

은 레이블링된 트레이닝 데이터 세트를 나타내고,

은 레이블이 없는 데이터 세트이다(

). 만약 선택한 샘플 x가 y로 레이블링되고 LD에 추가되면,

로 표시된다.

는 LD의 EER모델과

의

를 나타낸다. 가장 유익한 데이터 샘플은 레이블링되지 않은 데이터 세트를 사용하여 예상 엔트로피를 최소화하여 예상 오류 감소를 극대화하는 것으로 가정한다. 레이블링되지 않은 데이터 세트에서 가장 유익한 데이터 샘플이 다음의 방정식 (1)을 충족하는 샘플로 선택된다.First, an EER forward learning processor will be described. In the present invention, the EER method proposed and used in the pattern classification problem is adopted. The purpose of the EER method is to select samples that minimize the generalization error in future steps. Because test data is not available in advance, a portion of the streaming data set is used as a validation data set to estimate future errors. The true label of the unlabeled data set is unknown, and future errors are roughly estimated using the expected log loss for unlabeled data.

represents the labeled training data set,

is an unlabeled data set (

). If the selected sample x is labeled y and added to LD,

is displayed as

is the EER model of LD and

of

indicates It is assumed that the most informative data samples use unlabeled data sets to minimize expected entropy to maximize expected error reduction. The most informative data sample in the unlabeled data set is chosen as the sample satisfying the following equation (1).

은 현재 모델의 레이블 정보를 나타내고 두 번째 항은 모델

의 레이블이 없는 데이터 UD에 대한 예상 엔트로피의 합이다. 위 방정식 (1)은 직렬 모드 학습 프로세스를 공식화하는 반면 새 모델은 UD에서 각 새 데이터 샘플의 레이블링 직후에 업데이트된다. 그러나 학습의 철저한 반복은 실제로 무거운 계산 오버 헤드로 인해 어려움을 겪는다. 본 발명에서는 레이블이 없는 스트림 데이터 세트를 배치로 나누고, 각 배치는 도 3과 같이 UD에서 레이블이 없는 전체 데이터 샘플을 처리하는 대신 빈으로 분할된다.where C denotes the object class, the first term

represents the label information of the current model and the second term is the model

is the sum of the expected entropy for the unlabeled data UD of Equation (1) above formalizes the serial mode learning process, whereas the new model is updated immediately after labeling of each new data sample in the UD. However, exhaustive iterations of learning actually suffer from heavy computational overhead. In the present invention, the unlabeled stream data set is divided into batches, and each batch is divided into bins instead of processing the entire unlabeled data sample in the UD as shown in FIG. 3 .

를 고려하여 다음의 방정식 (2)와 같이 다시 작성된다.In rollback SSL, a bean of unlabeled data samples is used for each training step. For time step i, equation (1) is bin

Taking into account, it is rewritten as the following equation (2).

에서 모델

에 대한 예상 엔트로피의 합이다. 공동 샘플링을 적용한 후 방정식 (2)를 반복적으로 적용하는 빈(bin) 데이터 세트에 대한 유사 레이블 집합

을 결정할 수 있다. 그러나

의 각 데이터 샘플에 대한 모델을 빌드하려면 여전히 많은 계산 오버 헤드가 필요하다. 따라서 방정식 (2)는 다음의 방정식 (3)과 같이 유사 레이블링된 집합

의 선택된 샘플에 대한 모델을 구축하여 근사치를 구한다.Here, the first term represents the label information of the current model, and the second term is an unlabeled data bin.

model from

is the sum of the expected entropy for . A set of pseudo-labels for a bin data set iteratively applying Equation (2) after applying co-sampling

can be decided But

It still requires a lot of computational overhead to build a model for each data sample in Therefore, equation (2) is a pseudo-labeled set as in equation (3)

It is approximated by building a model for a selected sample of

의 선택된 샘플에 대한 현재 모델의 레이블 정보를 나타내고, 두 번째 항은 가중치 모델

과 레이블링 되지 않은 데이터

에 대한 예상 엔트로피의 합이다.where the first term is the labeled pseudoset

represents the label information of the current model for the selected sample of

and unlabeled data

is the sum of the expected entropy for .

레이블이

이고 LD에 추가 된 경우

로 표시된다. 방정식 (1)을 사용하여 레이블이 없는 전체 데이터 샘플에 대한 모델을 빌드하는 대신 모델을 한번 빌드한다는 것을 알 수 있다. EER 순방향 학습 프로세스는 CNN 검출기에 의해 실패한 빈의 재교육을 위한

의 재선택 알고리즘에 사용된다. 현재 레이블링된 데이터 세트에 추가 된 재 선택된 샘플과 결합 된 데이터 세트는 bin 기반 SSL 단계에 대한 CNN 모델을 유지하는 데 사용된다.selected sample

label

and added to LD

is displayed as Using Equation (1), we can see that we build the model once instead of building the model over the entire unlabeled data sample. The EER forward learning process is designed for retraining of failed bins by the CNN detector.

used in the reselection algorithm of The dataset combined with the reselected samples added to the current labeled dataset is used to maintain the CNN model for the bin-based SSL phase.

Next, the EER rollback learning processor will be described. The goal of this processor is to examine the most uncertain labeled samples that interfere with the current model and select new samples that either replace the most uncertain samples or relabel the recently pseudo-labeled samples. For rollback learning, we use EER estimation to minimize the expected entropy for pseudo-labeled data samples. When considering the computation time, only the most recent similarly labeled data set is examined without considering the entire unlabeled data set. The rollback process looks for candidate samples to be removed or relabeled from the most recent pseudo-labeled samples. The rollback learning process performs label authentication of the rollback samples by either relabeling the rollback samples or reselecting them from their neighbors. Rollback learning is divided into two types: 1) the removal process and 2) the relabel process.

First, looking at the removal process, the removal rollback process is to undo the bad effects of the data samples by removing them from the current pseudo-data set added to the labeled dataset because the rollback samples are suspected to interfere with the EER model. Models using the previously labeled data set are expected to be more stable than the last labeled data set, so they are used to update the model. While removing rollback training, the influence of rollback samples is removed from the model's pseudo-dataset. The removal rollback process deletes untrusted samples from the pseudo-labeled data set and untrusted samples are detected. The rollback removal process is composed of the following equations (4) and (5).

은

을 제거한 후 레이블링된 데이터 세트를 나타낸다. 방정식 (4)는 실제로 계산할 수 없는 많은 계산 시간이 필요하기 때문에 롤백 샘플,

은 레이블링되지 않은 전체 데이터 세트의 샘플 대신 현재 단계의 의사 레이블링된 데이터 샘플에서 선택된다. 트레이닝 데이터 세트는 LD에서

로 롤백하고 현재 기능 공간의 이웃에서 새 샘플을 대체한다. 제거로 인해 레이블링되지 않은 데이터 세트에 대한 엔트로피를 최소화하는 제거할 레이블링된 샘플 풀을 찾으며, 다른 유사 레이블링된 샘플은 재선택 프로세스에서 다시 선택된다. 제거를 위한 롤백 샘플은 다음과 같이 분류 모델인 방정식 (6)을 사용하여 마지막 의사 레이블링된 샘플의 빈에서만 선택된다.here

silver

represents the labeled data set after removing . Equation (4) requires a lot of computation time that cannot be calculated in practice, so the rollback sample,

is chosen from pseudo-labeled data samples of the current stage instead of samples from the entire unlabeled data set. The training data set is from LD.

rolls back to and replaces the new sample in the neighborhood of the current feature space. A pool of labeled samples to be removed is found that minimizes entropy for the unlabeled data set due to removal, and other similarly labeled samples are reselected in the reselection process. Rollback samples for removal are selected only from the bins of the last pseudo-labeled sample using Equation (6), which is a classification model as follows.

은 재선택 프로세스를 위해 제거할 롤백 샘플을 나타낸다. 선택한 롤백 샘플이 각각

에 의해 의사 레이블링된

이고 LD에서 제거 된 경우 집합 차이

으로 표시된다.here

represents the rollback samples to be removed for the reselection process. Each of the selected rollback samples

Physician labeled by

and the set difference if removed from LD

is displayed as

에서 선택한 레이블 재지정 샘플이 LD에 업데이트되거나 추가된다. 만약 롤백 학습 레이블 재지정 프로세스 후 롤백 샘플의 레이블이 변경되면 순방향 학습 레이블이 새 레이블로 대체된다. 이러한 방식으로 포워드 레이블링 오류가 수정된다. 만약 새 레이블이 정방향 학습 레이블과 동일하면 롤백 샘플이 새 샘플로 처리되고, LD에 추가될 것이다. 부스팅에 대한 유사한 아이디어가 논의되며, 잘못된 레이블이 붙은 샘플 후보가 더 집중 될 것이다. 한 샘플에 대한 레이블 재지정 롤백 학습은 다음의 방정식 (7)과 유사한 공식에 따라 수행된다.Next, take a look at the relabeling process:

The selected relabeling sample is updated or added to the LD. If the label of the rollback sample is changed after the rollback learning relabeling process, the forward learning label is replaced with the new label. In this way, forward labeling errors are corrected. If the new label is the same as the forward learning label, the rollback sample will be treated as a new sample and added to the LD. Similar ideas for boosting are discussed, and sample candidates with false labels will be more concentrated. The relabel rollback learning for one sample is performed according to a formula similar to the following equation (7).

는

에 의해 할당된 의사 레이블링된

를 나타낸다. Z는 다음에 의해 계산된 정규화 계수이다.here

Is

pseudo-labeled assigned by

indicates Z is the normalization coefficient calculated by

의 레이블이 재지정된 후보 풀 측면에서 아래의 방정식 (8)에서와 같이 레이블 재지정 롤백 학습 프로세스를 공식화한다.Taking into account the computational overhead of calculating the model for each relabeled candidate,

In terms of the relabeled candidate pool of , we formulate the relabel rollback learning process as in equation (8) below.

는

에 의해 할당된 의사 레이블링된

를 나타낸다. Z는 다음에 의해 계산된 정규화 계수이다.here

Is

pseudo-labeled assigned by

indicates Z is the normalization coefficient calculated by

Hereinafter, the algorithm of the method (EER-ASSL) according to the present invention will be described.

는 공동 샘플링 후 현재 배치에서 채굴 된 샘플을 나타낸다. 여기서는 롤백 빈 기반 SSL 알고리즘에 중점을 둔다.

는 빈 기반 SSL의 기밀 배치 데이터 세트를 나타내며 기밀 배치 데이터 세트의 컨테이너로 사용된다.

의 카디널리티가 신뢰 매개변수

가 되면 신뢰 표본 선택 프로세스가 중지된다.

는

을 만족하는 샘플로 초기화된다. 기밀 샘플링 전략은

에서 샘플을 선택하고

을 사용하여 현재 딥 피처 공간의 거리 메트릭에 따라

에 추가한다. 여기서

는 딥 피쳐 공간에서 두 샘플

와

간의 유클리드 거리이다. CNN은

에 있는 기밀 샘플의 빈 시퀀스를 사용하여 다시 트레이닝 된다. 기밀 샘플은 빈으로 분할되고

으로 표시된 빈 풀에 저장된다. 각 롤백 빈 기반 SSL 단계에서 신뢰도 점수는 현재 CNN 감지기에 의해 의사 샘플에 할당된다. 레이블링된 데이터

은 각각 처음에 CNN 검출기 모델

및 EER 모델

을 초기화하는데 사용된다.

은 검증 데이터를 사용하여

에 의해 계산된다. 각 빈에 대해 각각

를 사용하여 CNN 모델

을 세운다.

은

에 의해 계산 된 빈 점수 중 최대 정확도 즉,

를 나타낸다. 만약 성능이 향상 즉

이면,

및

를 업데이트하여 다음 단계로 이동한다. 시간 단계 i에서 각 빈에 대해

및

를 사용하여 CNN 모델

을 세우고, 세 가지 사례인 사례 1)

, 사례 2)

및 사례 3)

로 나뉜다. 여기서

은 탐사 가능성에 대한 허용 한계이다.The main work of EER-ASSL proposed in the present invention consists of joint sampling-based AL and rollback bean-based SSL. In the AL process, batches of data samples are collected from the input data stream, processed by a joint sampling algorithm for information samples with minimal redundancy, and partitioned into bins. In rollback SSL, EER-based rollback learning and bean-based SSL are combined for fast adaptive learning. Restricted labeled samples are used to initialize the CNN model and the EER model. The model is trained on a bean-based incremental SSL scheme. In CNN model training, if the performance criterion is violated, the EER method is activated for fast rollback learning. The volume of reliable labeled data sets is increased by adding similarly labeled data samples. The enlarged LD is then used to build CNN and EER models. This process is repeated until convergence. It can be seen that the EER rollback model provides fast short-term adaptation and the CNN detector model provides progressive long-term performance improvement, respectively.

represents the samples mined in the current batch after co-sampling. The focus here is on the rollback bean-based SSL algorithm.

represents the confidential batch data set of bean-based SSL and is used as a container for the confidential batch data set.

The cardinality of the trust parameter

, the confidence sample selection process is stopped.

Is

It is initialized to a sample that satisfies Confidential sampling strategy

select a sample from

according to the distance metric in the current deep feature space using

add to here

are two samples in deep feature space.

Wow

is the Euclidean distance between CNN

It is retrained using an empty sequence of confidential samples in Confidential samples are divided into bins and

It is stored in an empty pool marked with . At each rollback bean-based SSL step, a confidence score is assigned to the pseudo-sample by the current CNN detector. labeled data

Each is initially a CNN detector model

and EER models

used to initialize

using the validation data

is calculated by each for each bin

CNN model using

set up

silver

The maximum accuracy among the blank scores computed by

indicates If the performance improves i.e.

back side,

and

update to go to the next step. For each bin at time step i

and

CNN model using

, and three cases, Case 1)

, case 2)

and case 3)

is divided into here

is an acceptable limit for exploration possibilities.

및

;

을 업데이트한다. 선택한 빈을 제거하면 빈 풀

이 줄어드는 것을 알 수 있다.Case 1: Get the best bean for the next step

and

;

update If you remove the selected bean, the empty pool

It can be seen that this decreases.

에서 제거 샘플을 찾고, 2) 레이블 재지정 샘플을 찾고 방정식 (5)를 기반으로 한 롤백 학습 프로세스를 사용하여

에서 새 레이블을 할당하고, 방정식 (3)에 기반한 EER 순방향 학습 프로세스를 사용하여 재선택하여

을 업데이트한다. Case 2: Perform the following substeps. 1) using the rollback learning process using equation (7)

find the sample removed from, 2) find the relabel sample and use a rollback learning process based on equation (5)

by assigning a new label from , and reselecting using the EER forward learning process based on equation (3)

update

또는

조건을 만족하기 전 또는 시간 제한까지 반복한다. 만약 조건

이 충족되면,

와

를 업데이트한다.The forward rollback learning process above is

or

Repeat before the condition is satisfied or until the time limit. if condition

When this is met,

Wow

update

의 레이블링된 데이터를 잘못 레이블하고

을 업데이트한다. Case 3: Oracle

incorrectly label the labeled data of

update

는 시간 t에서

및

의 트레이닝에 사용되는 트레이닝 데이터 세트

을 세우는데 사용된다. 이 과정은 수렴 될 때까지 반복된다. 마지막으로 롤백 빈 기반 SSL은 두 가지 모델 f 및 g와 확장된 레이블 데이터 세트 LD를 생성한다. EER 기반 롤백 학습과 빈 기반 SSL을 결합하면 동적으로 변화하는 환경에서 노이즈가 있는 스트리밍 샘플에서도 신속 적응형 객체 감지기를 얻을 수 있다. 롤백 빈 기반 SSL 알고리즘은 아래의 알고리즘 1에 요약된다.The rollback process in Case 2 can significantly reduce the Oracle labeling step.

is at time t

and

training data set used for training in

used to build This process is repeated until convergence. Finally, rollback bean-based SSL produces two models f and g and an extended label data set LD. Combining EER-based rollback learning with bean-based SSL yields fast adaptive object detectors even on noisy streaming samples in dynamically changing environments. The rollback bean based SSL algorithm is summarized in Algorithm 1 below.

Hereinafter, experimental examples such as the method (EER-ASSL) according to the present invention will be described.

The method according to the present invention (EER-ASSL) performs extensive experiments using benchmark data sets such as PASCAL VOC and local data sets and compares the performance with state-of-the-art detector technologies such as Faster RCNN, SSD300 and YOLOv2. The experimental implementation was performed using a single server with a single NVIDIA TITAN X with cuDNN and Tensorflow. The experimental setup was used as a Darknet-19 CNN model, and the default detector is YOLOv2, a state-of-the-art object detector.

<benchmark data set>

- PASCAL VOC data set: There are two versions of the famous PASCAL VOC benchmark, Pascal VOC 2007 and 2012. Pascal VOC 2007 consists of 20 classes with a total of 9963 images (train/validation/test) and 24,640 annotated objects. Pascal VOC 2012 has 20 classes with 11,530 images (train/validation/test) containing 27,450 annotated objects. The YOLOv2 model was trained using the PASCAL VOC 2007 data set and the PASCAL VOC 2012 data set. There are four superclasses in the Pascal VOC 2007 data set: people, animals, vehicles, and indoors. The experiments of the present invention focused on the class of subjects in the indoor environment (eg bottles, chairs, tables, flowerpots, sofas and TV monitors).

- Local data set: 450 chair images, 450 flowerpot images, 450 turnstile images and 450 table image data sets are selected from the local area. The present invention uses an input image resolution of 416 x 416 pixels. Twelve chair images were used as initial label data samples. 98 images were randomly selected and used for the validation data set. The remaining 340 images from each class were used in the unlabeled data set for the experiment. When training using the PASCAL VOC data set, the local data set produces very poor detection results in YOLOv2.

<Setting experimental parameters>

The object detection method of the present invention used PASCAL VOC challenge assessment. This applies if the average precision in equation (10) below is calculated by averaging the precision over a set of evenly spaced recall levels. Equation (10) below is the expression for average precision.

은 r보다 큰 모든 재현율에 대해 최대 정밀도를 취하는 보간 정밀도이다. 본 발명 등에서는 실측 자료와 예측이라는 두 경계 사이의 중첩을 계산하기 위해 IoU (Intersection over Union)를 사용한다. 도 7에서와 같이 적색 경계 박스는 Ground Truth, 검은색 경계 박스는 EER-ASSL, 노란색 경계 박스는 YOLOv2 VOC 모델을 나타낸다. 실험에서 IoU 임계 값은 0.5(IOU => 0.5)로 미리 정의된 것으로 간주한다. 예측 성능이 임계값을 초과하면 올바른 것으로 간주하고 그렇지 않으면 잘못된 것으로 간주된다. 최상의 성능을 얻기 위해 기울기 기반 최적화 방법 Adam과 SGD(stochastic gradient descent)를 적용한다. 이 두 옵티마이저에 대해 학습률 0.001을 선택한다. 그러나 SGD 옵티마이저는 도 3에서 볼 수 있듯이 500 epoch의 Adam에 비해 실험에서 훨씬 느리다.here

is the interpolation precision taking the maximum precision for all recalls greater than r. In the present invention and the like, Intersection over Union (IoU) is used to calculate the overlap between two boundaries, namely, actual data and prediction. As shown in FIG. 7 , the red bounding box represents Ground Truth, the black bounding box represents EER-ASSL, and the yellow bounding box represents the YOLOv2 VOC model. In the experiment, the IoU threshold is assumed to be predefined as 0.5 (IOU => 0.5). If the prediction performance exceeds the threshold, it is considered correct, otherwise it is considered incorrect. To get the best performance, we apply the gradient-based optimization method Adam and stochastic gradient descent (SGD). We choose a learning rate of 0.001 for these two optimizers. However, the SGD optimizer is much slower in the experiment compared to Adam at 500 epochs, as shown in FIG. 3 .

3 shows experimental results of various learning rates while training a local data set using Adam and the SGD optimizer. Since the present invention has a higher learning rate of 0.001 and stable performance, we select the Adam optimizer with learning rates of 0.01 and 0.001. The training process is divided into several stages, such as stage 1 and stage 2. The SGD and Adam optimizers were used to train different numbers of bins. For both steps, the total number of bins is 20. This experiment shows that the Adam optimizer has faster (time) convergence than the SGD optimizer with higher AP. For this reason, the Adam optimizer with a learning rate of 0.001 was chosen. The full labeled data set was divided into two steps. The parameter combinations in step 1 are uncertainty, variability and reliability [0.8, 0.8, 0.8]. Change the parameter combination to [0.8, 0.6, 0.8] or [0.8, 0.8, 0.6] depending on the performance.

<Experimental effect of the method according to the present invention (EER-ASSL)>

The performance of EER-ASSL, IASSL, and simple SSL proposed in the present invention is compared in FIG. 4 . For uncertainty, variability, and reliability for all experiments, the collaborative sampling parameters are set to [0.8, 0.6, 0.8]. The Adam optimizer was used with a learning rate of 0.001. It can be seen that EER-ASSL has much improved performance compared to incremental ASSL. Simple SSL's performance improved little.

<Test local image with noise>

The much improved EER-ASSL, IASSL, and Iterative Cross Learning (ICL) were compared in the noisy image using the local noise image of FIG. 6 . ICL is tested for image classification performance, while EER-ASSL is evaluated for object detection performance where both the bounding box location and class label are noisy. The learning rate of the Adam Optimizer is set to 0.001. The experimental results are shown in FIG. 3 . At first, EER-ASSL shows low performance (AP) affected by noise data, but after the 4th bin of the first stage, it outperforms other methods. The experimental results are reflected in FIG. 7 . EER-ASSL shows excellent performance in various lighting changes. 7 shows the detection results of IASSL for the local data sets of chairs, sofas, and tables. In all cases, the red bounding box represents the actual data, the black bounding box represents EER-ASSL, and the yellow bounding box represents the YOLOv2 VOC model.

<Comparison between the method according to the present invention (EER-ASSL) and the state-of-the-art technology>

The method according to the invention (EER-ASSL) is compared with several advanced object detectors. Each of the four subjects had 100 labeled and 300 unlabeled data samples, and for fair evaluation, local chair, sofa and table images were mixed with PASCAL VOC test data. The detectors were trained on the same benchmark data set and local data set for fair evaluation. Table 1 below shows the comparison results in which each column represents the composition ratio of benchmark data and local data.

Table 1. Comparison of performance of state-of-the-art object detectors and EER-ASSL in terms of mAP measurements

YOLOv2 trains a network using the ImageNet 1000 class classification dataset and then modifies the network to perform detection. This joint training classification and detection data is much larger than the local data set, which is limited to 100 images for each class, such as a sofa or a turnstile. As a result, reducing the local training data does not significantly affect the YOLOv2 model in the experiment, whereas the EER-ASSL model of the present invention has already adopted the local data, and when the proportion of the constituent data is more than 50%, the EER-ASSL adds additional columns (10,90 ) and (25,75) outperform other state-of-the-art methods.

Table 2 below shows the results of comparing the state-of-the-art object detectors such as EER-ASSL and Faster RCNN, SSD 300 and YOLOv2 in terms of mAP. Experiments were performed similarly to the previous work, considering progressive learning using Fast RCNN and Faster RCNN. Similarly, the present invention uses both local and benchmark data sets as the reference detector as YOLOv2. Both PASCAL VOC 2007 test data and local data are used. The joint sampling parameters are set to 0.8, 0.6 and 0.8 for uncertainty, variability and reliability, respectively. EER-ASSL is a significant improvement over other object detectors.

Table 2. EER-ASSL performance in mAP measurements on local data sets

Table 2 above summarizes the average precision of the state-of-the-art method on the local test data set. The EER-ASSL method proposed in the present invention shows performance improvement at higher mAP results by using a new object (object) such as a sofa or a ticket gate in a similar environment. As shown in Table 2, the adaptive properties of the EER-ASSL method significantly improve the detection performance with faster computational speed on local data sets and corresponding environments.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations are possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. The scope of protection should be interpreted by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

Claims (4)

a first step of selecting streaming image samples that measure uncertainty and diversity in an input data stream through active learning (AL); and
Through semi-supervised learning (SSL), the selected streaming image sample is divided into bins, and each bin repeatedly transmits the identification information of EER and CNN models to the next bin until convergence with the EER rollback learning algorithm. A rollback deep learning method for fast adaptive object detection comprising a second step.
The method of claim 1,
The rollback deep learning method for rapid adaptive object detection, characterized in that the first step is a step in which a batch of data samples is collected and a streaming image sample is selected through joint sampling by a joint sampling algorithm.
The method of claim 1,
The second step is fast adaptive, characterized in that the sample is reselected and relabeled by performing forward learning and rollback learning based on EER through a semi-supervised learning (SSL) algorithm with a rollback function. A rollback deep learning method for object detection.
The method of claim 1,
The rollback learning is a rollback deep learning method for fast adaptive object detection, characterized in that sample removal, label change, and reselection from the bin.

Images