JP2024514813A - Learning ordinal representations for object localization based on deep reinforcement learning - Google Patents

Abstract

A reinforcement learning based approach to the problem of query object localization is provided. An agent is trained to localize objects of interest specified by a small set of examples. A transferable reward signal is formulated using the example set by ordinal metric learning. This outperforms fine-tuning approaches limited to annotated images by enabling test-time policy adaptation to new environments where the reward signal is not readily available. Furthermore, the transferable reward allows the trained agent to be reused for new tasks such as annotation refinement and selective localization from multiple common objects across a set of images. Experiments on corrupted MNIST and CU-Birds datasets demonstrate the effectiveness of our approach. [Selected Figure] Figure 2

Description

This disclosure relates generally to image processing and image recognition. More specifically, this disclosure describes systems and methods for learning ordinal representations for object localization based on deep reinforcement learning.

As one skilled in the art can readily appreciate, in many fields it is often important to automatically discover one or more types of common objects in an image or set of images. In particular, fully supervised object detection or localization requires a large amount of human annotations (i.e., bounding boxes around target objects) during training, which is costly and impractical for cost-sensitive applications. For example, in distributed fiber optic sensing and digital pathology, high-quality annotations from experienced experts are quite limited, while weakly supervised object detection or localization (WSOD or WSOL) methods require only image-level annotations (classes). However, such learned annotations are often partial, referring to regions where the target object is most discernible, rather than the entire region. Finally, existing methods for co-localization are unsupervised and may provide unnecessary common objects as output when an image dataset contains multiple types of common objects.

Aspects of the present disclosure that are directed to systems and methods for addressing the above problems provide an advance in the art. Advantageously, our approach requires only a "seed data set" with accurate bounding box annotations.

In clear contrast to conventional fully supervised object detection/localization approaches, our algorithm requires a much smaller size for the seed dataset. Starting from the seed dataset, a large number of perturbed boxes are sampled as the reinforcement learning agent explores the image environment. The priority of these perturbed boxes is necessarily determined based on the intersection over union (IoU) with respect to the bounding boxes of the image ground truth. We encode this information into an ordinal representation that is trained together with the reinforcement learning annotation agent. Existing deep reinforcement learning-based object localization approaches cannot encode this information, resulting in much less efficient sampling.

Furthermore, in contrast to WSOD/WSOL methods, our approach explicitly focuses on similarity between common objects across different images in the same image class, rather than discrimination between different classes. Image-level class labels can be incorporated, but are not required.

More specifically, ambiguity regarding the class of the target object in co-localization is avoided by explicitly specifying the target object in the seed dataset. The algorithm works in a human-in-the-loop manner: in particular, given an image dataset, a human starts annotating some data and a reinforcement learning agent automatically labels the remaining data following the human guidance.

The framework of the present invention is motivated by the common problem of image data annotation in fiber sensing tasks, which are very time-consuming and labor-intensive. However, our approach can also be applied to other data modalities/applications, such as images for digital pathology, object tracking for video, temporal constraints for sound event detection, etc.

Operationally, each image is viewed as an environment with which the annotation agent can interact by moving its bounding box. The learned localization strategy is assumed to be generalizable to new environments (images). To encourage information sharing across multiple learning stages and different images, rewards are not given directly via IoU but indirectly via the distances of the learned latent representations.

With our approach, ordered representation learning and deep reinforcement learning (RL) are trained together, benefiting each other. Representation learning models are trained not only on accurately annotated data but also on augmented data with perturbations. Existing representation learning methods cannot directly generate more compact clusters from correctly annotated data. Therefore, rewards can only be defined for the original data, not for potential embeddings. In our approach, a latent embedding function is trained to maintain order relationships between pairs of incomplete annotations of the same image. That is, a higher IoU bounding box embedding is closer to the ground truth bounding box embedding than a lower IoU box embedding. As a result, RL rewards can be defined based on embedding distance.

If the ordered embedding is trained separately on the deep RL agent, it is redundant and inefficient because the perturbation samples are randomly generated and most of the samples are not in the search path of the RL agent. In the proposed joint training scheme, ordered embeddings can be trained more efficiently because box pairs are sampled while the RL agent is exploring the embedding space. Supervision is customized at various stages of learning. In a post-training stage, the model learns to assign priority to pairs of boxes that are better annotated.

As a by-product, the embedding distance also provides a metric for assessing annotation quality: given a set of images with both high- and low-quality annotations, well-annotated data fall into compact clusters in the regular embedding space and can therefore be selected. The quality of annotations can be ranked according to the distance of the filtered data to the cluster centroids.

Finally, our recurrent neural network (RNN) based method allows for a search starting from the whole image. This makes our approach applicable to the problem of large-scale single-image co-localization containing multiple common objects of the same class, even when the target objects are of different sizes and the images are at high resolution. The interaction process between the human and the RL annotator works as follows: The human starts the annotation process by labeling one or two target objects of interest. The annotation agent starts by examining the whole image at a coarse resolution and, following a top-down scheme, takes a set of repetitive actions to locate the objects in the remaining images. The human can accept or reject the selected object and/or run the annotator again until no new objects are found.

A more complete understanding of the present disclosure may be realized with reference to the accompanying drawings.

FIG. 1 is a schematic diagram illustrating a co-training framework for annotation agents and data representations in accordance with aspects of the present disclosure.

FIG. 2 is a schematic flow diagram illustrating a model training process, according to one aspect of the present disclosure.

FIG. 3 is a schematic diagram illustrating application 1, i.e., human-guided automatic annotation of fiber sensing datasets, where properly annotated data can benefit downstream training of event classifiers, according to aspects of the present disclosure.

FIG. 4 is a schematic diagram illustrating Application 2, i.e., worker quality assessment and improvement for a crowdsourcing-based image annotation platform in which high-quality annotations can be identified by trained agents and low-quality data can be corrected, according to an aspect of the present disclosure.

FIG. 5 is a schematic diagram illustrating ordered representation learning of embedding nets and triplet loss, according to aspects of the present disclosure.

FIG. 6 is a schematic diagram illustrating reward and action space based on order embedding, according to aspects of the present disclosure.

FIG. 7 is a schematic diagram illustrating an architecture based on a recurrent neural network (RNN) suite of RL agents and ordinal representation learning, according to aspects of the present disclosure.

FIG. 8(A) is a diagram illustrating a plot of convergence of behavioral sequences and learning of an RL agent and convergence of co-localization of digit 4 from a cluttered background and embedding distance to ground truth, in accordance with aspects of the present disclosure. . FIG. 8(B) is a diagram illustrating a plot of the convergence of the behavioral sequence and learning of the RL agent and the convergence of the co-localization of the digit 4 from the clutter background and the embedding distance to the ground truth, in accordance with aspects of the present disclosure. . FIG. 8(C) illustrates a plot of the convergence of the RL agent's behavioral sequence and learning, as well as the convergence of the co-localization of the digit 4 from a cluttered background and embedding distance to ground truth, according to aspects of the present disclosure.

FIG. 9 illustrates a dataset comparing fixed embeddings and training embeddings during RL updates, according to aspects of the present disclosure.

FIG. 10 is a diagram illustrating a dataset showing agents trained and tested on the number 4 and other new numbers 0-9, in accordance with aspects of the present disclosure.

FIG. 11 is a schematic diagram illustrating RL-based query object localization with reward signals defined on an example set rather than bounding boxes, according to an aspect of the present disclosure.

FIG. 12 is a schematic diagram illustrating an exemplary RoI encoder and projection head in accordance with aspects of the present disclosure.

FIG. 13(A) is a dataset showing random sampling and anchor sampling (%) in OrdAcc, according to an aspect of the present disclosure. FIG. 13(B) is a dataset illustrating a comparison of signed and unsigned IoU rewards (%) in CorLoc, in accordance with aspects of the present disclosure.

FIG. 14(A) is a plot showing a comparison table under different training set sizes, according to an aspect of the present disclosure. FIG. 14(B) is a plot showing a comparison table under different training set sizes, in accordance with aspects of the present disclosure.

FIG. 15(A) is a diagram illustrating a data set of CorLoc (%), according to aspects of the present disclosure. FIG. 15(B) is a diagram illustrating a dataset for comparison of four training strategies depending on the anchors used, in accordance with aspects of the present disclosure.

FIG. 16 illustrates a data set of performance at different numbers depending on the anchor used, in accordance with aspects of the present disclosure.

FIG. 17 is a diagram illustrating plots illustrating pre-adaptation, post-adaptation, and adaptation fine-tuning depending on the anchor used, in accordance with aspects of the present disclosure.

FIG. 18(A) is a diagram illustrating the performance of bounding boxes with inaccurate annotations to strict annotations, according to aspects of the present disclosure. FIG. 18(B) is a diagram illustrating performance when transferring to other backgrounds, according to aspects of the present disclosure.

FIG. 19 is a diagram illustrating a list of Algorithm I for training and rewarding location agents depending on the anchors used, according to aspects of the present disclosure.

The following merely illustrates the principles of the present disclosure. Accordingly, those skilled in the art will be able to devise various configurations embodying the principles of this disclosure that are within the spirit and scope of this disclosure, even if not explicitly described or illustrated herein. I want you to understand.

Further, all examples and conditional terms mentioned herein are for educational purposes only to assist in understanding the principles of the present disclosure and concepts presented by the inventors in furtherance of the present technology. meaning and should not be construed as being limited to the specifically mentioned examples and conditions.

Furthermore, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Moreover, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., elements developed that perform the same function, regardless of structure.

Thus, for example, those skilled in the art will appreciate that any block diagram herein is a conceptual illustration that illustrates an example of a circuit implementing the principles of the present disclosure.

FIG. 1 is a schematic diagram illustrating a joint training framework for annotation agents and data representations in accordance with aspects of the present disclosure.

FIG. 2 is a schematic flow diagram illustrating a model training process in accordance with aspects of the present disclosure.

FIG. 3 illustrates Application 1, i.e., human-guided automatic annotation of fiber sensing datasets, where properly annotated data can benefit downstream training of event classifiers, according to aspects of the present disclosure. FIG.

FIG. 4 illustrates Application 2, a worker for a crowdsourcing-based image annotation platform in which high-quality annotations can be identified and low-quality data can be corrected by trained agents, according to aspects of the present disclosure. FIG. 2 is a schematic diagram showing quality evaluation and improvement of

As we will now explain, the method/algorithm of the present invention involves three steps in training:

Step 1: Identify a set of seed images. This can be obtained from human experts, pre-selected heuristics, or a third-party dataset.

Step 2: Pre-train the ordered embedding. Given a seed dataset, we pre-train by randomly perturbing the ground truth bounding box at various levels. The level of perturbation is indicated by the parameter p. Ordered embeddings require ordering constraints to be satisfied locally for each pair of perturbed data extended from the same image. FIG. 5 is a schematic diagram illustrating ordered representation learning for embedding nets and triplet losses in accordance with aspects of the present disclosure.

Step 3: Reinforcement learning. Given the embedding function, the RL agent starts from the entire image and recursively samples behavior from the discrete behavior space. FIG. 6 is a schematic diagram illustrating reward and behavior spaces based on ordered embeddings, in accordance with aspects of the present disclosure. The reward for an action is calculated from the embedding distance. The policy network (behavioral head) is updated together with the embedded network. The neural network architecture is illustrated in detail in FIG. 7, which is a schematic diagram illustrating an architecture based on a set of RL agents and ordinal representation learning recurrent neural networks (RNNs), in accordance with aspects of the present disclosure.

The effectiveness of the proposed approach is evaluated on the Clutter MNIST benchmark dataset. Figures 8(A), 8(B) and 8(C) show plots of the convergence of the RL agent's action sequence and learning, as well as the convergence of the co-localization of the digit 4 from a cluttered background and the embedding distance to the ground truth, according to aspects of the present disclosure. The figures show the benefit of co-training in terms of final localization performance, showing that the agent is trained on the co-localization task for one order of magnitude and adapts to find new classes of common objects (0-3, 5-9) that are not visible in the training phase.

The system and method of the present invention performs ordered representation learning and deep reinforcement learning together to overcome the lack of high quality annotated data. The systems and methods of the present invention are broadly applicable to fully supervised tasks, weakly supervised tasks, and co-location tasks.

The systems and methods of the present invention effectively utilize a limited amount of high-quality, reliable human-annotated data to identify the quality of low-quality annotated data. , adopt a human-in-the-loop paradigm to improve.

As one skilled in the art can readily understand and appreciate, the systems and methods of the present invention may be of benefit in many applications, namely: (1) as a tool for automatically annotating unlabeled datasets in cost-sensitive applications, including but not limited to fiber sensing; (2) as a tool for enhancing the interpretability of deep neural networks, such as class activation maps (CAM) methods; (3) as a tool for assessing annotation quality and improving poor quality annotations in crowdsourcing platforms; and (4) as a tool for locating multiple common target objects within the same image, such as crops from satellite images in intelligent farming or cells from whole slide images in digital pathology.

The exemplary embodiments are more fully described in the drawings and detailed description. However, embodiments according to the present disclosure may be embodied in many different forms and are not limited to the specific or exemplary embodiments set forth in the drawings and detailed description.

Figure 9 is a dataset comparing fixed embeddings and training embeddings during RL updates according to aspects of the present disclosure.

FIG. 10 is a dataset illustrating agents trained and tested on the number 4 and other new numbers 0-9, in accordance with aspects of the present disclosure.

Here, a method based on reinforcement learning for the query object localization problem will be described. In this approach, an agent is trained to locate objects of interest specified by a small set of examples. By ordinal metric learning, we learn a transferable reward signal formulated using an example set. This allows test-time policy adaptation to new environments where reward signals are not readily available, thus outperforming fine-tuning methods that are limited to annotated images. Additionally, transferable rewards allow trained agents to be reused for new tasks, such as refining annotations or selective localization from multiple common objects across a set of images. Experiments on corrupted MNIST and CU-Birds datasets demonstrated the effectiveness of our approach.

In this disclosure, we focus on a reinforcement learning (RL) formulation for the query object localization problem, where an agent is trained to localize a specified target object in a small set of example images. Vision-based agents can be viewed as active information gatherers that actively exchange information with the image environment according to a class-specific localization policy, making them more suitable for robotic operations and embodied AI tasks.

During the test time, the objects queried for location may be new or the background environment may change significantly, which precludes the applicability of class-independent agents with fixed policies. When reward signals are available, fine-tuning methods can effectively adapt agents to new environments and improve performance. Unlike standard RL settings, the bounding box annotation is detected by the localization agent in the test image, so no reward signal is available to the application during testing.

To address this issue, we describe a framework based on ordinal metric learning to learn an implicitly transferable reward signal defined on a small example set. An ordinal embedding network is pre-trained with data augmentation under a loss function designed to be relevant to the RL task. The reward signal enables explicit updating of the controller in the policy network with continuous training during test time. Compared to fine-tuning approaches, the agent has unlimited access to test images, thus exposing it to a wider range of novel environments. By accurately deriving information from the example set, the agent can flexibly adapt to changes in localization targets.

FIG. 11 is a schematic diagram illustrating RL-based query object localization with reward signals defined in an example set rather than a bounding box, in accordance with aspects of the present disclosure.

Compared to bounding box regression-based methods, off-policy RL-based object localization methods have the advantage of having a search path customized for each image environment and no region-proposals are required. The specificity of the agent purely depends on the class of the bounding box used in the reward. Agents can also be class-specific, but it is necessary to train an agent for each class separately.

Despite the proliferation of crowdsourcing platforms, obtaining a sufficient amount of bounding box annotations remains costly and error-prone. Furthermore, the quality of annotations often varies, and accurate annotation for a particular object class may require special expertise on the part of the annotator. The advent of weakly supervised object localization (WSOL) methods alleviates this situation, making use of image class labels in obtaining bounding box annotations. It is known that the WSOL method relies too much on the discrimination function between classes and has the drawback of not being able to generalize to classes that are invisible during the training stage.

Note that intra-class similarity is a more natural objective for the problem of locating objects belonging to the target class. A similar problem is image co-localization, where the task is to identify common objects within a set of images. Co-localization techniques exploit common characteristics across images to locate objects. Since the co-localization method is unsupervised, ambiguity may occur when multiple common objects or common parts (such as the head and body of a bird) are present, and unnecessary common objects may be provided as output. There is.

There is an apparent conflict between the goal of training agents with high task specificity and at the same time training agents with better generalization performance to new situations. The key to reconciling these two goals is to use a small set of examples. There has been a paradigm shift from training static models defined by parameters to models defined together with a support set, which has proven to be very effective in few-shot training.

In addition to efforts to meta-learn implicitly tunable models, fine-tuning of pre-trained models has also been exploited in transferring knowledge from data-rich to data-poor tasks. When no reward signal is available, policy adaptation techniques can be employed to fine-tune the intermediate representation by optimizing a self-supervised auxiliary loss while keeping the controller fixed. Our disclosure shares the same motivation as test-time training, but focuses on settings where the controller needs to be adapted or reused for new tasks.

In query object localization, we are given a set of images I and a small set of example images E. Image annotations are available in the form of bounding boxes g. Our goal is to find the location of a bounding box that contains the queried object in each image without using candidate boxes.

Considering each image I _i as the environment, existing RL methods for object localization use its ground truth object bounding box g _i as the reward signal.

である。マッピング

を学習するバウンディングボックス回帰型手法と同様に、画像とボックスをペアにする必要がある。但し、アノテーションが付与された画像とボックスのペア（Ｉ，ｇ）は、訓練段階とテスト段階の両方で不足している可能性がある。（？？）における報酬信号は、潜在的なドメインシフトを伴うテスト画像はもちろん訓練画像間でも転送できない。 Here, IoU (b _t , g _i ) indicates the IoU (Intersection-over-Union) between the current window b _t and the corresponding ground truth box g _i ,

It is. mapping

Similar to the bounding box regression method that learns, images and boxes need to be paired. However, the annotated image-box pair (I, g) may be missing in both the training and testing stages. The reward signal in (??) cannot be transferred between training images as well as test images with potential domain shifts.

で生成されたＭ次元表現ｂ_t及びｇが与えられると、距離関数

は埋め込み距離ｄ（ｂ_t，ｇ）を返す。但し、エージェントがグラウンドトゥルースボックスｇに近づくにつれて、単調に減少しない可能性がある。結果として、埋め込み距離に基づく報酬信号は（？？）よりも効果が低い可能性がある。 To address this problem, a natural idea is to define the reward signal based on the distance between the image cropped by the current window b _t and the ground truth window g. Embedding function from D-dimensional image feature vector

Given the M-dimensional representation b _t and g generated by , the distance function

returns the embedding distance d(b _t , g). However, as the agent approaches the ground truth box g, it may not decrease monotonically. As a result, reward signals based on embedding distance may be less effective than (??).

Furthermore, we propose to use a reward signal based on an ordinal embedding: for any two perturbed boxes _bj , _bk from g in a constraint set C, an embedding _bj , _bk , g is learned, preserving the relative priorities between any pair of boxes in Euclidean space.

Here, p _j and p _k represent the priorities (derived from the ground truth box or IoU for order feedback from the user). This problem was originally posed as non-metric multidimensional scaling. Here we apply a very simple pairwise-based approach, but other extensions also exist, such as listwise-based approaches, quadruplet-based approaches, landmark-based approaches, etc.

で置き換えることができる。ここで、ｂ_iは、グラウンドトゥルースボックスｇ_iで切り取られた画像Ｉ_iの埋め込みである。複数のクラスの画像が利用可能な場合、プロトタイプをさらにクラス依存またはクラスタベースにすることができる。幾つかの実験において、アンカーとしてプロトタイプに基づく埋め込みがｇよりも一般化性能が優れている可能性があることを見出した。この選択により、クラス毎の訓練画像が少ないサブセットのみにアノテーションが付与される場合、本発明者らの手法はフューショット訓練にも適している。順序報酬はメタ情報として見ることができる。さらに、テストタイム中の例示的なセットが切り取られたオブジェクトのみを含む場合でも、テストタイムのポリシー適応は、画像ボックスのペアなしで依然として実現可能である。 Anchor g in equation (2) is not limited to embedding from the same image. For example, this is the prototype embedding for the exemplary set E,

can be replaced with Here, b _i is the embedding of the image I _i cropped by the ground truth box g _i . If images of multiple classes are available, the prototype can be made more class-dependent or cluster-based. In some experiments, we found that embeddings based on prototypes as anchors can have better generalization performance than g. Due to this selection, if annotations are provided only to a small subset of training images for each class, our method is also suitable for few-shot training. Order rewards can be viewed as meta-information. Furthermore, even if the exemplary set during test time only includes cropped objects, the test time policy adaptation is still feasible without image box pairs.

During training time, we assume that the example set E contains both images I and boxes g. We employ a customized data augmentation scheme, box perturbation, in which C is constructed by sampling box pairs around g. We find that using an IoU-based partitioning scheme is more effective than random sampling. This can be seen as a procedure to enhance the robustness of the neural network against box perturbations and to safeguard the special purpose of its use: to distinguish between increasing or decreasing rewards. Pre-training with data augmentation also allows the downstream task of training the policy network to be more efficient.

In this disclosure, we define p as the IoU of box b with respect to ground truth box g, i.e., p = IoU(b, g). We learn an embedding space that is consistent with the local order constraints specified on the image pairs obtained by data augmentation.

ここで、ｆ_aは「アンカー」埋め込みである。ｆ_pは、グラウンドトゥルースボックスｇを備えたより大きいＩｏＵとより小さいＩｏＵを有する「正」の埋め込みであり、ｆ_nは、グラウンドトゥルースボックスｇを備えたより大きいＩｏＵとより小さいＩｏＵを有する「負」の埋め込みである。報酬を定義するための適切な表現は、必ずしも同時に適切な状態表現であるとは限らないことに注意されたい。エージェントが正しい行動を取得するための十分な情報が含まれていない可能性がある。表現と対照的な損失との間に投影ヘッドを追加すると、学習された表現の品質が大幅に向上することを示唆している。 We choose to optimize the triplet loss to learn the desired embedding.

where f _a is the "anchor" embedding, f _p is the "positive" embedding with larger and smaller IoU with the ground truth box g, and f _n is the "negative" embedding with larger and smaller IoU with the ground truth box g. Note that a good representation for defining the reward is not necessarily a good state representation at the same time; it may not contain enough information for the agent to obtain the correct action. We suggest that adding a projection head between the representation and the contrastive loss can significantly improve the quality of the learned representation.

の下では、状態表現ｓはｂに対する順序教師ありから間接的に恩恵を受けることができるが、それでも満足のいく画像再構成結果をレンダリングする必要がある。オートエンコーダスキームの他に、ＲｏＩエンコーダは事前訓練されたネットワークも使用できる。 We have found that the use of a projection head is crucial in balancing the two objectives in our task. The network architecture is shown in FIG. In FIG. 12, the MLP projection head is mounted after the RoI (Region of Interest) encoder. According to the given image and RoI, the RoI encoder extracts RoI features s that are used in the state representation for localization. The projection head learns the ordered embedding b to calculate the reward. The ROI alignment module processes boxes of various sizes. joint loss function

Under , the state representation s can indirectly benefit from order supervisedness on b, but is still required to render a satisfactory image reconstruction result. Besides autoencoder schemes, RoI encoders can also use pre-trained networks.

ここで、ａはプロトタイプ埋め込みである。順序埋め込みは、事前訓練されたＲｏＩエンコーダ及び投影ヘッドによってＥにおけるグラウンドトゥルースボックスで囲まれた画像領域から抽出され、プロトタイプが平均ベクトルとして計算される。さらに、本発明では、履歴行動及び状態のベクトルを含むＤｅｅｐＱＮｅｔｗｏｒｋではなく、回帰型ニューラルネットワーク（ＲＮＮ）（Ｍｎｉｈ ｅｔ ａｌ., ２０１４）を備えたポリシー勾配を使用する。画像ピクセル全体を入力として開始し、エージェントは、割引報酬和を最大化することで、各ステップで現在のバウンディングボックスを変換する行動を選択するように訓練される。エージェントは、現在のボックスからプールされた特徴を状態として取得すると同時に、履歴観察からの情報をエンコードするＲＮＮの内部状態も維持する。行動セットは、トップダウン検索を容易にする個別の行動で定義される。行動セットには、５つのスケーリング、８つの変換変換及び１つの停止行動が含まれる。 Localization is formulated as a Markov Decision Process (MDP) using raw pixels in each image as the environment. As described here, we use ordinal embeddings rather than bounding box coordinates to calculate improvements made by the agent. The reward for an agent moving from state s ⁰ to s is of the form:

Here, a is the prototype embedding. The ordered embedding is extracted from the image region bounded by the ground truth box in E by a pre-trained RoI encoder and projection head, and the prototype is computed as the mean vector. Moreover, we use policy gradients with recurrent neural networks (RNNs) (Mnih et al., 2014) rather than DeepQNetworks, which include vectors of historical behavior and states. Starting with the entire image pixel as input, the agent is trained to choose an action that transforms the current bounding box at each step by maximizing the discounted reward sum. The agent obtains the pooled features from the current box as state, while also maintaining the internal state of the RNN that encodes information from historical observations. Behavior sets are defined with individual behaviors that facilitate top-down searching. The behavior set includes 5 scaling, 8 transformation transformations, and 1 stop behavior.

In test-time adaptation during test time, the agent has the option to further update the policy network with the reward received from (4) including a as a prototype for the set of test examples E _test . To match the test conditions, the training batch is split into two groups and a is computed on a small subset that does not overlap with the training images to be localized. During test adaptation, a becomes a prototype for the example set. The complete algorithm is outlined in Algorithm 1, which is exemplarily shown in Figure 19.

The transferability of the reward signal from training to testing critically depends on the generalization ability of the learned ordinal representation. If ordinal priors do not hold in the test domain, our proposed test-time policy adaptation scheme will not work. Adapting the representation using a self-supervised objective could solve this problem. Although our approach does not directly handle the special cases of multiple or no queried objects in an image environment, it can be easily modified to accomplish these tasks.

にしたがって正しく位置が特定された画像のパーセンテージとして定義される。ここで、ｂ_pは予測ボックスであり、ｇはグラウンドトゥルースボックスである。 We evaluate our approach using several tasks on the MNIST and CUB bird datasets. For MNIST, we use three convolutional layers activating ReLU after each layer as the image encoder and the same but mirror structure as the decoder to learn the autoencoder. Next, following the two fully connected layers, we connect an RoI align layer as a projection head for ordinal reward learning. For the CUB data set, we employ layers prior to conv5_3 of VGG16 that have been pre-trained with the ImageNet encoder. The projection head has the same structure as before, but there are more units for each fully connected layer. To evaluate the learned ordered structure, we use OrdAcc, which is defined as the percentage of images in which the order of pairs of perturbed boxes is predicted correctly. The present invention uses the CorLoc (Correct Localization) metric. This is the criterion

is defined as the percentage of images that are correctly located according to Here b _p is the prediction box and g is the ground truth box.

We analyze the effectiveness of using ordinal embeddings in terms of representation and reward in Cluttered MNIST. 28x28 digits are randomly placed on a cluttered background of 84x84. Here, we compare embeddings trained with only the autoencoder and embeddings trained together with a normal projection head. Furthermore, we compare the IoU-based rewards used with embedding-based rewards. The agent is trained with a certain number of images of the digit 4. It is tested with all images in the test set. Results under different training set sizes are shown in Figures 13(A) and 13(B). Figure 13(A) is a dataset showing random sampling anchor sampling of OrdAcc(%) according to aspects of the present disclosure, and Figure 13(B) is a dataset showing the results of a comparison of signed and unsigned IoU rewards in CorLoc(%) according to aspects of the present disclosure. When ordinal embedding is present in both the representation and the reward ("AE+Ord+Embed"), the model consistently outperforms other settings, especially when the training set size is small.

Figures 14(A) and 14(B) are plots showing comparison results under different training set sizes according to aspects of the present disclosure.

To efficiently learn ordinal rewards, we conduct experiments to compare sampling strategies that generate pairs of augmented bounding boxes. The first strategy is random sampling, where pairs of boxes are generated completely randomly. Another strategy is anchor-by-anchor sampling, which first generates a high density of anchors at various scales and then uses a ground truth box to divide them into 10 groups according to IoU. The interval between each group is 0.1. Sampling is first done at the group level, ie two groups are sampled. Next, sample two boxes corresponding to each group. Therefore, the sampled box can cover more cases compared to random sampling. The OrdAcc obtained as a result of the two strategies is shown in FIG. 13(A). Anchor sampling can be used to learn better ordered embeddings.

Reward {+1, -1}, presence or absence of sign, uses equation 1 as the reward for training the agent. However, it can be seen from FIGS. 14(A) and 14(B) that there is a large gap between this IoU reward and the embedding reward, especially when the size of the training set is small. Since the ordinal reward approximates the characteristics of the IoU in the embedding space, it should have lower accuracy than the IoU as a reward, which is somewhat contrary to common sense. To analyze this problem, we remove the sign operation in Equation 1 to train the model on images of the digit 4. As shown in FIG. 13B, the sign operation increases the localization accuracy by 3.4% for the number 4 and by 6.2% for the test set of other numbers.

FIG. 15(A) shows a dataset of CorLoc (%), according to aspects of the present disclosure, and FIG. 15(B) shows comparison results of four training strategies depending on the anchors used, according to aspects of the present disclosure. This is a dataset showing the following.

In contrast to training the agent using Deep Q-Network, we apply policy gradients to optimize the agent. Furthermore, we employ a top-down search strategy with RNNs, in which vectors of historical behavior are used to encode the memory. As shown in FIG. 15(A), the design choices are evaluated using a model trained and tested on digit 4 or a model tested on other digits. As can be seen, the agent achieves the best performance with “PG+RNN”. When using historical behavior vectors, the accuracy decreases when the agent is trained with DQN.

We conducted experiments to evaluate the effect of ordinal reward learning and different training strategies on localization on a subset of the CUB dataset, where the training and test sets contained 15 and 5 different fine-grained classes, resulting in 896 images for training and 294 images for testing. Fig. 15(B) shows the OrdAcc and CorLoc for the four settings "Self", "Proto", "Shuffle self" and "Shuffle proto". "Self" uses the ground truth from this instance as an anchor for both embedding pre-training and agent training. "Proto" uses the prototype of the subgroup containing the instance in the batch for both embedding pre-training and agent training. "Shuffle self" uses the ground truth from another instance for both embedding pre-training and agent training. Shuffle proto uses the prototypes of the subgroup that do not have this instance in the batch for both embedding pre-training and agent training. The RoI encoder is trained with loss _trips only. Therefore, the entire training set can be viewed as one class. The results show that Shuffle proto has a lower OrdAcc than the others, but its CorLoc is the best by a large margin. This phenomenon suggests that this training strategy brings compactness to the training set and builds an order structure around the clusters. Note that OrdAcc is calculated using the instances as anchors.

As will be appreciated by those skilled in the art, we disclose a reward based on ordinal representation learning to train a localization agent to search for queried objects of interest in potentially new environments. In particular, we use a small exemplar set as a guidance signal to achieve the learning goal, which avoids ambiguity in learning. Meanwhile, we use a test image environment to inform the agent about domain shifts during testing without requiring image box pairs. Our algorithm takes raw image pixels as input without the need to propose candidate boxes.

Although our approach is based on feature similarity with an exemplary set, it is fundamentally different from bounding box regression and bounding box RL approaches. To generalize to different object classes and background scenarios, previous methods need to be trained as class recognizers on large datasets that cover foreground and background variations. In contrast, we will be able to train professional agents with the ability to adapt policies during test times.

Instead of training a localization model together with a classification model, we explore the annotations of learning boxes from image class labels, in a similar spirit to weakly supervised learning. Given the image labels from the classification model, our localization model is able to identify box regions with enhanced interpretability. Empirically, our approach has been shown to work in transfer learning settings from data-rich single source tasks to data-poor test tasks. Furthermore, our approach also applies to a few-shot learning setting where limited annotations across a large number of tasks are available during training. Future work will include curricular learning using attribute-based cross-modality or zero-shot queries and a designed set of targets in an exemplary set.

Collecting annotations plays an important role when building machine learning systems. This is one task that could greatly benefit from automation, especially in cost-sensitive applications. The present inventors aim to reduce the labor of human labeling in terms of the number of annotation samples for each class, the number of annotation classes, and the required accuracy level. Our approach allows objective assessment and iterative improvement of data quality.

FIG. 16 is a dataset illustrating performance at different numbers depending on the anchor used, according to aspects of the present disclosure.

Figure 17 is a plot showing pre-adaptation, post-adaptation, and fine-tuning of adaptation depending on the anchor used, according to an aspect of the present disclosure.

FIG. 18(A) is a dataset illustrating the performance of bounding boxes with inaccurate annotations to strict annotations according to aspects of the present disclosure, and FIG. This is a dataset that shows the performance when transferring to a background.

FIG. 19 is a listing of algorithm I for training and rewarding a localization agent depending on the anchors used, according to aspects of the present disclosure.

Although the present disclosure has been presented herein using several specific examples, those skilled in the art will recognize that the present teachings are not limited thereto. Accordingly, the present disclosure should be limited only by the scope of the claims appended hereto.

Claims (2)

A deep reinforcement learning (RL) method for object localization, comprising:
Obtain a seed dataset containing a set of seed images each with a ground truth bounding box annotation,
pre-training an ordered embedding that satisfies the ordering constraint locally for each pair of perturbed data augmented from the same image by randomly perturbing the ground truth bounding box at different levels denoted by a parameter p;
The pre-training is performed through a backbone network, a region of interest (RoI) head and the effect of a triplet loss;
Construct an RL agent using an embedding function to recursively sample actions from a discrete action space such that a reward is generated, starting from the entire image;
The reward of the sample action is determined from the embedding distance, and the policy network is updated based on the determined reward,
A method for outputting annotation policies and embedded functions. 2. The method of claim 1, wherein the annotation of the seed image bounding box is initially provided by human behavior.

Images