KR20160128869A - Method for visual object localization using privileged information and apparatus for performing the same - Google Patents

Abstract

A method and apparatus for performing an image object search using dictionary information are disclosed. The method of searching an image object includes a step of creating a learning framework combining preliminary information with a structured prediction framework, a step of executing alternate optimization learning on a learning framework, a step of generating a prediction model from a learning framework in which alternate optimization learning is executed And predicting or searching an object in the specific image from the test sample or the input image using the predictive model.

Description

TECHNICAL FIELD The present invention relates to a method of searching for an image object using dictionary information and an apparatus for performing the method.

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a learning algorithm that combines advance information, and more particularly, to an image object searching method using dictionary information and an apparatus for performing the same.

Object localization often results in a binary classification problem. Conventional learning classifiers determine the presence or absence of a target object within a candidate window of all positions, sizes, and aspect ratios. In recent years, SVM (Support Vector Machine) has been applied to the object search problem.

An optimal bounding box containing the target object is obtained through a classifier trained in structured prediction techniques. This approach provides a unified framework for detection and post-processing, and can handle issues related to objects with different aspect ratios.

However, object search is an inherently difficult task due to many changes in objects and scenes, such as shape changes, color differences, attitude changes, occlusion, viewpoint changes, background congestion, and the like. This issue is particularly acute when the size of the training data set is small.

If additional high-level information on the object of interest is available at training time, a less reliable training sample can be used to generate a more reliable model. This high level of information may be referred to as prior information. Dictionary information basically describes useful meaningful features such as parts, attributes, and partitions of objects. This idea is a typical example of Learning Using Privileged Information (LUPI). Some LUPIs use additional information to improve the performance of the training prediction model. The existing LUPI framework is coupled to the SVM in the form of an SVM + algorithm. However, the application of SVM + is often limited to the problem of binary classification.

SUMMARY OF THE INVENTION An object of the present invention is to provide a new structured SVM (Support Vector Machine) (SSVM +) framework using preliminary information that can apply a learning algorithm to a problem of object search There is. That is, an object of the present invention is to provide a method and apparatus for performing an image object search using advance information.

It is also an object of the present invention to improve the performance of an algorithm or apparatus for learning or object searching by combining pre-information with a framework including a binary learning framework and adjusting the model parameters to perform better generalization A method for searching for an object using dictionary information, and an apparatus for performing the method.

According to one aspect of the present invention, there is provided a method for generating a learning framework by combining a preliminary information with a structured prediction framework, Generating a prediction model from a learning framework in which alternate optimization learning has been performed; and predicting or searching an object in a particular image from an input image (which may include a test sample) using the prediction model And provides an image object search method.

According to another aspect of the present invention, there is provided a computer-readable medium having recorded thereon a program for performing the above-mentioned method for searching an object.

According to still another aspect of the present invention, there is provided a method for constructing a learning framework for a learning framework, the method comprising: generating a framework for learning by combining preliminary information with a structured prediction framework; A learning unit for performing alternate optimization learning, a model generating unit for generating a prediction model from a learning framework in which the alternate optimization learning is executed, and a search unit for predicting or searching an object in a specific image from the input image using the prediction model The object searching apparatus comprising:

Here, the framework generating unit may combine the first function of the first space based on the priori information and the second function of the second space based on the training sample. Herein, the dictionary information includes a segmentation, a part, an attribute, or a combination thereof of a training sample, and the combination of the first function and the second function includes a space including images and attributes of the training sample, To the space of the bounding box coordinates.

Here, the structured prediction frame may comprise a structured SVM (Structured Support Vector Machine) classifier.

Here, the learning unit can process the item of the objective function corresponding to the advance information through the alternating loss-augmented inference.

Here, the learning unit includes a first learning unit that alternately performs an Efficient Subwindow Search (ESS) in a first space based on the priori information and a second space based on the training sample through the alternate loss addition estimation .

Here, the learning unit may further include a second learning unit for extracting all the possible bounding boxes from the target image of the training sample through the alternate loss addition estimation, and estimating the bounding box coordinates of the object.

Here, the learning unit may further include a third learning unit that creates a connection feature point map that associates the bounding box coordinates with the alternate loss addition estimation to associate the relationship between the input and output variables.

Here, the searching unit can find the optimal bounding box given by the learned weighted vector of the prediction model and the image characteristic in the specific image from the input image.

Here, the video object search apparatus may further include a verification unit coupled to the framework generation unit or disposed between the framework generation unit and the model generation unit, for verifying the learning framework based on actual image information (Groundtruth) including a specific object .

Here, the video object search apparatus includes a memory system for storing a program or a command for operation of a framework generating unit, a learning unit, a model generating unit, a searching unit or a combination thereof, and a memory system connected to the memory system and executing a program or a command And searching for an object previously designated in the input image.

In the case of using the dictionary information according to the present invention and the apparatus for performing the image object search through the Structured Support Vector Machine (SSVM) and the apparatus performing the image object, it is possible to search for an object using the dictionary information, A framework can be provided. In other words, the performance of algorithms or devices for learning or object searching can be improved by combining the preliminary information into the initial learning framework and adjusting the model parameters for better generalization.

In addition, according to the present invention, an SSVM + framework can be created that can handle prior information together with conventional visual features by linking an alternate loss addition estimation method for efficient sub-window search.

Further, according to the present invention, a performance gain can be achieved in searching and classifying objects in an image, and a performance gain can be achieved particularly even for a small training data set. For example, in searching for a bird in the CUB-2011 dataset, performance can be improved by utilizing attributes of the standard image and split masks as prior information, in addition to the standard image features.

Further, according to the present invention, the conventional transition learning; Learning through side information or domain application; Binomial constraints or multi-kernel or metric-based learning; There is an advantage that image classification or search performance can be improved in conventional methods such as Zero-shot learning.

1 is a flowchart of an image object searching method according to an embodiment of the present invention.
2 is a schematic view of an object search framework using advance information that can be employed in the video object search method of FIG.
FIG. 3 is a flowchart illustrating a method of performing SSVM + learning of the object search framework of FIG. 2 alternately through efficient sub-window search (ESS) in a pri- vary space and a visual space, Augmented Inference < / RTI >
FIGS. 4A and 4B are graphs showing the performance of the SSVM of the image object searching method and the comparative example of FIG. 1, respectively, with an average overlay ratio and the number of detection for 100 classes.
5 is a block diagram of an image object search apparatus according to another embodiment of the present invention.
6 is a block diagram of an image object search apparatus according to another embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as being consistent with the meanings in the context of the relevant art and are not to be construed as ideal or overly formal meanings unless explicitly defined in the present application.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

1 is a flowchart of an image object searching method according to an embodiment of the present invention. 2 is a schematic view of an object search framework using advance information that can be employed in the video object search method of FIG. FIG. 3 is a flowchart illustrating a method of performing SSVM + learning of the object search framework of FIG. 2 alternately through efficient sub-window search (ESS) in a pri- vary space and a visual space, Augmented Inference). 4A and 4B are graphs showing the performance of the SSVM of the image object search method and the comparative example of FIG. 1, respectively, by an average overlay and a number of detection for 100 classes.

1 to 3, an image object searching method according to an embodiment of the present invention includes a step S11 of creating a learning framework in which dictionary information is combined with a structured prediction framework, an alternating optimization for a learning framework, A step S12 of generating a prediction model from the learning framework in which the alternate optimization learning is performed, and a step S12 of generating a prediction model by using the prediction model, (Step S14). An image object search method can be performed by a device capable of digital signal processing.

In the present embodiment, the term " Privileged Information " refers to a high level of information useful for understanding an image, and thus, using the dictionary information, it is helpful to learn a reliable model Can be obtained.

Actually, prior information is only available during training, so it is difficult to efficiently obtain prior information from visual data without user supervision. Accordingly, in the present embodiment, dictionary information including advance information, such as parts, attributes, and segmentation, which is prepared in advance in a specific framework, is linked with learning of a prediction function for searching for a structured object (See Figures 2 and 3). The high level of information associated with the framework can be used during testing as well as during training. This framework-based learning algorithm employs an efficient branch and a branch-and-bound loss-ar- ranged subwindow search method to create joints in the original visual space and privileged space, Inference by joint optimization can be performed. If additional information is not used when testing, the speculation in the test phase may be similar to that of standard structured SVM (SSVM).

Normally a standard learning algorithm requires a lot of data to construct a robust model, but zero-shot learning does not require any training samples. The general Learning Learning Privileged Information (LUPI) framework is intended to learn good models with a small number of training data taking advantage of the advance information available for training, so it is in the midst of traditional data-based learning and zero-shot learning . Here, prior information has been considered for face recognition, face feature detection, and event recognition, but learning methods and object search methods using prior information have not been generalized yet. In this embodiment, the LUPI framework is applied to object search based on the SSVM. That is, a technique using an SSVM for object search has already been known, and in recent years it has been known to employ SSVM as part of a search method. However, none of these prior art techniques employs a method of combining dictionary information or a similar method.

Hereinafter, each step of the above-described method of searching for an image object will be described in more detail from the background thereof.

Learning using dictionary information

일 때, 추가 정보

가 각 훈련 샘플에 구비된다.The Learning Using Privileged Information (LUPI) paradigm is a framework for combining additional information that is not used during testing during training. What is included in this information is used to find a better model, resulting in lower generalization errors. Unlike typical supervised learning, in the LUPI paradigm,

When,

Is provided for each training sample.

이다.For example,

to be.

On the other hand, such advance information is not required during the execution of the test. In the LUPI paradigm, a task is to find a specific function that is closest to the decision hidden from the then given data among the collected functions.

와

를 결합하여 학습함으로써 LUPI 프레임워크 내에서 물체 탐색을 수행한다. 여기서,

는 예측을 위해서만 사용된다. 예를 들어, 전술한 기능들은 이미지들과 속성들의 공간을 바운딩 박스 좌표들의 공간

에 연관시킨다. 결정 함수

와 보정 함수

는 다음의 [수학식 1]과 같은 관계에 의해 서로 의존한다.In particular, in this embodiment,

Wow

To perform object search within the LUPI framework. here,

Is used only for prediction. For example, the functions described above may be used to transform the space of images and properties into the space of the bounding box coordinates

. Decision function

And correction function

Are mutually dependent on each other according to the following relationship (1).

와

는 영상 공간(

)과 사전 공간(

)의 경험 손실 함수(empirical loss functions)를 각각 나타낸다. 이러한 두 공간에서의 불균등은, LUPI 패러다임으로부터 영감을 받은 것으로, 모든 훈련 샘플에 있어서 모델

가 사전 정보에 관한 모델

보다 더 작은 데이터 손실을 갖도록 항상 보정함으로써 해결될 수 있다. [수학식 1]에서의 이러한 제약은 동일한 개수의 훈련 샘플에서 영상 정보와 사전 정보의 조합이 단독 영상 정보보다 더 우수한 모델을 학습하는 공간을 제공한다고 추정할 때 유의미하다.In Equation (1)

Wow

The image space (

) And the dictionary space (

), Respectively, as the empirical loss functions. The inequality in these two spaces, inspired by the LUPI paradigm,

A model for prior information

Can be solved by always correcting to have less data loss. This constraint in Equation (1) is significant when it is assumed that the combination of image information and dictionary information in the same number of training samples provides a space for learning a model that is superior to single image information.

를, 이를 보정한 함수

로 대체하고, 대체된 함수에 의해 사전 정보의 값을 평가한다. 그 결과는 다음의 [수학식 2]와 같다.To actually apply these generalized learning ideas, the SVM + algorithm used for binary classification has been developed. The SVM + algorithm is a slack variable in the standard SVM formula,

, The corrected function

, And evaluates the value of the prior information by the replaced function. The result is shown in the following equation (2).

,

및

는 새로운 보정 공간

내에서만 전형적인 SVM의

,

및

와 동일한 역할을 담당한다. 또한,

는

에 대한 조정화 파라미터(regularization parameter)를 나타낸다. 가중 벡터

가

뿐만 아니라

도 의존하도록 유지하는 것은 중요하다. 이러한 이유로 슬랙 함수

를 대체하는 함수가 보정 함수로 지칭된다. 사전 정보가 슬랙 함수의 값을 추정하는데만 사용되므로 사전 정보는 훈련하는 동안뿐 아니라 테스트하는 동안에도 요구된다. 이론적인 분석은 전술한 SVM+ 알고리즘의 수렴속도의 한계가 실질적으로 표준 SVM을 더 좋게 할 수 있음을 보여준다.Here,

,

And

Lt; RTI ID =

Only within a typical SVM

,

And

. Also,

The

Lt; RTI ID = 0.0 > regularization < / RTI > Weighted vector

end

As well as

It is also important to keep it dependent. For this reason,

Is referred to as a correction function. Since the dictionary information is only used to estimate the value of the slack function, the dictionary information is required during testing as well as during training. The theoretical analysis shows that the convergence speed limit of the SVM + algorithm described above can substantially improve the standard SVM.

Structuring SVM  ( SSVM )

에 대한 가중 벡터

를 이산적으로 학습한다. 한번 학습된 예측 함수

는 다음의 [수학식 3]과 같이 모든 가능성

에서

를 최소화하여 얻어진다.The Structural Support Vector Machine (SSVM) is a scoring function in the data set of training input /

≪ / RTI >

. Once learned function

As shown in the following equation (3)

in

.

는 입력

와 구조화 출력

간의 관계를 모형으로 만든 연결 특징점 맵(joint feature map)이다. 가중 벡터

를 학습하기 위해 다음의 최적화 문제(마진 리스케일링, margin-rescaling)를 풀면 다음의 [수학식 4]와 같다.here,

Input

And structured output

The joint feature map is a model of the relationship between the joint feature map. Weighted vector

The following optimization problem (margin rescaling) is solved to obtain the following equation (4).

및

는 그라운드 트루스(ground-truth)

에 대한 예측(예측 함수)

의 품질인 태스크 특정 손실(task-specific loss)이다. 예측을 얻기 위해 주어진 입력

에 대한 응답 변수(response variable)에서의 [수학식 3]을 최대화한다. SSVM은 예측 작업의 다양성을 풀기 위한 일반적인 방법이다. 각 응용에 있어서, 연결 특징점 맵

, 손실 함수

및 효율 손실 추가 추정 기술이 맞춤형으로 꾸며진다.here,

And

Ground-truth < / RTI >

(Prediction function)

Specific loss of quality. Given input to get prediction

(3) in the response variable for < / RTI > SSVM is a common method for solving a variety of forecasting tasks. For each application, a link feature point map

, Loss function

And additional efficiency loss estimation techniques are tailored.

Object navigation through dictionary information

In the object search method using the dictionary information according to the present embodiment, a set of training images of the objects, their positions, their attributes, and segmentation information are given, and the object of interest is searched I want to learn the function to do. Unlike the existing method, the learned function does not require explicit or inference attribute and partition information at the time of prediction.

Structuring using dictionary information SVM ( SSVM +)

및

의 학습을 위해 세 쌍의 훈련 셋을 토대로 구조

를 예측하는 것을 학습한다. 세 쌍의 훈련 셋은

로 가정하며, 여기서

는 가변 영상 특징점에 대응하고,

는 속성과 분할에 대응하며,

는 모든 가능성 있는 바운딩 박스들의 공간을 지칭한다. 한번 학습된 함수

는 예측에만 사용된다. [수학식 3]에서와 같이, 함수

는 입력

과 출력

를 토대로 표준 SSVM과 동등하게 모든 가능성 있는 연결 특징점들에 대해 학습된 함수를 최대화하여 얻어진다.Let's extend the above-mentioned structuring prediction problem to the use of advance information. According to the above-described expression (1), the apparatus of the present embodiment includes a pair of functions

And

Based on three pairs of training sets for learning

Is predicted. Three pairs of training sets

, Where

Corresponds to a variable image feature point,

Corresponds to attributes and partitions,

Quot; refers to the space of all possible bounding boxes. Once learned function

Is used only for prediction. As in Equation (3), the function

Input

And output

To maximize the learned function for all possible connection feature points equal to the standard SSVM.

및

의 연결 학습을 위해, 본 실시예에서는 SSVM 프레임워크를 실질적으로 확장한다. 두 함수들

및

는 파라미터 벡터들

와

에 의해 특징지어질 수 있다. 이를 각각 나타내면 [수학식 5]와 같다.On the other hand, if the two functions < RTI ID = 0.0 >

And

The SSVM framework is substantially extended in this embodiment. Both functions

And

≪ / RTI >

Wow

. ≪ / RTI > These are expressed by Equation (5).

와

의 동시 학습을 위해, 본 실시예에서는 [수학식 1]에서의 제약을 포함하는 새로운 최대 마진 구조화 예측 프레임워크를 제안한다. 이러한 프레임워크는 도 2에 도시한 바와 같이 SSVM+로 지칭될 수 있다. SSVM+는 다음의 [수학식 6]과 같이 두 모델들을 결합하여 학습(learning)한다.The two weighted vectors

Wow

The present embodiment proposes a new maximum margin structured prediction framework including the constraints in Equation (1). Such a framework may be referred to as SSVM + as shown in FIG. SSVM + combines the two models to learn as shown in Equation (6) below.

이고, [수학식 1]에서의 불균등은 "Dmitry Pechyony and Vladimir Vapnik. On the theory of learning with privileged information. NIPS, pages 1894-1902, 2010."의 논문에서 유도된 대용 태스크 특정 손실(surrogate task-specific loss)

을 통해 소개되고 있다. 이러한 대용 손실(surrogate loss)은 다음의 [수학식 7]과 같이 정의될 수 있다.here,

, And the inequality in Equation (1) is the surrogate task-specific loss derived from the article of Dmitry Pechyony and Vladimir Vapnik. On the theory of learning and privileged information, NIPS, pages 1894-1902, specific loss)

. This surrogate loss can be defined as the following equation (7).

이고,

는 [수학식 1]에서의 제약에 대응하는 패널티화 파라미터(penalization parameter)이며, 태스크 특정 손실 함수들

및

는 [수학식 10]에서 정의된다. 본 실시예에서는 대용 손실을 통해 [수학식 1]에서의 불균등을 보통의 최대 마진(ordinary max-margin) 최적화 프레임워크에 적절하게 적용할 수 있다.here,

ego,

Is a penalization parameter corresponding to the constraint in Equation (1), and the task specific loss functions

And

Is defined in Equation (10). In this embodiment, the inequality in Equation (1) can be suitably applied to the ordinary max-margin optimization framework through the substitution loss.

)이 항상 영상 특징에 관해 훈련된 모델(

)을 보정하도록 강제 동작한다. 이것은 결과적으로 영상 특징점 단독의 경우보다 더 우수한 일반화 모델을 생성하도록 한다. SSVM과 유사하게, 본 실시예에서는 손실 추가 추정 및 최적화 방법들을 통해 전술한 문제들에 존재하는 지수함수적인 개수의 가능성 있는 제약을 다루기 쉽게 취급할 수 있다. 상기의 방법들은 예를 들어, 절단면 알고리즘(cutting plane algorithm)이나 더 최근의 블록 좌표 프랭크 울프(block-coordinate Frank Wolfe) 방법을 지칭한다. 절단면 방법을 이용하여 [수학식 6]을 풀기 위한 의사코드는 아래의 표 1에 기재된 알고리즘 1로 표현될 수 있다.The framework of the present embodiment is a model that is learned about attributes and division

) Is always a trained model for image features (

). This results in the generation of a better generalization model than the case of image feature points alone. Similar to the SSVM, this embodiment can handle the exponential number of possible constraints that are present in the above-mentioned problems in a manageable manner through loss addition estimation and optimization methods. The above methods refer to, for example, a cutting plane algorithm or a more recent block-coordinate Frank Wolfe method. The pseudo code for solving Equation (6) using the section method can be expressed by Algorithm 1 described in Table 1 below.

The algorithm 1 described above is shown in Table 1 below.

Table 1 is an example of an algorithm for solving Equation (6) using an intersection method.

The algorithm of this embodiment has a general form conforming to the SSVM framework. This implies that [Equation 6] is independent of the definitions of joint feature point maps, task specific loss and loss addition estimates. As such, the present embodiment can be applied to various other problems in addition to object search. Only what is required is a definition of the specific components of the three problems, and this definition is also required in the standard SSVM. As described later, the additional loss estimation step only becomes difficult to compare with the SSVM by including the advance information.

Joint Feature Map

The SSVM + of this embodiment estimates the bounding box coordinates by considering all possible bounding boxes in the target images as an extended structured output regressor. The structured output space,

로서 정의된다.

.

는 물체의 존재/부재(presence/absecne)를 나타내고,

는 바운딩 박스의 위, 왼쪽, 아래 및 오른쪽 코너들의 좌표들에 각각 대응된다. 입력과 출력 변수들 간의 관계를 연관짓기 위해, 본 실시예에서는 연결 특징점 맵을 정의한다. 연결 특징점 맵은

로 정의되는 바운딩 박스들에 대한

에서의 특징점들을 인코딩한다. 이를 모형으로 만들면 [수학식 8]과 같다.here,

Represents presence / absences of an object,

Correspond to the coordinates of the upper, left, lower and right corners of the bounding box, respectively. In order to relate the relationship between input and output variables, a connection feature point map is defined in this embodiment. The connection feature point map

For the bounding boxes defined by

To encode the minutiae points. If this is modeled, it can be expressed as [Equation 8].

는 좌표

을 갖는 바운딩 박스 내의 이미지 영역을 나타낸다.here,

Coordinate

Lt; / RTI > in the bounding box.

Similarly, the apparatus performing the method of this embodiment also defines other connection feature point maps for the dictionary space. Instead of the image feature points, the dictionary space can operate in the space of attributes with the help of the division information as in Equation (9).

The definition of the connection feature point map is to specify the problem, and thus, for the object search, the proposed document 1 "Matthew B. Blaschko and Christoph H. Lampert. Learning to localize objects with structured output regression. In ECCV, pages 2-15, 2008. " Detailed embodiments of the two connection feature point maps will be described below.

Task-Specific Loss

와 실제 구조화 레벨

사이의 불일치 수준을 측정하기 위해, 본 실시예에서는 불일치 레벨을 능률적으로 측정하는 손실 함수(loss function)를 정의한다. 본 실시예의 물체 탐색 문제에 있어서, 파스칼 VOC 중첩도(Pascal VOC overlap ratio) 기반의 태스크 특정 손실은 [수학식 10]에서와 같이 두 공간들에서 구해질 수 있다.Predicted output

And the actual structured level

In this embodiment, a loss function for efficiently measuring the level of inconsistency is defined. In the object search problem of the present embodiment, the task specific loss based on the Pascal VOC overlap ratio can be obtained in two spaces as in Equation (10).

는 i번째 이미지에서 객체의 존재(+1) 또는 부재(-1)를 나타낸다.here,

Represents the presence (+1) or absence (-1) of the object in the ith image.

및

인 경우, 0은 모든 제로 벡터에 대응된다.

와

로 정의된 바운딩 박스들이 동일할 때, 손실은 0이며, 그것들이 연결되지 않거나

일 때 손실은 1과 같다.

And

, 0 corresponds to all zero vectors.

Wow

≪ / RTI > are equal, then the loss is zero and they are not connected

The loss is equal to one.

Loss-Augmented Inference

로 인하여 본 실시예에서는 SSVM+ 프레임워크의 훈련과 테스트가 다른 효율적 추정 기술을 필요로 한다.The exponential number constraints that occur during the learning of < RTI ID = 0.0 > [Equation 6] < / RTI &

The training and testing of the SSVM + framework requires different efficient estimation techniques.

Prediction

와 영상 특징

에 의해 주어지는 최적의 바운딩 박스를 찾는 것이다. 사전 정보는 테스트할 때 사용할 수 없고, 추론은 시각적 특징들만으로 실행된다. 따라서, 표준 SSVM에서와 같은 동일한 최대화 문제는 예측 시에 해결될 필요가 있다. 이를 나타내면 [수학식 11]과 같다.In the present embodiment, the object search method is to use a learned weighted vector

And image features

To find the optimal bounding box given by. Dictionary information can not be used in testing, and inference is performed only with visual features. Therefore, the same maximization problem as in the standard SSVM needs to be solved at the time of prediction. This can be expressed by Equation (11).

This maximization problem spans the space of the bounding box coordinates. However, such a problem includes a very large search space and can not be solved thoroughly. In the object search operation, an Efficient Subwindow Search (ESS) algorithm can be employed to effectively solve the optimization problem. The ESS algorithm is described in " Christoph H. Lampert, Matthew B. Blaschko, and Thomas Hofmann. Efficient subwindow search: A branch and bound framework for object localization. TPAMI, 31 (12): 2129-2142, 2009. " can do.

Learning

Compared with the estimation problem required during the prediction process shown in [Equation 11], the optimization that is the main task of the present embodiment at the time of training includes a more complex reasoning process. The learning algorithm of the present embodiment needs to perform maximization as shown in Equation (12) through additional information, information, and substitution loss corresponding to the dictionary space during the iterative process.

과

는 상수이며, 최적화에 영향을 미치지 않는다. 손실 추가 추정으로 지칭되는 [수학식 12]에서의 문제는 절단면 방법(cutting plane method)의 각 반복 중에 두 함수들

와

를 학습하는데 이용하여 가중 벡터

와

를 푸는 것이다.In Equation (12)

and

Is a constant and does not affect optimization. The problem in equation (12), referred to as loss addition estimation, is that during each iteration of the cutting plane method,

Wow

The weight vector

Wow

.

가 [수학식 13]과 같이 정의되어 고정된 해(fixed solution)를 주는 사전 공간에서

를 구한다.The present embodiment employs an alternating approach for reasoning. That is, the original space,

Is defined as < EMI ID = 13.0 > and a fixed solution is given in a dictionary space

.

를 구하면서 [수학식 14]로 나타낸 바와 같이 원본 공간의 최적화를 수행한다.Then, the method of this embodiment

And optimizes the original space as shown in Equation (14).

와

를 얻게 된다.In the method of this embodiment, the two sub-processes of (13) and (14) are repeated until convergence,

Wow

.

에서 사각형 셋으로 전술한 목표 함수들의 상부 경계들을 유도하는 것은 중요하다. [수학식 7]의 대용 손실 항목들의 상부 경계들에 대해서만 유도하고, 나머지 항목들에 대한 유도는 문헌 2를 참조하여 얻을 수 있다.In object search, two problems can be solved by ESS, a branch-bound optimization technique. here,

It is important to derive the upper bounds of the above-mentioned target functions with a square set. The derivation for the upper bounds of the substitution loss items in (7) is derived only, and the derivation for the remaining items can be obtained by referring to Document 2.

의 상부 경계만을 고려한다.

은

의 단조 증가 함수(monotonically increasing function)이므로, 그것의 상부 경계는

의 상부 경계로부터 직접 유도된다. 특히,

의 상부 경계는 하기의 [수학식 15]와 같다.Once the solution is fixed in the pre-space, to obtain the upper boundary of the substitution loss

Lt; / RTI >

silver

Is a monotonically increasing function, its upper boundary is < RTI ID = 0.0 >

Lt; / RTI > Especially,

The upper boundary of Equation 15 is expressed by Equation 15 below.

의 대용 손실의 상부 경계는 하기의 [수학식 16]과 같다.Then,

The upper limit of the substitutional loss of Equation (16) is expressed by Equation (16).

에서 V-형상 함수가 되기 때문에 전술한 문제는 복잡하게 된다. 이 경우,

의 상부 및 하부 경계들에서 함수의 출력을 확인해야 한다.

의 상부 경계는

의 상부 경계와 동일하게 유도되고,

의 하부 경계는 하기의 [수학식 17]과 같이 된다.If the source space is fixed,

The above-described problem becomes complicated. in this case,

Lt; / RTI > the upper and lower bounds of the function.

The upper boundary of

Lt; RTI ID = 0.0 > upper < / RTI &

Is expressed by the following equation (17).

의 상부 경계와 하부 경계를 각각

와

라고 하면, 고정

를 포함한 대용 손실의 상부 경계는 하기의 [수학식 18]과 같이 된다.here,

Lt; RTI ID = 0.0 >

Wow

If you say,

The upper boundary of the substitutional loss is expressed by the following equation (18).

In the method of this embodiment, the target function of Equation (12) is optimized through the alternating procedure based on the standard ESS algorithm by proving the boundaries of the substitute loss as in Equations (17) and (18) .

Experiments

Dataset ( Dataset )

An empirical evaluation of the object search method according to the present embodiment was performed based on Caltech-UCSD Birds 2011 (CUB-2011). CUB-2011 includes 200 categories for different species of birds. The position of each bird is specified using a bounding box. In addition, many dummy dictionary information is provided in the form of 15 different part annotations and 312 attributes and split masks, which can be manually sorted by human annotations in each image. Each category includes 30 training images and about 30 test images.

Visual and Privileged Feature Extraction

In this embodiment, the feature descriptor of the video space includes the Speeded Up Robust Features (SURF) (Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Using the bag-of-visual-words (BoVW) model based on the CVIU, 110 (3): 346-359, In addition, attributes and partition masks are employed as prior information. The information about the attributes is described by the 312 dimensional vector. The elements of the 312-dimensional vector correspond to each attribute and have a binary value according to visibility and relevance. The segmentation information is used to restore the split masks within each image, and the resulting image will contain original background pixels including uniform foreground pixels.

Then, in the method of this embodiment, a 3000 dimensional feature descriptor is extracted based on the same BoVW model as in the video space. The intuition after this approach is to generate a series of feature points that provide a strong and strong response in the foreground area. Such a response can occur more strongly in the original space, thereby making it easier to search in the dictionary space. For each subwindow, a histogram is generated based on the presence of the attribute and the frequency of the dictionary codeword corresponding to the additional image space.

Evaluation

을 포괄하는

공간상에 하이퍼 파라미터들

,

및

를 준비하였다. SSVM 탐색 방법에서, 파라미터

에 대응하는 검색 공간의 단일 차원을 검색하였다.In order to evaluate the SSVM + algorithm according to this embodiment, several training scenes were compared with the original SSVM search method of Blaschko and Lampert. In all experiments,

Encompassing

Hyperparameters on space

,

And

Were prepared. In the SSVM search method,

A single dimension of the search space corresponding to the search space is searched.

First, we examined the effect of small training sample sizes on search performance. These settings are described in "Ryan Farrell, Om Oza, Ning Zhang, Vlad I. Morariu, Trevor Darrell, and Larry S. Davis in Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance in Literature 3. In ICCV, pages 161-168 , 2011. ".

이미지에 대해 훈련되었고, 그 결과

훈련용 이미지를 각각 얻었다. 또한, 본 실시예에서는 100개의 새 클래스들(bird classes)에 대응하고 각각은 10개의 훈련용 이미지들을 포함하는 1000개(n=1000)의 이미지들에 대한 모델을 훈련하였다. 확인 세트로서, 훈련용으로 사용된 이미지들 이외에 카테고리들에서 임의로 선택된 500개의 훈련용 이미지들을 사용하였다.In training, we focused on 14 new categories corresponding to two major new groups. In this embodiment, four different models were trained, each model having a different number of training images,

I was trained on images, and the results

Training images were obtained, respectively. Also, in the present embodiment, we trained a model for 1000 (n = 1000) images corresponding to 100 new classes and each containing 10 training images. As a confirmation set, we used 500 training images randomly selected from the categories in addition to the images used for training.

For testing, we used all the test images of the entire CUB-2011 dataset. The results of the above-described experiment are shown in Table 2 below. Table 2 shows the average overlay (A) and detection result (B) between the structured learning method (SSVM +) and the standard structured learning method (SSVM) through the dictionary information of this embodiment for 100 classes of the CUB- The comparison results are shown.

As can be seen from Table 2, in all cases, the object search method according to the present embodiment showed better results than the SSVM method of the comparative example in the mean detection (PASCAL VOC overlap ratio> 50%) as well as the mean superposition. This indicates that the object search method of the present embodiment converges to a model having better generalization performance than the SSVM of the comparative example, for the same number of training samples. In addition, Table 2 clearly shows the tendency of the benefits of prior information to be reduced by increasing the training data set in the case of the comparative example.

In order to further evaluate the merit of the SSVM + of this embodiment, the performance of the overlap ration and the number of detection for all 100 classes is compared with the performance of 10 classes per class Images (n = 140).

As shown in FIGS. 4A and 4B, SSVM + of the present embodiment shows relatively superior performance in both superposition and detection rates in most of the bird classes. The difference (diff) between the method (SSVM +) of this embodiment, shown in blue, and the method of comparison (SSVM), shown in gray, is indicated by the black area below these. Each class typically includes 30 test images, but some classes may contain 18 or fewer images. The average degree of overlap in this embodiment is 45.8% and the average number of detections is 12.1 (i.e., 41.5%).

According to the above-described embodiment, a structured prediction algorithm for object search based on SSVM including advance information is provided. The algorithm first involves combining the pre-information into a structured prediction frame. In addition, the algorithm of this embodiment may utilize various types of additional information during training to improve generalization performance in testing. These algorithms can be applied to object search problems, which can be solved by a new structured SVM model using prior information. That is, the object search method of the present embodiment employs an alternate loss addition estimation process to process items in the target function corresponding to the prior information. The method of the present embodiment also provides useful results that suggest a positive advantage of applying the proposed algorithm to the Caltech-UCSD Birds 200-2011 dataset and utilizing additional information available only during training. Unfortunately, the benefit of prior information tends to decrease as the number of training samples increases. However, the SSVM + framework of the present embodiment may be particularly useful when there is little training data or the cost of annotation is very large.

5 is a block diagram of an image object search apparatus according to another embodiment of the present invention.

Referring to FIG. 5, the video object searching apparatus 11 according to the present embodiment includes a framework generating unit 111, a learning unit 112, a model generating unit 113, And a search unit 114, and may further include a verification unit 115 according to an implementation.

In this case, the video object search apparatus 11 may be implemented as a processor such as a microprocessor. In this case, the video object search apparatus 11 may include a framework generation unit 111, a learning unit 112, a model generation unit 113, And verification unit 115 may include a framework generation module, a learning module, a model generation module, a search module, and a verification module, respectively, or correspond to each module in the described order.

The framework generation unit 111 generates a learning framework in which the preliminary information is combined with the structured prediction framework. Here, the structured prediction frame may include a structured SVM (Structured Support Vector Machine) classifier. The framework generation unit 111 may combine the first function of the first space based on the prior information and the second function of the second space based on the training sample. Herein, the dictionary information includes a segmentation, a part, an attribute, or a combination thereof of a training sample, and the combination of the first function and the second function includes a space including images and attributes of the training sample, To the space of the bounding box coordinates.

The learning unit 112 executes alternate optimization learning for the learning framework. The learning unit 112 may process the item of the objective function corresponding to the advance information through the alternating loss-augmented inference. The learning unit 112 can learn the first model based on the training example and the second model based on the prior information of the training example, as in Equation (6).

The model generation unit 113 generates a prediction model from the learning framework in which the alternate optimization learning is executed. The prediction model can operate to always correct the model trained by attributes and segmentation, which is always trained by the image feature. In this way, the model generation unit 113 can easily handle the exponential number of possible constraints associated with the optimization as compared to the existing loss addition estimation or optimization methods (such as the intersection algorithm or the block coordinate Frankwolf method) .

The search unit 114 predicts or searches an object in a specific image from a test sample or an input image using a prediction model. The search unit 114 can find the optimal bounding box given by the learned weighted vector of the prediction model and the image feature in the specific image from the input image.

The verification unit 115 can verify the learning framework of the learning unit 112 based on the actual image or the actual image information (groundtruth). The verification unit 115 may be coupled to the learning unit 112 or may be coupled between the learning unit 112 and the model generation unit 113 to verify the framework during or after learning.

FIG. 6 is a detailed block diagram of a configuration that can be employed in the learning unit of the video object search apparatus of FIG. 5;

6, the video object search apparatus according to the present embodiment may include a first learning unit 1121, a second learning unit 1122, and a third learning unit 1123 as a learning unit 112 .

The first learning unit 1121 alternately performs Efficient Subwindow Search (ESS) in the first space based on the advance information and the second space based on the training sample through the alternate loss addition estimation. According to the first learning unit 1121, it is possible to optimize loss-augmented inference.

The second learning unit 1122 estimates the bounding box coordinates of the object by extracting all possible bounding boxes from the target image of the training sample through the alternate loss addition estimation. The second learning unit 1122 can correspond to a concrete embodiment of the operation of the first learning unit 1121. [ That is, the operation of the second learning unit 1122 may correspond to the equations (13) and (14) being repeated until convergence and thereby obtaining solutions.

The third learning unit 1123 creates a connection feature point map that associates the bounding box coordinates through the additional loss addition estimation and associates the relationship between input and output variables.

According to the first learning section 1121 and the third learning section 1123 or the second learning section 1122 and the third learning section 1123 described above, the structured prediction model is estimated using the advance information, It is possible to generate a prediction model with excellent performance even for a small number of data samples.

On the other hand, in the video object search apparatus 11 according to the above-described embodiment, the framework generating unit, the learning unit, the model generating unit, the searching unit, the verifying unit, A processor stored in the system and coupled to the memory system may be used to efficiently search for a particular object in the input image when executing the program.

That is, in the above-described embodiment, the elements (including 111 to 114) of the video object search apparatus 11 may be implemented as a module or a functional unit mounted on a processor of a mobile device or a computer apparatus, but the present invention is not limited thereto . The above-mentioned components are stored in a computer-readable medium (recording medium) in the form of software for implementing a series of functions (multiple multiple emotion recognition methods) performed by them, or transmitted to a remote place in a carrier form so as to operate in various computer devices Can be implemented. In this case, the computer-readable medium may be coupled to a plurality of computer devices or a cloud system connected via a network, and at least one of the plurality of computer devices or the cloud system may be connected to the memory system A program or a source code for performing an object search method can be stored.

The computer-readable medium may be embodied in the form of program instructions, data files, data structures, and the like, alone or in combination. Programs recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. The computer-readable medium may also include hardware devices that are specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Program instructions may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. The hardware device may be configured to operate with at least one software module to perform the variance estimation method of the present invention, and vice versa.

7 is a block diagram of an image object search apparatus according to another embodiment of the present invention.

Referring to FIG. 7, the video object searching apparatus 10 using the dictionary information according to the present embodiment includes a processor 11 and a memory system 12 as an embodiment of the apparatus for performing the video object searching method described above. . In addition, the video object search apparatus 10 may include a network interface 13 according to an implementation, and may further include a display device 14 (hereinafter, simply referred to as a display) and an interface 15.

The processor 11 may be connected to the memory system 12 to perform the program 12a stored in the memory system 12. [ The program 12a may be an implementation of the video object searching method using the dictionary information of the embodiment described above. That is, the processor 11 may be mounted on a mobile terminal or a computer device capable of image processing to search for an object in an image, and may operate to predict or search for a specific object in the input image.

More specifically, the processor 11 may be configured to sample an object to be searched in an input image from a camera (which may be included in an interface) or a memory system 12 included in the system 10, such as a mobile device, It is possible to learn through the alternate optimization learning method and effectively search the desired object in the input image by using the prediction model having learned parameters.

The processor 11 may include a framework generation module, a learning module, a model generation module, a search module, and the like. The processor 11 may include one or more processors on which the above-described modules are mounted. The processor 11 may also include any type of computing circuit, such as a microprocessor, microcontroller, graphics processor, digital signal processing processor, or any other type of processing circuitry. In addition, the processor 11 may include embedded controllers, such as general purpose or programmable logic devices or arrays, application specific integrated circuits, single chip computers, smart cards, and the like.

When the processor 11 is a microprocessor, a microcontroller, a graphics processor, or a digital signal processing processor, the processor 11 includes an arithmetic logic unit (ALU) for performing calculations, a register for temporary storage of data and instructions And a controller for controlling or managing the interface device between the middleware. When at least one of the modules described above is mounted on the processor 11 in the form of an application program, the processor 11 may include a high-level command processing unit and a module control unit. The module control unit may include a mapping unit and a module interface unit, and may control each module through a module control unit. Here, the high-level command processing unit converts a signal or an instruction input through an API (Application Programming Interface) to output a high-level command, and the mapping unit maps a high-level command into a device level command that can be processed by each module, The module interface can pass device level commands to the module.

The memory system 12 may be a volatile memory such as a random access memory (RAM) or a read only memory (ROM), a storage medium in the form of a nonvolatile memory, and a storage medium such as a floppy disk, a hard disk, a tape, a CD- long-term storage medium. In addition, the memory system 12 may store programs, data, or a set of commands that perform the image object search algorithm of the present embodiment according to the implementation.

The network interface 13 is connected to a network and can perform data communication with other communication devices on the network. When using the network interface 13, assuming that alternate optimization learning has been performed in advance by advance information, the video object search apparatus 10 of the present embodiment can generate data, commands or signals for performing the steps of the video object searching method It is possible to efficiently search for a desired object in the input image by downloading or receiving it in real time. The network interface 13 described above can be implemented to support one or more communication protocols for performing data communication in one or more single or combination networks selected from a wireless network, a wired network, a satellite network, a power line communication, and the like.

The display 14 is connected to the processor 11 and displays at least a part of a process of searching for an object in a training sample, an authentication sample, a test sample, or an input image used in the video object searching apparatus 10, Refers to a component that performs a function corresponding to the means. The display 14 may be directly connected to the processor 11 but is not limited thereto and may be connected to the remote site via the network interface 13. [ The display 14 may be a liquid crystal display (LCD) device, an organic light emitting diode (OLED) display device, a plasma display panel (PDP) device, or a cathode ray tube (TV) equipped with a modem.

The interface 15 may be connected to the processor 11 and may include means for communicating between the video object search apparatus 10 and the outside (including an external user) or a device performing a function corresponding to this means . The interface 15 may include a user interface. For example, the interface 15 may include at least one input device selected from a keyboard, a mouse, a touch screen, a touch panel, a microphone, a camera, and the like. And may include at least one or more.

According to the present embodiment, a reliable model can be learned even with a small number of training sample data through an apparatus that performs a learning algorithm combining prior information, and an object to be searched in the input image can be efficiently predicted through the learned model There is an advantage to be able to search.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

10: Image object search device
11: Processor
12: Memory system
13: Network interface
14: Display
15: Interface

Claims (20)

A method performed by an apparatus capable of digital signal processing,
Generating a learning framework in which the structured prediction framework is combined with prior information;
Executing alternate optimization learning for the learning framework using training samples including the dictionary information;
Generating a prediction model from the learning framework in which the alternate optimization learning is performed; And
Searching for an object in a specific image from the input image using the prediction model;
A method of searching an embedded video object. The method according to claim 1,
Wherein the generating the learning framework comprises:
And combining a first function of the first space based on the dictionary information and a second function of the second space based on the training sample,
Wherein the combination of the first function and the second function is a combination of an image of the training sample and a space containing attributes of the object in the image, And concatenating them into a space of box coordinates. The method according to claim 1,
Wherein the structured prediction frame comprises a structured SVM (Structured Support Vector Machine) classifier. The method according to claim 1,
Wherein the step of performing the alternate optimization learning comprises:
Processing an item of an objective function corresponding to the dictionary information through an alternating loss-augmented inference. The method of claim 4,
Wherein the step of performing the alternate optimization learning comprises:
And alternately performing Efficient Subwindow Search (ESS) in a second space based on the original image of the training sample and a first space based on the advance information through the alternate loss addition estimation. Search method. The method of claim 4,
Wherein the step of performing the alternate optimization learning comprises:
Further comprising estimating the bounding box coordinates of the object by extracting all possible bounding boxes from the target image of the training sample through the alternate loss addition estimation. The method of claim 6,
Wherein the step of performing the alternate optimization learning comprises:
And creating a connection feature point map that links the bounding box coordinates through the alternate loss addition estimation to associate the relationship between input and output variables. The method according to claim 1,
The step of predicting or searching the object may include:
And finding an optimal bounding box given by a learned weighted vector of the predictive model and an image feature within a particular image from the input image. The method according to claim 1,
Further comprising: verifying the learning framework based on actual image information after the step of generating the learning framework and before the step of generating the prediction model. A computer-readable medium having recorded thereon a program for performing an image object searching method according to any one of claims 1 to 9. A framework generation unit for generating a learning framework in which dictionary information is combined with a structured prediction framework;
A learning unit for executing alternate optimization learning for the learning framework using training samples including the dictionary information;
A model generation unit that generates a prediction model from the learning framework in which the alternate optimization learning is performed; And
A search unit for searching for an object in a specific image from the input image using the prediction model;
Comprising: a video object search device; The method of claim 11,
Wherein the framework generating unit comprises:
Combining the first function of the first space based on the dictionary information and the second function of the second space based on the training sample,
Wherein the combination of the first function and the second function includes a space including images and attributes of the training sample as a space of bounding box coordinates Wherein the video object is a video object. The method of claim 11,
Wherein the structured prediction frame comprises a structured SVM (Structured Support Vector Machine) classifier. The method of claim 11,
Wherein the learning unit processes the item of the objective function corresponding to the advance information through an alternating loss-augmented inference. 15. The method of claim 14,
The learning unit includes a first learning unit that alternately performs an Efficient Subwindow Search (ESS) in a first space based on the advance information and a second space based on the training sample through the alternate loss addition estimation Wherein the object searching unit comprises: 15. The method of claim 14,
Wherein the learning unit further comprises a second learning unit for extracting all possible bounding boxes from a target image of the training sample through the alternate loss addition estimation and estimating bounding box coordinates of the object. 18. The method of claim 16,
Wherein the learning unit further comprises a third learning unit for creating a connection feature point map that connects the bounding box coordinates through the alternate loss addition estimation to associate the relationship between input and output variables. The method of claim 11,
Wherein the search unit finds an optimal bounding box given by a learned weighted vector of the predictive model and an image feature in a specific image from the input image. The method of claim 11,
And a verification unit coupled to the framework generation unit or arranged between the framework generation unit and the model generation unit and verifying the learning framework based on actual image information including a specific object. The method of claim 11,
A memory system for storing a program or an instruction for operation of the framework generating unit, the learning unit, the model generating unit, the searching unit, or a combination thereof; And
A processor coupled to the memory system and executing the program or command to search for an object previously designated in the input image
Comprising: a video object search device;

Images