KR101898648B1 - Apparatus and Method for Detecting Interacting Groups Between Individuals in an Image - Google Patents

Abstract

The present invention relates to a device and a method for detecting an interacting group between characters in an image, capable of accurately determining an interacting group by considering not only position information of the individual in the image but also the emotional interrelation between the individuals. The device for detecting an interacting group between characters in an image includes: a candidate group ingathering generating unit generating a candidate group ingathering by inferring the relative distances in a three-dimensional space between people in the image; a geometric relation scale calculation unit calculating the geometric relation scale by inferring the distance between two faces in an actual three-dimensional space in the candidate group ingathering; a social relation scale calculation unit calculating the social relation scale by analyzing social relation information by considering face information of the people and the position information of the people; and an interacting group determining unit performing optimization to determine the interacting group by selecting an ingathering of mutually exclusive groups among the groups which can be generated so that the total of a weighted value of a geometric relation scale value and a social relation scale value becomes maximum.

Description

[0001] Apparatus and Method for Detecting Interaction Group Between Individuals [

The present invention relates to image analysis, and more specifically, it relates to a method and apparatus for detecting an interaction group between image characters, which can more accurately determine an interactive group considering not only the positional information of an individual in the image but also the emotional relationship between individuals And more particularly,

The development of social networks has generated a lot of images of people's social activities. These images contain important information about interaction and social relations among people.

Such information is very useful for analyzing customers in market transactions, which is an important area of modern society, and for monitoring objects in social security services. In order to obtain relevant information for this purpose, it is essential to find an interaction group from images of everyday life containing social activities.

Several techniques have been proposed to determine the interaction group in an image with the need for such a technique.

However, these techniques mainly focused on images acquired through surveillance cameras. The images acquired through the surveillance camera have a wide angle of view and have a lot of information about the position of a person because they acquire images at a fixed location at a high position.

However, the images of various daily lives generated in the recent social network service (SNS) have taken various angles and objects, and some objects are also occluded. Therefore, applying the previous techniques to the images of everyday life is not a self-evident process.

There are two main types of techniques for analyzing the interaction of people in a video.

One is based on human behavior and the other is based on the formation of people in the image.

The method based on people's behavior usually starts with finding pre-defined motion in the video. Then, it finds the people who interact with it and determines the interaction group based on them.

Techniques based on behavior to date use continuous information from video rather than single images.

And behavior-based technologies do not find interactions among people who do not act in advance. Therefore, these techniques have limitations in finding interaction groups in images of everyday life generated from SNS.

Studies based on people's structures can analyze a wider variety of interactions, including static situations, than behavior-based ones.

The methods based on the structure of people first determine the plane position of the people in the top view image and deduce the face direction for each individual. Then we find an interaction group among the predefined structures of people.

However, these methods assume that people can be easily located in the images captured in Topview. This assumption can be satisfied only in the surveillance camera image, not in the images originating from the SNS.

Another way is to automatically infer the location of people in the top view of the video, but this method has the advantage of being able to find the interaction group even in various situations including the static situation, but the structure of the group to be searched must be defined in advance .

Also, since the direction of the face plays an important role in determining the structure to be searched in the image, the final result is very sensitive to the error of the estimated face direction.

Therefore, there is a limit to apply to a large number of people and daily life images photographed under various conditions.

Korea Patent Publication No. 10-2009-0024969 Korean Patent No. 10-1782590 Korean Patent No. 10-1715708

The present invention solves the problem of image analysis and interaction detection of the related art, and it is an object of the present invention to provide an image processing apparatus and method that can more accurately determine an interactive group considering not only position information of an individual in an image but also emotional relationship among individuals It is an object of the present invention to provide an apparatus and method for detecting an interaction group between video characters.

It is an object of the present invention to provide an apparatus and method for detecting an interaction group between image characters that detect an interactive group in a daily life image by designing a new interaction scale.

The present invention relates to an interactive group of image characters to enhance the performance through the verification of the efficiency of image analysis and interaction detection by producing a new data set for evaluating the performance of group detection that interacts in the everyday life image And to provide a device and method for detection.

The present invention provides an apparatus and method for detecting an interaction group among image characters so as to more accurately detect an interacting group using semantic clues such as social relations and geometric information of a point where people are located It has its purpose.

The present invention can be applied to socioeconomic analysis such as strategy establishment, camera security and various sociological analysis by making MLPA Social Group dataset, which is a new data set for detecting interactive groups, The present invention provides an apparatus and method for detecting an interaction group among a plurality of video characters.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided an apparatus for detecting an interaction group among a plurality of video character sets, the apparatus comprising: a candidate group set generation unit for generating a candidate group set by inferring a relative distance in a three- A geometric relation scale calculation unit that calculates a geometrical relationship scale by inferring the distance between two faces in a real three-dimensional space in a candidate group set, and calculates social relation information by considering the face information of people together with the location information of people A social relation scale calculation unit for calculating a social relation scale and selecting a group of mutually exclusive groups among the groups that can be generated so as to maximize the weight sum of the values of the geometrical relationship scale and the social relation scale to be the interaction group And an interaction group determination unit for performing optimization for .

According to another aspect of the present invention, there is provided a method for detecting an interaction group between video characters, the method comprising: estimating a relative distance in a three- A step of generating a geometric relationship measure by inferring the distance between two faces in an actual three-dimensional space in the image, analyzing the social relationship information by considering the face information of the people together with the location information of the people, Performing optimization for determining an interaction group by selecting a set of mutually exclusive groups among groups that can be generated such that a sum of a weight of a geometric relation measure value and a social relation measure value is maximized; And a control unit.

The apparatus and method for detecting an interaction group among video characters according to the present invention have the following effects.

First, it allows the user to more accurately determine the interacting group by taking into account the emotional interrelationship among the individuals as well as the location information of the individual in the image.

Second, by designing new interaction measures, interactive groups can be effectively detected in daily life images.

Third, we can improve the performance by verifying the effectiveness of image analysis and interaction detection by creating a new data set to evaluate the performance of group detection that interacts in daily life images.

Fourth, we use semantic clues such as social relations and geometric information of the points where people are located to more accurately detect interacting groups.

Fifth, MLPA Social Group dataset, which is a new data set for detecting interactive groups, is produced and performance evaluation using dataset is applied to socio-economic analysis such as strategy establishment, camera security and various sociological analysis. do.

1 is a block diagram of an apparatus for detecting an interaction group between video characters according to the present invention.
Figure 2 is a flow chart illustrating a method for detecting an interaction group between video characters according to the present invention.
3 is a block diagram showing an example of generation of a candidate group set
FIG. 4 is a detailed block diagram of an apparatus for detecting an interaction group among video characters according to the present invention.
Figs. 5A to 5C are diagrams showing an example of geometric relationship measure calculation
6 is a block diagram illustrating an optimization process for determining an interaction group
7 is a diagram showing an example of an MLPA Social Group dataset
8 is a diagram showing a comparison result in the MLPA Social Group data set
FIG. 9 is a diagram showing a comparison of performance by relational metrics

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of an apparatus and method for detecting an interaction group among video characters according to the present invention will be described in detail.

The features and advantages of an apparatus and method for detecting an interaction group between video characters according to the present invention will be apparent from the following detailed description of each embodiment.

FIG. 1 is a block diagram of an apparatus for detecting an interaction group among video characters according to the present invention, and FIG. 2 is a flowchart illustrating a method for detecting an interaction group among video characters according to the present invention .

An apparatus and method for detecting an interaction group among video characters according to the present invention can detect a group interacting more accurately using semantic clues such as social relations and geometric information of a point where people are located will be.

To this end, the present invention may include a configuration for creating candidates for groups that are actually interacting using geometric relationships between faces of people in an image.

The present invention quantitatively calculates the extent to which individuals interact with each other in the generated group and considers the physical information such as the location and gaze of each individual as well as the emotional relevance deduced through facial expression, sex, and age Configuration.

In order to analyze the geometric relationship, the present invention calculates the distance between people considering the straight line distance between the point where the individual is located and the relative face size of the individual, and analyzes the social relationship based on the interpersonal circle of sociology, Can be classified into eight categories, and SRN (Social Relations Network) can be used to apply the results to social relationship measurement.

As shown in FIG. 1, an apparatus for detecting an interaction group among video characters according to the present invention generates a candidate group set by inferring a relative distance in a three-dimensional space between people using face size information of people A geometric relation scale calculation unit 20 for estimating a distance between two faces in an actual three-dimensional space in the daily life image to calculate a geometrical relation scale, A social relation scale calculating unit 30 for calculating a social relation scale by analyzing the social relation information by considering the face information of the people together, and a group of mutually exclusive groups among the groups that can be generated, An interaction that performs optimization for the determination of an interaction group by selecting the weighted sum of the relational measure values to be maximal And a determination unit 40. The

A method for detecting an interaction group among video characters according to the present invention will be described in detail as follows.

2, the candidate group set is created by inferring the relative distance in the three-dimensional space between people using the face size information of the people (S201)

Next, the geometric relationship scale is calculated by inferring the distance between two faces in the actual three-dimensional space in the daily life image (S202)

Then, the social relation scale is calculated by analyzing the social relationship information in consideration of the location information of the people and the face information of the people (S203)

Then, optimization is performed for determining an interaction group by selecting a set of mutually exclusive groups from among the groups that can be generated so that the sum of the weights of the geometric relation measure and the social relation measure is maximized (S204)

3 is a block diagram showing an example of generation of a candidate group set.

In the following description, "interaction" means actions in which two or more people directly affect each other.

For example, the present invention includes grouping people who interact with each other in an image containing a given social activity of everyday life, including conversation, eating together, or looking at each other.

내에 존재하는

명의 얼굴 정보를 가지고 있는 집합을

라고 정의한다.Here,

Existing within

A set of face information

.

는 각각

번째 사람의 얼굴 정보, 영상에서 얼굴의 중심 좌표 그리고 영상에서 해당 얼굴의 크기를 나타낸다. 그 후 상호 배타적인 상호작용 하는 사람들의 그룹들을 찾는다. here,

Respectively

The center coordinates of the face in the image, and the size of the face in the image. It then looks for groups of mutually exclusive interacting people.

의 부분 집합

를 임의의 그룹들의 집합인 후보 그룹 집합이라고 하며 다음 조건을 만족한다.here,

Subset of

Is a set of candidate groups, which is a set of arbitrary groups, and satisfies the following conditions.

는

에 속하는 그룹을 나타내고 그룹의 수(

)의 범위는 1부터

사이의 값을 갖는다.here,

The

And the number of groups (

) Ranges from 1 to

Lt; / RTI >

를 정의하여 그룹

에 속한 개체들 간의 상호 작용 정도를 수치화하고 수학식 2의 목적 함수를 충족하는 최적의 그룹 집합

를 결정한다.Interaction scale

Group to define

And the optimal group set satisfying the objective function of Equation (2)

.

를 찾는 예를 보여준다.FIG. 3 illustrates a method of generating a candidate group set from an input,

.

(a) shows an input image and face information as an example. (b), the input image and information are represented by a graphic model. (c) shows a candidate group set generated from (b), and each group set is composed of two or more groups.

및

에서처럼 공통 그룹을 가질 수 있다. Some group sets

And

As shown in FIG.

The present invention largely consists of three steps: generation of a candidate group set, calculation of an interaction scale for each group, and search for an optimal group set.

로부터

를 생성한다.In the candidate group set generation step, a geometric constraint is defined,

from

.

Then, calculate the degree of geometric and social relations for each group on a relational scale.

Finally, in the optimal group set search process, an optimal group set satisfying the objective function of Equation (2) is determined through an optimization process.

FIG. 4 is a detailed block diagram of an apparatus for detecting an interaction group among video characters according to the present invention.

Create a set of candidate groups that satisfy the geometric constraints and calculate the interaction scale based on geometric and social relations for each group.

를 결정한다.The optimal interaction group set

.

)가 상호 작용하는 사람들로 잘 그룹화 되었는지 평가하기 위한 상호 작용 척도는 다음과 같이 기하학적 관계 척도

와 사회적 관계 척도

로 계산된다.Any group (

) Were grouped well into interacting people, the interaction scale was used to measure the geometric relationship scale

And social relations scale

.

는 두 관계의 균형을 맞추기 위한 설계 매개 변수로서, 0에서 1 사이의 값을 갖는다.here,

Is a design parameter for balancing the two relations and has a value between 0 and 1.

The generation of the candidate group set will be described in detail as follows.

People in distant places in the image are unlikely to interact with each other, but it is theoretically possible to collectively group all people in the image.

Therefore, when detecting an interactive group, it is effective to exclude people who are far from the computation and performance in grouping.

To this end, the present invention creates a pair of persons satisfying the geometric constraint, and then performs a grouping process.

In order to measure the distance between people, information about the position of an individual in a three-dimensional space is needed. However, it is not easy to deduce the position of people in three-dimensional space in images taken in everyday life.

To solve this problem, we use people's face size information to deduce the relative distance in three-dimensional space between people.

과

의 얼굴 크기의 비율이 일정 크기보다 크다면, 두 사람이 3차원 공간에서 아주 멀리 떨어져 있다고 가정한다. 이 가정을 기반으로 두 얼굴의 비율

을 수학식 4에서와 같이 정의하여 두 얼굴이 그룹화 될 만큼 가까운 지의 여부를 판단한다.Two random faces

and

Is larger than a certain size, it is assumed that the two persons are far away from the three-dimensional space. Percentage of two faces based on this assumption

Is defined as shown in Equation (4) to determine whether the two faces are close enough to be grouped.

의 값이 설계 파라미터

보다 작거나 같을 때만 두 얼굴

과

을 그룹화 할 수 있다고 하고 이 조건을 기하학적 제약 조건이라고 한다. For example,

Lt; RTI ID = 0.0 >

Two faces only when less than or equal to

and

Can be grouped, and this condition is called a geometric constraint.

은 두 사람의 쌍에 대한 집합으로 각 쌍은 그룹화 될 수 있는 사람들로 구성된다. 그 후

의 원소들을 상호 배타적으로 선택하여 그룹

를 생성하고 생성된

를 통해 후보 그룹 집합

을 생성 한다.

Is a set of pairs of two, each pair consisting of people that can be grouped. After that

Of the elements of the group < RTI ID = 0.0 >

Lt; RTI ID = 0.0 >

Through the candidate group set

.

A detailed description of the calculation of the geometric relationship scale is as follows.

Figs. 5A to 5C are diagrams showing an example of geometric relationship measure calculation. Fig.

The position of each person in a video is an important clue to understanding interpersonal interactions. Interacting people are usually located at close distances, and the location of people represents a particular structure depending on the type of interaction.

The methods of using people's location information mainly deduce their interaction based on people's location structure in the top view image. However, since the top view image can not be easily acquired in everyday life, there are limitations on the types of images that can be analyzed.

과

사이의 거리

를 추론한다.In order to overcome these limitations, the present invention has been applied to a method for detecting two faces

and

Distance between

.

As in Equation (5), the distance between the faces is calculated by correcting the straight line distance between the faces to the face size.

는 다음과 같이 임의의 그룹

에 대한 기하학적 척도를 정의하는 데 사용된다.Inferred distance

Is an arbitrary group

Is used to define the geometric metric for.

는

에 있는 모든 쌍의 집합이다.here

The

Is a set of all pairs in.

A detailed description of the calculation of the social relationship scale is as follows.

Personal location information is an important clue to determine which groups interact. Interactions can occur, however, even among people who are at a certain distance from static situations, such as emotional empathy, or without specific behavior.

In order to solve this problem, the present invention analyzes social relation information by considering face information of people along with location information of people for various types of interaction detection.

The present invention uses SRN in the form of an expanded Siamese neural network.

SRN uses additional features based on the Deep Convolutional Network (DCN) to determine the degree of social relationship between the two.

In SRN, social relations are defined as 'trust', 'attachment', 'revealing emotions', 'assurance', 'dominant', 'competitive', 'warm', 'intimate' '.

)을 입력으로 부분 모듈인 DCN을 통해 나이, 성별, 얼굴 표정, 얼굴 방향 정보를 포함한 20 차원 특징 벡터를 추출한다.SRN is a pair of face images (

), The 20-dimensional feature vector including age, gender, facial expression, and face direction information is extracted through the partial module DCN.

를 출력으로 하며 이때

은

를 만족한다.The feature vector extracted is a 8-dimensional vector representing the social relation through the neural network constructed in the form similar to Siamese neural network

And outputs

silver

.

의 쌍

에 대한

을 사용하여 다음과 같이 그룹

에 대한 사회적 관계 척도

를 정의한다.Satisfy geometric constraints

Pair of

For

Use the following groups

Social Relations Scale for

.

The optimization process for determining the interaction group will be described in detail as follows.

6 is a block diagram illustrating an optimization process for determining an interaction group.

로부터 생성 될 수 있는 그룹들 중에서 상호 배타적인 그룹들의 집합을 기하학적 관계 척도 값과 사회적 관계 척도 값의 가중치 합이 최대가 되도록 선택 한다.The objective function of Equation (2)

A group of mutually exclusive groups is selected such that the weighted sum of the values of the geometric relationship measure and the social relation measure is maximized.

Equation 2 is therefore a type of combinatorial optimization problem. Based on this, we re-formulate the problem of Binary Inter- gram Programming (BIP) mathematically equivalent to the original problem to solve the equation (2).

를 다음과 같이

에서 생성 할 수 있는 모든 그룹들의 집합으로 정의한다.first,

As follows

Is defined as a set of all groups that can be generated by the user.

는

의 원소들로 생성되어 그룹화 될 수 있는 얼굴들의 쌍들로 구성되는 그룹을 나타내며,

은 가능한 모든 가능한 유효한 그룹들의 수를 나타내며 최대

의 값을 갖는다.here,

The

≪ / RTI > and pairs of faces that can be created and grouped into elements of < RTI ID = 0.0 >

Represents the number of all possible possible groups,

Lt; / RTI >

의 원소들 중에서 특정 그룹이 최적의 그룹 집합에 포함되는지의 여부를 결정하는 이진 결정 벡터(Binary decision vector)를

로 정의한다.

A binary decision vector that determines whether a particular group is included in the optimal group set

.

로 정의한다.In addition, a vector having an interaction measure value of a group corresponding to a binary decision vector

.

를 찾는 것과 수학적으로 동일한 것이 된다.Thus, solving Equation (2) can be done in the BIP of Equation (9) using the optimal binary decision vector

Is mathematically equivalent to finding.

로부터

는 다음과 같이 얻어 진다:(9) < / RTI >

from

Is obtained as follows:

To solve the combinatorial optimization problem, we use the widely used branch and bound (B & B) method.

The MLPA Social Group dataset will be described in detail below.

FIG. 7 is a block diagram showing an example of an MLPA Social Group data set, and FIG. 8 is a block diagram showing a comparison result in an MLPA Social Group data set.

And FIG. 9 is a diagram showing a comparison of the performance of each relational scale.

The MLPA Social Group dataset is a set of group data consisting of images collected from the web. It consists of about 250 images of various situations, angles, and numbers of people. It is made.

The performance of the apparatus and method for detecting an interaction group among video characters according to the present invention is evaluated using the MLPA Social Group data set as follows.

Table 1 shows that performance drops when one of the two scales is not used.

가 2/3 일 때 사회적 관계는 기하학적 관계에 비해 성능에 더 많은 영향을 미친다. 그러나

가 1 일 때, 두 관계 사이에는 큰 차이가 없다.As shown in Table 1

Is 2/3, social relations have more impact on performance than geometric relationships. But

Is 1, there is no significant difference between the two relations.

This is because geometric relationships have a greater impact on the evaluation of stricter standards because most interactions in everyday life depend on distance.

As shown in FIG. 9, it can be seen that both geometric and social relations play an important role in the interaction measurement.

The apparatus and method for detecting an interaction group among the video characters according to the present invention can detect the position of an individual in a video image, It is.

The present invention provides a method and apparatus for detecting groups interacting more accurately using semantic clues such as social relations and geometric information of points where people are located, It is possible to improve the performance by verifying the efficiency of image analysis and interaction detection by creating a data set.

As described above, it will be understood that the present invention is implemented in a modified form without departing from the essential characteristics of the present invention.

It is therefore to be understood that the specified embodiments are to be considered in an illustrative rather than a restrictive sense and that the scope of the invention is indicated by the appended claims rather than by the foregoing description and that all such differences falling within the scope of equivalents thereof are intended to be embraced therein It should be interpreted.

10. Candidate group set generation unit 20. Geometrical relation scale calculation unit
30. Social Relative Scale Calculation Unit 40. Interaction Group Decision Unit

Claims (11)

A candidate group set generation unit for generating a candidate group set by deducing a relative distance in a three-dimensional space between people in an image;
A geometric relationship measure calculating unit for calculating a geometrical relation measure by deducing a distance between two faces in an actual three dimensional space in a candidate group set;
A social relation scale calculation unit for calculating the social relation scale by analyzing the social relation information by considering the face information of the people together with the location information of the people;
And an interaction group determination unit for performing optimization for determining an interaction group by selecting a set of groups mutually exclusive among the groups that can be generated so that the sum of the weights of the geometric relationship measure and the social relation measure is maximized and,
The candidate group set generation unit generates a candidate group set

Existing within

A set of face information

≪ / RTI >

Respectively

And the face coordinates of the face in the image, and the size of the face in the image. delete 2. The apparatus of claim 1, wherein the candidate group set generation unit comprises:

Subset of

Is a set of candidate groups, which is a set of arbitrary groups, the following condition is satisfied,

here,

The

And the number of groups (

) Ranges from 1 to

, And the interaction scale

Group to define

And the optimal group set satisfying the objective function of the above equation

To

Wherein the determination of the interaction group is based on the determination of the interaction group. 4. The method of claim 3,

Is determined,
Any group (

) Were grouped well into interacting people, the interaction scale was used as a geometric relationship scale

And social relations scale

in,

Lt; / RTI >
here,

Is a design parameter for balancing the two relationships and has a value between 0 and 1. A device for detecting an interaction group between video characters. The apparatus of claim 1, wherein the candidate group set generation unit comprises:
Two random faces

and

Is larger than a certain size, the ratio of the two faces

of

And determining whether grouping of the two faces is to be performed. 6. The method of claim 5,

Lt; RTI ID = 0.0 >

Two faces only when less than or equal to

and

Can be grouped,

Is a set of pairs of two, each pair consisting of people who can be grouped,

Of the elements of the group < RTI ID = 0.0 >

Lt; RTI ID = 0.0 >

Through the candidate group set

And generating an interactive group of video characters. 2. The apparatus according to claim 1,
Two faces in real 3D space in video

and

Distance between

In addition,

The distance between the faces is calculated by correcting the straight line distance between the faces to the face size,
Inferred distance

The

As an arbitrary group

Lt; RTI ID = 0.0 > geometric < / RTI >
here

The

The set of all pairs in the set of video characters. 4. The method of claim 1, wherein the social relational metric calculator uses SRN in the form of an expanded Siamese neural network,
SRN is a pair of face images (

), And 20-dimensional feature vectors including age, gender, facial expression, and facial direction information are extracted through DCN, which is a partial module, and extracted feature vectors are classified into 8 groups representing social relations through a neural network composed of Siamese neural network Dimension vector

And outputs

silver

Wherein the first group comprises a plurality of video characters. 9. The method of claim 8, further comprising:

Pair of

For

Using

Group by

Social Relations Scale for

Wherein the first group of images is defined as a group of images. 2. The apparatus according to claim 1,

Set of all groups that can be created in

Lt;

Lt; / RTI >
here,

The

≪ / RTI > and pairs of faces that can be created and grouped into elements of < RTI ID = 0.0 >

Represents the number of all possible possible groups,

Lt; / RTI >

A binary decision vector that determines whether a particular group is included in the optimal group set

, And a vector having an interaction measure value of the group corresponding to the binary decision vector is defined as

Wherein the group is defined as a group consisting of a plurality of characters. Generating a candidate group set by inferring a relative distance in a three-dimensional space between people using face size information of people in the image;
Calculating a geometric relationship scale by inferring a distance between two faces in an actual three-dimensional space in the image;
Analyzing the social relationship information in consideration of the location information of the people and the face information of the people, and calculating the social relation scale;
Selecting a set of mutually exclusive groups among the groups that can be generated so as to maximize the weighted sum of the values of the geometric relation measure and the social relation measure so as to optimize for the interaction group determination,
In order to generate a candidate group set in the step of generating a candidate group set,

Existing within

A set of face information

≪ / RTI >

Respectively

A face coordinate of the face of the first person, a face coordinate of the face, and a size of the face in the image.

Images