KR20180073851A - Apparatus and method for selecting image based on human affects - Google Patents

Abstract

The present invention relates to an apparatus for selecting an image based on emotion, which can comprise: an emotional feature extracting unit extracting emotional features of each search image corresponding to query; an image cluster forming unit forming a plurality of image clusters based on the emotional features; a ranking calculating unit calculating ranking of the plurality of image clusters; an image extracting unit extracting a representative image among images included in an image cluster belong to predetermined cluster ranking; and an output unit outputting the extracted representative image. Accordingly, images considering user emotion can be more effectively provided.

Description

[0001] APPARATUS AND METHOD FOR SELECTING IMAGE BASED ON HUMAN AFFECTS [0002]

The present invention relates to a sensitivity-based image selection apparatus and method.

Recently, the number of images has increased exponentially due to the rapid growth of the Internet and social networks. Accordingly, there is a demand for a technique that can provide an image related to a search keyword input from a user in a more efficient manner.

The search server for the conventional image search and providing service only searches for images that are directly matched one-dimensionally to the search keywords received from the user terminal, but does not provide any images considering the emotion of the user at all.

For example, if the search keyword is 'deciduous', the conventional search server only provides images containing various 'deciduous leaves' that are directly matched to 'deciduous leaves', for example, There has been a problem in that the conventional technology does not satisfy the needs of such users when it is desired to provide images sharing sensibility in connection with emotional feelings such as 'solitude', 'loneliness' and 'loneliness'.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 10-2007-0009833 (published on January 19, 2007).

SUMMARY OF THE INVENTION It is an object of the present invention to provide an apparatus and method for selecting images based on emotion that can constitute and provide images in consideration of emotion of a user in a more efficient manner.

It is an object of the present invention to provide a sensitivity-based image selection apparatus and method capable of providing a highly satisfactory search result in providing a search image required by a user.

It is to be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to another aspect of the present invention, there is provided a method of selecting an image based on emotion, comprising the steps of: (a) extracting emotional characteristics of each of search images corresponding to a query; (C) calculating a ranking of the plurality of image clusters; (d) extracting a representative image among images included in the image cluster belonging to the predetermined cluster ranking; And (e) outputting the extracted representative image.

The step (a) may further include the steps of: (a1) dividing an image into a plurality of regions based on mean-shift clustering for each of the search images; (a2) Extracting M seed areas from among the plurality of divided areas on the basis of an importance value calculated for each of the M seed areas, Extracting a plurality of color compositions using color information of a plurality of adjacent regions; and (a4) based on probabilistic latent semantic analysis (PLSA) And calculating an emotional vector of each of the search images using the search results.

In addition, the step (b) may form the plurality of image clusters by performing k-means clustering based on the sensed vectors calculated for each of the search images.

In the step (c), a plurality of characteristic values may be calculated for each of the plurality of image clusters, and a ranking of the image cluster may be calculated based on the calculated characteristic value and the weight of each characteristic value.

In addition, the plurality of characteristic values may include a coverage value considering the number of images included in the image cluster, an affective coherence value using the coverage value and the cluster variance, and a distance between the plurality of image clusters And may include a distinctiveness value.

The step (d) may further include a step of calculating, based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster and the number of images within a predefined distance from each image, The image ranking for the images included in the image cluster may be calculated, and the image belonging to the predetermined image ranking may be extracted as the representative image.

In the step (d), a distance value is calculated between images belonging to the predetermined image ranking using L ₂ -norm in extracting the representative image, and an image having a predetermined distance value or less is determined as a duplicate image Can be removed.

Meanwhile, the emotion-based image selection apparatus according to an exemplary embodiment of the present invention includes an emotion feature extraction unit that extracts emotion features of each of search images corresponding to a query, An image extracting unit for extracting a representative image among the images included in the image cluster belonging to the predetermined cluster ranking, and an output unit for outputting the extracted representative image And an output unit.

The emotion feature extracting unit may divide the image into a plurality of regions based on Mean-shift clustering for each of the search images, and calculate an importance value calculated for each of the divided regions extracting M number of seed areas from among the plurality of divided areas based on color information of the M number of seed areas and color information of a plurality of areas adjacent to each of the M number of seed areas, The emotion vectors of each of the retrieved images can be calculated using the plurality of color combinations based on the extraction of color compositions and the probabilistic latent semantic analysis (PLSA).

The image cluster forming unit may form the plurality of image clusters by performing k-means clustering based on the sensed vectors calculated for each of the search images.

The cluster ranking calculation unit may calculate a plurality of characteristic values for each of the plurality of image clusters, and calculate a ranking of the image cluster based on the calculated characteristic value and the weight of each characteristic value.

In addition, the plurality of characteristic values may include a coverage value considering the number of images included in the image cluster, an affective coherence value using the coverage value and the cluster variance, and a distance between the plurality of image clusters And may include a distinctiveness value.

In addition, the image extracting section may calculate, based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster, and the number of images within a predefined distance from each image, And the image belonging to the predetermined image ranking can be extracted as the representative image.

The image extracting unit may calculate a distance value between images belonging to the predetermined image ranking using L ₂ -norm in extracting the representative image, and judge an image having a predetermined distance value or less as a redundant image and remove the image .

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-described task solution of the present invention, a plurality of image clusters are formed based on the calculated emotion characteristic for each of the search images corresponding to the query, a ranking of a plurality of image clusters is calculated, It is possible to construct and provide an image in consideration of the emotion of the user in a more efficient manner by extracting and providing a representative image out of the images to which the user belongs.

The above and other objects, features, and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

However, the effects obtainable here are not limited to the effects as described above, and other effects may exist.

1 is a schematic block diagram of an emotion-based image selection apparatus according to an exemplary embodiment of the present invention.
2 is a diagram illustrating steps performed by the emotion feature extraction unit in the emotion-based image selection apparatus according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram illustrating a color scale based on the PLSA-learning proposed in the emotion-based image selecting apparatus according to the embodiment of the present invention.
4 is a diagram illustrating an image of an image cluster calculated based on Equation (14) in an emotion-based image selecting apparatus according to an embodiment of the present invention.
FIG. 5 is a schematic view illustrating a process of outputting a representative image extracted by the image-based image selection apparatus according to an exemplary embodiment of the present invention.
FIG. 6 is a flowchart illustrating an image-based image selection method according to an exemplary embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when an element is referred to as being "connected" to another element, it is intended to be understood that it is not only "directly connected" but also "electrically connected" or "indirectly connected" "Is included.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as " including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

The present invention relates to a technique for effectively providing an image in which a user's emotion is taken into consideration in providing a search image corresponding to a query input from a user.

1 is a schematic block diagram of an emotion-based image selection apparatus according to an exemplary embodiment of the present invention.

1, an emotion-based image selecting apparatus 100 according to an exemplary embodiment of the present invention includes an emotion feature extracting unit 110, an image cluster forming unit 120, a cluster ranking calculating unit 130, (140) and an output (150).

The emotion-based image selection apparatus 100 may be a user terminal that receives a query from a user, receives an image corresponding to the input query from a search server (e.g., a web server), and outputs the received image . Alternatively, the emotion-based image selection device 100 may be a server that receives a query from a user terminal (not shown), retrieves the image corresponding to the received query, and provides the retrieved image to the user terminal.

The user terminal may be a Personal Communication System (PCS), a Global System for Mobile communication (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), a Personal Digital Assistant (PDA), an International Mobile Telecommunication (IMT) Such as CDMA (Code Division Multiple Access) -2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smart pad, tablet PC, desktop PC, Etc., and may include any type of wire / wireless communication device.

The emotion feature extraction unit 110 may extract the affective features of each of the search images corresponding to the query inputted from the user.

In extracting the emotion feature, the emotion feature extraction unit 110 may express the search images corresponding to the query in an affective space. To this end, the visual features of the image may be expressed in an emotional space . ≪ / RTI >

Here, the emotion feature extraction unit 110 may use an emotional corpus defined by Kobayashi, whereby the emotion-based image selection apparatus 100 according to an exemplary embodiment of the present invention You can automatically select standard images for landscape-related queries and products. Kobayashi in Japan defined an image scale consisting of pairs of adjectives with opposite connec- tions, such as warm-cool, hard-soft, etc. in the "Color Image Scale" published in 1990. In addition, And associated emotional adjectives with a single color. In addition, the color scheme of some colors has been associated with the assumption of color schemes. Accordingly, the emotion feature extraction unit 110 can extract the emotion feature based on the image scale defined by Kobayashi.

The emotion feature extraction unit 110 may define affective classes for emotion feature extraction as follows. The emotion feature extracting unit 110 can tag and rank the one million scenery images in order to examine the emotional words used in the flickr's 1 million landscape images. Based on the result, for example, the emotional feature extraction unit 110 may select 15 affective classes, and match the selected emotional class with the emotional class of Kobayashi. The fifteen emotional classes chosen as an example are pretty, colorful, dynamic, gorgeous, wild, romantic, natural, graceful, It can be defined as graceful, quiet, classic, modern, majestic, pure, cool, dandy. have.

Hereinafter, the process of mapping the search images corresponding to the query to the emotional space by the emotional feature extraction unit 110 will be described in more detail.

Conventionally, the related art has proved that there is a strong relationship between colors and affects. On the basis of this, the emotion feature extraction unit 110 extracts color-based descriptors, color compositions (CCs) can be used. The color combinations (CCs) include both the characteristics of the individual colors present in the image and the configuration for combining these parts in their entirety here.

에 의하여

로 표현되는 감성 벡터로 변형할 수 있다. 여기서, 감성 특징인

는 이미지 (I)가 m번째(m_th) 감성 부류(

)에 속할 확률을 나타낸다. 즉, 이미지 (I)의 감성 벡터인 E(I)는 하기 수학식 1과 같이 표현될 수 있다.The emotion feature extraction unit 110 may perform an image segmentation process, a seed area extraction process, a color combination extraction process, and a PLSA-based emotional mapping process to annotate a search image using emotion of the user. Through this process, the emotion feature extraction unit 110 extracts the image I as the emotion category

By

As shown in FIG. Here,

(I) is the mth (m _th ) emotional class (

). ≪ / RTI > That is, the emotion vector E (I) of the image I can be expressed by the following equation (1).

[Equation 1]

는 감성 부류 각각에 대하여 계산될 수 있다. 이러한 확률을 계산하기 위해, 감성 특징 추출부(110)는 검색 이미지들 각각에 대하여 앞서 말한 이미지 분할 과정, 시드 영역 추출 과정, 컬러 조합 추출 과정 및 PLSA 기반의 감성 맵핑 과정을 수행할 수 있으며, 이는 도 2를 통해 보다 쉽게 이해될 수 있다.Here,

Can be calculated for each of the emotional classes. In order to calculate the probability, the emotion feature extraction unit 110 may perform the image segmentation process, the seed area extraction process, the color combination extraction process, and the PLSA-based emotion mapping process for each of the search images, Can be more easily understood with reference to FIG.

2 is a diagram illustrating steps performed by the emotion feature extraction unit 110 in the emotion-based image selection apparatus 100 according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the emotion feature extraction unit 110 extracts a feature extraction process S21, a seed area extraction process S22, a color combination extraction process S23, and an emotion vector calculation process using a PLSA- S24).

In step S21, the emotion feature extraction unit 110 may divide the image into a plurality of regions based on mean-shift clustering for each of the search images corresponding to the query.

)으로 나누기 위해 평균 이동 클러스터링을 이용할 수 있다. 여기서, 평균 이동 알고리즘은 클러스터의 수를 고려하지 않으며, 또한 형상 분포(shape distribution)를 제약하지 않는다.Specifically, the image may be divided into clusters using similar colors. At this time, the emotion feature extraction unit 110 performs color quantization in advance to convert the RGB color space into 130 basic colors used in Kobayashi's ontology can do. Then, the emotion feature extraction unit 110 extracts each of the search images from the plurality of regions R,

Lt; RTI ID = 0.0 > clustering < / RTI > Here, the average moving algorithm does not consider the number of clusters and does not limit shape distribution.

Next, in step S22, the emotion feature extraction unit 110 extracts M seed regions from a plurality of regions based on the importance value calculated for each of the plurality of regions divided in step S21 Can be extracted.

Specifically, the emotional feature extraction unit 110 may extract all the color combinations (CCs) from the image through a RAG (region adjacency graph) analysis for each of the search images. Here, the color combination (CCs) of one area can be constituted by the color of two areas adjacent to the color of the corresponding area. Thus, if _Rj has three neighboring regions, it may have a configuration of ₃ C ₂ . Then, considering that it is computationally intensive to extract all the color combinations (CCs) from the image, the emotion feature extraction unit 110 extracts color combinations (" CCs) can be extracted. The emotion feature extraction unit 110 may extract the seed area by calculating the importance values of the divided areas using the following equation (2).

&Quot; (2) "

Here, A (R _j ) represents the area (area) of the current area, and G (R _j ) represents the Gaussian distance from the center of the current area to the center of the image. Therefore, it can be said that as the area is large and closer to the center of the image, the importance value of the area becomes larger.

Based on the importance value calculated for each of the plurality of divided regions, the emotion feature extraction unit 110 extracts the upper M regions as the seed region S in the descending order of the importance value among the plurality of regions (i.e., M (I.e., extracting two seed areas S).

Next, in step 23, the emotion feature extraction unit 110 extracts a plurality of color combinations (CCs) using the color information of the M seed areas and the color information of the plurality of areas adjacent to each of the M seed areas .

)를 획득할 수 있다. 여기서, C(·)는 해당 영역과 그의 이웃 영역, 즉

의 세 개의 컬러 조합으로 계산된 맵핑 함수를 나타낸다. 따라서, 수학식 1에서

는 집합 Λ에 따라 하기 수학식 3과 같이 표현될 수 있다.Specifically, the emotion feature extraction unit 110 sums the color combinations (CCs) for the M seed areas,

Can be obtained. Here, C (·) denotes a corresponding region and its neighboring region, that is,

≪ / RTI > is a mapping function calculated with three color combinations of < RTI ID = 0.0 > Therefore, in Equation 1,

Can be expressed by Equation (3) according to the set Λ.

&Quot; (3) "

는 컬러 조합

가 감성 단어인

에 맵핑될 확률을 나타낸다.here,

Color combination

Is the emotional word

Lt; / RTI >

Next, in step S24, the emotion feature extraction unit 110 calculates the emotion vector of each of the search images using a plurality of color combinations based on probabilistic latent semantic analysis (PLSA) .

를 계산하기 위해서는, 먼저 컬러 조합과 감성 부류 간에 상관 관계를 나타낼 수 있는 감성 공간을 구축할 필요가 있다. 이때, 실제 적용에서 제대로 작동하기 위해서는, 감성 공간(affective space)이 다른 시각적 특징과 감성 부류의 통합을 위해 보다 융통성이 있어야할 필요가 있다. 이를 위해, 감성 특징 추출부(110)는, 스파스 특징 공간(sparse feature space)에서 동시 발생 데이터(co-occurrence data)를 사용하는 잠재적 관계 분석을 위한 PLSA를 이용할 수 있다.Specifically,

It is necessary to establish emotional space that can express the correlation between the color combination and the emotional class. At this time, in order to function properly in actual applications, the affective space needs to be more flexible for integrating different visual characteristics and emotional classes. For this, the emotional feature extraction unit 110 may use the PLSA for potential relationship analysis using co-occurrence data in a sparse feature space.

PLSA is an effective analytical technique for modeling documents with a mix of several potential meanings, a technique that is of great interest to the community of text analytics. PLSA introduces a variation on the way concurrent information is captured in a particular home, and if coincidence information is observed, the potential semantic representation can potentially be extracted from the data. Recently, these models have been applied to computer vision, where images serve as documents and pixels or regions serve as words. The emotion feature extraction unit 110 may calculate an emotion vector of an image using the PLSA.

)에서 묘사된 일련의 이미지(즉, 이미지 집합,

)가 주어졌을 때, 감성 부류는 잠재적 토픽(

)에 의해 생성되도록 모델링될 수 있다. 감성 특징 추출부(110)는 이미지 I_i로부터 추출된 컬러 조합(

)을 잠재적 토픽으로서 이용할 수 있다. 이때, 잠재적 토픽의 각각의 속성(

)은 컬러 조합 발생(

)의 정규화된 히스토그램(histogram )일 수 있다.Emotion Class

) (I.e., a set of images,

), The emotional class is a potential topic (

). ≪ / RTI > The emotion feature extraction unit 110 extracts the color combination ( _i . E.

) Can be used as a potential topic. At this time, each property of the potential topic (

) Is the color combination occurrence (

) ≪ / RTI >

Considering observations of co-occurrence patterns that are sub-composed of images and affects, if PLSA is able to model the probability of each coincidence as a mixture of independent polynomial distribution conditions, the process can be expressed as: .

&Quot; (4) "

는 감성 부류 W_j에서 관찰되지 않은 조건에서 특정 컬러 조합(CC)의 분류-조건 확률(class-conditional probability)을 나타낸다. 또한,

는 이미지 I_i에서 컬러 조합

가 얻어지는 확률을 나타낸다. 이와 같은 경우, 감성 특징 추출부(110)는 확률

에 초점을 맞춰, P(I_i)가 균일한 분포로 주어진다고 가정할 수 있다. 단계S24에서는, 확률

를 가장 잘 묘사한 컬러 조합을 찾는 것이 목표이며, 이 과정은 분포

와 이미지 특정 혼합 비율인

에서

의 행렬 분해로 간주될 수 있다. 감성 특징 추출부(110)는 이미지에서 관찰할 수 있는 모드 컬러 조합(CCs)의

리스트를 나열할 수 있으며,

는 본원에서 찾고자 하는 정보, 즉 감성과 컬러 조합 간에 관계를 포함할 수 있다. 이러한 분포는, 하기 수학식 5에 기초하여, EM(Expectation-Maximization) 알고리즘을 사용하여 log-likelihood 함수를 최대화함으로써 명확하게 설명될 수 있다.Where P (I _i ) represents the probability that a particular image will be randomly selected,

Indicates the class-conditional probability of a particular color combination (CC) under conditions not observed in the emotional class W _j . Also,

The color combination in the image I _i

Is obtained. In this case, the emotion feature extraction unit 110 extracts the probability

, It can be assumed that P (I _i ) is given a uniform distribution. In step S24,

The goal is to find the color combination that best describes the process,

And the image specific blending ratio

in

Lt; / RTI > The emotion feature extraction unit 110 extracts a feature of the mode color combination (CCs)

You can list the list,

May include the relationship between emotion and color combinations, i.e., the information sought herein. This distribution can be clarified by maximizing the log-likelihood function using an EM (Expectation-Maximization) algorithm based on Equation (5) below.

&Quot; (5) "

Where N, L, and K represent the number of images, emotional classes, and color combinations, respectively, and n (I _n , w _j ) represents the frequency at which color combinations occur for each image.

에 대한 사후 확률을 계산할 수 있다.In the EM algorithm, E-step is calculated from the previous estimate of the arbitrarily initialized model parameter of the model parameter,

Can be calculated.

&Quot; (6) "

,

및 기대값

를 업데이트할 수 있다. 여기서, 파라미터

는 하기 수학식 7과 같이 표현되고, 파라미터

는 하기 수학식 8과 같이 표현될 수 있다.Then, the maximizing M-step is a parameter

,

And expected value

Can be updated. Here,

Is expressed by the following Equation (7), and the parameter

Can be expressed by the following equation (8).

&Quot; (7) "

&Quot; (8) "

는 이미지 I_q로부터 계산될 수 있다. 이때, 이전에 훈련된

와 함께 이 새로운 이미지가 주어질 경우, j번째 감성 부류의 사후 확률은 하기 수학식 9로부터 추론될 수 있다.When the image I _q is given,

Can be calculated from the image I _q . At this time,

, The posterior probability of the jth emotional class can be deduced from the following equation (9).

&Quot; (9) "

FIG. 3 is a diagram illustrating a color scale based on the PLSA-learning proposed in the emotion-based image selection apparatus 100 according to an embodiment of the present invention.

를 나타낸다.Referring to Fig. 3, FIG. 3 shows a scale between 1170 color combinations defined by the emotional class and Kobayashi, where all visual characteristics are colored according to their importance corresponding to the emotional class. This gray level can be divided into ten levels depending on the influence of the color combination associated with the emotional class. If the color is close to white, this may mean that the color combination has a strong relationship with the emotional class. In the present application, this scale (scale) can be constructed by using the proposed PLSA learning. Thus, each gray level has a distribution of all combinations of emotional classes and color combinations

.

In general, it is natural that some color combinations may be more related to some emotional classes, and other color combinations may be associated with only one emotional class.

The correlation of Kobayashi is derived from the knowledge that man has devised through a long investigation, where they limited a color combination to a single emotion class for convenience. In contrast, the PLSA learning-based color scale proposed herein suggests a more diverse relationship between color combinations and emotional classes, which indicates that all color combinations are associated with one or more emotional classes that have different influences . Thus, the PLSA model is a mixture of independent polynomial distribution conditions, modeling the probability of each coincidence of color combinations and emotions, which can take into account the potential implications between emotional classes and color combinations more clearly. Accordingly, the emotion feature extraction unit 110 can calculate emotion vectors for each of the search images through the PLSA-based emotion mapping.

The image cluster forming unit 120 may form a plurality of image clusters based on the emotion feature extracted by the emotion feature extracting unit 110. [

Hereinafter, a process of clustering in an emotional space to select a standard image (a representative image for a query, an image to be visualized) by the image cluster forming unit 120 will be described first. A new criterion for selecting a cluster having < RTI ID = 0.0 > a < / RTI > Based on the thus-defined new reference characteristic (i.e., a plurality of characteristic values to be described later), the cluster ranking calculation unit 130, which will be described later, can calculate the ranking of a plurality of image clusters.

The image cluster forming unit 120 may use the emotion feature to perform clustering in the emotional space.

Conventionally, various algorithms such as k-means clustering and mean-shift clustering have been considered for clustering, and the sensitivity-based image selection apparatus 100 according to an embodiment of the present invention has been proposed, May use k-means clustering in forming image clusters. Accordingly, the image cluster forming unit 120 can form a plurality of image clusters by performing k-means clustering based on the sensed vectors calculated for each of the search images.

The cluster ranking calculation unit 130 may calculate a ranking of a plurality of image clusters formed by the image cluster formation unit 120. [ The cluster ranking calculation unit 130 may calculate a plurality of characteristic values for each of the plurality of image clusters, and calculate a ranking of the image cluster based on the calculated characteristic value and the weight of each characteristic value. Here, the plurality of characteristic values include a coverage value in consideration of the number of images included in the image cluster, an affinity coherence value using the coverage value and the cluster variance, and a distinction considering the distance between the plurality of image clusters ( lt; / RTI > value.

Specifically, the cluster ranking calculation unit 130 calculates how much the clusters are related to the search query, how various they are, when the cluster (c _k? C) is given, Can be evaluated. The standard image may be selected in an image cluster belonging to an upper ranking among a plurality of image clusters by an image extracting unit 140 to be described later. This is because the image cluster belonging to the upper ranking may contain more representative images. On the other hand, there may be fewer image clusters belonging to the lower ranking.

The reference characteristics to be considered when the cluster ranking calculation unit 130 calculates the ranking of the image cluster will be described below. For effective standard image selection, the image cluster must satisfy the conditions for the desired property values, which include the coverage value, the affective coherence value, and the distinctiveness value .

In the case of coverage, the com- mnon concepts present in the scene can be expressed in the amount they deal with. Accordingly, the cluster considered in calculating the cluster rankings may include cases in which a large number of images are handled. In the case of emotion consistency, images within a cluster can share a common feeling. Accordingly, images in the cluster must have cohesion in terms of emotion. In the case of distinction, the diversity of the upper image clusters may be a measure of non-redundancy. Accordingly, similar clusters may not be included in the parent image cluster.

The three attributes are as follows.

First, the coverage value for the image cluster can be expressed as: < EMI ID = 10.0 >

&Quot; (10) "

In consideration of the coverage value, the cluster ranking calculation section 130 can select an image cluster having a large number of images among a plurality of image clusters, for ranking calculation of the image cluster. In Equation (10), num () represents the number of images belonging to the k-th cluster.

According to Equation (10), the larger the number of images included in the image cluster, the larger the coverage value of the image cluster. However, as mentioned above, it can be assumed here that the images in the cluster share a common sensibility. In particular, images with some noise that are not related to the query can affect cluster size.

)을 사용할 수 있으며, 이는 커버리지 값이 높더라도, 너무 큰 분산을 갖는 이미지 클러스터에 페널티를 부과하는데 이용될 수 있다. 그러나, 더 큰 클러스터는(즉, 커버리지가 상대적으로 큰 이미지 클러스터) 더 작은 클러스터(즉, 커버리지가 상대적으로 작은 이미지 클러스터) 보다 감성 공간에서 더 드문드문 분포되어 있을 가능성이 높기 때문에, 커버리지와 감성 일관성 간에는 상호 절충(trade-off)적이다. 즉, 커버리지가 상대적으로 큰 이미지 클러스터는 커버리지가 상대적으로 작은 이미지 클러스터 보다 감성 일관성 속성 값이 낮을 수 있다. 그러므로, 이미지 클러스터에 대한 감성 일관성 값은 하기 수학식 11과 같이 표현될 수 있다.In order to solve this problem, in ranking calculation of image clusters, affective coherence based on their distribution within an image cluster can be considered. Here, the cluster ranking calculation unit 130 calculates the cluster variance

), Which can be used to penalize image clusters with too large variance, even if the coverage value is high. However, because larger clusters are more likely to be distributed sparsely in emotional space than smaller clusters (i.e., image clusters with relatively small coverage), the coverage and emotional consistency There is a trade-off between the two. That is, an image cluster having a relatively large coverage may have a lower value of the emotional consistency property than an image cluster having a relatively small coverage. Therefore, the emotion consistency value for the image cluster can be expressed by Equation (11).

&Quot; (11) "

는 본원의 일 실시예에 따른 실험에 따라, 일예로 0.5로 결정될 수 있다.In this case, the emotion consistency can be adjusted nonlinearly according to the size of the image cluster, and in Expression (11)

May be determined to be 0.5, for example, according to an experiment according to one embodiment of the present application.

Next, the upper image clusters should avoid repetition and provide redundant information. To achieve this goal, a plurality of image clusters of a higher ranking need to be distinguished from each other. Accordingly, the cluster rank calculating unit 130 may use the distinction value of the image cluster at the time of ranking calculation of the image cluster, and the distinction value may be expressed by the following equation (12).

&Quot; (12) "

The discrimination value for one image cluster can be modeled considering the distance between the plurality of image clusters. That is, the discrimination value may be modeled as a set of mutual distances of the cluster pairs, wherein, when calculating the discrimination value of the image cluster, the minimum pair distance value of the image cluster may be used.

At this time, the cluster ranking calculation unit 130 can measure the distance between two image clusters using the distance between the centers of the image clusters. As this value is larger, it can be assumed that the two image clusters are dispersed to each other. However, this assumption can only be valid if the image cluster has a dense distribution. If the image clusters are rarely distributed, the image clusters may overlap each other even if the distance between the centers of the image clusters is large. In consideration of this problem, the cluster ranking calculation unit 130 may calculate the rank of the image cluster considering the distances between the image clusters as well as the distances between the center of the image clusters. Thus, the distance between the image cluster and the image cluster, i. E., The distance of the image cluster pair, can be given as the cluster center distance as it scales using the spread of the image cluster, i.e., the standard deviation of c _k and c _j .

The cluster ranking calculation unit 130 calculates a characteristic value for each of the plurality of image clusters through Equations 10 to 12, and thereafter, calculates a cluster rank of the image cluster based on the calculated characteristic value and the weight of each characteristic value Can be calculated. The cluster ranking calculation unit 130 calculates the ranking of the image clusters so that a plurality of image clusters are divided into image clusters belonging to a predetermined cluster ranking (i.e., an upper image cluster) and image clusters not belonging to a predetermined cluster ranking Image cluster).

More specifically, ideally, the standard image should represent the query and need to show various aspects. Accordingly, the cluster ranking calculation unit 130 may consider a cluster that optimizes the proposed characteristic values (the coverage value, the emotion consistency value, and the discrimination value) at the time of ranking calculation of the image cluster.

First, since it is apparent that a cluster larger than a small cluster is more representative, the cluster ranking calculation section 130 can calculate the rank of image clusters using the coverage value. However, due to the rarity of emotional space, there may be some agreement between images belonging to the same image cluster. Accordingly, the cluster ranking calculation unit 130 may consider the coverage value and the emotion consistency value as shown in Equation (13) when calculating the rank of the image cluster.

&Quot; (13) "

및

는 이미지 클러스터 c_k의 대표성에 대한 커버리지 값과 감성 일관성 값의 기여도를 조절할 수 있는 가중치로 정의될 수 있다.here,

And

Can be defined as a weight that can control the coverage value and the contribution of the emotion consistency value to the representativeness of the image cluster c _k .

Next, in ranking calculation of the image cluster, the selected image cluster must not overlap with the other image cluster in order to obtain various sets of representative images (i.e., image to be visualized, image to be provided corresponding to the query). Accordingly, the cluster ranking calculation unit 130 may define a ranking model for rank calculation of a plurality of image clusters as shown in Equation (14) by combining Equations (12) and (13).

&Quot; (14) "

는 이미지 클러스터 c_k의 대표성에 대한 구별도 값의 기여도를 조절할 수 있는 가중치로 정의될 수 있다.here,

Can be defined as a weight that can control the contribution of the discrimination value to the representativeness of the image cluster c _k .

On the other hand, scaling is necessary when comparing the property values in the image cluster and the property values between the image clusters. Hence, according to Equation (14), as the distinction value is expanded to a smaller range, the distinction value can be multiplied by 10 ³ . The cluster ranking calculation unit 130 may calculate the rankings of the plurality of image clusters based on Equation (14).

In this regard, FIG. 4 is a diagram showing an image of an image cluster calculated based on Equation (14) in the emotion-based image selecting apparatus 100 according to an embodiment of the present invention.

Specifically, FIG. 4 shows a query-related image for a coast, a rest and a mountain, and in particular, FIGS. 4 (a) to 4 (c) FIG. 4 (d) to FIG. 4 (f) show images included in the lower image cluster belonging to the lower ranking.

Referring to FIG. 4, it can be seen that the images contained in the upper image cluster are artfully stronger, and the images contained in the lower image cluster are not artificially strong despite being relevant to the query. Further, it can be seen that an image which can be referred to as noise is included in Fig. 4 (f).

4, it can be seen that the image clusters belonging to the upper ranking among the ranking of the image clusters calculated on the basis of the expression (14) are higher in the probability of providing images of higher satisfaction related to the query than the image clusters belonging to the lower ranking, Accordingly, the image extracting unit 140, which will be described later, can extract the representative image based on the upper image cluster belonging to the upper ranking.

On the other hand, the cluster ranking calculation unit 130 may perform bi-directional correlation analysis between parameters in order to investigate whether there are additional parameters. As a result of the correlation analysis, it was found that the cohesion of large image clusters is less in their image clusters and is less distinguishable from other image clusters. However, it can be seen that clusters with large emotional cohesion are largely distinguished between them. As a result, there are no significant differences between the types of scenes in the image, which is the ranking model proposed herein, which can be used to rank a plurality of image clusters, regardless of the type of scene in the image It can be said that it can be used as a general model.

The image extracting unit 140 may extract a representative image among the images included in the image cluster belonging to the predetermined cluster ranking. Here, the image cluster belonging to the predetermined cluster ranking may mean the above-mentioned upper image cluster.

Based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster, and the number of images within a predefined distance from each image, the image extracting unit 140 extracts, The image ranking for the images can be calculated, and then the image belonging to the predetermined image ranking can be extracted as the representative image.

Specifically, in extracting the representative image, the image extracting unit 140 may measure representative values (i, c _k ) defining how well the image i represents the image cluster c _k . At this time, the image extracting unit 140 may consider two distance measurement values, and each of the two distance measurement values may include a distance from the center of each image to the center of the cluster and an image selected within a predefined distance (diameter, diameter) Lt; RTI ID = 0.0 > a < / RTI >

The image extracting unit 140 may calculate the ranking for the images included in each image cluster and may calculate the ranking using the Euclidian distance value from the center of each cluster to the corresponding image .

)와 동일한 이미지 클러스터에 속하는 i와 o 사이의 거리 d(i, 0)으로 정의될 수 있다. 또한, 이미지 i의 k-거리 이웃인 N_{k-distance(i)}는 이미지 i로부터의 거리가 K-거리보다 크지 않은 모든 이미지를 포함하는 수를 나타내며, 이 값이 클수록 이미지 i가 더 대표적임을 나타낼 수 있다. 이때, N_{k-distance(i)}는 주어진 이미지에서 인접한 이미지로부터의 거리에 따라 비선형적으로 패널티가 적용될 수 있다. 따라서, 이미지 추출부(140)는 하기 수학식 15에 기초하여 이미지 클러스터 내에 포함된 이미지들의 랭킹(순위)을 산출할 수 있다.Then, the image extracting unit 140 may calculate the number of adjacent images within a predefined distance, and for this, the k-distance (i) may be used. Given an arbitrary positive integer k, the k-distance (i) is i (

) And the distance d (i, 0) between i and o belonging to the same image cluster. Also, the k-distance neighbor of the image i, Nk _{-distance (i)} , represents the number of all images that are not greater than the K-distance from the image i, and the larger this value, the more representative the image i . At this time, N _{k -distance (i)} can be nonlinearly penalized according to the distance from the adjacent image in a given image. Accordingly, the image extracting unit 140 can calculate the ranking (rank) of the images included in the image cluster based on the following expression (15).

&Quot; (15) "

는 각 클러스터의 중심으로부터 해당 이미지까지의 유클리드 거리를 나타낸다. 이와 같은 수학식 15에 의하면, 유클리드 거리가 작고 N_{k-distance(i)}의 값이 클수록 이미지 클러스터 내에 포함된 이미지들의 랭킹 순위는 높게 산출될 수 있다.here,

Represents the Euclidean distance from the center of each cluster to the image. According to Equation (15), as the Euclidean distance is small and the value of Nk _{-distance (i)} is large, the rank ranking of the images included in the image cluster can be calculated to be high.

When the ranking of the images included in the predetermined image cluster is calculated through Equation (15), the image extracting unit 140 calculates a list R of representative images belonging to a high ranking (that is, a list of images to be visualized) can do. At this time, the number of images belonging to the list R, that is, the number of images selected in each of the upper image clusters may be proportional to the rank ranking of the image clusters. In other words, the lower ranked image cluster (i.e., lower image cluster) is discarded, and in the higher ranked image cluster (i.e., the upper image cluster), the higher the rank ranking of the image cluster according to their ranking ranking, Can be selected.

On the other hand, when extracting the representative image, the image extracting unit 140 calculates a distance value between images belonging to a predetermined image ranking using L ₂ -norm, and judges an image having a predetermined distance value or less as a redundant image to be removed .

Specifically, image clusters of limited size should avoid duplicate information while avoiding repetition. Accordingly, in order to measure the diversity of one image in comparison with other images, the image extracting unit 140 extracts the minimum value of the distance between the image and the image (in other words, the distance between the pairs of images The minimum value of < / RTI >

&Quot; (16) "

At this time, the image extracting unit 140 may use L ₂ -norm to measure the distance between two images. At this time, the distance between the two images can be given by the dot product of the two image vectors.

The output unit 150 may output (or visualize) the representative image extracted by the image extraction unit 140. [ Here, the visualization may mean displaying the selected or extracted representative image on the screen so that the user can view it.

Meanwhile, the process of extracting the representative image based on the extracted emotion characteristic for each of the search images corresponding to the query can be more easily understood from FIG.

FIG. 5 is a diagram schematically illustrating a process of outputting a representative image extracted by the image-based image selecting apparatus 100 according to an exemplary embodiment of the present invention.

Referring to FIG. 5, the emotion feature extraction unit 110 extracts emotion features through an image segmentation process S21, a seed area extraction process S22, a color combination extraction process S23, and a PLSA- Through the vector calculation process (S24), the emotion vector E (I) can be calculated as the emotion characteristic of each of the images corresponding to the query.

Then, the image cluster forming unit 120 can form a plurality of image clusters by performing k-means clustering based on the emotion characteristic (emotion vector)

Thereafter, the cluster ranking calculation unit 130 calculates a characteristic value (i.e., a coverage value, an emotion consistency value, a discrimination value) for each of the plurality of image clusters, c ₁ , c ₂ , c ₃ , c ₄ , ..., c _n ) can be calculated. At this time, the cluster ranking calculation unit 130 may define c ₁ to c ₃ among the plurality of image clusters as an upper image cluster and c ₄ to c _n as lower image clusters based on the calculated ranking.

Thereafter, the image extracting unit 140 extracts representative images among the images included in the image clusters belonging to the predetermined cluster ranking (i.e., c ₁ to c ₃ as the upper image clusters) based on the ranking of the calculated image clusters Can be selected or extracted. Here, for example, a c ₁ image cluster includes I ₁₁ to I ₁₄ images, a c ₂ image cluster includes I ₂₁ to I ₂₄ images, and a c ₃ image cluster may include I ₃₁ to I ₃₄ images have.

The image extracting unit 140 may extract a representative image in each of the upper image clusters. In this case, the number of images extracted from each of the image clusters may be determined in proportion to the ranking ranking of each image cluster. As an example, the image extraction unit 140 as a representative image, c ₁ in the image clusters of three images (i. E., I _11, I _12, I _13), and, c ₂ images in the cluster, the two images to extract (I _21, I ₂₂ ), and one image (I ₃₁ ) can be extracted in the c ₃ image cluster.

In addition, the image extracting unit 140 may remove duplicate images from the images extracted from each image cluster.

The output unit 150 may output the representative images extracted from the image extracting unit 140 (i.e., the images from which the redundant images have been removed) to a display screen or the like, And provide the extracted representative images to the user terminal.

 Hereinafter, the operation flow of the present invention will be briefly described based on the details described above.

FIG. 6 is a flowchart illustrating an image-based image selection method according to an exemplary embodiment of the present invention.

The sensibility-based image selection method shown in FIG. 6 can be performed by the sensitivity-based image selection apparatus 100 described above. Therefore, even if the contents are omitted in the following description, the description of the image selection apparatus 100 based on emotion can be similarly applied to FIG.

Referring to FIG. 6, in step S61, the emotion characteristic of each of the search images corresponding to the query can be extracted.

Also, in step S61, for each of the retrieved images, the image may be divided into a plurality of areas based on the average moving clustering. Further, M seed areas can be extracted from among the plurality of divided areas based on the importance value calculated for each of the plurality of divided areas. In addition, a plurality of color combinations can be extracted by using the color information of the M seed areas and the color information of the plurality of areas adjacent to each of the M seed areas. Further, based on probability-based potential semantic analysis (PLSA), emotion vectors of each of the retrieved images can be calculated using a plurality of color combinations.

Next, in step S62, a plurality of image clusters can be formed based on the emotion characteristic extracted in step S61.

Further, in step S62, a plurality of image clusters can be formed by performing k-means clustering based on the emotion vector calculated for each of the search images.

Next, in step S63, ranking of a plurality of image clusters formed in step S62 can be calculated.

Further, in step S63, a plurality of characteristic values may be calculated for each of the plurality of image clusters, and the rank of the image cluster may be calculated based on the calculated characteristic value and the weight of each characteristic value. Here, the plurality of characteristic values may include a discrimination value in consideration of the coverage value considering the number of images included in the image cluster, the sensitivity value using the coverage value and the cluster variance, and the distance between the plurality of image clusters.

Next, in step S64, a representative image among the images included in the image cluster belonging to the predetermined cluster ranking can be extracted.

Further, in step S64, based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster, and the number of images within a predefined distance from each image, , And extracts an image belonging to the predetermined image ranking as a representative image.

In step S64, a distance value may be calculated between images belonging to the predetermined image ranking using L ₂ -norm when extracting the representative image, and an image having a predetermined distance value or less may be determined as a redundant image and removed.

Next, in step S65, the representative image extracted in step S64 may be output (or visualized), and the output image may be an image from which the redundant image has been removed.

In the above description, steps S61 to S65 may be further divided into further steps, or combined in fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The sensibility-based image selection method according to one embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Further, the emotion-based image selection method described above can also be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: Sensibility based image selection device
110: Emotion characteristic extracting unit
120: image cluster forming section
130: Cluster Ranking Calculation Unit
140: image extracting unit
150:

Claims (15)

(a) extracting emotional characteristics of each of the search images corresponding to the query;
(b) forming a plurality of image clusters based on the emotional characteristic;
(c) calculating a ranking of the plurality of image clusters;
(d) extracting representative images among the images included in the image cluster belonging to the predetermined cluster ranking; And
(e) outputting the extracted representative image,
Based on the sensed images. The method according to claim 1,
The step (a)
(a1) for each of the search images, dividing the image into a plurality of regions based on mean-shift clustering;
(a2) extracting M seed areas of the plurality of divided areas based on the importance value calculated for each of the plurality of divided areas;
(a3) extracting a plurality of color compositions using color information of the M seed areas and color information of a plurality of areas adjacent to each of the M seed areas; And
(a4) calculating an emotional vector of each of the search images using the plurality of color combinations, based on probabilistic latent semantic analysis (PLSA); and
Based on the sensed images. 3. The method of claim 2,
The step (b)
Wherein the plurality of image clusters are formed by performing k-means clustering based on the emotion vectors calculated for each of the search images. The method of claim 3,
The step (c)
Calculating a plurality of feature values for each of the plurality of image clusters and calculating a ranking of the image clusters based on the calculated feature value and the weight of each feature value. 5. The method of claim 4,
The plurality of characteristic values may include:
A coverage value considering the number of images included in the image cluster, an affective coherence value using the coverage value and the cluster variance, and a distinctiveness value considering a distance between the plurality of image clusters Based image selection method. The method according to claim 1,
The step (d)
Based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster and the number of images within a predefined distance from each image, Calculating an image ranking, and extracting an image belonging to a predetermined image ranking as the representative image. The method according to claim 6,
The step (d)
Calculating a distance value between images belonging to the predetermined image ranking using L ₂ -norm in extracting the representative image, and judging an image having a predetermined distance value or less as a redundant image and removing the image, How to choose. An emotion feature extraction unit for extracting emotion features of each of the search images corresponding to the query;
An image cluster forming unit forming a plurality of image clusters based on the emotional characteristic;
A cluster ranking calculation unit for calculating a ranking of the plurality of image clusters;
An image extracting unit for extracting a representative image among the images included in the image cluster belonging to the predetermined cluster ranking; And
An output unit for outputting the extracted representative image,
Based on the sensed image. 9. The method of claim 8,
Wherein the emotional feature extraction unit comprises:
For each of the search images, dividing the image into a plurality of regions based on mean-shift clustering,
Extracting M seed areas out of the plurality of divided areas based on the importance value calculated for each of the plurality of divided areas,
Extracting a plurality of color compositions using color information of the M seed areas and color information of a plurality of areas adjacent to each of the M seed areas,
Based on the probabilistic latent semantic analysis (PLSA), calculates an emotional vector of each of the retrieval images using the plurality of color combinations. 10. The method of claim 9,
Wherein the image cluster forming unit comprises:
Wherein the plurality of image clusters are formed by performing k-means clustering based on the sensed vectors calculated for each of the search images. 11. The method of claim 10,
The cluster ranking calculation unit calculates,
Calculate a plurality of characteristic values for each of the plurality of image clusters, and calculate a rank of the image cluster based on the calculated characteristic value and the weight of each characteristic value. 12. The method of claim 11,
The plurality of characteristic values may include:
A coverage value considering the number of images included in the image cluster, an affective coherence value using the coverage value and the cluster variance, and a distinctiveness value considering a distance between the plurality of image clusters Based image selection device. 9. The method of claim 8,
Wherein the image extracting unit comprises:
Based on the Euclidean distance value between each image included in the image cluster belonging to the predetermined cluster ranking and the center of the image cluster and the number of images within a predefined distance from each image, The image ranking is calculated, and an image belonging to a predetermined image ranking is extracted as the representative image. 14. The method of claim 13,
Wherein the image extracting unit comprises:
Calculating a distance value between images belonging to the predetermined image ranking using L ₂ -norm in extracting the representative image, and judging an image having a predetermined distance value or less as a redundant image and removing the image, Selection device. A computer-readable recording medium on which a program for executing the method of any one of claims 1 to 7 is recorded.

Images