KR20060083138A - Method and apparatus for category-based photo clustering using photographic region templates of digital photo - Google Patents

Abstract

The present invention relates to a clustering method and apparatus using a region partition template, the method comprising: segmenting a photo into regions using a region partition template; Modeling a semantic concept included in the divided region; Merging the semantic concepts of each region according to the confidence of the local meaning measured from the modeling when the image of the region includes a corresponding semantic concept and represents a degree of reliability; Modeling the full meaning concept included in the photograph by using the final local meaning concept determined by merging; And determining one or more categories included in the input photograph according to the reliability of the overall semantic concept measured from modeling.

According to the present invention, in order to reliably extract the semantic concept included in a photograph, a multi-content-based feature value extracted from a segmented region image is extracted using a region segmentation template, and the region defined using the region is extracted. Measure the reliability of the input image with respect to the semantic concept, and extract the more reliable local meaning concept by merging the regional meaning concept of the picture according to the reliability, and measure the reliability of the overall meaning concept using the merged local meaning concept Therefore, by extracting the concept of multiple categories included in the input photo accordingly, it is possible to quickly and effectively album the photo data.

Description

Method and apparatus for category-based photo clustering using photographic region templates of digital photo}

1 is a block diagram illustrating a configuration of a photo clustering system using a geographic division template according to an embodiment of the present invention.

2 is a flowchart illustrating a photo clustering method using a geographic division template according to the present invention.

3 illustrates a regional partition template.

4 shows an example of photo segmentation by a regional partition template.

5 illustrates an embodiment of an entity concept and an attribute concept of a divided region.

Figure 6 illustrates the regional concept modeling in more detail.

FIG. 7 illustrates a process of grouping conceptual merged target areas performed by a regional conceptual merger.

8 is a block diagram illustrating a category-based clustering process of a digital photo album according to the present invention.

The present invention relates to a digital photo album, and more particularly, to a method and apparatus for category-based photo clustering using a region segmentation template.

A general digital photo album is for transferring a picture taken by a user from a digital camera or a memory card to a user's local storage device and managing the picture on a computer. In general, users want to use a photo album to index several photos in chronological or randomly created photo categories, browse accordingly, or share the pictures with other users.

Among them, the ability to automatically cluster photos into categories that contain them is one of the main functions of the photo album. Categorization reduces the scope of the search in the process of searching for a picture desired by the user, thereby improving the accuracy of the search as well as the speed of the search. In addition, by automatically classifying photos into categories that users want, it makes it easy for users to manage large amounts of photos into albums.

The existing categorization methods are mostly text-based methods, and photos are categorized by using metadata described by text. However, this method is not useful in the case that a large amount of pictures is difficult for a person to describe the category information of the pictures one by one, and that the text information is not effective in describing the meaning concept of the pictures. Therefore, a method of categorizing multimedia content using content-based features such as color, shape, texture, etc. extracted based on the content of a photo may be proposed.

To date, many studies have been performed for clustering by category using content-based feature values of photographic images. However, since photography includes various semantic concepts, automatic extraction of multiple semantic concepts is still a very difficult research field. As a means to solve this problem, studies have been conducted to extract main objects from photographs (images) and to index or categorize photographs according to the semantics of the objects. Since it is very difficult to extract the various semantic concepts included in the photograph, it is a method of extracting only the main semantic concepts included in the photograph.

Among them, Jiebo Luo's Method for Automatic Determination of Main Subjects in Photographic Images (Eastman Kodak Company) extracts the main subjects from the semantic objects included in the photographs and identifies and indexes them. Were mainly performed. That is, in categorizing a photo, a study of segmenting an object included in the photo and a study of indexing or categorizing the separated object have been conducted.

However, as described above, since most photographic images include many semantic concepts, categorization by the main object extraction method results in missing other semantic concepts included in the photograph.

In general, a picture is divided into a foreground and a background. In categorizing photographic data, the semantic concept of the foreground is important, but the semantic concept of the background is also important. Therefore, as a method of categorizing photo data, there is a need for a method of extracting various semantic concepts included in a photo by simultaneously considering the concept of foreground and background, rather than separating objects.

As a method of extracting various semantic concepts from a photo, there is a method of extracting a semantic concept for each region by dividing an image into small regions. Dividing an image into small pieces makes it easier to extract a single semantic concept. However, if the size of the divided image is too small, even a single semantic concept may be difficult to extract. In other words, it is not easy to determine the size to divide the image. Therefore, an effective image segmentation method for extracting various semantic concepts of a photo and a method of extracting an accurate semantic concept from the segmented image are essential for category concept detection using localization of the image.

The technical problem to be achieved by the present invention is to provide a category-based clustering method and system for digital photo albums to automatically classify photos into various categories by extracting various semantic concepts included in the photos based on the content-based feature values of the photos. As the image data is effectively divided into regions, the semantic concepts of the divided regions are extracted, and the semantic concepts included in the photos are categorized through efficient merging of the semantic concepts of the regions to find the meaning of the entire photograph. It is to provide a method and apparatus for category-based photo clustering using a region segmentation template.

According to an aspect of the present invention, there is provided a category-based clustering method of a digital photo album, comprising: (a) dividing a photo into regions using a region division template; (b) modeling a semantic concept included in the divided region; (c) merging the semantic concepts of each region according to the confidence of the local meaning measured from the modeling when the image of the region includes a corresponding semantic concept and a measure indicating the degree of reliability; (d) modeling the full meaning concept included in the photograph by using the final local meaning concept determined by merging; And (e) determining one or more categories included in the input photo according to the reliability of the overall semantic concept measured from modeling.

The regional partition template of step (a)

[Equation 3]

(Where T is the template of the photo, w is the width of the photo, and h is the height of the photo).

It is preferable that the area is represented by the equation (3).

In the step (b), it is preferable to model the semantic concept by extracting the content-based feature value of the photograph. Preferably, the content-based feature value includes color, texture, and shape information of an image. The semantic concept of step (b) is composed of an item ( L _entity ) indicating an entity of a semantic concept included in a photograph and an item ( L _attribute ) indicating an attribute of a body of a semantic concept. It is desirable to. The semantic concept modeling preferably models a concept of an entity and an attribute of a divided region.

In the semantic concept modeling of step (b), it is preferable to model a local concept of a region-divided input photograph by using a support vector machine (SVM). In the reliability measurement of step (c), it is preferable to measure the reliability of each local semantic concept by using one SVM for each defined local semantic concept.

Merging of the semantic concepts of the regions of step (c)

[Equation 14]

(Where T (1), T (2), T (3), T (4), and T (5) represent the base region to which the final meaning of the region semantics is assigned, and C _L 'represents the confidence vector of the region. .)

It is preferable to merge the regional concept reliability of the five basic regions by Equation 14 based on the reliability of the regional concept allocated to the ten regions divided by the region partition template.

The reliability C ' _local of the _local concept obtained after the merging is

[Equation 15]

(Where C _local (T) represents a vector of confidence sets for the semantic concept L _local merged in the partition region T).

It is preferably expressed as shown in Equation 15. In the step (d), it is preferable to model the entire concept of the regionally divided input picture I by using a support vector machine (SVM). In the clustering method of a digital photo album using a regional partition template according to the present invention, the reliability of the entire concept is measured using the reliability of the local concept as an input.

In the step (e), it is preferable to determine the overall meaning concept having the largest value among the reliability of the overall meaning concept measured from the modeled overall meaning concept as the category of the photograph. In the step (e), it is preferable to determine the overall meaning concepts having a reliability greater than a predetermined threshold value among the reliability of the overall meaning concepts measured from the modeled overall meaning concepts as the category of the picture.

According to an aspect of the present invention, there is provided a category-based clustering device for a digital photo album according to the present invention. An area conceptual modeling unit for modeling a semantic concept included in the divided area; When the image of the area includes a corresponding meaning concept and a measure indicating the degree of reliability, the local concept merging unit for merging the semantic concept of each area according to the confidence of the local meaning measured from the modeling; An overall meaning concept modeling unit modeling an overall meaning concept included in a photograph by using the final local meaning concept determined by merging; And a category determination unit that determines one or more categories included in the input picture according to the reliability of the overall semantic concept measured from modeling.

The category-based clustering apparatus of the digital photo album according to the present invention preferably further includes a photo input unit for receiving photo data for category-based clustering.

The local meaning modeling unit extracts a content-based feature value of a photo to model a semantic concept, and the content-based feature value includes color, texture, and shape information of an image. The local sense concept is preferred to contain the columns (L entity) and the entry (L attribute) indicating the attribute (attribute) of a body of a sense concept displaying unit (entity) of the sense concept, which includes the picture .

In the semantic conceptual modeling of the local conceptual modeling unit, it is preferable to model a regional concept of a region-divided input photograph by using a support vector machine (SVM). In the reliability measurement of the regional meaning concept merging unit, it is preferable to measure the reliability of each local semantic concept by using one SVM for each defined local semantic concept.

A computer readable recording medium having recorded thereon a program for executing the invention described above is provided.

Hereinafter, with reference to the accompanying drawings will be described in detail a method and system for clustering photos using a regional segmentation template according to the present invention.

1 is a block diagram illustrating a configuration of a photo clustering system using a region segmentation template according to an embodiment of the present invention. The region segmentation unit 110, the region conceptual modeling unit 120, and the region conceptual concept merging unit ( 130, the overall meaning modeling unit 140 and the category determination unit 150. The photo clustering system may further include a photo input unit 100.

The photo input unit 100 receives photo data for category-based clustering as an input. For example, a picture string is input from an internal memory device or a portable memory device of a digital camera. Photo data is based on general still image data, and the format of the photo data includes any image data format such as JPEG, TIFF, or RAW. The format of the picture data is not limited to the example described above.

The region dividing unit 110 divides a photo into regions using a region dividing template.

The local conceptual modeling unit 120 models a semantic concept included in the divided region and uses a local concept SVM (support vector machine) 160.

When the image of the region includes the meaning of the measured image representing the degree and reliability of the image of the region, the meaning concept of each region according to the confidence (confidence) of the region meaning measured from the modeling Merge

The overall meaning concept modeling unit 140 models the overall meaning concept included in the photograph by using the final local meaning concept determined and merged, and uses the global concept SVM (support vector machine) 170.

The category determiner 150 determines one or more categories included in the input photo according to the reliability of the overall semantic concept measured from the overall meaning conceptual modeling.

FIG. 2 is a flowchart illustrating a photo clustering method using a region segmentation template according to the present invention. Referring to FIGS. 1 and 2, the photo clustering method using the region segmentation template and the operation of the system will be described. Shall be.

The image string is received from the internal memory device or the portable memory device of the digital camera (step 200). The input picture is first divided using the region division template of the picture (step 210). 3 illustrates a region segmentation template of a photo according to an embodiment of the present invention. The present invention includes dividing a photo into ten base templates, as shown in FIG. 3. Ten regional partition based templates are represented by the following equation.

Here, T (t) represents the t-th regional partition template.

If the input photo I has a size of horizontal w and vertical h, the coordinates of each region division template are expressed by the following equation.

Here, left (t) represents the x-axis coordinate of the left side of the t-th template, top (t) represents the y-axis coordinate of the upper left of the t-th template, and right (t) represents the x-axis of the right side of the t-th template Bottom (t) represents the y-axis coordinate of the bottom right of the t-th template. According to Equation 2, the coordinates of each template are as follows.

The input picture I divided according to the region partition template is represented by the following equation.

4 illustrates an example of picture division performed by the area divider 110. In FIG. 4, it can be seen that a local meaning concept is included in each divided area. For example, in the case of the first picture, it can be seen that the sky is at the top, the water is at the bottom left, and the grass is at the bottom right. That is, it fully expresses the semantic concept information contained in the photograph.

Next, the local semantic concept is modeled by extracting multiple content-based feature values from each of the divided regions (step 220). The multiple content based feature values used are represented by equation (5).

Where N _f represents the number of feature values used. The present invention includes a method of extracting content-based feature values based on color, texture, shape information, etc. of an image, and basically includes a method of extracting feature values using MPEG-7 descriptors. do. However, the method of extracting content-based feature values is not limited to MPEG-7 descriptors.

The multiple content-based feature values extracted from the region of the image segmented by the template T are represented by equation (6).

The local semantic concept included in each segmented region of the image is modeled based on the region-based feature values.

To this end, we first define local semantic concepts that can be included in the target category of category-based clustering. Local semantic concept L local is composed of L entity which is an item indicating an entity of a semantic concept included in a picture, and an L attribute which is an item indicating an attribute of a main body of a semantic concept. FIG. 5 is a table illustrating an embodiment of an attribute concept representing an entity concept of an divided region and an attribute of an entity concept.

The item L entity representing the main body of the semantic concept is represented again by Equation (7).

Where L _entity (e) represents the e-th body meaning concept, and Ne represents the number of defined body meaning concepts.

The item L _attribute representing an attribute of a semantic concept is represented again by Equation 8.

Here, L _attribute (a) represents the a-th attribute semantic concept, and Na represents the number of defined attribute semantic concepts.

The above-described local semantic concept L local is expressed as in Equation (9).

Here, L (l) denotes the l-th meaning concept and may be a body meaning concept or an attribute meaning concept.

Next, based on the regional semantic concepts described above, training sample images having respective regional semantic concepts are collected. The content-based feature values are extracted from the collected images, and the extracted feature values are learned using a support vector machine (SVM). The SVM local learned for each local semantic concept is represented by Equation 10.

Here, SVML represents the SVM learned for the semantic concept L. The input of the SVM local is the content-based feature value vector F described above.

Next, using the learned SVM local, model the local concept of the regionally divided input picture I. That is, the input photo I is segmented by the above-described method, and the segmented local image is modeled using the learned SVM local. Modeling the local concept includes inputting content-based feature values extracted from the segmented local image into SVML of the defined semantic concept L, and performing a process of extracting the reliability of the semantic concept.

6 illustrates a more process of regional semantic concept modeling (step 220). That is, the regional concept modeling includes regional body concept modeling (step 600) and regional attribute concept modeling (step 650).

The reliability of the semantic concept for the divided region T can be obtained by Equation (11).

Here, F _T represents a content-based feature value vector of the divided region T, and CL (T) represents a reliability of the semantic concept L of the divided region T. Reliability represents a measure of how much segmented-region images contain the semantic concept.

The reliability vector obtained by performing the SVM of all the defined local semantic concepts is expressed by Equation 12.

Here, C _local (T) represents each reliability vector for all local semantic concepts modeled for the divided region T.

As a result, the reliability of the local semantic concept for the 10 divided regions can be obtained, and the local semantic concept reliability vector obtained for the 10 divided regions is expressed by the following equation.

The partitioned regions defined above include regions that overlap each other spatially. That is, the divided area T (1) overlaps with T (10), and T (2) overlaps with T (6), T (8) and T (10); T (3) overlaps with T (6), T (9), T (10); T (4) overlaps with T (7), T (8), T (10); Finally, T (5) overlaps with T (7), T (9) and T (10). Thus, there will be a total of five overlapping regional groups. The present invention includes a process of merging the reliability of the region concept of the overlapping region group to extract a more reliable region semantic concept (step 230).

In the present invention, a method for merging semantic concepts of overlapping regional groups, wherein the divided regions T (1) and T (10) merge into T (1); T (2), T (6), T (8), and T (10) merge into T (2); T (3), T (6), T (9), and T (10) merge into T (3); T (4), T (7), T (8), and T (10) merge into T (4); T (5), T (7), T (9), and T (10) include merging into T (5). The region semantic concept merging process includes assigning the most reliable value to the merge region among semantic concepts assigned to the divided regions belonging to each divided region group. 7 illustrates a process of grouping local concept merging objects performed by the regional meaning concept merging unit 130. The process of merging local semantic concepts is expressed as in Equation 14.

As a result, the final divided regions are determined by T (1), T (2), T (3), T (4), and T (5), and the reliability of the local semantic concept assigned to each divided region, C _local It is expressed as in Equation 15.

Here, each C ' _local (T) represents a vector of confidence sets for the semantic concept L _local determined in the divided region T.

Next, based on the reliability of the local semantic concept measured as described above, a global semantic concept included in the input photograph I, that is, a category concept, is modeled (step 240). After collecting the sample image of the belonging photo, the reliability C ' _local of the _local semantic concept is obtained by the above process on the collected sample image. Based on this reliability, the learning process is performed using SVM. First, the full meaning concept, that is, the category concept, is expressed as in Equation 16.

Here, L (g) means the g-th category concept, and Ng represents the number of defined category concepts.

The SVM _global learned for each category concept is expressed as in equation (17).

Here, SVM _G represents the SVM learned for category concept G. As the input of SVM _global , C _{local which} is a reliability set of semantic concepts extracted and merged from the divided region is used.

Next, the category concept of the input photo I is modeled using the learned SVM _global (step 240). The reliability of each category concept for the input picture I is obtained by inputting C _local, which is a set of local concept reliability of the input picture, into the SVM for modeling each category concept. Reliability for the modeled category concept G is expressed as in equation (18).

Where C _G represents the confidence in category concept G. The reliability set C _global of the whole category concept obtained based on the above method is expressed as in Equation 19.

The final category concept for the input picture I is determined by selecting a category with a high value among the reliability of the category concept L _global defined above. The present invention includes a method of selecting a category concept having the highest reliability, and a method of selecting a category concept having a certain reliability or higher.

The method of selecting the category concept with the highest reliability is expressed as in Equation 20.

Where L _target represents the last selected category concept.

The method of selecting a category concept whose reliability is above a certain value is expressed as in the following equation.

Here, C _th is a threshold value of the confidence value for selecting the final category concept.

8 is a block diagram illustrating a flow of category-based clustering of a digital photo album according to the present invention as described above.

The present invention can be embodied as code that can be read by a computer (including all devices having an information processing function) in a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording devices include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

According to the category-based clustering method and apparatus of the digital photo album according to the present invention, not only the information that can be basically obtained from the photo such as camera information and file information stored in the photo, but also the color, texture, By using content-based feature value information such as shape at the same time to effectively categorize a large volume of photos, photo data can be albumed more quickly and effectively.

Claims (27)

(a) dividing the photo into regions using a region partition template; (b) modeling a semantic concept included in the divided region; (c) merging the semantic concepts of each region according to the confidence of the local meaning measured from the modeling when the image of the region includes a corresponding semantic concept and a measure indicating the degree of reliability; (d) modeling the full meaning concept included in the photograph by using the final local meaning concept determined by merging; And (e) determining one or more categories included in the input photo according to the reliability of the overall semantic concept measured from modeling. According to claim 1, wherein the region partition template of step (a) is [Equation 3]

(Where T is the template of the photo, w is the width of the photo, and h is the height of the photo.) Clustering method of digital photo album using a region partition template, characterized in that consisting of the region represented by the equation (3). The method of claim 1, wherein step (b) A method of clustering digital photo albums using a geographic partition template, characterized by modeling semantic concepts by extracting content-based feature values. The method of claim 3, wherein the content-based feature value is A method for clustering digital photo albums using a geographic partition template, which includes color, texture, and shape information of an image. The meaning concept of step (b) is An area division template, characterized in that configuring the entry property (attribute) of the main body of the (L entity) and, means concept displaying unit (entity) of the sense concept, which includes the picture to the item (L attribute) indicating Clustering method of digital photo album using. The method of claim 5, wherein the semantic concept modeling A method of clustering digital photo albums using a geographic partition template, characterized by modeling the concept of entities and attributes of divided regions. The method of claim 1, wherein the semantic modeling of step (b) A method of clustering digital photo albums using a segmentation template characterized by modeling a region concept of region-divided input photos using a support vector machine (SVM). The method of claim 7, wherein the measuring of the reliability of step (c) A method of clustering digital photo albums using a geographic division template characterized by measuring reliability of each local semantic concept using one SVM for each defined local semantic concept. The method of claim 2, wherein the merging of semantic concepts of the region of step (c) [Equation 14]

(Where T (1), T (2), T (3), T (4), and T (5) represent the base regions to which the final concept of local semantics is assigned, and C _L 'represents the region's reliability vector. Indicates.) The digital photograph using the region division template, wherein the region concept reliability of five basic regions is merged by Equation 14 based on the reliability of the region concept allocated to the ten regions divided by the region division template. How to cluster albums. 10. The method of claim 9, wherein the reliability of the local concept obtained after the merging C ' _local is [Equation 15]

(Where C _local (T) represents a vector of confidence sets for the semantic concept L _local merged in the partition region T). Clustering method of a digital photo album using a segmentation template, characterized in that represented by the equation (15). The method of claim 1, wherein step (d) A clustering method of digital photo albums using a segmentation template, characterized by modeling the entire concept of region-divided input photo I using a support vector machine (SVM). The method of claim 11, A method of clustering digital photo albums using a geographic division template characterized by measuring the reliability of the entire concept using the reliability of the regional concept as an input. The method of claim 1, wherein step (e) The clustering method of a digital photo album using a regional segmentation template, characterized in that the overall meaning concept having the highest value among the reliability of the overall meaning concept measured from the modeled overall meaning concept is determined as the category of the picture. The method of claim 1, wherein step (e) The clustering method of the digital photo album using the segmentation template, characterized in that by determining the overall meaning concepts having a reliability greater than a predetermined threshold value among the reliability of the overall meaning concept measured from the modeled overall meaning concept. A region division unit that divides a photo into regions using a region partition template; An area conceptual modeling unit for modeling a semantic concept included in the divided area; A non-concept merging unit for merging a semantic concept of each region according to the confidence of the local meaning measured from the modeling when the image of the region includes a corresponding semantic concept and a degree indicating reliability; An overall meaning concept modeling unit modeling an overall meaning concept included in a photograph by using the final local meaning concept determined by merging; And And a category determination unit for determining one or more categories included in the input photo according to the reliability of the overall semantic concept measured from the modeling. The method of claim 15, And a photo input unit for receiving photo data for category-based clustering. The method of claim 15, wherein the local meaning modeling unit Model semantic concepts by extracting content-based feature values The content-based feature value is A clustering device for digital photo albums using a geographic partition template, characterized by including color, texture, and shape information of an image. 18. The method of claim 17, wherein the local meaning is An area division template, characterized in that configuring the entry property (attribute) of the main body of the (L entity) and, means concept displaying unit (entity) of the sense concept, which includes the picture to the item (L attribute) indicating Clustering device of digital photo album using. The semantic concept modeling of claim 5, wherein A clustering apparatus for digital photo albums using a segmentation template, characterized by modeling the region concept of region-divided input photos using a support vector machine (SVM). The reliability measure of claim 19, wherein A clustering device for digital photo albums using a regional segmentation template, characterized by measuring the reliability of each regional semantic concept using one SVM for each defined regional semantic concept. 16. The method of claim 15, wherein merging the semantic concepts of the region [Equation 14]

(Where T (1), T (2), T (3), T (4), and T (5) represent the base region to which the final meaning of the region semantics is assigned, and C _L 'represents the confidence vector of the region. .) Digital photo album using a region partition template, wherein the region concept reliability of five basic regions is merged by the above equation based on the reliability of the region concept assigned to the ten regions divided by the region partition template. Clustering device. 22. The method of claim 21, wherein the reliability of the local concept obtained after the merging C ' _local is [Equation 15]

(Where C _local (T) represents a vector of confidence sets for the semantic concept L _local merged in the partition region T). Clustering apparatus of digital photo album using a regional partition template, characterized in that expressed as the equation. The method of claim 15, wherein the concept modeling unit as a whole means A clustering device for digital photo albums using a segmentation template, characterized by modeling the overall concept of region-partitioned input photo I using a support vector machine (SVM). 24. The method of claim 23, wherein the overall concept reliability measure of the category determiner is A clustering apparatus for digital photo albums using a regional segmentation template, characterized by measuring the reliability of the entire concept using the reliability of the local concept as an input. The method of claim 15, wherein the category determiner Lustering apparatus of a digital photo album using a regional partition template, characterized in that the overall meaning concept having the highest value among the reliability of the overall meaning concept measured from the modeled overall meaning concept is determined as the category of the picture. The method of claim 15, wherein the category determiner The clustering apparatus of the digital photo album using the area segmentation template, characterized by determining the overall meaning concepts having a reliability greater than a predetermined threshold value among the reliability of the overall meaning concept measured from the modeled semantic concept. A computer-readable recording medium having recorded thereon a program for executing the invention according to any one of claims 1 to 14.

Images