KR101563238B1 - Apparatus and method for creating closeness between people based on photos, and computer-readable recording medium with program therefor - Google Patents

Abstract

In the present invention, a photograph-related graph is constructed using photographed persons, photographing time, and photographing place information, and then a person-based close-up graph is newly defined through the same to thereby perform character-based photo clustering, A device and method for establishing person familiarity, comprising: a photo input unit for inputting photo information from a terminal; and an input unit for analyzing and describing the inputted photo to provide photo description information including a photo identifier, A photograph description information generating unit for generating analysis information of the photographs based on the analyzed information of the photographs including the generated various photograph description information, You can build associations between photos to more efficiently search and group A photo group information generating unit for generating group description information based on the association between photographs and the intimacy between the people based on the output of the clustering unit; And a photo album information generation unit for generating photo album information by matching the photographic description information generated by the generation unit and the group description information generated by the photo group information generation unit to each other to combine them.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer-readable recording medium storing a computer-readable recording medium,

The present invention relates to an apparatus and method for establishing a personality of a person based on a photograph, and more particularly, it relates to an apparatus and method for constructing a relationship between photographs using photographed characters, a photographing place and photographing time information, And more particularly, to an apparatus and a method for displaying and managing information.

Recently, the demand of digital cameras is increasing more and more. In particular, with the development of memory technology, the use of highly integrated / miniature memory is becoming common, and as digital image compression technology which does not impair image quality is developed, users can store hundreds to thousands of pictures in one memory device . Accordingly, a device and a tool for managing many pictures more effectively have become necessary. Therefore, there is an increasing demand of users for efficient digital photo albums.

A typical digital photo album is for transferring photographs taken by a user from a digital camera or a memory card to a user's local storage device, and managing pictures and the like on a computer. Users use photo albums to cluster and index multiple photos by time, by person, or by specific categories. The resulting photo album allows the user to later browse the photo data or easily share the photo with other users.

David Frohlich investigated the features of photo albums that people demand from a survey of many users in Requirement for photoware (ACM CSCW, 2002). Most people agree with the need for digital photo albums, but they have been inconvenienced by the time and effort involved in grouping and labeling many photos, and they have had a lot of trouble sharing pictures with others.

So far, there have been many studies suggesting clustering of photographic data. Basically, there were methods of clustering according to the time of photographing, and methods of clustering using content-based feature information of photographs. Furthermore, clustering based on photo events and methods for clustering photos based on photographed characters were studied.

Among them, one of the most useful and frequently used methods of clustering is clustering photo data by person. Automatically clustering photos by a large number of photographic data on a photo makes it easy for users to share photo data with others or browse photo data at a later date.

Face recognition technology is the most important technique in clustering photo data by person. Face recognition can be described as a process of collecting different faces for each person to create a face database, and comparing the inputted face images with the face images of the database that have already been created.

That is, a face image is detected from the photographic data, and the detected face is recognized by matching with the face images of the database. For this purpose, the process is almost the same as the pattern recognition in the image processing method, and it goes through processes such as image acquisition, preprocessing, face extraction, and face recognition.

There have been so many researches on methods of detecting faces or recognizing faces from images. The face recognition field developed mainly in the security system. For example, research on face extraction and recognition for intelligent unmanned surveillance systems, access control systems, and criminal suspect search systems was conducted. In this application area, research focuses on extracting face of person more accurately and developing robust face recognition method for external illumination, face expression and pose of person.

On the other hand, research on recognizing faces in photographic data is very early. The process of extracting or recognizing a face from photographic data is much more difficult than the process of extracting or recognizing a face from a captured image in a security system. In the security system, since the image is acquired using the fixed camera installed in the fixed space, the face image extracted from the image has a relatively standardized background. On the other hand, since the picture data is taken by the user carrying the camera, the background in the picture generally varies and varies very often. In addition, the user can have various backgrounds in a fixed space depending on how the camera is operated (for example, using the camera zoom or using the flash) or the direction of the camera at the time of taking the picture, The change in illumination is much greater.

Due to the above-described reasons, in the method of clustering photographic data by persons, very low performance is generated when only face images are used as in an existing security system application method. In order to better cluster the photograph data by the person, a method and system for using together with various additional information obtained from the photograph as well as facial image information extracted from the photograph are required.

As a precedent study to solve these problems, the following studies are representative.

In this paper, we propose a face recognition system which recognizes face images with relatively no interference from light sources. Respectively. However, this system has a problem that the user has to prepare less noisy facial images in advance.

Second, in Automatic Cataloging of People in Digital Photographs (US 6,606,398 B2, 2003/08), an identification parameter related to each facial image stored in the database is defined and used for clustering of face-based photographs Respectively. Identification parameters include name, relationship, gender, and age. New incoming images are searched and clustered in the database using the identification parameters entered by the user. However, since the user has to input the identification parameters directly, it takes a long time to input the identification parameters, which is not practical.

Third, automatic Face-based Image Grouping for Albuming (IEEE, 2003) automatically detects the age and gender of each face, and proposes a method of clustering a collection of faces using faces. This method has the advantage of using age information and gender information of a person in addition to the face information of a photograph, but it can be a more effective grouping of pictures if various additional information of the photograph are used together.

In this way, the existing person-based digital photograph clustering method appropriately combines various additional information obtained from the photographic data, thereby obtaining the factors such as external brightness change, pose change, facial expression change, time change, Based clustering of photographs using only the information extracted from the information or photographs stored in the photographs, and considering the reliability (personal closeness) among the individuals who own the photographs, It is possible to construct a more effective grouping of photographs.

10-0601997: METHOD AND APPARATUS FOR CLUSTERING PHOTO-BASED DIGITAL PHOTOGRAPHS, AND METHOD AND APPARATUS FOR DIGITAL PHOTO ALPHABETING USING THE SAME

10-0624465: DEVICE AND METHOD FOR DETECTING CONTEXT CHANGE OF DIGITAL PHOTOGRAPHY AND METHOD AND APPARATUS FOR CLUSTERING STATUS BASED ON DIGITAL PHOTO ALBUM

Korean Publication No. 2013-0126529: Method and system for analyzing association among users using photographing information

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in an effort to solve the problems as described above, and it is an object of the present invention to construct a photograph-related graph using a photographed person, a photographing time, Based photos clustering, and to provide a device and method for constructing a photo-based person familiarity for effective grouping of photos.

Another object of the present invention is to provide a photo-based person familiarity building method capable of generating a virtual album in which only the addresses of corresponding photographs are stored by using not only photographs stored locally but also photos uploaded on a social network Apparatus and method.

Other objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to another aspect of the present invention, there is provided a device for establishing a person relationship with a person based on photographs, the device comprising: a photo input unit for inputting photo information from a terminal; a display unit for analyzing and describing the input photo, A photographic description information generating unit for generating photographic description information and analysis information of the photographed image, a photographed person, a photographing place, a photographing time, Based clustering unit for performing photo-based clustering for associating photographs for searching and grouping of photographs using at least one of photographing location information, A picture group information generating unit for generating picture group description information of the picture description information May combine to each other, matching the picture group description information generated by the picture descriptive information and picture group information generator generating in part is composed by including a photo album information generator for generating a photo album information.

According to another aspect of the present invention, there is provided a device for establishing a close relationship with a person based on photographs, the device comprising: a photo input unit for inputting photo information from a terminal; and an input unit for analyzing and describing the input photo, A photographic description information generating unit for generating photographic description information and photographic analysis information of the photographic subject information based on the analysis information of the photographic image including the generated photographic description information; Based clustering unit for associating the intimacy between the persons with each other through association; a photo group information generating unit for generating predetermined person group description information using the intimacy between the person output from the person familiarity-based clustering unit; The photograph description information generated by the description information generation unit and the image group information generated by the photo group information generation unit Combine with each other to match the character group consists of descriptive information is to include a photo album information generator for generating a photo album information.

According to another aspect of the present invention, there is provided a device for establishing a close relationship between a photographer and a person, the device comprising: a photo input unit for inputting photo information from a terminal; an analyzing unit for analyzing and describing the input photo, A photographic description information generating unit for generating photographic description information and analysis information of a photograph including the generated photographic description information; and a photographed person, a photographing place, A clustering unit configured to associate photographs to search and group the pictures more efficiently using photographing time and photographing place information, and to perform character-based photo clustering by defining personality intimacy; The group description information is generated using the association between photographs and the intimacy between people And a photo album information generation unit for generating photo album information by matching and combining the photo description information generated by the photo description information generation unit and the group description information generated by the photo group information generation unit, .

Preferably, the system further includes an album database for storing the photo album information. The photo physical / semantic association graph and the person familiarity graph obtained from various user terminals are integratedly managed through the album database, And analyzing the intimacy.

Preferably, the clustering unit includes: a photo-based clustering unit for generating a photo-related graph that expresses association between photos by using information extracted or analyzed from information stored directly in a photograph; And a personality affinity-based clustering unit for generating a person affinity association graph indicating the degree of reliability among individuals who own the photograph by using the photo-related information extracted through the graph.

According to another aspect of the present invention, there is provided a method of establishing a personality based on photographs, the method comprising: (A) receiving photo information from a terminal; (B) analyzing and describing the input photo, The method comprising the steps of: (a) generating photographic description information including a photographic identifier and analysis information of the photographic image; (C) capturing a photographed person of the photographed image based on analysis information of the photographed photographed description information A method of clustering a group of photographs, comprising the steps of: (a) clustering photographs to search and group the photographs more efficiently by using the location, photographing time, and photographing location information; (D) (E) combining the generated photo description information and group description information with each other to combine the photo album information And storing the generated data.

Preferably, the step (C) searches the photographs using at least one of the photographed person, the photographing place, the photographing time, and the photographing place information based on the analysis information of the photograph including the generated photograph description information, And performing photo-based clustering for establishing an association between the photographs.

According to another aspect of the present invention, there is provided a method of establishing a familiarity with a person based on photographs, the method comprising: inputting a user's photograph through a user terminal to a server; analyzing the input photograph to include photograph description information; A step of classifying the photographed person, the photographing place, the photographing time, and the photographing place information based on the analysis information of the generated photograph to perform clustering for more efficiently searching and grouping the photographs And generating and storing photo album information by combining the generated photo description information and the group description information generated through the clustering so as to be matched with each other.

According to another aspect of the present invention, there is provided a method for establishing a familiarity with a person based on a photograph, comprising the steps of: analyzing a picture held by a user using a picture analysis program (Apps) A step of inputting analysis information of a photograph including the photographic description information to the server; and a step of classifying the photographed person, the photographing place, the photographing time, and the photographing place information based on the analyzed information of the inputted photograph, Generating group description information by performing clustering for searching and grouping, combining the generated description description information and group description information generated through clustering so as to match each other, and generating and storing photo album information .

According to another aspect of the present invention, there is provided a method of constructing a familiarity to a person based on photographs, comprising: analyzing a photograph held by a user using a graph construction program (Apps) Analysis information of photographs including descriptive information is extracted and clustering is performed to search for photographs and group them by classifying the photographed persons, photographing place, photographing time, and photographing place information based on the analyzed information of the extracted photographs And a person affinity graph are input to a server, and a comparison is made between the graphs input from a plurality of users using the input photo correlation graph and / or the person familiarity graph, Generate group description information by extending association graph or / and person affinity graph The steps and, in combination with each other to match the generated picture information and describing the generated group descriptive information through the execution of clustering it may makin comprises the step of storing the information to create a photo album.

Preferably, the photo-based clustering method further comprises the steps of: generating a photo-related graph expressing relevance between photographs using information extracted or analyzed from information or photographs directly stored in the photo; And generating predetermined picture group description information using the association.

Preferably, the person affinity clustering method is a method of clustering a person, which defines an intimacy between the persons based on the generated photographic description information, such as a personality association, a place relation, and a temporal association, Generating an affinity association graph; and generating a predetermined person group description information using the affinity between the persons through the generated person affinity association graph.

Preferably, the person affinity association graph is generated by using the photo association information extracted through the photo association graph generated through the photo-based clustering method.

Preferably, the content of the photo-related graph or the person affinity graph is encrypted and processed.

The above-described apparatus and method for constructing a photo-based person familiarity according to the present invention have the following effects.

First, it is possible to classify the values into a range of values that is suitable for a range in which a person is considered to be semantically similar, instead of classifying them into a fixed range of values.

Second, unlike the existing patents, the photos included in the album are not actual physical data, and thus, a virtual album can be created, so that one photo can be duplicated in many albums.

Third, the existing social network analysis service depends on the text information of the users, but the present invention can measure the intimacy using the photo information.

FIG. 1A is a block diagram showing a configuration of a photograph-based person affinity building apparatus according to a first embodiment of the present invention; FIG.
1B is a block diagram showing a configuration of a photograph-based person affinity building apparatus according to a second embodiment of the present invention
1C is a block diagram illustrating a configuration of a photograph-based person affinity building apparatus according to a third embodiment of the present invention
FIG. 2 is a diagram showing a physical photograph-related graph in a photograph-based person affinity building apparatus according to the present invention
3 is a graph showing a person affinity graph in a photograph-based person affinity building apparatus according to the present invention
4 to 6 are flowcharts for explaining a method for constructing a photograph based personality intimacy according to an embodiment of the present invention

Other objects, features and advantages of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an apparatus and method for establishing a personality based on photographs according to the present invention will now be described with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. It is provided to let you know. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

FIG. 1A is a block diagram illustrating a configuration of a photograph-based person affinity building apparatus according to a first embodiment of the present invention.

1A, a photograph-based person familiarity building apparatus includes a photo input unit 100, a photo description information generating unit 200, a photo-based clustering unit 310, a photo group information generating unit 400, And an information generating unit 500. The photograph-based person familiarity building apparatus further includes an album database 600.

The photograph input unit 100 receives photographs from an internal memory device or a portable memory device of a digital camera. At this time, the input photograph includes photograph information stored in a smart device such as a smart phone or a tablet PC, and may also include a photograph included in a social network on the Internet accessible to a smart device user.

The photograph description information generating unit 200 analyzes and describes a photograph input from the photograph input unit 100 to generate photo description information including photograph IDs and analysis information of photographs. More specifically, the photograph description information generating unit 200 checks whether the camera information and the photographing information stored in the photo file are present from the respective photographs, and extracts information if the information exists in the photo file It is expressed according to the photo description structure. The photograph description information is input to a clustering unit 300 for grouping photographs. At this time, the photograph description information is set as a photographing date, a time, etc. in a unique name (ID) for distinguishing between photographs.

The photo-based clustering unit 310 may search the photographs more efficiently by using the photographed person, the photographing place, the photographing time, and the photographing place information based on the analyzed information of the photograph including the various photograph description information And perform photo-based clustering to build associations between photos to group them. At this time, the photo-based clustering unit 310 generates a photo-related graph expressing relevance between photographs by analyzing information extracted directly from photographs or information extracted from the photographs. The photographic association graph will be described in detail below again. And the correlation between the photographs is related to the photographing time, the photographing place, and the photographs according to the person information in the photograph. The photographic association graph is classified into a photographic physical association graph based solely on the physical information of time, place and person, and a photographic semantic association graph imparting semantic analysis on time, place and person.

The photo group information generation unit 400 generates predetermined photo group description information based on the correlation between the photos based on the output of the photo-based clustering unit 310. [ At this time, the photograph group description information is a unique name (ID) for identifying a group of photographs having a relation between photographs.

The photo album information generating unit 500 may combine the photo description information generated by the photo description information generating unit 200 and the photo group description information generated by the photo group information generating unit 400 to combine the photo album information, . The generated photo album information is stored in the album database 600. At this time, the album database 600 may store the photo album information together with the photo information input through the photo input unit 100.

FIG. 1B is a block diagram illustrating a configuration of a photograph-based person affinity building apparatus according to a second embodiment of the present invention.

1B, the photograph-based person familiarity building apparatus includes a photo input unit 100, a photo description information generating unit 200, a person familiarity clustering unit 320, a photo group information generating unit 400, And an information generating unit 500. The photograph-based person familiarity building apparatus further includes an album database 600.

The photograph input unit 100 receives photographs from an internal memory device or a portable memory device of a digital camera. At this time, the input photograph includes photograph information stored in a smart device such as a smart phone or a tablet PC, and may also include a photograph included in a social network on the Internet accessible to a smart device user.

The photograph description information generating unit 200 analyzes and describes a photograph input from the photograph input unit 100 to generate photo description information including photograph IDs and analysis information of photographs. More specifically, the photograph description information generating unit 200 checks whether the camera information and the photographing information stored in the photo file are present from the respective photographs, and extracts information if the information exists in the photo file It is expressed according to the photo description structure. The photograph description information is input to a clustering unit 300 for grouping photographs.

The person affinity clustering unit 320 is a person affinity clustering unit based on analysis information of the photographs including the generated various photograph description information, a person familiarity degree based on the personality association, Clustering is performed. At this time, the person affinity-based clustering unit 320 generates a person affinity association graph indicating how high the reliability of the individual who owns the photograph is. That is, it is defined that people who are frequently appearing in photographs have high intimacy with each other through personality relation, place relation, and temporal relation. The person affinity association graph will be described in detail below again.

The photo group information generation unit 400 generates predetermined person group description information using the intimacy between the characters output from the person familiarity clustering unit 320. [ At this time, the person group description information is a unique name (ID) for distinguishing a person group having intimacy between the persons.

The photo album information generating unit 500 may combine the photo description information generated by the photo description information generating unit 200 and the person group description information generated by the photo group information generating unit 400 to provide photo album information . The generated photo album information is stored in the album database 600. At this time, the album database 600 may store the photo album information together with the photo information input through the photo input unit 100.

1C is a block diagram illustrating a configuration of a photograph-based person affinity building apparatus according to a third embodiment of the present invention.

1C, the photograph-based person familiarity building apparatus includes a photo input unit 100, a photo description information generation unit 200, a clustering unit 300, a photo group information generation unit 400, (500). The photograph-based person familiarity building apparatus further includes an album database 600.

The photograph input unit 100 receives photographs from an internal memory device or a portable memory device of a digital camera. At this time, the input photograph includes photograph information stored in a smart device such as a smart phone or a tablet PC, and may also include a photograph included in a social network on the Internet accessible to a smart device user.

The photograph description information generating unit 200 analyzes and describes a photograph input from the photograph input unit 100 to generate photo description information including photograph IDs and analysis information of photographs. More specifically, the photograph description information generating unit 200 checks whether the camera information and the photographing information stored in the photo file are present from the respective photographs, and extracts information if the information exists in the photo file It is expressed according to the photo description structure. The photograph description information is input to a clustering unit 300 for grouping photographs.

The clustering unit 300 searches the photographs more efficiently by using the photographed persons, the photographing place, the photographing time, and the photographing place information based on the analysis information of the photograph including the various photograph description information, In order to construct the association between photographs, personality based clustering is performed by defining personality intimacy. For this, the clustering unit 300 includes a photo-based clustering unit 310 and a personality-based clustering unit 320.

At this time, the photo-based clustering unit 310 generates a photo-related graph expressing relevance between photographs by analyzing information extracted directly from photographs or information extracted from the photographs. The photographic association graph will be described in detail below again. And the correlation between the photographs is related to the photographing time, the photographing place, and the photographs according to the person information in the photograph. The photographic association graph is classified into a photographic physical association graph based solely on the physical information of time, place and person, and a photographic semantic association graph imparting semantic analysis on time, place and person.

Based on the photo-related information extracted through the photo-related graph generated by the photo-based clustering unit 310, the person affinity-based clustering unit 320 calculates a person-affinity-related graph indicating how high the reliability of the individual possessing the photograph is . That is, in the photographic association graph, the association includes personality association, place association, and temporal association, and the intimacy among the persons frequently appearing on the high association photograph is considered to be high. Therefore, the intimacy between individuals who have high-related photos in the photographic association graph is set high. The person affinity association graph will be described in detail below again.

The photo group information generator 400 generates group description information based on the output of the clustering unit 300 using the associations between photographs and the intimacy between the people. At this time, the group description information refers to a unique name (ID) for identifying a group of photographs having an association between photographs and a group of persons having an intimacy between the persons.

The photo album information generating unit 500 combines the photo description information generated by the photo description information generating unit 200 and the group description information generated by the photo group information generating unit 400 to generate photo album information do. The generated photo album information is stored in the album database 600. At this time, the album database 600 may store the photo album information together with the photo information input through the photo input unit 100.

Meanwhile, the photo-related graph generated by the photo-based clustering unit 310 is a graph expressing the relevance of photographs using information extracted from analyzing information or photographs directly stored in photographs. Photographic associative graphs are classified into physical photographic associative graphs when used photograph information is physical information, and semantic photographic associative graphs when semantic information is generally understood by humans.

First, as shown in FIG. 2, the physical photographic association graph is composed of nodes and edges of a graph in the same manner as a general graph. As shown in Fig. 2, the nodes of the graph are the respective photographs P1 to P10, and the edges of the graph are the photographing time T of the photographs, the photographing place L, Personality relevance (H). As a reference, the photographing time (T), which is a physical time correlation (PTA), is an association according to the difference in time between two photographs, and gives a different degree of association to each photograph according to the time difference from the reference photograph. The photographing location (L), which is a physical location association (PLA), is an association diagram according to the place where the two photographs were taken. The physical distance between the GPS coordinates stored in the EXIF information of the photograph is calculated, To each other. The figure of association (H), which is the physical figure of association (PPA), is the degree of association according to the characters appearing in the two photographs, and is calculated as the proportion of the characters commonly appearing in the two photographs of the two photographs . That is, the degree of association is given to each photograph using the persons included in the reference photograph. At this time, the photographed person can distinguish each person through a photo recognition technique such as face recognition or the like and find the same person in another photograph.

Referring to FIG. 2, the physical association (PA) between the photographs P1 and P2 can be defined as the following equation (1).

[Equation 1]

PPA (p1, p2) + w2 * PLA (p1, p2) + w3 * PPA (p1,

At this time, w is a weight, and w1 + w2 + w3 = 1.

Next, the semantic photographic association graph consists of the nodes and edges of the graph, the nodes of the graph are the individual photographs, and the edges of the graph are the associations in terms of semantics between the two photographs , Semantic temporal associations, semantic place associations, and semantic personality associations. For reference, a semantic time association (STA) is a time-based unit such as seconds, minutes, hours, days, months, years, or a semantic step such as breakfast, lunch, And assigns semantic temporal associations between two different photographs to each photograph according to the time difference between them. In addition, the Semantic Locality Association (SLA) classifies photographs into units such as buildings, administrative districts, and specific areas using GPS coordinates stored in the EXIF information of photographs, And gives a degree of association according to different semantic distances. In addition, the semantic associativity (SPA) is related to the physical personality, that is, the ratio of the characters that appear in common in both photographs, . For example, it is possible to assign a higher degree of association to a person in a pleasant expression, such as a smiling expression recognizing a person's facial expression in a photograph, and the closer the physical distance between the persons in the photograph is, the higher the degree of association can be given.

Similarly, the semantic association (SA) between the pictures P1 and P2 can be defined as the following equation (2).

&Quot; (2) "

SPA (p1, p2) + w2 * SLA (p1, p2) + w3 * SA (p1, p1) = w1 * STA

At this time, w is a weight, and w1 + w2 + w3 = 1.

The person affinity association graph generated by the person affinity-based clustering unit 320 is a graph expressing intimacy between the characters based on the photographic physical / semantic association graph. For example, the familiarity among the people frequently appearing in high-related photographs in the photographic association graph can be considered high.

As shown in FIG. 3, the person affinity graph is composed of nodes and edges of a graph in the same manner as a general graph. At this time, the graph nodes are figures (H1 to H7) appearing in the photographic physical / semantic association graph, and the graph edges indicate the intimacy between the two persons indicated in the node. At this time, based on the photographic physical / semantic association graph, the intimacy between the two persons is determined.

In one embodiment, when the personal closeness of two persons P1 and P2 is referred to as PC (p1, p2), PC (p1, p2) can be calculated through the following rule. At this time, if the photographs in which the person p1 and the person p2 appear are photo1 and photo2 respectively,

The higher the number of paths between photo1 and photo2, the higher the intimacy of p1 and p2, and is determined by the distance between photo1 and photo2. For example, when p1 and p2 appear in the same photograph, they are more intimate than when they are indirectly related through other photographs on the photographic physical / semantic graph. And, the larger the degree of semantic personality in the path between photo1 and photo2, the higher the affinity between p1 and p2.

In addition, the place relation (PLC) shows that the relationship between p1 and p2 is high in relation to the relationship between the photo1 and the photo2, and the relationship between the place1 and the photo2 in the public place The affinity of p1 and p2 are higher than those of p1 and p2.

And the temporal correlation (PTC) shows that the greater the semantic temporal relationship in the path between photo1 and photo2, the higher the affinity between p1 and p2, and the more the temporal associativity due to private time (holiday, The familiarity of p1 and p2 is higher than that of time (day, time, etc.).

Therefore, the personal closeness (PC) between the person p1 and the person p2 can be defined by the following Equation 3.

&Quot; (3) "

PC (p1, p2) + w3 * PTC (p1, p2) + w2 *

At this time, w is a weight, and w1 + w2 + w3 = 1.

As described above, the photo group description information generated using the photo physical / semantic association graph and the person familiarity graph generated by the smart device of each user is stored and integrated in the album database 600 of the photo-based person familiarity building apparatus, The photo / semantic association graph and the person affinity graph obtained from the smart device can be managed in an integrated manner, so that it is possible to analyze the relationship between more photographs and more people.

The method for constructing the photograph-based person familiarity based on the photograph information, the photograph relevance graph and the person familiarity graph existing in each smart device using the apparatus for establishing a personality close to the figure according to the present invention can be classified into the following three methods Can be used.

In the first method, only the process of transferring the photographs from the user's terminal is performed, and the photographs are analyzed using the information of the photos uploaded from the photograph-based person familiarity building device (hereinafter referred to as 'server') and clustering And / or a person-affinity graph, and stores the generated photo album information in the album database 600. [0064] FIG. At this time, the server refers to a photo / person relation management server including a photo-based person familiarity building device including a cloud server, a personal server, a hosting server, etc., such as Amazon Web Services.

Referring to FIG. 4, a user creates a graph through advance transmission, which is a first method, in step S10. The user uploads his or her own photograph to the server through the photo input unit 100 in step S10. At this time, the picture is not limited to the picture held in the terminal, and includes all pictures on the internet such as social network such as Facebook and Cacao Story.

Then, the server analyzes the photograph inputted through the photograph description information generating unit 200 to generate analysis information of the photograph including the photograph description information, and transmits the analysis information to the person The photographing place, the photographing time, and the photographing place information, and performs clustering for searching and grouping the photographs more efficiently (S11). At this time, the analysis information of the photograph includes the photographed person, the photographing place, the photographing time, and the photographing place information.

The method of performing the clustering can be divided into a photo-based clustering and a person-based clustering, and clustering can be performed using one or both of them.

First, the photo-based clustering method generates a photo-associative graph that expresses associations between photos using information extracted from the information or photos directly stored in the photographs. At this time, the correlation between the photographs is the degree of association between the two photographs according to the photographing time, the photographing place, and the person information in the photograph. The photographic association graph is classified into a photographic physical association graph based solely on the physical information of time, place and person, and a photographic semantic association graph imparting semantic analysis on time, place and person. The detailed description according to the photographic association graph is omitted above because it is described above.

Then, the predetermined photograph group description information is generated using the association between the photographs through the generated photograph association graph. At this time, the photograph group description information is a unique name (ID) for identifying a group of photographs having a relation between photographs.

Next, the person affinity clustering method defines the intimacy between the persons based on the generated photographic description information, such as the personality association, the site association, and the temporal association, Generate an affinity graph. That is, it is defined that people who are frequently appearing in photographs have high intimacy with each other through personality relation, place relation, and temporal relation. At this time, the affinity affinity graph is set to have a high reliability among individuals who have high relevance photographs. The detailed description based on the person affinity association graph is omitted above because it is described above.

On the other hand, in the process of creating the person-affinity association graph in the affinity clustering method, a person who shows how high the reliability of the individual possessing the photograph is by using the photo-related information extracted through the photo- You can also create an affinity graph.

Then, predetermined person group description information is generated using the intimacy between the persons through the generated person affinity association graph. At this time, the photograph group description information is a unique name (ID) for identifying a person group having intimacy between the persons.

As described above, the clustering method may perform clustering for performing grouping by independently performing photo-based clustering and person-based clustering, but it is also possible to perform the clustering in which the photo-based clustering and the person- No clustering may be performed to perform clustering for grouping.

The photograph description information generated through the photo album information generating unit 500 and the photo group description information or the character group description information (hereinafter, referred to as 'group description information') generated through the clustering through the clustering unit 300, To match each other to generate photo album information. The generated photo album information is stored in the album database 600 (S12).

In the second method, when a photograph is transmitted from a user terminal, analysis information of photographs is extracted by analyzing the photographs stored in advance through a photo analysis program (Apps.) Included in the terminal, and the analysis The information is transmitted from the user's terminal to the server. Then, the server generates the photo album information by using the photo-related graph and / or the person-familiarity graph generated through the clustering through the transmitted photo analysis information, and stores the information in the album database 600. At this time, the analysis information transmitted by the user is encrypted and transmitted, and the type of information includes meta information such as EXIF, analysis information such as color information and feature value, and information of the photographed person.

5, a user analyzes a photograph held by a user using a photo analysis program (Apps) included in the terminal. Then, Extracts the analysis information of the photograph including the photograph description information, and uploads the extracted analysis information to the server through the photo input unit 100 (S20). At this time, the photograph is not limited to the photographs held on the terminal, and includes all the images on the Internet such as social networks such as Facebook and Cacao Story. The analysis information of the photograph includes the photographed person, And includes shooting location information.

Then, the server classifies the photographed person, the photographing place, the photographing time, and the photographing place information based on the analysis information of the photograph inputted through the clustering unit 300, and performs clustering for searching and grouping the photographs more efficiently S21).

At this time, the method of performing the clustering can be divided into a photo-based clustering and a person-based clustering, and clustering is performed using one or both of them to generate group description information. The detailed description of the clustering method will be omitted because it is described above.

The photo description information generated through the photo album information generating unit 500 and the group description information generated through the clustering through the clustering unit 300 are combined to be matched with each other to generate photo album information. The generated photo album information is stored in the album database 600 (S22).

The third method is to analyze in advance the pictures that are held through the graph building program (Apps.) Included in each terminal in advance, and then, based on the analyzed image analysis information, the photo correlation graph generated by clustering and / Build the graph. Then, when the constructed photographic association graph or / and person affinity graph are transmitted to the server, the server generates the photographic album information using the corresponding graph on the server using the received photographic association graph or / and person affinity graph information And stores it in the album database 600.

6, the user analyzes a photograph held by a user using a graph construction program (Apps) included in the terminal, and displays the photo description Extracts analysis information of photographs containing information. Then, clustering is performed to search and group the photographs more efficiently by classifying the photographed person, the photographing place, the photographing time, and the photographing place information based on the extracted analysis information, and the photograph correlation graph or / And the person affinity graph are uploaded to the server (S30).

At this time, the photograph is not limited to the photographs held on the terminal, and includes all the images on the Internet such as social networks such as Facebook and Cacao Story. The analysis information of the photograph includes the photographed person, And includes shooting location information. The detailed description of the clustering method will be omitted because it is described above.

The server then generates a group description information by expanding the photo correlation graph and / or the person familiarity graph through comparison and matching with the respective graphs input from a plurality of users by using the inputted photograph relevance graph or / and person affinity graph. (S31).

The photo description information input through the photo album information generating unit 500 and the generated group description information are combined to be matched with each other to generate photo album information. The generated photo album information is stored in the album database 600 (S32).

Meanwhile, in the method of constructing the photograph-based person familiarity, it is necessary to encrypt and process the contents of the information used in the association operation to prevent the contents of the photo-related graph and the person-affinity graph from being used maliciously.

Therefore, the time information and the place information in the picture information encode the picture taking time information and the picture taking place information (GPS) contained in the EXIF information of the picture, respectively. And the person information uses a method of synthesizing and encrypting the unique information of the person like the face recognition system. At this time, the photographing time, the photographing place, and the person information can be divided into respective unique values after encryption is performed.

And the picture taking time is encrypted in such a way that two hours difference can be calculated even after encryption, and the picture taking place is encrypted in such a form that the distance of two places can be calculated even after encryption. If the characters in the different photos are the same person, the values for the two characters after encoding are the same.

Meanwhile, the embodiment of the present invention can be made into a program that can be executed in a computer. That is, the various steps involved in the method according to the present invention may be stored in a computer readable recording medium. The medium may be a magnetic storage medium such as a ROM, a floppy disk, a hard disk, etc., an optical reading medium such as CD-ROM or DVD, a digital storage medium such as a USB memory, XD), etc.) and carrier waves (e.g., transmission over the Internet).

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It will be apparent to those skilled in the art that various modifications may be made without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (44)

A photo input unit for receiving photo information from the terminal,
A photo description information generating unit for analyzing and inputting the input photograph to generate photo description information including a photo identifier and analysis information of the photo,
Based on the analysis information of the photograph including the generated photograph description information, constructs an association between the photographs for searching and grouping the photographs using at least one of the photographed person, the photographing place, the photographing time, and the photographing place information of the photograph A photo-based clustering unit for performing photo-based clustering,
Based on the output of the photo-based clustering unit, a picture group information generating unit for generating predetermined picture group description information by using a correlation between pictures,
And a photo album information generation unit for generating photo album information by matching the photo description information generated by the photo description information generation unit and the photo group description information generated by the photo group information generation unit,
Wherein the photo-based clustering unit generates a photo-related graph that expresses associations between the pictures by using information extracted from the information or photographs directly stored in the photographs. A photo input unit for receiving photo information from the terminal,
A photo description information generating unit for analyzing and inputting the input photograph to generate photo description information including a photo identifier and analysis information of the photo,
An affinity affinity-based clustering unit for associating the intimacy between the persons based on the analysis information of the photographs including the generated photograph description information,
A photograph group information generating unit for generating predetermined person group description information by using an intimacy between the persons output from the person affinity-based clustering unit;
And a photo album information generator for generating photo album information by matching the photo description information generated by the photo description information generator and the person group description information generated by the photo group information generator with each other to generate photo album information A device for constructing familiarity with a photograph based person. A photo input unit for receiving photo information from the terminal,
A photo description information generating unit for analyzing and inputting the input photograph to generate photo description information including a photo identifier and analysis information of the photo,
Based on the analysis information of the photographs including the generated various description information, the association between the photographs is constructed to search and group the pictures by using the photographed persons, the photographing place, the photographing time, and the photographing place information of the photographs A clustering unit for defining personality intimacy to perform person-based photo clustering,
A photograph group information generating unit for generating group description information using the associations between photographs and the intimacy between figures based on the output of the clustering unit;
And a photo album information generator for generating photo album information by matching and combining the photo description information generated by the photo description information generator and the group description information generated by the photo group information generator,
At this time, the clustering unit
A photo-based clustering unit for generating a photo-related graph expressing associativity between photographs by using information extracted or analyzed by directly storing information or photographs stored in the photographs,
And a personality affinity-based clustering unit for generating a person affinity association graph indicating the degree of reliability among individuals who own the photograph by using the photo-related information extracted through the photo-related graph generated by the photo-based clustering unit Based photo intimacy device. 4. The method according to any one of claims 1 to 3,
Wherein the photo information includes a photograph stored in a user's personal terminal and a photograph included in a social network on the Internet accessible through the personal terminal. 4. The method according to any one of claims 1 to 3,
Wherein the photograph description information is set to include at least one of a photographing date and a time in a unique name (ID) for distinguishing between photographs. delete 2. The method of claim 1,
A photographic physical association graph based solely on physical information about time, place, and person,
Wherein the image is classified into a photograph semantic association graph to which a semantic analysis of time, place, and person is given. The method according to claim 1,
Wherein the association between the photographs is represented by a predefined number of degrees of association between the two photographs according to the photographing time, the photographing place, and the person information in the photograph. The method according to claim 1,
Wherein the photograph group description information is a unique name (ID) for identifying a group of photographs having a relation between photographs. 4. The method according to any one of claims 1 to 3,
Further comprising an album database for storing the photo album information,
Wherein the photograph database is used to collectively manage photographic physical and semantic association graphs and person familiarity graphs obtained from a plurality of user terminals through the album database to analyze associations between photographs and intimacy between the persons. . 3. The method of claim 2,
Wherein the person affinity-based clustering unit generates a person affinity association graph indicating how high the reliability of the person who owns the photograph is. 12. The method of claim 11,
Wherein the intimacy degree of the person is closely related to each other as the number of appearances in the photograph is high due to the personality relation, the place relation, and the temporal relation. 3. The method of claim 2,
Wherein the person group description information is a unique name (ID) for identifying a person group having an intimacy between the persons. delete The method of claim 3,
Wherein the intimacy between the individuals having the high-related photographs in the photo-related graph is set higher than the intimacy between the individuals having the high-related photographs in the photo-related graph. The method of claim 3,
The photograph-related graph is a graph expressing the relevance of photographs using information extracted from analyzing information or photographs directly stored in photographs. When the used photograph information is physical information, it is classified into a physical photograph- Wherein the semantic information is classified into a semantic photographic association graph when the semantic information is generally understood. 17. The method of claim 16, wherein the physical photo-
It consists of nodes and edges of the graph,
Wherein the nodes of the graph are respective photographs and the edge of the graph is composed of photographing time of the photographs, location of photographing, and personality correlation between the photographs calculated using the person information in the photographs. 18. The method of claim 17,
The photographing time (T) is a physical time correlation (PTA) that constitutes a personality relationship according to a difference in time of photographing two photographs, and gives a different degree of association to each photograph according to a time difference from the reference photograph ,
The photographing location L is a physical location association (PLA) that constitutes personality association according to the place where the two photographs are photographed, and calculates a physical distance between GPS coordinates stored in the EXIF information of the photograph, Each picture is assigned a different degree of association,
The person information (H) in the photograph is a physical personality association (PPA) constituting a personality association according to a person appearing in two photographs. The proportion of a person Wherein the degree of association of each of the photographs is calculated by using figures included in the reference photograph, that is, the figures included in the reference photograph. 19. The method of claim 18,
The physical association (PA) between the pictures P1 and P2 is expressed by the following equation PA (p1, p1) = w1 * PTA (p1, p2) + w2 * PLA (p1, p2) + w3 * Wherein w is a weight, and w1 + w2 + w3 = 1. 17. The method of claim 16, wherein the semantic photographic association graph
It consists of nodes and edges of the graph,
The nodes of the graph are each photographs, and the edge of the graph is a relation in terms of semantics between two photographs calculated using physical photograph information, and the semantic time relation, the semantic place association degree, And the degree of association is determined by the degree of association. 21. The method of claim 20,
The semantic temporal relationship may be classified into a time unit such as seconds, minutes, hours, days, months, and sun, or a semantic step such as breakfast, lunch, And the semantic temporal associations between two different photographs are given to each photograph,
The semantic locality association classifies photographs into units such as buildings, administrative districts, and specific areas using GPS coordinates stored in the EXIF information of photographs, The degree of association according to the distance is given,
The above-mentioned semantic personality degree is a relation of physical personality, that is, not only the ratio of the characters appearing commonly in the two photographs but also the degree of association according to the distance between the semantic figures based on the distances between the persons in the photographs and photographs Wherein said means for establishing a person-friendly relationship with a photographer comprises: 22. The method of claim 21,
The above-mentioned semantic personality association is to recognize the facial expression of the person in the photograph and give a higher degree of association between the person and the person in a pleasant expression such as a laughing expression and give a higher degree of association as the physical distance between the persons in the photograph is closer Feature-based photo intimacy builder. 22. The method of claim 21,
The semantic association (SA) between the pictures P1 and P2 is expressed by the following equation SA (p1, p1) = w1 * STA (p1, p2) + w2 * SLA ), Wherein w is a weight, and w1 + w2 + w3 = 1. 4. The method according to claim 2 or 3, wherein the person affinity association graph
It consists of nodes and edges of the graph,
Wherein the graph node is a person appearing in a photo-related graph, and the graph edge is composed of an intimacy between two persons pointed to a node. 25. The method of claim 24,
Letting PC1 (p1, p2) be the personal closeness of two persons P1 and P2 and photo1 and photo2 photographs of the person p1 and p2, respectively,
As the number of paths between photo1 and photo2 increases, the affinity between p1 and p2 is high, and the distance between photo1 and photo2 is determined by the distance between photo1 and photo2. the affinity of p2 is set high,
The place-associativity (PLC) shows that the more intimate the relationship between p1 and p2 in the path between photo1 and photo2, the higher the affinity between p1 and p2, and the more the place- The affinity of p1 and p2 is set higher than that of place,
The temporal associations (PTC) show that the closer the semantic temporal relationship between the photo1 and the photo2, the higher the intimacy of p1 and p2, and the temporal associations by the private time in the path between photo1 and photo2 are greater than the temporal associations by p1 and the intimacy degree of the person p2 is set high. 26. The method of claim 25,
The personal closeness (PC) between the person p1 and the person p2 is calculated by the following formula PC (p1, p1) = w1 * PPC (p1, p2) + w2 * Wherein w is a weight, and w1 + w2 + w3 = 1. (A) receiving photo information from a terminal,
(B) analyzing and describing the input photograph, generating photo description information including the photo identifier and analysis information of the photo,
(C) performing photo clustering to search for and group photos using photographing person, photographing place, photographing time and photographing place information based on analysis information of photographs including the generated photograph description information Wow,
(D) generating group description information using association between photographs and intimacy between figures based on the result of the photo clustering;
(E) generating and storing photo album information by matching and combining the generated photo description information and group description information with each other,
At this time, the step (C)
Based on the analysis information of the photograph including the generated photograph description information, constructs an association between the photographs for searching and grouping the photographs using at least one of the photographed person, the photographing place, the photographing time, and the photographing place information of the photograph And performing photo-based clustering on the photo-based image. delete 28. The method of claim 27, wherein step (C)
Based on the analysis information of the photograph including the generated photograph description information, associating the intimacy of the person with the person based on the personality relation, the place relation, and the temporal relation of the photograph, How to build. 28. The method of claim 27, wherein step (C)
Based on the analysis information of the photographs including the generated photograph description information, an association is established between the photographs to search and group the photographs using the photographed person, the photographing place, and the photographing time information, And performing a person-based photo clustering on the basis of the feature information. The method comprising the steps of: inputting, by the user terminal,
Analyzing the input photograph to generate analysis information of the photograph including the photograph description information;
A step of classifying the photographed person, the photographing place, the photographing time, and the photographing place information based on the analyzed information of the generated photograph to perform clustering for more efficiently searching and grouping the photographs to generate group description information;
And generating and storing the photo album information by combining the generated photo description information and the group description information generated through the clustering so as to match each other,
At this time, the clustering method is performed by at least one of photo-based clustering and person-based clustering,
The method of performing clustering
Comprising the steps of: generating a photo-associative graph expressing associativity between photos using information extracted or analyzed from information stored directly in the photo;
And generating a predetermined photo group description information by using the association between the photos through the generated photo association graph. A step of analyzing a picture held by a user using a picture analysis program (Apps) included in a user terminal and inputting analysis information of the picture including the extracted picture description information to a server,
Generating group description information by performing clustering for searching and grouping photographs by classifying the photographed persons, photographing locations, and photographing time information based on the analyzed information of the inputted photographs;
And generating and storing the photo album information by combining the generated photo description information and the group description information generated through the clustering so as to match each other,
At this time, the clustering method is performed by at least one of photo-based clustering and person-based clustering,
The method of performing clustering
Comprising the steps of: generating a photo-associative graph expressing associativity between photos using information extracted or analyzed from information stored directly in the photo;
And generating a predetermined photo group description information by using the association between the photos through the generated photo association graph. The analysis of the photographs held by the user using the graph construction program (Apps.) Included in the user terminal is performed to extract the analysis information of the photograph including the photograph description information. Based on the analysis information of the extracted photograph, A step of inputting at least one of a photo correlation graph and a person familiarity graph generated by performing clustering for searching and grouping pictures by classifying a person, a photographing place, and a photographing time information,
And generating at least one of a photo correlation graph and a person familiarity graph by comparing and matching with each graph input from a plurality of users using at least one of the input photo correlation graph and the person familiarity graph, , ≪ / RTI &
And generating and storing photo album information by combining the generated photo description information and group description information generated through clustering to match each other. 34. The method according to any one of claims 31 to 33,
Wherein the photograph includes a photograph on the terminal and a photograph on the Internet such as a social network. 34. The method according to any one of claims 31 to 33,
Wherein the analysis information of the photograph includes a photographed person, a photographing place, a photographing time, and a photographing place information. 34. The method of claim 33,
Wherein the clustering is performed by at least one of photo-based clustering and person-based clustering. delete 33. The method according to claim 31 or 32,
Wherein the association between the photographs is a degree of association between two photographs according to the photographing time, the photographing place, and the person information in the photograph. The method of claim 31 or 32, wherein the person familiarity clustering method
A step of generating a person affinity association graph indicating the degree of reliability among the individuals who own the photograph by defining the intimacy between the persons through the personality association, the place association, and the temporal association based on the generated photograph description information; ,
And generating a predetermined person group description information by using an intimacy between the persons through the generated person affinity association graph. 40. The method of claim 39,
Wherein the person affinity association graph is generated using the photo association information extracted through the photo association graph generated through the photo-based clustering method. 33. The method of claim 31 or 32, wherein the photo-
A photographic physical association graph based solely on physical information about time, place, and person,
And a photograph semantic association graph to which a semantic analysis of time, place, and person is given. 34. The method of claim 33,
Wherein the contents of the photo-related graph and the person affinity graph are encrypted and processed. 34. The method according to any one of claims 31 to 33,
Among the analysis information of the photograph, the time information and the place information are respectively processed by encrypting the photographing time information and the photographing place information (GPS) included in the EXIF information of the photograph,
The person information is synthesized by encrypting unique information of a person, such as a face recognition system,
The photographing time information is encrypted and processed in such a form that two time differences can be calculated even after encryption,
Wherein the photographing location information is encrypted and processed in such a form that the distance between two places can be calculated even after encryption. A computer-readable recording medium having recorded thereon a program for executing each step of a method for establishing a personality of a photograph based on any one of claims 27, 31, 32 and 33.

Images