KR101086773B1 - Method and apparatus for clustering of digital photos obtained from multiple cameras - Google Patents

Abstract

The present invention relates to a method and an apparatus for clustering digital photographs taken with multiple cameras to enable effective classification of photographs based on situation by structuring photographs taken with multiple digital cameras and applying a graph algorithm. A method of clustering photographs obtained from a plurality of cameras and photographers, the method comprising: extracting content-based feature values from image contents of a digital photograph and extracting additional information feature values from metadata and file information of the digital photograph; Grouping them and structuring them according to feature values; generating clusters using feature values of a photo or feature values of a group including the corresponding photo.

Digital Photography, Clustering, Context Based, Graph Algorithms, Structured, Feature Values

Description

Method and apparatus for clustering of digital photos taken with multiple cameras

The present invention relates to the clustering of digital photographs. Specifically, digital photographs photographed by multiple cameras can be effectively classified based on situation by structuring photographs photographed by several digital cameras and applying graph algorithms. A method and apparatus for clustering of an apparatus are provided.

In the case of digital photo clustering, in the previous method, clustering was mainly applied to photographs taken by one camera or one individual user using additional information such as image content itself or metadata of the photo. In particular, it is common to use shooting time as the main criterion for clustering.

This is based on the premise that in the case of photographs taken by ordinary individuals, the photographs adjacent to each other in time are likely to be similar in meaning. In addition, one of the main criteria people use to manage their photos is to classify them according to when and under what circumstances and events.

These methods mainly use the method of arranging photos in chronological order and linearly dividing the section where the contents change much with time difference.

Although the processing time is fast because it does not require much computation to access and analyze the metadata, this premise is limited to general photographs taken by individuals or by one camera.

Therefore, when the camera is popularized and the penetration rate is increased in recent years, when shooting with multiple cameras at the same event or travel, shooting different objects at the same time point, or taking the same object or objects with the same meaning. Even when shooting, the time difference between the pictures occurs, so it is difficult to cluster on the basis of time only.

In particular, when taking pictures with multiple cameras, it is common to take pictures of the other party with one's own camera. In order to manage these pictures, in the future, when collecting pictures of multiple cameras or photographers and classifying pictures There are many difficulties.

As such, in the related art, the clustering target is limited to a photograph taken by a single camera of a specific individual, so even if the photograph is taken in the same situation, the photographing time or other metadata contents obtained from multiple cameras are not consistent. There is a problem that is not applicable.

The present invention is to solve such a problem of the prior art, to organize the photos taken with multiple digital cameras in the clustering of digital photos and to effectively classify the photos based on the situation by applying a graph algorithm It is an object of the present invention to provide a method and apparatus for clustering digital photographs taken with a multi-camera.

The present invention constructs a graph based on the feature value extracted from the image of the photo and the feature value extracted from the file information and metadata, and assigns the weight of the edge to the graph with the photo as a vertex, and then applies the graph algorithm to obtain the most connection strength. It is an object of the present invention to provide a method and apparatus for iteratively clustering the divided portions.

The present invention enables the user to set the criteria of the cluster to be classified and the time interval to be allowed to obtain a more suitable cluster picture, and not only to derive the photos collected by several photographers as a single classification result, respectively. The purpose of the present invention is to provide a method and apparatus for clustering digital photographs, which can be classified according to their tastes so that photo management can be performed in a desired manner.

In order to achieve the above object, a method for clustering digital photographs taken by multiple cameras according to the present invention is a method of clustering photographs obtained from a plurality of cameras and photographers. Extracting a value and extracting additional information feature values from metadata and file information of the digital picture; grouping the pictures and structuring them according to the feature value; characterizing the picture or feature value of the group including the corresponding picture Generating a cluster using the same;

Here, the content-based feature value of the picture is characterized by including the color, texture, and shape values extracted by using the pixel information of the picture.

And the content-based feature value of the picture, characterized in that it comprises a statistical analysis value based on the color, texture, shape value extracted from the pixel information.

The statistical analysis value may include histogram information of an image, texture analysis information using a wavelet, and frequency information using a Fourier transform.

The additional information feature may include file information provided by an operating system and a picture information field value provided by metadata such as EXIF or IPTC provided by a digital picture.

In the structuring of the photographs, the photographs may be converted into a graph structure according to a feature value.

In converting the photo into a graph structure, the photographs are taken as vertices by the photographer or the photographed camera, and the neighboring vertices are connected by edges in a chronological order.

In addition, a graph algorithm for calculating the maximum flow is applied to the graph structure in which the pictures are converted to divide the graph with the minimum cut at the edge having the maximum flow.

And it is characterized by dividing the graph until the reference value is satisfied by repeatedly applying the maximum flow calculation by inputting the divided graph.

When the photo vertices are connected by edges, edge weights are set to a similar degree based on feature values between the two photographs.

And when the photo vertices are connected by edges, the pictures taken by the same camera are connected to each other by edges in order of time so that the division does not occur in several places when applying the graph algorithm by applying a certain value other than similarity. do.

In converting the picture into a graph structure, each picture becomes a node of the graph, and in order to form a direction graph that becomes an edge between nodes in consideration of the similarity between the pictures, the picture is divided by roll according to the camera where the picture was taken. Sorting according to the shooting time; Photos in the same roll is the process of connecting the edges with time similarity weights from neighboring nodes in time to the next node in time; individual photos take time and content similarity together Making the edges connect with the closest one of the later shots on different rolls; connecting the start and end nodes to the first and last photo of each roll to flow and accept the flow. The process of; characterized in that it comprises a.

An apparatus for clustering digital photos taken with multiple cameras for achieving another object may include: a content-based feature value extractor for extracting content-based feature values from an image content of a digital photo; An additional information feature value extracting unit for extracting additional information feature values from metadata and file information; a group information generation unit for grouping pictures into a graph structure using image content feature points of the pictures and additional information feature points of file information and metadata; And a graph structure converter for converting; a cluster generator for generating a cluster using feature values of a picture or feature values of a group including the corresponding picture.

Such a method and apparatus for clustering digital photos taken with multiple cameras according to the present invention have the following effects.

First, by structuring pictures taken by multiple digital cameras and applying graph algorithms, pictures can be effectively classified based on context.

That is, digital photographs taken by multiple cameras by multiple photographers can be effectively clustered on a situation-based basis.

Second, digital photographs photographed by multiple cameras by multiple photographers may be converted into a graph structure by using image content feature points of the picture and additional information feature points of file information and metadata.

Therefore, it is possible to effectively classify the pictures taken by a plurality of cameras whose shooting time is not constant using a graph algorithm.

Third, it is possible to automatically calculate a large number of photographed pictures without the user hassle of classifying each one can facilitate user convenience.

Fourth, the user can set the criteria of the cluster to be classified and the time interval to be allowed to obtain a cluster picture more suitable for their purpose.

Therefore, not only the photos collected by the various photographers are derived as a single classification result, but also the photos can be managed according to their desired manner by classifying them individually according to their tastes.

Hereinafter, a preferred embodiment of a method and apparatus for clustering digital photos taken with multiple cameras according to the present invention will be described in detail.

Features and advantages of the method and apparatus for clustering digital photographs taken with multiple cameras according to the present invention will become apparent from the following detailed description of each embodiment.

1 is a time similarity function graph of a digital picture, and FIGS. 2 to 4 are flow networks for digital picture clustering according to the present invention.

5 is a block diagram of an apparatus for clustering digital photos taken with multiple cameras according to the present invention, and FIG. 6 is a flowchart for clustering digital photos taken with multiple cameras according to the present invention.

The present invention enables to classify photographs effectively based on situation by structuring photographs taken by several digital cameras and applying a graph algorithm when clustering digital photographs.

The apparatus for clustering digital photographs taken with multiple cameras according to the present invention includes a content-based feature value extractor 51 for extracting content-based feature values from image contents of a digital photograph as shown in FIG. An additional information feature value extracting unit 52 for extracting additional information feature values from the data and file information, a group information generating unit 53 for grouping pictures by a photographer or a camera used for photographing, and image contents of the obtained picture A cluster structure converting unit 54 converts the graph into a graph structure by using the feature points and additional information feature points of the file information and metadata, and a cluster that generates a cluster using the feature values of the photo or the feature values of the group including the photo. It comprises a generator 55 and is configured.

Hereinafter, a process of clustering photos using the apparatus for clustering digital photos taken with multiple cameras according to the present invention having the above configuration will be described in detail.

The present invention performs the same process as in FIG. 6 to cluster the photos obtained from multiple cameras or multiple photographers.

When the photo input is made large (S601), the content-based feature value is extracted from the image content of the digital picture, and the additional information feature value is extracted from the metadata and file information of the digital picture (S602).

The photographs are grouped by the photographer or the camera used for photographing, and are converted into a graph structure by using the image content feature points, the file information, and the additional information feature points of the metadata (S604).

In order to generate the cluster, the graph algorithm is applied and the graph segmentation is repeated until the reference value is satisfied (S604).

Here, the content-based feature value extracted from the digital photograph includes a numerical value obtained by analyzing the color, texture, and shape of the digital photograph image.

In addition, it may include a histogram statistically analyzing these numerical values, frequency analysis values through Fourier transform, statistical analysis values using wavelets, and the like.

Extracting the additional information feature value from the digital picture includes a file information attribute provided by an operating system, that is, a file creation time, a modification time, a size, a file attribute, and the like.

It also includes metadata information provided by digital photographs, for example, photographing information and content description information about photographs provided by standards such as EXIF and IPTC.

EXIF's information includes information about the camera's settings at the time of shooting, including the time the picture was taken, as well as the aperture setting, shutter speed, focal length, flash setting, and white balance. Can be.

In addition, the additional information feature may include a numerical value obtained by statistically analyzing metadata information. Through the attribute values of the metadata, the numerical value of the result of analogy of the contents of the photograph obtained through probabilistic analysis and machine learning, such as a Bayesian network, may be included in the additional information feature value. These values are applicable as reference values when analyzing digital photographs and comparing image content.

Grouping and structuring the above pictures by cameras is possible by constructing the input pictures in the form of a graph.

The graph algorithm can be applied, and the graph wants to divide the graph into two regions with the minimum cutting edges representing the similarity between the pictures in each graph.

In other words, it finds the parts with the lowest similarity between the pictures and divides them into two pieces, which corresponds to finding the minimum cut in the graph.

The minimum cut problem is the same problem as finding the max flow in the graph.

Therefore, in the present invention, a graph consisting of nodes of each input picture should be configured first in a graph form to which a maximum flow algorithm can be applied.

At maximum flow, the graph can be likened to a network of pipes capable of flowing water, and each directional pipe connection (edge) has throughput and flow values.

Capacity (cap, capacity) is a number that can flow water at the edge, the flow of water below the throughput flowing through each pipe is a flow (flow).

Finding the maximum flow of water when flowing water from the start node (s) to the end node (t) is the same as the minimum cut problem with cuts at the edges that flow the maximum flow. .

The method of organizing pictures in graph form is as follows.

Each picture becomes a node of the graph, and the similarity between the pictures constitutes a direction graph that becomes an edge between the nodes.

(1) First, divide the picture by roll according to the camera where the picture was taken and sort according to the shooting time.

(2) Pictures belonging to the same roll (pictures taken by the same camera) connect edges with time similarity weights between nodes in time with neighboring nodes from the node having the quickest time to the next node.

(3) The individual photos are connected with the edges of the most similar ones of the slower shooting time on different rolls, and the similarity takes into account the time and content similarity.

(4) The start and end nodes for flowing and accepting flows are connected to the first and last pictures of each roll.

The time similarity among the similarities between the pictures in connecting the similarity between the pictures by the weight of the edge is calculated as follows.

라고 정의한다. 이때 u번째 롤에 해당하는 사진은 명시적으로

와 같이 표시한다

.A set of all the photos

It is defined as. In this case, the photo corresponding to the u roll is explicitly

Display as

.

The time similarity ST between the two pictures pi and pj uses a logistic distribution function as shown in Equation 1 so that the similarity value decreases as the time interval increases.

Where μ represents the position of the median, real s greater than zero is the scale, and ts (.) Represents the timestamp of the photo. The output value of the function is from 0 to 1 and has a value of 0.5 when there is a μ time difference. In the present invention, μ is used as a user input value, and scale s is mainly used as μ / 5 value, but may be used as any other value according to the user's intention.

In the temporal similarity comparison, by adjusting the μ value and the s value, it is possible to adjust how much time-lapsed pictures are accepted as the same picture and the degree of change in the similarity value according to time difference.

1 shows an example of a time similarity function.

The content similarity SH between two photographs is obtained through comparison using the feature values of the digital photograph obtained above. The integrated similarity $ S_C $ between the two pictures considers the time similarity and the content similarity as in Equation 2.

The flow throughput setting and the edge connection process of the graph will be described in detail as follows.

In order to reflect the shooting order of the pictures, the process of constructing a graph of neighboring pictures (nodes) belonging to the same role in connecting each node (picture) of the graph G (V, E) is as follows.

Here, the set V of nodes in the graph includes the input pictures, the source (s) and the sink (t) nodes to construct the graph.

First, neighboring nodes connect edges in time, and edge weights (flow throughput, cap) are applied only with time similarity.

This is based on a situation where a photograph taken with a single camera is often similar in meaning to a temporally adjacent photograph, and many previous studies have used this time analysis method [Loui et al. (2003). , Cooper et al (2005). Grapham et al. (2002)].

The following shows the edge connections between neighboring nodes belonging to a certain role u. If the front and back pictures on the same roll are more than the reference value Ts, it is assumed that the same thing is taken in succession, adding a constant value Cs that is large enough to prevent a cut between the two.

를 만족하는 각 사진에 대해서 수학식 3에서와 같이 에지 가중치를 부여한다.

Edge weights are assigned to each photo satisfying the equation as in Equation 3.

Next, as shown in Equation 4, the edges between the pictures of each roll similar to the specific picture are connected. In this case, the two pictures are compared using the integrated similarity $ S_C $.

This is to increase the cohesion between several cameras when the pictures included in different rolls are connected so that the cuts do not pass between the pictures so that they are included in the same cluster.

However, since the network flow graph is a directional graph, the connection direction of the edge is to take a cut in consideration of the shooting time of the photo and to connect the time stamp (ts) from the quick photo to the late photo so as not to cause a cycle on the graph.

Edge concatenation with SC weights in a particular photo ensures that only one of the most similar photographs is attached to each roll. It also links the photos that are on the same roll (u = v).

Now that the connection between the pictures is complete, we need to connect the source (s) and sink (t) nodes to send and receive the flow (s, t ∈ V). The source and sink are connected with the first and last pictures of each roll as shown in FIG.

The minimum cut is obtained by applying the maximum flow algorithm on the flow network configured as described above. In general, the cut is expected as shown in FIG. 2, but depending on the configuration of the graph, the cut is sometimes divided into several parts within one roll as shown in FIG. 3. The case occurs.

Of course, such a cut is possible if the similar pictures are scattered back and forth in the graph of the pictures in the narrow time zone, but if you want to maintain the order in which the pictures are taken in the roll as usual, the graph does not occur as shown. We need to adjust the edge weights of.

One method is to increase the flow through the neighboring nodes in the roll by significantly overlapping edges with constant values Cd from the source to the sink, with sufficient weights in addition to the edge connections between neighbors of the same roll as in Figure 4 Is to prevent the cut from splitting several places.

The application of the maximum flow algorithm in the method for clustering digital photographs taken with multiple cameras according to the present invention will be described below.

In order to obtain the maximum flow in the graph configured as described above, the maximum flow implementation method that is already used may be utilized.

Some implementation methods such as reflowing and gap heuristics are known as goldberg and Tarjan preflow-push methods, and the meaning and operation of the present invention are the same using any implementation method.

The maximum flow algorithm shows the maximum flow amount and the amount of flow for each edge flowing through the network. Using them, you can also find the minimum cut and the nodes split into two based on it.

The maximum flow is divided by the maximum amount of flows that can be entered into the graph to normalize the result to [0, 1].

If the maximum flow result is larger than the user input reference value Tm, the pictures constituting the graph are considered to be the final photo clusters considering that the similarities between each other are large enough, and if the size is smaller than the reference value, the graph is divided into F and R.

Each of the pictures belonging to F and R is separately formed in a flow network graph, and it is repeatedly applied until the maximum flow algorithm satisfies the reference value Tm or there is only one segmented graph remaining outside the s or t nodes.

Photo nodes that remain in the subgraph that are no longer segmented form photo clusters and, when actually displayed to the user, provide them in chronological order or similarity of the contents of the photo or in the user's order.

As described above, the present invention constructs the graph based on the feature value extracted from the image of the picture and the feature value extracted from the file information and metadata, and assigns the weight of the edge to the graph with the picture as a vertex, and then applies the graph algorithm. The clustering is repeated by dividing the portion having the least connection strength.

In addition, by allowing the user to set the criteria of the cluster to be classified and the time interval to allow, it is possible to obtain a cluster picture more suitable for their purpose.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention.

Therefore, the technical scope of the present invention should not be limited to the contents described in the embodiments, but should be defined by the claims.

1 is a time similarity function graph of a digital photograph

2-4 are flow networks for digital photo clustering according to the present invention.

5 is a block diagram of an apparatus for clustering digital photos taken with multiple cameras according to the present invention;

6 is a flow chart for clustering digital photos taken with multiple cameras in accordance with the present invention.

Explanation of symbols for the main parts of the drawings

51. Content-based feature value extractor 52. Additional information feature value extractor

53. Group information generator 54. Graph structure converter

55. Cluster Generator

Claims (13)

In a method for clustering photos obtained from a plurality of cameras, photographers, Extracting content-based feature values from the image content of the digital picture and extracting additional information feature values from metadata and file information of the digital picture; A structured step of grouping photos, using the content-based feature value and the additional information feature value to align pictures in time order by taking photographs as photographic or photographed cameras as vertices and connecting neighboring vertices with edges to convert them into a graph structure. ; And generating a cluster by using the content-based feature value and the additional information feature value of a group including a picture or a corresponding picture. The content-based feature of claim 1, wherein A method for clustering digital photographs taken with multiple cameras comprising color, texture, and shape values extracted using pixel information of a photograph. The content-based feature of claim 1, wherein A method for clustering digital photographs taken with multiple cameras, comprising statistical analysis based on color, texture, and shape values extracted from pixel information. The method of claim 3, wherein the statistical analysis value, A method for clustering digital photographs taken with multiple cameras, comprising histogram information of an image, texture analysis information using wavelets, and frequency information using Fourier transform. The method of claim 1, wherein the additional information feature value, A method for clustering digital photographs taken with multiple cameras, comprising: file information provided by an operating system and picture information field values provided by EXIF or IPTC metadata provided by a digital photograph. delete delete The method of claim 1, wherein a graph algorithm for calculating the maximum flow is applied to the graph structure in which the pictures are converted to divide the graph with the minimum cut at the edge having the maximum flow. Way. 10. The method of claim 8, wherein the divided graph is input and the maximum flow calculation is repeatedly applied to divide the graph until the reference value is satisfied. The method of claim 1, wherein the weight of an edge when connecting the photo vertices by a trunk line is set based on the content-based feature value and the additional information feature value between two pictures. Way. 11. The method of claim 10, wherein when the photo vertices are connected by edges, the photographs taken by the same camera are adjusted by applying a constant value to connect neighboring vertices to the edges in time order to adjust the splitting when the graph algorithm is applied. A method for clustering digital photos taken with multiple cameras. The method of claim 1, wherein in converting the photograph into a graph structure, Dividing the picture by roll according to the camera where the picture was taken and sorting according to the shooting time; A photograph belonging to the same roll is a process of connecting edges having weights based on the photographing time from neighboring nodes to nodes from the node having the fastest photographing time in time; Allowing individual pictures to be edge-connected with one of the photographs having a later recording time on another roll based on time and content extracted using the content-based feature value and the additional information feature value; Connecting the start node and the end node to the first and last pictures of each roll to flow and accept the flow; and constructing a direction graph that is an edge between the nodes, where the nodes of the graph are each picture Method for clustering digital photos taken with multiple cameras, characterized in that. In the clustering device of digital photography, A content-based feature value extraction unit for extracting content-based feature values from the image contents of the digital photograph; An additional information feature value extracting unit for extracting additional information feature values from metadata and file information of the digital photograph; Group information generator for grouping photos Using the content-based feature value of the photo and the additional information feature value of the file information and metadata, the photographs are taken as vertices for each photographer or photographed camera and arranged in chronological order to connect neighboring vertices with edges to form a graph. A graph structure converter to convert; Apparatus for clustering digital photos taken with multiple cameras comprising a; cluster generation unit for generating a cluster by using the content-based feature value of the photo or group, the additional information feature value.

Images