JP2014142810A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To construct a person relation database closer to reality.SOLUTION: An image processing apparatus includes: image grouping means which determine whether a plurality of images belong the same image group on the basis of difference of shooting time of images and difference of person configuration recognized from the images; image-group person relation degree determination means which defines a relation degree between persons in images determined to belong the same image group by the image grouping means, as a first value; and person relation database combining means which combines an image-group person relation degree determined by the image-group person relation degree determination means with the person relation database.

Description

  The present invention relates to a technique for constructing a person relationship from image data including a face.

  In recent years, with the widespread use of digital cameras, it has become possible to take pictures easily, and the number of shots has increased. In addition, a photographed image is shared among family members and friends. As the number of opportunities to shoot increases, the number of images owned increases, and it is becoming an issue to efficiently manage photographs. On the other hand, examples include face detection technology that detects faces in a photo, and face detection technology that recognizes who the detected faces are, grouped by person, and used for image search. Are known.

However, people belong to various communities such as family, relatives, companies, schools, circles, etc., and take pictures in each scene. In addition, there are many people in the image, and in order to share the image efficiently with an appropriate person, it is necessary not only to recognize the person in the image but also to consider human relations. It becomes.
Conventionally, when an image is shared, a person relation database is constructed using face images in the image, and a photo sharing destination is calculated based on the database information (for example, see Patent Document 1). ).

  In addition, there is also known a method of performing image search that matches a user's tendency by providing a new search key by generating a group from information on a person shown in an image and adding a tag.

JP 2011-155385 A

However, in the prior art, all the target images are processed in the same manner. For example, when a plurality of groups of photographers are replaced and continuously photographed, the images are commonly displayed in the plurality of images. The degree of relationship between people is higher than the original. On the other hand, since the persons who took turns are not shown in a common photo, the degree of relationship is not added.
In this way, there is a problem that a personal relationship database different from the original personal relationship is constructed.

  Therefore, an object of the present invention is closer to reality by analyzing the continuity of photographs and analyzing a person relationship with respect to the entire photograph group in the group for photograph groups determined to be the same group. The purpose is to build a personal relationship database.

The image processing apparatus according to the present invention includes an image grouping unit that determines whether or not a plurality of images belong to the same image group based on a difference in photographing date / time of each image and a difference in person configuration recognized from each image. ,
A person relationship degree determining means in an image group having a first value as a degree of relationship between persons appearing in an image group determined to belong to the same image group by the image grouping means;
And a person relation database synthesizing unit that synthesizes the person relation degree in the image group determined by the person relation degree determining unit in the image group with a person relation database.

  According to the present invention, a person relation database closer to reality can be constructed.

FIG. 3 is a block diagram illustrating a configuration of an image processing apparatus according to first and second embodiments. The block diagram which shows the structure of the image processing apparatus which builds the person relationship database which concerns on 1st Embodiment. The flowchart which shows the procedure of the person relationship database creation process which concerns on 1st, 2nd embodiment. FIG. 3 is a diagram illustrating an example of an image according to the first embodiment. 9 is a flowchart showing a procedure of image grouping processing according to the first and second embodiments. 5 is a flowchart showing a procedure of person relationship degree determination processing within an image group according to the first embodiment. The flowchart which shows the procedure of the process which synthesize | combines the person relationship degree in an image group which concerns on 1st, 2nd embodiment to a person relationship database. The figure which shows the conditions of grouping based on 1st Embodiment. The figure which shows the person relationship database calculation result which concerns on 1st Embodiment. The block diagram which shows the structure of the image processing apparatus which builds the person relation database which concerns on 2nd Embodiment. The figure which shows an example of the information of the face image in the image and 3rd image which concern on 2nd Embodiment. 12 is a flowchart showing a procedure of person relationship degree determination processing within an image group according to the second embodiment. The flowchart which shows the procedure of the person grouping process which concerns on 2nd Embodiment. The flowchart which shows the procedure of the relationship degree calculation process between person groups which concerns on 2nd Embodiment. The flowchart which shows the procedure of the relationship degree calculation process between the uncalculated persons which concerns on 2nd Embodiment. The figure which shows the person relationship database calculation result which concerns on 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments do not limit the scope of the claims of the present invention, and all combinations of features described in the following embodiments are indispensable for constituting the present invention. Is not limited.

<First Embodiment>
First, the first embodiment will be described.
FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to the first embodiment. In FIG. 1, a CPU 101 is a system control unit and controls the entire image processing apparatus. The ROM 102 is a read-only memory that stores the control program of the CPU 101 and various fixed data. The RAM 103 is a rewritable memory composed of SRAM, DRAM, and the like, and stores program control variables and the like. Various setting parameters, various work buffers, and the like are also stored in the RAM 103. The display unit 104 notifies the operator of display using an LCD or the like. The operation unit 105 includes a keyboard, a pointing device, and the like, and is used by the operator to perform various input operations. The system bus 106 connects the CPU 101, the ROM 102, the RAM 103, the display unit 104, and the operation unit 105 so that they can communicate with each other.

FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus that constructs the personal relationship database according to the first embodiment. The CPU 101 functions as each unit shown in the block diagram by reading and executing data and programs stored in the ROM 102 and RAM 103.
The image grouping unit 210 determines whether or not a plurality of images belong to the same image group based on the difference in shooting date / time of each image and the difference in person composition recognized from each image.
The person-in-image group degree determining unit 220 uses the first value (in this embodiment, “1”) as the degree of relationship between persons in the image group determined to belong to the same image group by the image grouping unit 210. To do.
The person relation database synthesizing unit 230 synthesizes the person relation degree in the image group determined by the person relation degree determining unit 220 in the image group with the person relation database 240.

  FIG. 3 is a flowchart showing a processing procedure of the image processing apparatus according to the present embodiment, in which the image groups are grouped, and the person relations of the grouped image groups are analyzed and reflected in the person relation database. .

FIG. 4 is a diagram illustrating an example of an image group stored in the RAM 103. In the present embodiment, description will be given using two images (first image 310 and second image 320) shown in FIG.
In the first image, there are four circles B, C, D, and E. This is a simplified representation of the person structure in the photograph, and each alphabet is associated with a person. . That is, there are four persons B, C, D, and E in the first image. Similarly, there are four persons A, C, D, and E in the second image.
Each circle corresponds to the position where the face is shown.
Further, the shooting date and time of the first image is 13:10 on November 25, 2011, and the shooting position indicates that (longitude, latitude) is (139.88, 35.63).
The shooting date and time of the second image is 13:11 on November 25, 2011, and the shooting position indicates that (longitude, latitude) is (139.88, 35.63).

  Hereinafter, the flow of processing for constructing a personal relationship database for the first image and the second image will be described. In the present embodiment, the degree of relationship between persons existing in the same image and the same group is 1, and the sum of the degree of relationship between each image and group is used as a person relationship database. Further, it is assumed that the image groups are arranged in time series in advance, and the processing is also performed in this order.

[Image grouping]
In the flowchart shown in FIG. 3, first, the CPU 101 (image grouping unit 210) performs an image grouping process (step S201).
In this embodiment, image grouping processing is performed according to the flowchart shown in FIG. The CPU 101 repeats the following process for the target image (step S401). In this embodiment, the shooting date and the person configuration are compared as a grouping condition.
The specific conditions are the same as those shown in FIG. 8, and the difference in shooting date and time is less than 5 minutes and the person composition difference is 50% or more compared to the previous image. I will do it.

In accordance with the above conditions, the CPU 101 first performs person recognition processing to acquire person information in the image (step S402).
In this embodiment, it is assumed that face information of a person who may be added to the database is held in advance. Therefore, it is possible to identify which person the face image detected in step S402 is. There are various known techniques for the person recognition technique, and this processing is not limited to the type of person recognition technique. For the first image 310, personal information of four persons B, C, D, and E is extracted.
Subsequently, the CPU 101 adds new person information in the person relation database (person information not yet stored in the person relation database) to the person relation database (step S403).
Since the person relation database is empty at the time when the first image 310 is processed, all persons B, C, D, and E are added to the person relation database.
Subsequently, the CPU 101 determines whether or not the difference between the shooting date and the previous image is within a threshold (step S404).
Since there is no previous image when the first image 310 is read, the determination result is NO.
If NO is determined in step S404, the CPU 101 creates a new group (first group) and adds person information to the group (step S407).
When the first image 310 is processed, four names B, C, D, and E are added. Then, the next second image 320 is read.

Regarding the processing of the second image 320, in step S402, personal information of four persons A, C, D, and E is acquired.
In subsequent step S403, since the new person in the person relation database is A, the new person A is added to the person relation database.
In step S404, since the difference in shooting date and time from the first image 310 is 1 minute, the determination result in step S404 is YES in order to satisfy the condition of less than 5 minutes shown in FIG.
Subsequently, the CPU 101 determines whether or not the difference in person configuration from the previous image is within a threshold value (step S405).
The person composition of the first image 310, which is the previous image, is four persons B, C, D, and E, and the person composition of the second image 320 is four persons A, C, D, and E. Therefore, 3 out of 4 people (75%) are common. Therefore, since the condition that 50% or more people are common as shown in FIG. 8 is satisfied, the determination result in S405 is YES.
Subsequently, the CPU 101 adds a new person in the group to the first group (step S406). Since the new person in the first group is A, A is added to the first group.

  When the first image and the second image are subjected to the image grouping process, five persons A, B, C, D, and E are registered in the person relation database, and A, B, and A are also registered in the first group. The five names C, D, and E are registered. The above is the image grouping process.

In the flowchart shown in FIG. 3, the CPU 101 subsequently repeats the following processing for all groups (step S202).

[Individual relationship determination processing within image group]
First, the CPU 101 (in-image group person relationship determination means 220) performs an in-image group person relationship determination process (step S203). In the present embodiment, processing is performed according to the flowchart shown in FIG. That is, the degree of relationship between the persons in the group is set to 1 (step S504). In the group composed of the first image and the second image, the degree of relationship between AB, between AC, between AD, between AE, between BC, between BD, between BE, between CD, between CE, and between DE Each becomes 1.

[Compositing to Person Relation Database]
Subsequently, the CPU 101 (person relation database synthesizing unit 230) synthesizes the person relation degree in the image group into the person relation database (step S204).
In this embodiment, processing is performed according to the flowchart shown in FIG.
The CPU 101 repeats the following processing for the calculated degree of person relation in the image group (step S601).
The CPU 101 adds the person relation level in the image group to the person relation database (step S602).
In this embodiment, since there is only one group in which the first image and the second image are grouped, the degree of relationship calculated in the above process is added to the person relationship database, and as a result, the first image and the second image are added. It agrees with the degree of group relation by the two images.
The person relationship database can be constructed by the above processing.

The resulting personal relationship database in this embodiment after processing the first image and the second image is as shown in FIG. 9A.
On the other hand, FIG. 9D is a personal relationship database constructed when the same processing as in this embodiment is not grouped. The processing procedure of the method without grouping will be briefly described.
First, at the time after processing the first image, as shown in FIG. 9B, the degree of relationship between B, C, D, and E becomes 1, and these degree of relationship is added to the person relation database. .
In the second image processing, as shown in FIG. 9C, the degree of relationship between A, C, D, and E becomes 1, and this is added to the person relationship database.
As a result, as shown in FIG. 9D, the degree of relationship between A and B that does not appear in the same image is zero. In addition, since C, D, and E appear in both the first image and the second image, the degree of relationship between C, D, and E is 2.
However, the first image and the second image have a short difference in shooting date and time, and there are many common people, so it is highly likely that the people in these images belong to a common group. The degree of relationship is likely to be equal.
Therefore, it is desirable that A, B, C, D, and E have the same degree of relationship 1 as in the person relation database according to the present embodiment.

  According to the present embodiment, a person relation database closer to reality can be constructed by analyzing a person configuration after grouping image groups that satisfy a certain condition.

<Second Embodiment>
Next, a second embodiment will be described.
The image processing apparatus according to the second embodiment is the same in that processing is performed according to the flowchart shown in FIG. Further, the details of the image grouping process S201 and the details of the synthesizing process S204 to the person relation database of the person relation degree in the image group are the same. The difference is in the details of the person relationship degree determination process S203 in the image group.

FIG. 10 is a block diagram illustrating a configuration of an image processing apparatus that constructs a personal relationship database according to the second embodiment. The CPU 101 functions as each unit shown in the block diagram by reading and executing data and programs stored in the ROM 102 and RAM 103.
The in-image group person relationship degree determination means 220 includes a person grouping means 221, a person group relationship degree calculation means 222, and an uncalculated person relationship degree determination means 223.
The person grouping means 221 determines whether or not the person corresponding to each face image belongs to the same person group for the face images recognized from the same image.
In the image group person relationship determination means 220, the person grouping means 221 determines that they belong to the same person group, and sets the person relationship between each face image as the first value (“1” in this embodiment). .
The degree-of-person-group relationship calculation means 222 performs the processing on the face images that are not determined to belong to the same person group by the person grouping means 221.
The face image of the person determined to belong to one person group and the face image of the person determined to belong to the other person group are selected from the two closest face images, and the two selected face images are selected. The degree of relationship between the face images is calculated, and the calculated degree of relationship is set as the degree of relationship between each person belonging to one person group and each person belonging to the other person group.
The uncalculated person relationship determination means 223
The degree of relationship between persons whose relationship degrees have not been calculated because they are recognized from different images belonging to the same image group,
In the case where both share the person whose degree of relationship is the first value in the image in which each person is shown, the degree of relationship between the two is also the first value,
In the case where both persons do not share the person whose degree of relationship is the first value in the image in which each person is shown, the degree of relation between the two is the second value (“0” in this embodiment). And

  In the present embodiment, a description will be given using the third image and the fourth image shown in FIGS. Similar to the images shown in FIGS. 4A and 4B, the circles in FIGS. 11A and 11B represent face images, and the internal alphabet represents a person. In the third image, there are four persons, A, B, C, and E, and in the fourth image, there are three persons B, C, and D. The shooting date / time and shooting position are the same as those of the first image and the second image, respectively.

[Image grouping]
First, the CPU 101 (image grouping unit 210) performs an image grouping process (S201). Similar to the first embodiment, when the processing is performed according to the image grouping process shown in FIG. 5, the third image and the fourth image have a difference in shooting date and time of 1 minute, and the person composition is common to C and D. Therefore, it becomes the same group.
Next, the CPU 101 repeats the following processing for all groups (S202).

[Individual relationship determination processing within image group]
First, the CPU 101 (image group person relationship determination means 220) performs image group person relationship determination processing according to the flowchart shown in FIG.
In the flowchart shown in FIG. 12, the CPU 101 repeats the following processing for all images in the group (S1001).
-Person grouping process First, CPU101 (person grouping means 221) performs a person grouping process. In the present embodiment, the person grouping process is performed according to the flowchart shown in FIG.

In the flowchart shown in FIG. 13, first, the CPU 101 repeats the following processing for all person combinations in the image.
First, the CPU 101 performs person group determination processing based on the size and distance information of face images between persons (S1102). In the present embodiment, the person group is determined by the following processing. For a person M and a person N to be determined, the distance between the center coordinate of the face image in the image of the person M and the center coordinate of the face image in the image of the person N is D, and the radius of the circle that almost includes the face image of the person M Is assumed to be Rm, and the radius of a circle that almost includes the face image of the person N is assumed to be Rn. At this time, the person M and the person N are set to the same group when both of the following two conditions of Expression 1 and Expression 2 are satisfied.

  Expression 1 represents the closeness of the positions of the two face images, and Expression 2 represents the degree of similarity between the sizes of the two face images. That is, when the positions of the two face images are closer than a certain threshold value and the sizes of the two face images are closer than a certain threshold value, the same group is obtained.

  This is the same as determining whether or not the subject is close by the position of the face image and determining how far the subject is from the camera based on the size of the face image. That is, even if the position in the image is close, if the size of the face image is far away, it is determined that the distance from the camera is greatly different and not the same group.

FIG. 11C represents the radius of a circle that includes the face image of each person in the third image, and FIG. 11D represents the distance between the persons in the third image.
When the above conditions are applied to these values, person A and person B become the same person group, person B and person C become the same person group, and E becomes a single group.

Finally, the groups to which the common person belongs are integrated (S1106). By this processing, persons who are close to each other can be made into one group.
Since the person group to which person A and person B belong and the person group to which person B and person C belong are person groups to which common person B belongs, these person groups are integrated, and person A, person B, and person C are combined. It becomes the same person group.
The above is the person grouping process (S1002).

Subsequently, in the flowchart shown in FIG. 12, the CPU 101 sets the degree of relationship between persons belonging to the same person group to 1 (S1003).
In the third image, since person A, person B, and person C belong to the same person group, the degree of relationship between persons A-B, between persons A-C, and between persons B-C is shown in FIG. As shown in FIG.

-Relationship degree calculation process between person groups Subsequently, the CPU 101 (relationship degree calculation means 222 between person groups) performs a relationship degree calculation process between person groups (S1004). This process follows the flowchart shown in FIG.
In the flowchart shown in FIG. 14, first, the CPU 101 repeats the combinations between all person groups in the same image. In the third image, there are a group to which person A, person B, and person C belong and a group to which only person E belongs, so the following processing is performed between these two person groups.
The CPU 101 calculates the degree of relationship between the closest persons in each person group (S1202). According to FIG. 11D, the closest relationship between the group to which the person A, the person B, and the person C belong and the group to which only the person E belongs is between the persons CE, so the relationship between the persons CE. Calculate the degree.
The degree of relationship between persons in the present embodiment is calculated using the following formula 3 or formula 4.

According to this equation, the degree of relationship between the persons CE is 0.33. The calculation process is as follows.
If Rm is the radius of a circle that includes the face image of the person C (= 7), and Rn is the radius of the circle that includes the face image of the person E (= 3), then Rn <Rm. From Expression 3, the degree of relationship between the persons CE = (3/7) * {(3 + 7) / 13} = 0.33
It becomes.
Then, the CPU 101 sets the calculated degree of relation as a person relation degree between person groups (between a person belonging to one person group and a person belonging to the other person group) (S1203).
Therefore, in the third image, as shown in FIG. 16B, the degrees of relationship between the persons CE, the persons A-E, and the persons BE are all 0.33.

The above processing is also performed on the fourth image.
-Person grouping process The person B, the person C, and the person D are reflected in the fourth image, and the circles including the face images partially overlap each other. Since the person group to which the person B and the person C belong and the person group to which the person C and the person D belong are the person group to which the common person C belongs, they are integrated into the same person group (S1106).
-Relationship degree calculation process between person groups Since there is only one person group in the fourth image, the relationship degree calculation process between person groups is not executed.
The degrees of relationship between the persons BC belonging to the same person group, between the persons BD, and between the persons CD are all 1 as shown in FIG. 16C (S1003).

-Relationship calculation process between uncalculated persons Finally, the CPU 101 (uncalculated person relationship determination means 223) determines the relationship between uncalculated persons (S1008).
At the time before this processing is performed, the degree of relationship between the person A and the person D and the person D and the person E that are not shown in the same image has not been calculated.
In the present embodiment, the degree of relationship is calculated under the following conditions.
FIG. 15 shows the flow of processing for calculating the degree of relationship between uncalculated persons.
When calculating the degree of relationship between the person M and the person N, if the person with the degree of relation 1 is shared in the image in which each person is shown, the degree of relation between the two is set to 1. In other cases, the degree of relationship is 0.

When this condition is applied, the degree of relationship between person A and person D is as follows.
The person A has a relevance of 1 with the person B and the person C in the third image in which the person A is shown (YES in step S1302).
The person D has a relationship 1 with the persons B and C in the fourth image in which the person D is captured (YES in step S1303).
That is, the person A and the person D share a person with a relationship degree of 1 (YES in step S1304). Therefore, the degree of relationship between person A and person D is 1 (step S1305).

Next, the degree of relationship between person E and person D is as follows.
There is no person whose degree of relationship with the person E is 1 in the third image in which the person E is shown (NO in step S1302).
Therefore, since the person D and the person E do not share the person with the degree of relationship 1, the degree of relation between the person D and the person E is 0.
By performing the above processing, the personal relationship database calculated from the third image and the fourth image becomes as shown in FIG.

  According to the present embodiment, the degree of relationship between persons can be calculated according to the relationship between the size and position of the face image shown in the image. Thereby, it is possible to reduce the degree of relationship with a person who has been unintentionally captured in a small size, and to construct a person relation database closer to reality.

  Each means and each step constituting the embodiment of the present invention described above can be realized by operating a program stored in a RAM, a ROM, or the like of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

210 Image grouping means 220 Image group person relation degree determining means 221 Person grouping means 222 Person group relation degree calculating means 223 Uncalculated person relation degree determining means 230 Person relation database composition means

Claims (5)

Image grouping means for determining whether or not a plurality of images belong to the same image group based on a difference in photographing date and time of each image and a difference in person composition recognized from each image;
A person relationship degree determining means in an image group having a first value as a degree of relationship between persons appearing in an image group determined to belong to the same image group by the image grouping means;
A person relation database synthesizing means for synthesizing the person relation degree in the image group determined by the person relation degree in the image group into a person relation database;
An image processing apparatus comprising: The image group person relationship determination means further includes a person grouping means, a person group relationship calculation means, and an uncalculated person relationship determination means.
The person grouping means determines whether or not the person corresponding to each face image belongs to the same person group for the face images recognized from the same image,
The image group person relation determining means is determined to belong to the same person group by the person grouping means, and the person relationship between each face image is the first value,
The person group relationship degree calculating means is for the face images that are not determined to belong to the same person group by the person grouping means.
The face image of the person determined to belong to one person group and the face image of the person determined to belong to the other person group are selected from the two closest face images, and the two selected face images are selected. The degree of relationship between face images is calculated, and the calculated degree of relationship is set as the degree of relationship between each person belonging to one person group and each person belonging to the other person group,
The uncalculated person-to-person relationship degree determining means includes:
The degree of relationship between persons whose relationship degrees have not been calculated because they are recognized from different images belonging to the same image group,
In the case where both share the person whose degree of relationship is the first value in the image in which each person is shown, the degree of relationship between the two is also the first value,
2. The image according to claim 1, wherein, in an image in which each person is photographed, when both of them do not share a person whose degree of relation is the first value, the degree of relation between the two is set as a second value. Processing equipment. The person grouping means includes:
The distance between each face image,
Based on the size of each face image, determine whether the person corresponding to each face image belongs to the same person group,
The person group relationship degree calculating means includes:
Selecting a face image of a person determined to belong to one person group and two face images that are closest to each other among face images of persons determined to belong to the other person group;
The two selected face images,
The distance between a face image belonging to one person group and a face image belonging to the other person group,
The size of face images belonging to one person group,
The image processing apparatus according to claim 2, wherein the degree of relationship between the two selected face images is calculated based on the size of the face image belonging to the other person group. An image grouping step for determining whether or not a plurality of images belong to the same image group based on a difference in photographing date and time of each image and a difference in person composition recognized from each image;
A person relationship degree determination step in an image group in which the degree of relationship between persons appearing in an image group determined to belong to the same image group in the image grouping step is a first value;
A person relation database synthesizing step for synthesizing the person relation degree in the image group determined in the person relation degree in the image group in a person relation database;
An image processing method comprising:   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 3.

Images