JP7188478B2 - SEARCH DEVICE, SEARCH METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a search device, search method, and program for searching for a specific person from image data.

There is an application for selling images taken at an event attended by a large number of people using a web-based system. In such applications, it is necessary for the user to check a large number of images in which many people are photographed one by one, and select an image in which a specific person is photographed (hereinafter referred to as a target image). is. However, when the number of images increases, such sorting work becomes troublesome. Therefore, there is a demand for a method of extracting a target image from a large number of images efficiently and with high accuracy.

When extracting an image containing a specific person from a large number of images, it would be possible to automatically extract candidate images containing the target person (hereinafter referred to as candidate images) using face recognition technology, thereby saving the user time and effort. omission. However, when a candidate image is extracted using a general face recognition technique, there is a possibility that the target image will not be extracted. For example, in a person search using face recognition technology, a face that is not facing the front tends to be omitted from the search results because recognition accuracy is low. In addition, it may not be possible to reliably detect the target person due to the effects of brightness and shadows in the photograph. Therefore, there is a demand for a technique for specifying a person using not only an image but also some information other than the image.

Patent Literature 1 discloses a method of using, in addition to visual similarity obtained from images, social connections obtained from various applications (hereinafter referred to as apps) for specific ranking of persons. there is Patent Literature 1 discloses an example of using social connection metrics obtained from a communication application, an SNS (Social Networking Service) application, a calendar application, a joint application, or the like to rank personal identification.

In Patent Document 2, in addition to a person's face image, attributes and classifications such as date of birth, place of work, educational background, hobbies, special skills, circles, etc. are used to estimate the strength of human relationships, and a person who should share content. is disclosed.

Patent Literature 3 discloses a device for specifying a search target person appearing in a specific target image. The device of Patent Document 3 estimates the movement pattern of a person in a plurality of images based on the imaging time and the imaging location of the images, and compares the movement pattern of the person in the image with the imaging time of the specific target image. Based on this, the location where the person is likely to exist is estimated. Then, the apparatus of Patent Document 3 estimates a person who is highly likely to have existed at the imaging location at the imaging time of the specific target image based on the estimation result of the location where the person is likely to be present, and the person is captured in the captured image. The person to be searched is identified by comparing the person to be searched with the person to be searched.

Japanese Patent No. 5557911 Japanese Patent No. 5477017 Japanese Patent No. 6139364

In the method of Patent Document 1, in order to promote identification of a person in an image by using social connection metrics obtained using an application such as an SNS application or a calendar application, an application constructed as such an external system is used. need to work with Therefore, the method of Patent Literature 1 has a problem that a person cannot be specified in an environment in which cooperation with an external system is not possible. In addition, the method of Patent Document 1 does not have sufficient security from the viewpoint of privacy because a person is specified using personal information such as social connection metrics.

In the method of Patent Literature 2, since the strength of human relationships is estimated using information related to attributes and classifications of people, it is not possible to accurately search for human relationships if pre-registered information is incorrect or outdated. I had a problem with the possibility.

In the method of Patent Document 3, in order to specify a search target person in a specific target image, it is necessary to extract a search target person candidate using an image captured before the specific target image. . Therefore, if a person in an image taken before the specific target image faces sideways or diagonally, the person's movement pattern cannot be estimated, and it may not be possible to accurately extract search target person candidates. There was a problem that there is

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-described problems and to provide a search device capable of safely and highly accurately searching for a specific person based on human relationships without cooperating with an external system.

A search device according to one aspect of the present invention includes an acquisition unit that acquires a first image representing a first target and a second image representing a target different from the first target; a judgment unit for judging whether the first object is shown in the image data based on the degree of similarity with the first image and the degree of similarity between the image data and the second image; an output unit for outputting the image data for display.

A search method according to one aspect of the present invention is a search method executed by a computer, in which a first image showing a first object and a second image showing an object different from the first object are obtained, Based on the degree of similarity between the image data to be determined and the first image and the degree of similarity between the image data and the second image, it is determined whether the first object is shown in the image data, Output to display the image data showing the object.

A program according to one aspect of the present invention includes processing for acquiring a first image showing a first target and a second image showing a target different from the first target, image data to be determined, and the first image. and the similarity between the image data and the second image, a process for determining whether the first object is shown in the image data, and the image data in which the first object is shown. The computer is caused to execute a process of outputting for display.

ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the search device which can search a specific person safely and highly accurately based on human relations, without cooperating with an external system.

1 is a block diagram showing an example of the configuration of a search device according to the first embodiment of the present invention; FIG. It is a table which shows an example of the search result table stored in similarity DB of the search device which concerns on the 1st Embodiment of this invention. FIG. 4 is a conceptual diagram showing an example of displaying output data of the search device according to the first embodiment of the present invention on a monitor; FIG. 4 is a conceptual diagram showing another example of displaying output data of the search device according to the first embodiment of the present invention on a monitor; 4 is a flow chart for explaining the operation of the search device according to the first embodiment of the present invention; It is a block diagram which shows an example of a structure of the search device based on the 2nd Embodiment of this invention. FIG. 10 is a conceptual diagram showing an example of displaying candidate images output by the search device according to the second embodiment of the present invention on a monitor; FIG. 11 is a conceptual diagram showing another example of displaying candidate images output by the search device according to the second embodiment of the present invention on a monitor. 9 is a flow chart for explaining the operation of the search device according to the second embodiment of the present invention; It is a block diagram showing an example of hardware constitutions which realize a search device concerning each embodiment of the present invention.

EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated using drawing. However, the embodiments described below are technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In addition, in all the drawings used for the following description of the embodiments, the same symbols are attached to the same portions unless there is a particular reason. Further, in the following embodiments, repeated descriptions of similar configurations and operations may be omitted.

(First embodiment)
First, a search device according to a first embodiment of the present invention will be described with reference to the drawings. The search device of the present embodiment uses images and similarities stored in a database (DB) to search for image data showing a person to be searched from at least one selection target image data.

(Constitution)
FIG. 1 is a block diagram showing an example of the configuration of a search device 10 of this embodiment. As shown in FIG. 1, the search device 10 includes a face recognition unit 11, an image DB 12, a similarity calculation unit 13, a similarity DB 14, and a search result output unit 15. The search device 10 is connected to an input device 110 and an output device 150 .

The search device 10 uses the target image 111 and the auxiliary image group 112 input from the input device 110 to search the image data stored in the image DB 12 for image data showing the person to be searched. The search device 10 generates output data (also referred to as search results) including searched image data, and outputs an output data group 151 that is a set of generated output data to the output device 150 .

The input device 110 is connected to the face recognition section 11 . The input device 110 is input by the user with a target image 111 that is an image representing a person to be searched, and an auxiliary image group 112 that is a set of at least one auxiliary image that assists the search for the target image 111 . The input device 110 outputs the input target image 111 and auxiliary image group 112 to the face recognition section 11 . The auxiliary image is an image showing a person related to the person to be searched. For example, the input device 110 is implemented by a terminal device having a monitor and peripheral devices such as a keyboard, mouse, and touch panel connected to the terminal device.

The face recognition section 11 is connected to the input device 110 . The face recognition unit 11 is also connected to the image DB 12 and the similarity calculation unit 13 . A target image 111 and a group of auxiliary images 112 are input from the input device 110 to the face recognition unit 11 .

When the target image 111 and the auxiliary image group 112 are input, the face recognition unit 11 sequentially extracts at least one selection target image data from the image DB 12 . The face recognition unit 11 detects faces from each image data extracted from the image DB 12 . The face recognition unit 11 calculates the matching degree between the face detected from the image data, the target image 111 and the auxiliary image group 112 . The face recognition unit 11 outputs to the similarity calculation unit 13 a calculation result (also referred to as an evaluation result) of the degree of matching between the face detected from the image data, the target image 111 and the group of auxiliary images 112 . For face detection and face recognition, general techniques may be used.

For example, the user uses the input device 110 to input a person image of a person to be searched as the target image 111, and inputs at least one person image to assist the search to the search device 10 as an auxiliary image. For example, assuming a site that sells school excursion photos, the image of the user's child is input to the search device 10 as the target image 111, and the images of the friend who was in the same group as the child are the auxiliary images. is entered as

An image DB 12 (also referred to as a first database) is connected to the face recognition section 11 , similarity calculation section 13 and search result output section 15 . The image DB 12 is a database in which image data to be selected is stored. For example, when the search device 10 is used to sell photographs taken on a school excursion, image data of images taken on the excursion are accumulated in the image DB 12 as image data to be selected.

For example, the image DB 12 stores image data to be selected and shooting conditions (metadata) under which the image data was shot in association with each other. For example, the image DB 12 stores image data in Exif (Exchangeable image file format) format. For example, the metadata of the image data to be selected includes the shooting date and time when the image data was shot and GPS (Global Positioning System) information corresponding to the position information of the shooting position.

The similarity calculation unit 13 is connected to the face recognition unit 11, the image DB 12, and the similarity DB . The similarity calculation unit 13 acquires the evaluation result of face recognition from the face recognition unit 11 . When obtaining the face recognition evaluation result, the similarity calculation unit 13 uses the metadata of the image data stored in the image DB 12 to determine whether or not the person to be searched appears in the image data. The similarity calculator 13 stores the determination result in the similarity DB 14 .

A similarity DB 14 (also referred to as a second database) is connected to the similarity calculation section 13 and the search result output section 15 . The similarity DB 14 is a database in which similarity determination results obtained by the similarity calculation unit 13 are stored.

The search result output unit 15 is connected to the image DB 12 and the similarity DB 14 . The search result output unit 15 acquires the determination result by the similarity calculation unit 13 from the similarity DB 14 . The search result output unit 15 acquires image data from the image DB 12 based on the determination result acquired from the similarity DB 14 . Then, the search result output unit 15 acquires image data corresponding to the determination result acquired from the similarity DB 14 from the image DB 12 . The search result output unit 15 uses the image data acquired from the image DB 12 to generate output data (also referred to as search results) in a format that can be processed by the output device 150 . Note that the search result output unit 15 may include metadata of the image data corresponding to the determination result in the output data. The search result output unit 15 outputs an output data group 151 that is a set of generated output data to the output device 150 .

The output device 150 is connected to the search result output section 15 . An output data group 151 is input from the search result output unit 15 to the output device 150 . The output device 150 is a device for presenting search results of the search device 10 to the user. For example, output device 150 is implemented by a display device having a display. When the output device 150 is implemented by a display device, the output device 150 displays the search results of the search device 10 on its own monitor. Also, for example, the output device 150 is realized by a printer. When the output device 150 is implemented by a printer, the output device 150 prints the search results of the search device 10 on paper. Note that the method of outputting the search results of the search device 10 is not particularly limited as long as the user can confirm the search results.

The above is the description of the configuration of the search device 10 of the present embodiment. Note that the configuration of the search device 10 in FIG. 1 is an example, and the configuration of the search device 10 of this embodiment is not limited to the form as it is.

〔judgment result〕
Next, an example of the determination result stored in the similarity DB 14 will be described. FIG. 2 is a determination result table 140 as an example of determination results stored in the similarity DB 14 . The determination result table 140 stores, for example, determination results regarding n pieces of image data (images 1 to n) to be selected.

In the determination result table 140, each column of the target image 111 and the auxiliary image group 112 (auxiliary image 1, auxiliary image 2, auxiliary image 3) indicates the degree of matching between the target image 111 and the auxiliary image group 112 and each image data. This is the calculation result.

In the determination result table 140, the time information column indicates information about the shooting time of the image data included in the metadata of the image data to be selected. In the determination result table 140, T1, T1', and T1'' are substantially the same time, and other times are separated from each other.

In the determination result table 140, each column of position information indicates information about the shooting location (geotag, etc.) of the image data included in the metadata of the image data to be selected. In the determination result table 140, P1 and P1' are at substantially the same position, and the other positions are separated from each other.

In the determination result table 140, the similarity determination column indicates the determination result as to whether or not the person to be searched appears in each image based on the degree of matching and metadata. The similarity determination indicates the degree of certainty that the person to be searched appears in the image data. The determination result table 140 shows an example in which the determination result is determined in three levels of "high", "middle", and "low".

Regarding the image 1, referring to the determination result table 140, the degree of matching with the target image 111 is 90% (percentage). Regarding image 1, it is determined that there is a high possibility that the person to be searched is shown, and the similarity determination is set to "high".

Regarding the image 2, referring to the determination result table 140, the degree of matching with the target image 111 is 70%. As for image 2, the degree of matching is relatively high at 70%. The probability is determined to be low and the similarity determination is set to "low".

Regarding the image 3, referring to the determination result table 140, the degree of similarity with the target image 111 is 50%. Image 3 has a low matching rate of 50%, but is captured at approximately the same time and at approximately the same position as image 1 . Also, a person having a high degree of similarity with the auxiliary images 1 to 3 is photographed at the same time. For example, in the case of images taken on an excursion of elementary school students, if the group of children to be searched is active at the relevant position at the relevant time, there is a high possibility that they are shown together in the same image. For such a reason, regarding image 3, the similarity determination of the image data is set to "middle".

Regarding the image n, referring to the determination result table 140, the degree of matching with the auxiliary images 2 and 3 is low. is judged to be Therefore, regarding image n, the degree of matching with the auxiliary image does not affect the similarity determination, and the degree of matching with the target image is as high as 80%, so the similarity determination is set to "high".

The search result output unit 15 refers to the similarity determinations stored in the similarity DB 14 and retrieves image data with similarity determinations of “high” and “medium” from the image DB 12 . The search result output unit 15 uses the image data extracted from the image DB 12 to generate output data that can be displayed on the monitor of the output device 150 . The search result output unit 15 outputs the generated output data to the output device 150 .

For example, the search result output unit 15 generates thumbnails of images whose similarity is judged to be "high" and "medium", and output data in html format, which enables display of full-size images from the thumbnails. Note that the full-size image referred to by the thumbnail may be an enlarged image that can be determined by the user, rather than the actual full-size image.

The output device 150 acquires output data from the search result output unit 15 . The output device 150 displays the search result of the search device 10 on its own monitor. For example, the output device 150 displays images with similarity determinations of “high” and “medium” in different display areas. A user viewing the monitor of the output device 150 can check the images according to the similarity determination, so that the user is relieved from the burden of checking all the images to be selected. In addition, if the classification of similarity judgment is subdivided into "high" and "medium", the user may feel a new burden of distinguishing between images with "high" similarity judgment and images with "medium" similarity judgment. There is also sex. Therefore, the similarity determination of the images to be confirmed by the user may be set to one level instead of two levels of "high" and "medium". On the other hand, there may be situations where it is better to subdivide the similarity determination categories in more detail. In that case, the classification for similarity determination may be set in more detail. That is, the similarity determination classification can be adjusted to an appropriate classification according to the application.

3 and 4 show an example of displaying a user interface 155 (also referred to as a first user interface) for selecting retrieved image data on the monitor of the output device 150 according to similarity determination. In FIGS. 3 and 4, a user interface 155 is displayed on the monitor for allowing the user to select image data that the user wishes to purchase by dividing the image data group into a group of image data with a "high" degree of similarity determination and a group of image data with a "medium" degree of similarity determination. Here is an example of what to do.

As an example, the user interface 155 in FIG. 3 enlarges and displays image data with a similarity determination of “high”. In FIG. 3, as an example, it is assumed that seven pieces of image data with a similarity determination of "high" are selected, and image data with a similarity determination of "medium" are not selected. If image data can be purchased for 100 yen per image, the total number of selected images is 7, so the purchase price is displayed as 700 yen.

As an example, the user interface 155 in FIG. 4 enlarges and displays the image data with the degree of similarity determination of "medium". In FIG. 4, as an example, it is assumed that three pieces of image data with a "medium" similarity judgment have been selected, and seven pieces of image data with a "high" similarity judgment have already been selected. If image data can be purchased for 100 yen per image, the total number of selected images is 10, so the purchase price is displayed as 1000 yen.

FIGS. 2 to 4 above show an example in which the similarity DB 14 stores three levels of similarity judgments of “high”, “middle”, and “low” for all image data. However, the similarity determination may be subdivided by percentage or the like instead of stepwise determination. By subdividing the similarity determination, the search accuracy can be set more finely.

For example, the search device 10 can be configured as a system local to the terminal device operated by the user. Further, as an example, the input of the target image 111 and the group of auxiliary images 112 and the display of the search results can be configured to be performed by a terminal device such as a personal computer or smartphone operated by the user. Further, the search device 10 may be configured as a computer resource connected via the Internet to a terminal device operated by a user.

(motion)
Next, the operation of the search device 10 of this embodiment will be described with reference to the drawings. FIG. 5 is a flowchart for explaining the operation of the search device 10. As shown in FIG. In the description according to the flowchart of FIG. 5, the search device 10 is the subject of the operation.

In FIG. 5, the search device 10 first acquires the target image 111 and the auxiliary image group 112 input from the input device 110 (step S11).

Next, the search device 10 retrieves image data to be selected from the image DB 12 (step S12). Note that when processing a plurality of image data collectively, the plurality of image data may be collectively extracted from the image DB 12 .

Next, the search device 10 detects a face from the image data extracted from the image DB 12 (step S13). When processing a plurality of selection targets collectively, faces may be collectively detected from a plurality of image data extracted from the image DB 12 .

Next, the search device 10 calculates the matching degree between the face detected from the image data to be selected, the target image 111 and the auxiliary image group 112 (step S14). Note that when a plurality of images are collectively processed, the degrees of matching between faces detected from a plurality of image data, the target image 111 and the auxiliary image group 112 may be calculated collectively.

Next, the search device 10 uses the calculated degree of matching and the metadata of the image data stored in the image DB 12 to determine whether or not the person to be searched appears in the image data (step S15). When determining whether or not a person to be searched appears in image data to be selected, if the metadata of the image data is not used, the search device 10 uses the calculated degree of matching for determination.

Next, the search device 10 stores the determination result as to whether or not the person to be searched appears in the selected image data in the similarity DB 14 (step S16).

If image data to be searched remains (Yes in step S17), the process returns to step S13. On the other hand, if image data to be selected does not remain (No in step S17), image data is obtained from the image DB 12 based on the determination result stored in the similarity DB 14 (step S18). Then, the search device 10 generates output data using the image data acquired from the image DB 12, and outputs the generated output data to the output device 150 (step S19).

The above is the description of the operation of the search device 10 according to the flowchart of FIG. The operation of the search device 10 according to the flowchart of FIG. 5 is an example, and the operation of the search device 10 of this embodiment is not limited to the procedure as it is.

As described above, the search device of this embodiment includes a first database, a face recognition section, a similarity calculation section, a second database, and a search result output section.

The first database stores at least one selection target image data. The face recognition unit detects a face from the image data to be selected stored in the first database. The face recognition unit detects a face detected from image data of a selection target when a target image indicating a person to be searched and an auxiliary image group including at least one auxiliary image for assisting the search for the person to be searched are designated. is included in the target image and the auxiliary image group. The similarity calculation unit determines whether the person to be searched appears in the image data using the evaluation result by the face recognition unit. The second database stores the determination result by the similarity calculation unit. A search result output unit obtains from the first database image data in which the person to be searched is determined to appear based on the determination result stored in the second database, and outputs data including the obtained image data. is generated and output.

As one aspect of the present embodiment, an image representing a person related to a person to be searched is input as an auxiliary image to the face recognition unit.

As one aspect of the present embodiment, the face recognition unit calculates the degree of matching between the face detected from the image data and the human face included in the target image and the auxiliary image group. The similarity calculation unit determines whether or not the person to be searched appears in the image data based on the degree of matching calculated for the target image and the group of auxiliary images.

As one aspect of the present embodiment, metadata of image data to be selected is stored in the first database. The similarity calculation unit acquires metadata of image data to be selected from the first database. The similarity calculation unit determines whether the person to be searched appears in the image data based on the acquired metadata of the image data to be selected and the degree of matching calculated for the target image and the group of auxiliary images.

As one aspect of the present embodiment, the similarity calculation unit calculates the probability that the person to be searched is shown based on the metadata of the image data to be selected and the degree of matching calculated for the target image and the group of auxiliary images. The similarity determination shown is given to the image data.

As one aspect of this embodiment, the search result output unit is connected to an output device having a monitor. The search result output unit outputs the output data to the output device. The output device causes the monitor to display a first user interface for selecting image data included in the output data.

The search device of this embodiment receives a target image, which is an image representing a person to be searched, and an auxiliary image group including an auxiliary image, which is an image of at least one person who assists the search for the target image. For example, a face image of a user's child is input to the search device as a target image, and a face image of the child's friend is input as an auxiliary image. When a target image and a group of auxiliary images are input, the search device of this embodiment performs face recognition on image data to be searched stored in the first database. The search device of the present embodiment uses the evaluation result of face recognition and the metadata of the image data of the search target image to determine whether or not the search target person appears in the image data based on the similarity evaluation. do.

For example, in a case where a Web-based system is used to sell excursion photos of children, there is a high possibility that the child, who is the person to be searched, will be photographed together with related persons such as close friends and group members. According to the present embodiment, even for image data in which a person to be searched is shown but whose face recognition matching degree is low, it is determined whether or not the person to be searched is shown based on the matching degree of face recognition of related persons. can be verified.

Further, according to the present embodiment, it is possible to verify whether or not the person to be searched appears in the image data including the meta information (time information and position information) of the image data. For example, the person to be searched does not appear in a plurality of image data shot at different locations at the same time. Therefore, any one of a plurality of image data shot at different locations at the same time can be excluded from the search results.

In this embodiment, the determination results of these evaluations are accumulated in the second database. According to this embodiment, an image to be confirmed by the user can be selected by selecting and outputting an image in which the person to be searched is shown from the image database according to the determination result accumulated in the second database. .

When extracting image data in which a search target person is shown from a large number of image data, the search device of the present embodiment uses determination results indicating the presence or absence of related persons of the search target person, time information, position information, and the like. Use metadata from Therefore, according to the search device of the present embodiment, it is possible to improve the search accuracy as compared with a search using only face recognition of a person to be searched.

For example, when purchasing an image taken at a school event in which the user's child participated, even if the child's face is not facing the front, the user may want to purchase the image. is. In this embodiment, not only the face image recognition result of the search target person, but also the search result of the related person who is highly likely to act together with the search target person, and the action history (time information and position information) are combined. to refer to. Therefore, according to the present embodiment, it is possible to highly accurately eliminate the possibility of overlooking an image in which a person to be searched is included, and to provide more accurate search results.

In addition, due to the social situation where privacy protection regulations are becoming stricter, there is a limit to improving search accuracy by using social graphs that use external systems such as social network services and shared calendar applications. be. According to this embodiment, since it is possible to improve the accuracy of searching human relations in a closed environment without requiring cooperation with an external system, it is safe from the viewpoint of privacy protection.

That is, according to the search device of the present embodiment, it is possible to search for a specific person safely and with high accuracy based on human relationships without cooperating with an external system.

(Second embodiment)
Next, a search device according to a second embodiment of the present invention will be described with reference to the drawings. The present embodiment is the first embodiment in that candidate images (also referred to as candidate images) of the target image and the auxiliary image are generated from the image data stored in the image DB, and the generated candidate images are proposed to the user. different from the form.

FIG. 6 is a block diagram showing an example of the configuration of the search device 20 of this embodiment. As shown in FIG. 6, the search device 20 includes a face recognition unit 21, an image DB 22, a similarity calculation unit 23, a similarity DB 24, a search result output unit 25, and a candidate image proposal unit . The search device 20 is connected to an input device 210 and an output device 250 .

The search device 20 detects a face from at least one selection target image data stored in the image DB 22 . The search device 20 outputs an image including the detected face to the input device 210 as a candidate image for a target image or an auxiliary image. The user selects a target image or an auxiliary image from the candidate images displayed on the input device 210, and designates the selected target image or auxiliary image. The search device 20 uses the target image 211 and the auxiliary image group 212 designated by the user via the input device 210 to search for image data in which the search target person is shown from among the image data stored in the image DB 22. do. The search device 20 generates output data including the searched image data, and outputs an output data group 251 that is a set of generated output data to the output device 250 .

The input device 210 is connected to the face recognition section 21 and the candidate image proposal section 26 . Candidate images for the target image and the auxiliary image are input to the input device 210 from the candidate image proposing section 26 . The input device 210 causes the monitor to display a user interface for selecting the target image 211 and the auxiliary image from among the input candidate images. The input device 210 designates the candidate image selected via the user interface as the target image 211 or the auxiliary image group 212 . At this time, the input device 210 outputs information indicating which candidate image is designated as the target image 211 or the auxiliary image group 212 to the search device 20 . Alternatively, the input device 210 may output image data designated as the target image 211 and the auxiliary image group 212 to the search device 20 .

FIG. 7 is a conceptual diagram showing an example of a user interface 215 (also referred to as a second user interface) for presenting candidate images to the user on the monitor of the input device 210. As shown in FIG. A plurality of candidate images are displayed on the user interface 215 of FIG. The user interface 215 displays check boxes associated with each candidate image for selecting a target image or an auxiliary image.

FIG. 8 shows an example of a state after the user who checked the user interface 215 of FIG. 7 selects the target image and the auxiliary image. In FIG. 8, as an example, No. 2 candidate image is selected as the target image 211, and No. 2 is selected as the target image 211; Candidate images 1, 3 and 7 are selected as auxiliary images. The input device 210 designates the candidate image selected on the user interface 215 as the auxiliary image forming the target image 211 or the auxiliary image group 212 .

The candidate image proposing unit 26 acquires from the face recognition unit 21 an image including the face detected from the image data to be selected. The candidate image proposing unit 26 generates candidate images for the search target image and the auxiliary image using the acquired images. The candidate image proposing section 26 outputs the generated candidate images to the input device 210 .

The face recognition section 21 is connected to the input device 210 . The face recognition unit 21 is also connected to the image DB 22 , the similarity calculation unit 23 and the candidate image proposal unit 26 .

The face recognition unit 21 acquires at least one selection target image data stored in the image DB 22 and detects a face from the acquired image data. The face recognition unit 21 outputs an image including the face detected from the image data to be selected to the candidate image proposal unit 26 . The face recognition unit 21 may store the image including the face detected from the image data to be selected in the image DB 22, or may temporarily delete the image.

Information indicating which candidate images have been designated as the target image 211 and the auxiliary image group 212 is input from the input device 210 to the face recognition unit 21 . When the target image 211 and the auxiliary image group 212 are designated, the face recognition unit 21 calculates the degree of matching between the face detected from the image data to be selected and the target image 211 and the auxiliary image group 212 . The face recognition unit 21 outputs to the similarity calculation unit 23 the calculation result (also referred to as the evaluation result) of the degree of matching between the face detected from the image data to be selected, the target image 211 and the auxiliary image group 212 .

An image DB 22 (also referred to as a first database) is connected to the face recognition section 21 , similarity calculation section 23 and search result output section 25 . The image DB 22 is a database in which image data to be selected is stored.

The similarity calculator 23 is connected to the face recognition unit 21, the image DB 22, and the similarity DB 24. FIG. The similarity calculation unit 23 acquires the evaluation result of face recognition from the face recognition unit 21 . When obtaining the face recognition evaluation result, the similarity calculation unit 23 uses the metadata of the image data stored in the image DB 22 to determine whether or not the person to be searched appears in the image data. The similarity calculator 23 stores the determination result in the similarity DB 24 .

A similarity DB 24 (also referred to as a second database) is connected to the similarity calculation section 23 and the search result output section 25 . The similarity DB 24 is a database in which similarity determination results obtained by the similarity calculation unit 23 are stored.

The search result output unit 25 is connected to the image DB 22 and the similarity DB 24 . The search result output unit 25 acquires the determination result by the similarity calculation unit 23 from the similarity DB 24 . The search result output unit 25 acquires image data from the image DB 22 based on the determination result acquired from the similarity DB 24 . Then, the search result output unit 25 acquires image data corresponding to the determination result acquired from the similarity DB 24 from the image DB 22 . The search result output unit 25 uses the image data acquired from the image DB 22 to generate output data (also referred to as search results) in a format that can be processed by the output device 230 . Note that the search result output unit 25 may include metadata of the image data corresponding to the determination result in the output data. The search result output unit 25 outputs an output data group 231 that is a set of generated output data to the output device 230 .

The output device 230 is connected to the search result output section 25 . An output data group 231 is input from the search result output unit 25 to the output device 230 . The output device 230 is a device for presenting search results of the search device 20 to the user. Note that the method of outputting the search results of the search device 20 is not particularly limited as long as the user can confirm the search results.

The above is the description of the configuration of the search device 20 of the present embodiment. The configuration of the search device 20 in FIG. 6 is an example, and the configuration of the search device 20 of this embodiment is not limited to the form as it is.

(motion)
Next, the operation of the search device 20 of this embodiment will be described with reference to the drawings. FIG. 9 is a flow chart for explaining the operation of the search device 20. As shown in FIG. In the description according to the flowchart of FIG. 9, the search device 20 is the subject of the operation.

In FIG. 9, first, the search device 20 generates candidate images from the image data to be selected stored in the image DB 22 (step S21).

Next, the search device 20 outputs the generated candidate images to the input device 210 (step S22).

Next, the search device 20 obtains from the input device 210 information indicating which candidate images have been designated as the target image 211 and the auxiliary image group 212 (step S23).

Next, the search device 20 calculates the degree of matching between the face detected from the image data to be selected, the target image 111 and the auxiliary image group 112 (step S24).

Next, the search device 20 uses the calculated degree of matching and the metadata of the image data stored in the image DB 22 to determine whether or not the person to be searched appears in the image data (step S25). When determining whether or not a person to be searched appears in image data to be selected, if the metadata of the image data is not used, the search device 20 uses the calculated degree of matching for determination.

Next, the search device 20 stores the determination result as to whether or not the person to be searched appears in the selected image data in the similarity DB 14 (step S26).

If image data to be searched remains (Yes in step S27), the process returns to step S24. On the other hand, if there is no image data to be selected (No in step S27), image data is obtained from the image DB 22 based on the determination result stored in the similarity DB 24 (step S28). Then, the search device 20 generates output data using the image data acquired from the image DB 22, and outputs the generated output data to the output device 230 (step S29).

The above is the description of the operation of the search device 20 according to the flowchart of FIG. The operation of the search device 20 according to the flowchart of FIG. 9 is an example, and the operation of the search device 20 of this embodiment is not limited to the procedure as it is.

As described above, the search device of this embodiment includes a candidate image proposal section in addition to the first database, face recognition section, similarity calculation section, second database, and search result output section.

The face recognition unit acquires at least one piece of image data stored in the first database, detects a face from the acquired image data, and outputs image data including the detected face to the candidate image proposal unit. The candidate image proposing unit generates candidate images for the target image and the auxiliary image using the image including the face detected by the face recognition unit, and outputs the generated candidate images.

As one form of this embodiment, the candidate image proposing unit is connected to an input device having a monitor and outputs candidate images to the input device. The input device causes the monitor to display a second user interface for selecting the target image and the auxiliary image from among the candidate images. The input device designates each of the candidate images selected via the second user interface as either the target image or the auxiliary image, and outputs information regarding the designated target image and auxiliary image to the face recognition unit.

In general, there are many cases where image data of the user's child is at hand, but appropriate image data of a friend to be used as an auxiliary image is not at hand. In this embodiment, candidate images generated using face images detected from image data to be selected stored in the first database are presented to the user. is selected by the user. As a result, the user can save the trouble of procuring an auxiliary image that is not at hand.

For example, the search device of the present embodiment extracts a plurality of person images assumed to be different persons from the first database, and outputs and displays thumbnails and full-size images of those person images as data in html format as links. . The user looks at the image displayed on the monitor, and clicks or taps a picture of the person to be searched or a person related to the person to select. At this time, if there are a large number of target persons, the candidate images are displayed on a plurality of pages. For example, by preferentially displaying a person image determined to be closely related by similarity calculation, it is possible to reduce the user's operation burden.

The method of each embodiment of the present invention involves a relatively closed interpersonal relationship, as in the case of selling photographs taken at school events such as excursions and school performances through a web-based system, but the search target is a large number of people. It is effective for person image retrieval ranging from

For example, it takes a lot of effort to search for a target image in which a specific person is shown while referring to a large number of image data in which an unintended person is shown one by one. Using the face recognition technology saves labor as compared to retrieving a target image while referring to all image data one by one. However, with the current detection accuracy of face recognition technology, it is not possible to accurately detect a target image when a specific person is facing sideways or obliquely.

Adults who have accumulated human relationships and action histories can search for a target image using human relationships and action histories extracted from social network services, shared calendar applications, and the like, so the search accuracy can be improved. However, when retrieving an image of a child whose human relationship or action history has not been accumulated, the use of a social network service, a shared calendar application, or the like does not necessarily improve the retrieval accuracy of the target image. In addition, due to the social situation where privacy protection regulations are becoming stricter, it is becoming difficult to publicly use information such as social network services and shared calendar applications.

Further, according to the method of each embodiment of the present invention, the face of the person to be searched and the faces of related persons of the person to be searched are used without using information from an external system such as a social network service or a shared calendar application. Target images can be searched.

That is, according to the search device of the present embodiment, it is possible to search for a specific person safely and with high accuracy based on human relationships without cooperating with an external system.

(hardware)
Here, the hardware configuration for executing the processing of the search device according to each embodiment of the present invention will be described by taking the information processing device 90 of FIG. 10 as an example. Note that the information processing device 90 of FIG. 10 is a configuration example for executing the processing of the search device of each embodiment, and does not limit the scope of the present invention.

As shown in FIG. 10, an information processing device 90 includes a processor 91 , a main memory device 92 , an auxiliary memory device 93 , an input/output interface 95 and a communication interface 96 . In FIG. 10, the interface is abbreviated as I/F (Interface). Processor 91 , main storage device 92 , auxiliary storage device 93 , input/output interface 95 and communication interface 96 are connected to each other via bus 99 so as to enable data communication. Also, the processor 91 , the main storage device 92 , the auxiliary storage device 93 and the input/output interface 95 are connected to a network such as the Internet or an intranet via a communication interface 96 .

The processor 91 expands a program stored in the auxiliary storage device 93 or the like into the main storage device 92 and executes the expanded program. In this embodiment, a configuration using a software program installed in the information processing device 90 may be used. The processor 91 executes processing by the search device according to this embodiment.

The main memory 92 has an area in which programs are expanded. The main memory device 92 may be a volatile memory such as a DRAM (Dynamic Random Access Memory). Also, a non-volatile memory such as MRAM (Magnetoresistive Random Access Memory) may be configured and added as the main storage device 92 .

The auxiliary storage device 93 stores various data. The auxiliary storage device 93 is configured by a local disk such as a hard disk or flash memory. It should be noted that it is possible to store various data in the main storage device 92 and omit the auxiliary storage device 93 .

The input/output interface 95 is an interface for connecting the information processing device 90 and peripheral devices. A communication interface 96 is an interface for connecting to an external system or device through a network such as the Internet or an intranet based on standards and specifications. The input/output interface 95 and the communication interface 96 may be shared as an interface for connecting with external devices.

The information processing apparatus 90 may be configured to connect input devices such as a keyboard, mouse, and touch panel as necessary. These input devices are used to enter information and settings. Note that when a touch panel is used as an input device, the display screen of the display device may also serve as an interface of the input device. Data communication between the processor 91 and the input device may be mediated by the input/output interface 95 .

Further, the information processing device 90 may be equipped with a display device for displaying information. When a display device is provided, the information processing device 90 is preferably provided with a display control device (not shown) for controlling the display of the display device. The display device may be connected to the information processing device 90 via the input/output interface 95 .

Further, the information processing device 90 may be equipped with a disk drive, if necessary. Disk drives are connected to bus 99 . Between the processor 91 and a recording medium (program recording medium) not shown, the disk drive mediates reading of data programs from the recording medium and writing of processing results of the information processing device 90 to the recording medium. The recording medium can be implemented by, for example, an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may be a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card, a magnetic recording medium such as a flexible disk, or other recording medium.

The above is an example of the hardware configuration for enabling the search device according to each embodiment of the present invention. Note that the hardware configuration of FIG. 10 is an example of a hardware configuration for executing arithmetic processing of the search device according to each embodiment, and does not limit the scope of the present invention. The scope of the present invention also includes a program that causes a computer to execute processing related to a search device according to each embodiment. Further, the scope of the present invention also includes a program recording medium on which the program according to each embodiment is recorded.

The components of the search device of each embodiment can be combined arbitrarily. Also, the constituent elements of the search device of each embodiment may be implemented by software or may be implemented by a circuit.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

10, 20 search device 11, 21 face recognition section 12, 22 image DB
13, 23 similarity calculator 14, 24 similarity DB
15, 25 search result output unit 26 candidate image proposal unit 110, 210 input device 150, 250 output device

Claims (12)

acquisition means for acquiring a first image showing a first object that is a person to be detected and a second image showing a second object related to the first object;
Determining whether or not the first object is shown in the image data based on the degree of similarity between the image data to be determined and the first image and the degree of similarity between the image data and the second image. means and
output means for outputting to display the image data showing the first object;
candidate image proposal means for outputting candidate images of the first image and the second image;
A search device comprising: The acquisition means is
2. The search device according to claim 1, wherein an image showing a related person of said first target is obtained as said second image. The determination means is
calculating the degree of matching between the face detected from the image data and the human face included in the first image and the second image;
Determining whether the first object appears in the image data based on the degree of matching calculated for the first image ;
3. The search device according to claim 1 , wherein it is determined whether or not the second object appears in the image data based on the degree of matching calculated for the second image . The acquisition means is
storing metadata for the image data;
The determination means is
4. The search device according to claim 3, wherein it is determined whether the first object appears in the image data based on the metadata and the degree of matching calculated for the first image and the second image. . The determination means is
3. Based on the metadata and the degree of matching calculated for the first image and the second image, similarity determination indicating a degree of probability that the first object is captured is added to the image data. 5. The search device according to 4. connected to an output device having a monitor;
The output means is
outputting the image data showing the first object and the object in the second image to the output device;
The output device is
6. The retrieval device according to claim 5, wherein the monitor displays a first user interface for selecting the image data showing the first target and the subject in the second image. The determination means is
outputting at least one of the image data to the candidate image proposing means;
The candidate image proposing means includes:
7. The search device according to any one of claims 1 to 6, wherein the candidate image is output using the image data output by the determination means. The candidate image proposing means includes:
connected to an input device having a monitor, outputting the candidate image to the input device;
The input device is
displaying on the monitor a second user interface for selecting the first image and the second image from among the candidate images, and displaying each of the candidate images selected via the second user interface; 8. The search device according to claim 7, wherein one of said first image and said second image is specified, and information on said specified first image and said second image is output to said determination means. Based on the degree of similarity between the image data and the first image, and the degree of similarity between the second image showing the related person of the first object and the image data, the object in the second image and the first image 9. The search device according to any one of claims 1 to 8, further comprising classifying means for classifying the image data in which the target is shown into a plurality of levels of priority. The output means is
outputting the image data showing the first object and the subject so as to be displayed for each of the priorities ;
The search device according to claim 9. A computer implemented search method comprising:
Acquiring a first image showing a first object that is a detection target person and a second image showing a second object related to the first object,
Determining whether the first object is shown in the image data based on the similarity between the image data to be determined and the first image and the similarity between the image data and the second image,
outputting to display the image data showing the first object ;
A search method for outputting candidate images for the first image and the second image . a process of acquiring a first image showing a first object that is a person to be detected and a second image showing a second object related to the first object;
Determining whether or not the first object is shown in the image data based on the degree of similarity between the image data to be determined and the first image and the degree of similarity between the image data and the second image. When,
a process of outputting to display the image data showing the first object ;
a process of outputting candidate images of the first image and the second image;
A program that makes a computer run

Images