JP7406757B1 - methods and programs - Google Patents

Abstract

[Problem] To select candidates for photos to be posted in an album in a well-balanced and efficient manner. [Solution] A program according to an aspect of the present disclosure includes means for classifying a plurality of photographs into a plurality of groups based on image feature amounts, and for each group, an upper limit of the reference number of photographs determined for the group. As a means for selecting at least a part of the photos belonging to the group, and as a means for presenting information including the selected photos. [Selection diagram] Figure 4

Description

Application of Article 30, Paragraph 2 of the Patent Act February 28, 2022, https://prtimes. jp/main/html/rd/p/000000008.000059552. html etc. posted on the website by the applicant.

The present disclosure relates to an information processing device, method, program, and system.

For example, in order to create an album such as a graduation album, it is necessary to select photos to be included in the album from among a large number of photos. Therefore, there is a need for technology that can reduce such work burden.

In Patent Document 1, a different reference range is set for each shooting scene based on the theme of the album, and the evaluation value given to each image data is set in the reference range corresponding to the shooting scene determined for the image data. A technique for normalizing is disclosed.

Japanese Patent Application Publication No. 2011-186715

When selecting photos to be included in an album, we not only focus on the high quality of each individual photo, but also on the balance of the album as a whole. For example, it is often considered undesirable to have an extremely large or small number of photos in which a specific student appears, or to have an extremely large or small number of photos taken in a specific scene.

In the technology described in Patent Document 1, the evaluation value of image data is normalized using a different reference range for each shooting scene based on the theme of the album, so the image data selected for each shooting scene may be biased. obtain.

An object of the present disclosure is to provide a technique for efficiently selecting photo candidates to be included in an album in a well-balanced manner.

A program according to an aspect of the present disclosure includes a means for classifying a plurality of photos into a plurality of groups based on image feature amounts, and a method for classifying a plurality of photos into a plurality of groups based on image feature amounts, and a method for classifying a plurality of photos into a plurality of groups based on an image feature amount. The present invention functions as a means for selecting at least some of the photographs belonging to the group, and a means for presenting information including the selected photographs.

FIG. 1 is a block diagram showing the configuration of an information processing system according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a server according to the present embodiment. FIG. 2 is an explanatory diagram of one aspect of the present embodiment. FIG. 3 is a diagram showing the data structure of a photo database according to the present embodiment. FIG. 2 is a diagram showing a data structure of a database of number of appearances by person according to the present embodiment. It is a flowchart of information processing of this embodiment. 8 is a flowchart of an example of step S134 in FIG. 7. FIG. 3 is a diagram illustrating an example of a screen displayed during information processing according to the present embodiment. FIG. 3 is a diagram illustrating an example of a screen displayed during information processing according to the present embodiment. 10 is a diagram showing a modification of the screen shown in FIG. 9. FIG.

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In addition, in the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted.

In the following description, when a common description is given for a plurality of elements of the same type, a common code such as "99" may be used. On the other hand, when describing these elements individually, numerals such as "99-1" or "99-2" in which a subscript is added to a common numeral may be used.

(1) Configuration of information processing system The configuration of the information processing system will be explained. FIG. 1 is a block diagram showing the configuration of an information processing system according to this embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer. The users of the client device 10 are, for example, a person who takes photos that are candidates for being posted in an album (for example, a photographer at a photo studio), and a person who determines photos to be posted in an album (for example, a school teacher). . Note that both may be the same person.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. Server 30 is, for example, a server computer.

(1-1) Configuration of client device The configuration of the client device will be explained. FIG. 2 is a block diagram showing the configuration of the client device of this embodiment.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface 14. Client device 10 is connected to display 21 .

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (for example, flash memory or hard disk).

The programs include, for example, the following programs.
・OS (Operating System) program ・Application program that executes information processing (for example, a web browser or an application for editing an album)

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is a computer that implements the functions of the client device 10 by activating a program stored in the storage device 11. The processor 12 is, for example, at least one of the following.
・CPU (Central Processing Unit)
・GPU (Graphic Processing Unit)
・ASIC (Application Specific Integrated Circuit)
・FPGA (Field Programmable Array)

The input/output interface 13 is configured to acquire information (for example, user instructions) from an input device connected to the client device 10 and output information (for example, an image) to an output device connected to the client device 10. be done.
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display 21, a speaker, or a combination thereof.

Communication interface 14 is configured to control communication between client device 10 and an external device (eg, server 30).

The display 21 is configured to display images (still images or moving images). The display 21 is, for example, a liquid crystal display or an organic EL display.

(1-2) Server configuration The server configuration will be explained. FIG. 3 is a block diagram showing the configuration of the server of this embodiment.

As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of ROM, RAM, and storage (eg, flash memory or hard disk).

The programs include, for example, the following programs.
・OS program ・Application program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is a computer that implements the functions of the server 30 by activating a program stored in the storage device 31. The processor 32 is, for example, at least one of the following.
・CPU
・GPU
・ASIC
・FPGA

The input/output interface 33 is configured to obtain information (e.g., user instructions) from an input device connected to the server 30 and output information (e.g., an image) to an output device connected to the server 30. .
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display.

Communication interface 34 is configured to control communication between server 30 and an external device (eg, client device 10).

(2) One aspect of the embodiment One aspect of the present embodiment will be described. FIG. 4 is an explanatory diagram of one aspect of this embodiment.

As shown in FIG. 4, the first user US11 (for example, a photographer at a photo studio) operates the client device 10-1 to send information to the client device 10-1 about candidates to be published in, for example, a school yearbook. The photo is uploaded to the server 30.

The server 30 acquires the uploaded photo and stores it in the storage device 31. The server 30 classifies the acquired photos into a plurality of groups, which will be described later, based on the image features of the photos. Note that uploading and classifying photos may be performed as a series of processes, or may be performed as independent processes. For example, classification of photos may not occur until requested by the second user US12 (eg, a school teacher). Here, each group corresponds to an element (for example, a scene) constituting the event for which the photo was taken. For example, when dealing with photos that are candidates to be published in a school yearbook, the event may be an unusual event such as a school event (for example, a sports day, a school trip, an excursion, a school festival, or an entrance ceremony). , or daily events such as classes, after school, club activities, commuting to and from school, etc. may be defined. Multiple groups may be defined for one event. Specifically, for an event called a sports day, a plurality of groups may be defined corresponding to the events that make up the sports day (for example, relay, cheering contest, ball toss, etc.).

The server 30 evaluates the acquired photos based on predetermined criteria. Then, for each group, the server 30 selects at least some of the photos belonging to the group as recommended photos for the group, with the reference number determined for the group being the upper limit.

The server 30 presents information including the selected photos (that is, the recommended photos of each group) to the second user US12. Specifically, the server 30 transmits the photo to be presented together with additional information to the client device 10-2 in response to an instruction from the second user US12. The additional information includes, for example, information regarding the classification results of the photos to be presented (group information to be described later), information regarding the evaluation results of the photos (evaluation information to be described later), and information indicating whether the photo has been selected as a recommended photo in the group. Information (status information to be described later), information about characters appearing in the photo (character information to be described later), or a combination thereof can be included.

The client device 10-2 acquires the photo and additional information from the server 30 and presents them to the second user US12 by displaying them on the screen. At this time, the client device 10-2 can organize (for example, filter) the photos using the additional information and present them. For example, the client device 10-2 may present recommended photos for each event (that is, for all groups corresponding to an event), or may present recommended photos for each group. The second user US12 can refer to the presented recommended photos and efficiently decide which photos to post in the album in a well-balanced manner.

(3) Database The database of this embodiment will be explained. The following database is stored in the storage device 31.

(3-1) Photo database The photo database of this embodiment will be explained. FIG. 5 is a diagram showing the data structure of the photo database of this embodiment.

The photo database stores photo information. The photo information is information regarding photos that are candidates to be posted in an album.

As shown in FIG. 5, the photo database includes a "photo ID" field, an "event" field, a "group" field, a "rating" field, a "character" field, and a "status" field. . Each field is associated with each other.

A photo ID is stored in the "photo ID" field. The photo ID is information that identifies a photo.

Event information is stored in the "event" field. The event information is information regarding the event at which the photo was taken. The event information may be specified by the user (for example, the photographer), may be automatically assigned according to the shooting date and time, the shooting location, or a combination thereof, or may be inferred using a trained model. Good too.

Group information is stored in the "group" field. Group information is information regarding groups into which photos are classified.

Evaluation information is stored in the "evaluation" field. The evaluation information is information regarding the evaluation result of the photo.

Character information is stored in the "Character" field. The character information is information related to the identification results of the people appearing in the photo.

Status information is stored in the "Status" field. The status information is information regarding the status of the photo. The status of the photo can include at least one of the following:
・It is scheduled to be published in the album (“Scheduled to be published”)
・The user is considering the photo as a candidate for the album (“Under consideration”)
・Recommended as a candidate for photos in the album (“Recommended”)
・None of the above statuses apply (“Other”)

In addition, as the photo information, information on the date and time of photography, information on the location of photography (for example, GPS information), etc. may be stored. Further, as the photo information, information that can specify the album associated with the photo (for example, which year's graduation album of which school) may be stored.

(3-2) Database of number of appearances by person The database of number of appearances by person of this embodiment will be explained. FIG. 6 is a diagram showing the data structure of the number-of-appearances database for each person according to this embodiment.

The number of appearances by person database stores information about the number of appearances by person. The number-of-appearance information for each person is information regarding the number of appearances for each person in each of at least one photo group. Each photo group may be a group of photos corresponding to a specific event, a group of photos corresponding to a specific group of a specific event, or (candidates to be posted in the album) It may also be a group consisting of all photos. Each photo group may be a group of photos that correspond to a specific status, a group of photos that correspond to a specific status and a specific event, or a group of photos that correspond to a specific status and a specific event. may be a group of photos corresponding to a particular group of events.

As shown in FIG. 6, the appearance count database for each person includes a "person ID" field and a "number of appearance" field. Each field is associated with each other.

A person ID is stored in the "person ID" field. The person ID is information that identifies the person specified in any of the photos (which are candidates for being published in the album). The person ID can be associated with information about the person's name and information about the feature amount of the person's face in a person-by-person appearance count database or another database (for example, a face database described later).

The “number of appearances” field stores information on the number of appearances. The appearance count information is information regarding the number of appearances (total number) of a person in each of at least one photo group.

(4) Information processing Information processing of this embodiment will be explained. FIG. 7 is a flowchart of information processing according to this embodiment. FIG. 8 is a flowchart of an example of step S134 in FIG. FIG. 9 is a diagram showing an example of a screen displayed in the information processing of this embodiment. FIG. 10 is a diagram showing an example of a screen displayed in the information processing of this embodiment.

The information processing of the present embodiment may be started in response to the server 30 receiving a start request from the client device 10 based on an instruction from a user who is a person who decides which photos to post in an album, for example. The start request includes, for example, information that can identify the album.

As shown in FIG. 7, the server 30 executes photo acquisition (S130).
Specifically, the server 30 acquires photos associated with the album to be processed. As an example, the server 30 reads photos stored in the storage device 31 (or may be an external storage device, not shown). These photos are taken in advance by a user corresponding to a person who takes photos that are candidates for publication in the album, and are uploaded to the server 30 using the client device 10.

After step S130, the server 30 performs photo classification (S131).
Specifically, for each event for which a plurality of groups are defined, the server 30 selects each photo taken during the event and acquired in step S130 based on the image feature amount of the photo. Classify into one of multiple groups. The server 30 updates group information in the photo database (FIG. 5) based on the classification results.

The server 30 may classify the photos based on the output of the trained model. As an example, the server 30 applies a group determination model corresponding to the event in which the photo was taken to input data based on the image feature amount of the photo. The group determination model generates output data corresponding to the inference result of the group to which the photo belongs. The group determination model includes, for example, input data for learning based on the image features of a large number of photographs prepared in advance, and the group to which the original photograph of each input data belongs (this may be determined by a human). It can be constructed by machine learning (supervised learning) using learning data including correct answer data indicating .

After step S131, the server 30 executes character identification (S132).
Specifically, the server 30 identifies the person in each photo based on the person's face. As an example, the server 30 performs facial recognition on each photo. A trained model can be used for face recognition.

As a first example of specifying a character appearing (S132), the user of the client device 10 stores information on a person whose number of appearances is to be managed (for example, a student or a school staff member including a teacher) in a database (hereinafter referred to as " be registered in advance in a face database (referred to as a "face database"). As an example, the client device 10 transmits facial photographs of each person to the server 30. The server 30 extracts the feature amount from the facial photograph, associates the feature amount with information that can identify the person (for example, ID or name), and stores the feature amount in the face database. Here, the user may specify information that can identify a person via the client device 10. As another example, the client device 10 transmits group photo data including a plurality of people to the server 30. The server 30 performs face recognition on the group photo and presents the result of the face recognition (for example, a face image) to the user via the client device 10. For each detected face, the user specifies information that can identify the person via the client device 10. The server 30 stores facial feature amounts in a face database in association with information that can identify the person designated by the user, in accordance with the user's designation.
The server 30 then identifies the characters using the face database constructed in this way. That is, the server 30 determines, based on the facial features of the person newly detected from the photo, which of the people already registered in the face database the person is the same as. In this way, for each person registered in advance in the face database, it is possible to specify which photo the person appears in. Similarly, it is possible to specify which person registered in the face database appears in each photo associated with the album to be processed. The server 30 updates the character information in the photo database (FIG. 5) and the number-of-appearance information in the person-specific appearance number database (FIG. 6) based on the result of character identification.

The second example of character identification (S132) does not require prior construction of a face database. Specifically, the server 30 determines whether the newly detected person (for example, a student or a school staff member including a teacher) is a known person (that is, is recognized as a different individual, and whose facial features are in the face database). It is determined which of the stored persons the face is the same as based on the facial features. Here, if the server 30 determines that the newly detected person is not the same as any of the known persons, the server 30 converts the facial features of the person into information that can identify the person (for example, ID or name). and stored in the face database. Here, the user may specify information that can identify a person via the client device 10. In this way, it is possible to specify, for each person recognized as a different individual, which photo the person appears in, without having to construct a face database in advance. Similarly, it is possible to specify which person registered in the face database appears in each photo associated with the album to be processed. The server 30 updates the character information in the photo database (FIG. 5) and the number-of-appearance information in the person-specific appearance number database (FIG. 6) based on the result of character identification.

After step S132, the server 30 performs photo evaluation (S133).
Specifically, the server 30 evaluates each photo and calculates an evaluation value. The server 30 updates the evaluation information in the photo database (FIG. 5).

As a first example of photo evaluation (S133), the server 30 evaluates the photo to be evaluated based on a standard common to the group.
As a first example of evaluation based on a common standard among groups, the server 30 determines the degree of smile of the person who is the subject in the photo to be evaluated, and evaluates the photo based on the degree of smile. good. As an example, the server 30 may be configured to determine whether statistics (e.g., mean value, median value, mode, maximum value, minimum value, first quartile, third quartile, or sum) of the degree of smiling are The larger the value, the higher the evaluation value may be calculated. The degree of smiling can be determined using, for example, a trained model.

As a second example of evaluation based on a common standard among groups, the server 30 determines the gesture of the person who is the subject in the photograph to be evaluated (for example, based on the shape of the hand), and based on the determination result, the server 30 determines the gesture of the person in the photograph to be evaluated. may be evaluated. As an example, the server 30 may calculate a higher evaluation value as the total number of specific gestures (for example, peace signs) appearing in the photo is larger. Gestures can be determined using, for example, a learned model.

As a third example of evaluation based on a common standard among groups, the server 30 determines whether or not the face of the person who is the subject is included in the photo to be evaluated, and whether the face of the person who is the subject is included or not. You can lower the rating of photos that do not. Whether or not a person's face is included in the photo can be determined using, for example, a trained model. Here, the server 30 determines whether the person in question is a main subject or a non-main subject based on the size of the face of the person in the photo, and the person ( (Typically, people whose faces are so small that they appear to have appeared only by chance) may be excluded from consideration in this evaluation.

As a fourth example of evaluation based on common criteria among groups, the server 30 can combine a plurality of the first to third examples.

As a second example of photo evaluation (S133), the server 30 evaluates the photo to be evaluated based on the individual criteria of the group to which the photo belongs. As an example, the server 30 determines an action taken by a person who is a subject in a photo belonging to a certain group (for example, "running", "throwing", etc.), and uses the determination result and the individual actions of the group. Evaluate the photo based on the following criteria: The actions taken by the person can be determined using, for example, a learned model. For example, as an individual criterion for a group, scores may be assigned for behaviors characteristic of the group. In this case, the server 30 stores statistics of scores (for example, mean, median, mode, maximum, minimum value, first quartile, third quartile, or total), a higher evaluation value may be calculated for the photo.

As a third example of photo evaluation (S133), the server 30 can combine the first and second examples described above.

After step S133, the server 30 executes photo selection (S134).
Specifically, for each group corresponding to the classification destination of the photos in step S131, the server 30 determines whether the photos belong to the group (i.e., are classified into the group in step S131) up to the reference number determined for the group. select at least some of the photos (received) as recommended photos for the group; The server 30 may select recommended photos for each group based on the evaluation (that is, evaluation value) of the photos belonging to each group in step S133. The server 30 updates the status information in the photo database (FIG. 5) regarding the photo selected as the recommended photo.

The server 30 may determine the reference number of sheets according to a user's instruction, or may determine it automatically.
As a first example of automatically determining the reference number of photos, the server 30 determines the reference number of photos for each group (eg, the total number multiplied by a predetermined ratio) based on the total number of photos belonging to each group. As a second example of automatic determination of the standard number of sheets, the server 30 determines the number of sheets to be published for each group based on the number of sheets to be published set for the event to which each group belongs or the number of sheets to be published for the entire album. The standard number of pages (for example, the number of pages multiplied by a predetermined ratio by one of the above scheduled number of pages) is determined. As a third example of automatic determination of the reference number, the server 30 determines the number of photos belonging to each group in past albums of the same type (for example, past yearbooks of the same school, or yearbooks of other schools that are being created or have been created in the past). The standard number of pages for the group will be determined based on the actual number of pages published. As a fourth example of automatic determination of the reference number of sheets, the server 30 determines the smallest number among the reference number of sheets determined in a plurality of the first to third examples as the final reference number of sheets.

As an example, the server 30 selects photos for each group as shown in FIG.
As shown in FIG. 8, the server 30 performs photo sorting (S1341).
Specifically, the server 30 sorts the photos belonging to the group to be processed in descending order of the evaluation values calculated in step S133.

After step S1341, the server 30 selects a target photo (S1342).
Specifically, the server 30 selects, as the target photo, the first one (that is, the photo with the highest evaluation value) among the photos that belong to the group to be processed and have not yet been selected in this step.

After step S1342, the server 30 executes similarity calculation (S1343).
Specifically, the server 30 calculates the degree of similarity between the target photo selected in step S1342 and each photo already selected as a recommended photo of the group to be processed.

If the maximum similarity calculated in step S1343 is less than the threshold, the server 30 selects a target photo (S1344).
Specifically, the server 30 selects the target photo selected in step S1342 as the recommended photo for the group to be processed.

On the other hand, if the maximum similarity calculated in step S1343 is less than the threshold, the server 30 skips selection of the target photo (S1344). This can prevent a plurality of extremely similar photos, such as a series of photos generated by the continuous shooting function, from being selected as recommended photos.

After executing or skipping step S1344, if the number of recommended photos has reached the reference number, the server 30 ends the process of FIG. 8. Furthermore, after executing or skipping step S1344, the number of recommended photos has not reached the reference number, but the target photo is the last photo (all photos belonging to the group to be processed have already been selected in step S1342). ), the server 30 ends the process of FIG.

After executing or skipping step S1344, the number of recommended photos has not reached the standard number and the target photo is not the last photo (photos that belong to the group to be processed and have not yet been selected in step S1342 remain) ), the server 30 re-executes step S1342.

After step S134, the server 30 presents information (S135).
The server 30 determines the information to be presented based on the photo acquired in step S130, the photo database (FIG. 5), and the database of number of appearances by person (FIG. 6). The server 30 transmits information to be presented to the client device 10. The client device 10 displays a screen based on the information acquired from the server 30 on the display 21.

Further, the client device 10 may obtain a user instruction on the screen and transmit it to the server 30. In this case, the server 30 re-determines the information to be presented based on the received user instruction, and transmits the information to be presented to the client device 10 as necessary. The client device 10 displays a screen based on the information acquired from the server 30 on the display 21 (updating the screen).

As an example, the client device 10 displays the screen shown in FIG. 9 on the display 21. The screen in FIG. 9 includes objects J20, J21, J22, J22a, J23, J24, J25, and J26.

Object J20 displays the selected event. On the screen of FIG. 9, among the photos acquired in step S130, photos corresponding to the event specified by object J20 are arranged.

Object J21 accepts a user instruction to switch groups. When the user of the client device 10 selects (activates) any object J21, photos belonging to the group corresponding to the object J21 are arranged on the screen. The contents of object J21 change depending on the event specified by object J20.

In the object J22, a photo (object) that belongs to the group corresponding to the active object J21 and whose status is "scheduled to be published" is placed. Note that the photos placed in the object J22 may be sorted based on evaluation information (evaluation value).

The object J22a has information about the number of pages scheduled to be published set for the group corresponding to the active object J21, and information about the number of pages placed in the object J22 (that is, if it belongs to the group corresponding to the active object J21 and has a status information on the number of photos (which are scheduled to be posted).

The object J23 is a photo that belongs to the group corresponding to the active object J21 and whose status is "recommended", that is, the (object) selected as the recommended photo for the group in step S134. Note that a photo whose status is "recommended" and "scheduled to be published" does not need to be placed in the object J23. Similarly, a photo whose status is "recommended" and "under consideration" does not need to be placed in the object J23. Further, the photos placed in the object J23 may be sorted based on evaluation information (evaluation value).

A photo (object) that belongs to the group corresponding to the active object J21 and whose status is "under consideration" is placed in the object J24. Note that the photos placed in the object J24 may be sorted based on evaluation information (evaluation value).

The object J25 is a photograph (object) that belongs to the group corresponding to the active object J21 and whose status is "other". Note that the photos placed in the object J25 may be sorted based on evaluation information (evaluation value).

Object J26 displays a photo (which may be a thumbnail of the photo). Object J26 can also display additional information. Additional information may include evaluation information, for example. Thereby, the user can efficiently decide which photos to post in the album by referring to the evaluation information provided by the server 30 in addition to his or her own subjective evaluation.

Furthermore, object J26 may accept at least one of the following user instructions. The server 30 may change the information to be presented or change the status information in the photo database (FIG. 5) in accordance with the received user instruction.
・User instructions for displaying detailed information about the photo ・User instructions for enlarging the photo ・User instructions for enabling or canceling "Planned to be posted" in the photo status ・User instructions for "Under consideration" in the photo status User instructions for activating or deactivating

In response to a user instruction for displaying detailed information of a photo, the client device 10 may display, for example, a screen shown in FIG. 10 on the display 21.

The screen in FIG. 10 includes objects J30 to J37.
Object J30 displays the group to which the selected photo belongs.
Object J31 displays the selected photo (which may be a thumbnail of the photo).

Object J32 accepts a user instruction to enable "scheduled publication" in the status of the selected photo. The server 30 changes the status information in the photo database (FIG. 5) according to the received user instruction.

Object J33 accepts a user instruction to enable "under consideration" in the status of the selected photo. The server 30 changes the status information in the photo database (FIG. 5) according to the received user instruction.

Object J34 displays information about the person specified in the selected photo (for example, a representative face image of the person). In addition to the object J34, an object that displays information on the identified person (for example, name information) may be arranged. Note that when the information of the identified person is not registered as in the second example of character identification (S132), the user can provide the identified person with a name (for example, real name (last name only or last name)). and part of their first name), a nickname, or an attendance number).

Object J35 displays the number of appearances of each person in photos taken at an event corresponding to the group to which the selected photo belongs (that is, displayed in object J30) and has a status of "scheduled to be published."

Object J36 displays the number of appearances of each person across all photos with the status of "scheduled to be published."

Note that the objects J35 and J36 are merely examples, and an object that displays at least one of the following information may be placed on the screen, or the information may be displayed on a separate screen.
・Information on the number of appearances of each person across all photos with a specific status ・Information on the number of appearances of each person across all photos with a specific status ・Information on a specific group that forms part of all photos with a specific status (for example, photos taken at a specific event, Information on the number of appearances of each person across all photos (e.g., photos taken at a specific event, Information on the number of times each person appears in photos belonging to a specific group, etc.)

Object J37 is an object that accepts user instructions for displaying photos in which each person appears. When one of the objects J37 is selected, the client device 10 displays on the display 21 a screen on which a photograph in which a person corresponding to the object J37 appears is arranged. On this screen, photos may be arranged and arranged by event, group, status, or date and time of shooting (for example, by year).

(5) Summary The server 30 of this embodiment classifies a plurality of photos into a plurality of groups based on the image feature amount, and for each group, the number of photos set for the group is set as the upper limit. Select at least some of the photos belonging to. The server 30 then presents information including the selected photo. Thereby, the user who is presented with the information can efficiently decide which photos to include in the album in a well-balanced manner.

The server 30 may evaluate a plurality of photos among groups based on common criteria, and select photos for each group based on the evaluation of photos belonging to the group. This makes it difficult for photos with low ratings to be selected in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

The server 30 may determine the degree of smile of the person who is the subject in each of the plurality of photographs, and may evaluate the photograph based on the degree of smile. This makes it easier to select photos with a high degree of smiling in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

The server 30 may determine the gesture of the person who is the subject in each of the plurality of photographs based on the shape of the person's hand, and may evaluate the photograph based on the determination result. This makes it easier to select photos that include a subject making a specific gesture in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

The server 30 may determine whether or not the face of the person as the subject is included in each of the plurality of photos, and lower the evaluation of the photo in which the face of the person as the subject is not included. This makes it difficult for photos that do not include a part of the subject's face to be selected in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

Server 30 may evaluate multiple photos according to individual criteria for the group. The server 30 determines the motion of the person who is the subject in a plurality of photographs, evaluates the photographs belonging to the group based on the determination results and the individual criteria of the group, and evaluates the photographs belonging to the group for each group. You may select photos based on This makes it difficult for photos with low ratings to be selected in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

The server 30 does not have to select, from among the photos belonging to each group, photos whose degree of similarity to already selected photos is equal to or greater than a threshold. This prevents a plurality of extremely similar photos from being selected all at once in each group, so the user who is presented with the information can efficiently decide which photos to post in the album.

The server 30 changes the status of a photo in accordance with a user's instruction, and displays information on the number of appearances of each person in all photos with a specific status or in a specific group that forms part of all photos with the specific status. may be presented. Thereby, the user who is presented with the information can efficiently decide which photos to post in the album in a well-balanced manner while considering the number of appearances of each person.

The server 30 may present the selected photo together with the evaluation results for the photo. Thereby, the user who is presented with the information can efficiently decide which photos to post in the album while considering the evaluation results of each photo.

The multiple photos may be taken during the same event, and each of the multiple groups may correspond to an individual scene that makes up the event. Thereby, the user can decide the photos to be posted in the album in a well-balanced manner, keeping in mind each scene that constitutes the same event.

The server 30 may determine the reference number of sheets according to a user instruction. This allows the user to control the number (upper limit) of photos to be selected for each group and efficiently determine photos to be posted in the album.

(6) Modification In the above description, the screen example shown in FIG. 9 was shown. However, recommended photos (and other status photos) may be presented on an event by event basis.

As an example, the client device 10 displays the screen shown in FIG. 11 on the display 21. The screen in FIG. 11 includes objects J40, J42, J42a, J43, J44, J45, and J46.

Object J40 displays the selected event. On the screen of FIG. 11, among the photos acquired in step S130, photos corresponding to all groups belonging to the event specified by object J40 are arranged.

In the object J42, a photo (object) that belongs to any group of the currently selected event and whose status is "scheduled to be posted" is placed. Note that the photos placed within the object J42 may be sorted based on evaluation information (evaluation value).

Object J42a has information about the number of pages scheduled to be posted for the currently selected event, and is placed within object J42 (that is, it belongs to any group of the currently selected event, and the status is "published"). information on the number of photos (planned).

The object J43 is a photo that belongs to any group of the selected event and has a status of "recommended", that is, a (object) selected as a recommended photo for any group in step S134. Note that a photo whose status is "recommended" and "scheduled to be published" does not need to be placed in the object J43. Similarly, a photo whose status is "recommended" and "under consideration" does not need to be placed in object J43. Further, the photos placed within the object J43 may be sorted based on evaluation information (evaluation value).

In the object J44, a photo (object) that belongs to any group of the selected event and whose status is "under consideration" is placed. Note that the photos placed in the object J44 may be sorted based on evaluation information (evaluation value).

In the object J45, a photo (object) that belongs to any group of the selected event and whose status is "other" is placed. Note that the photos placed in the object J45 may be sorted based on evaluation information (evaluation value).

Object J46 is similar to object J26 in FIG.

(7) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Display 21 may be built into client device 10 . The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30. For example, processing described as being performed by one device may be performed by another device, or processing described as being performed by interaction of multiple devices may be performed by a single device. Further, in the above description, an example was shown in which each step is executed in a specific order in each process, but the execution order of each step is not limited to the example described as long as there is no dependency relationship.

In the above explanation, an example was explained in which photos are selected for each group based on evaluation and similarity. However, the server 30 may select photos for each group based further on statistics of the number of times a person appears. This controls the statistics of the number of appearances of a person across a plurality of selected photos, so the user who is presented with the information can efficiently decide which photos to post in the album in a well-balanced manner.
Specifically, the server 30 stores statistics (for example, variance, maximum value, minimum value, average value, median value, mode, first value, Photos may be selected such that they meet criteria (such as a quartile, a third quartile, or a total). As a first example, the server 30 performs selection based on evaluation values and similarity so that the variance of the number of appearances of each person in photos selected across multiple groups (an example of "dispersion") is less than or equal to a threshold. The results may be corrected. In the above correction, an unselected second photo (a photo with a lower evaluation value than the first photo, but whose dispersion would improve if selected) may be selected instead of the selected first photo. As a second example, the server 30 selects the selection results based on the evaluation value and similarity so that the minimum number of appearances of each person in the photos selected across multiple groups is equal to or greater than a threshold value (for example, "1"). may be corrected. In the above correction, an unselected second photo (a photo of a person whose number of appearances is equal to or less than a threshold value) may be selected instead of the selected first photo. As a third example, the server 30 may correct the selection result based on the evaluation value and similarity so that the maximum number of appearances of each person in photos selected across multiple groups is equal to or less than a threshold value. . In the above correction, an unselected second photo (a photo that does not include such a person) may be selected instead of the selected first photo (a photo that includes a person whose number of appearances is equal to or greater than a threshold value).

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Moreover, various improvements and changes can be made to the embodiments described above without departing from the spirit of the present invention. Furthermore, the above embodiments and modifications can be combined.

1 : Information processing system 10 : Client device 11 : Storage device 12 : Processor 13 : Input/output interface 14 : Communication interface 21 : Display 30 : Server 31 : Storage device 32 : Processor 33 : Input/output interface 34 : Communication interface

Claims (2)

computer,
means for classifying multiple photos into multiple groups based on image features;
means for evaluating the plurality of photos according to individual criteria of the group;
means for selecting, for each group, at least a portion of the photos belonging to the group, up to a reference number determined for the group;
means for presenting information including selected photographs;
function as
The evaluating means determines the motion of the person who is the subject in the plurality of photographs, and evaluates the photographs belonging to the group based on the determination result and the individual criteria of the group,
The selecting means selects photos for each group based on evaluations of photos belonging to the group.
program. The computer is
classifying a plurality of photos into a plurality of groups based on image features;
evaluating the plurality of photos according to individual criteria of the group;
For each group, selecting at least some of the photos belonging to the group, up to a reference number determined for the group;
performing the steps of presenting information including the selected photo;
In the step of evaluating, the motion of the person who is the subject in the plurality of photographs is determined, and the photographs belonging to the group are evaluated based on the determination result and the individual criteria of the group;
In the selecting step, photos are selected for each group based on evaluations of photos belonging to the group.
Method.